SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
Authors: Haozhan Li
Yuxin Zuo
Jiale Yu
Yuhao Zhang
Yang Zhaohui
Kaiyan Zhang
Xuekai Zhu
Yuchen Zhang
Tianxing Chen
Ganqu Cui
Dehui Wang
Dingxiang Luo
Yuchen Fan
Youbang Sun
Jia Zeng
Jiangmiao Pang
Shanghang Zhang
Yu Wang
Yao Mu
Bowen Zhou
Ning Ding
Venue: ICLR 2026 Poster (2026)
Keywords: VLA Models
Reinforcement Learning
Bimanual Manipulation
Robot Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

Abstract: Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pretraining and supervised fine-tuning (SFT), these models face two fundamental challenges:
(i) the scarcity and high cost of large-scale robotic trajectories required for SFT scaling,
and (ii) limited generalization to tasks under distribution shift.
To overcome these limitations, we explore reinforcement learning (RL) as a pathway to scaling VLA training beyond limited datasets.
Inspired by LLM breakthroughs where RL with outcome rewards enhances step-by-step reasoning, we ask: Can outcome-driven RL improve long-horizon step-by-step action planning of VLA?
In this work, we introduce SimpleVLA-RL, an efficient RL framework tailored for VLA models.
Building upon veRL, we introduce VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering, and optimized loss computation.
Applied to OpenVLA-OFT, SimpleVLA-RL achieves 99\% of SoTA performance on LIBERO and 80\% relative
improvement on RoboTwin 1.0\&2.0, outperforming $\pi_0$ with our proposed exploration-enhancing strategies.
SimpleVLA-RL reduces dependence on large-scale data, enables robust generalization, and remarkably surpasses SFT in real-world tasks.
Moreover, we identify a novel phenomenon "pushcut'' during RL training, wherein the policy discovers unseen patterns beyond those seen in previous training process.

Published as a conference paper at ICLR 2026
SIMPLEVLA-RL: SCALING VLA TRAINING VIA RE-
INFORCEMENT LEARNING
Haozhan Li Yuxin Zuo"! Jiale Yu" Yuhao Zhang* Zhaohui Yang", Kaiyan Zhang 1.7 Xuekai Zhu', Yuchen Zhang,. Tianxing Chen', Ganqu Cui", Dehui Wang Dingxiang Luo', Yuchen Fan", Youbang Sun', Jia Zeng', Jiangmiao Pang" Shanghang Zhang,, Yu Wang", Yao Mu Bowen Zhou Ning Ding
"Tsinghua University Shanghaii AI Lab "Nature Will
*Shanghai Jiao Tong University Thhe University of Hong Kong FFonnts.AA
'Peking University
C Code: https: cOM/PRIME RL/ RL/SimpleVIA-B
ABSTRACT
Vision- Lioon-LLnguag-Acttinn (VLA) models have emerged a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pre- training and supervised fine-tuning (SFT), these models face two fundamental challenges: (i) the scarcity and high cost of large-scale robotic trajectories required for SFT scaling, and (ii) limited generalization tasks under distribution shift. To overcome these limitations, we explore reinforcement learning (RL) pathway to scaling VLA training beyond limited datasets. Inspired by LLM breakthroughs where RL with outcome rewards enhances step-by-step reasoning, ask: Can outcome-driven RL improve long-horizon step-by-step action planning of VLA? In this work, introduce SimpleVL.A-RI for VIAA models. Building upon veRI., we introduce VI.A-ppecfiic trajectory sampling, scalable parallelization, multi-enviromment rendering, and optimized loss computation. Applied to OpenVLA-OFT, SimpleVLA- achieves 99% of SoTA performance on LIBERO and 80% relative improvement on RoboTwin 1.0&2.0, outperforming Ty with our proposed exploration cellorationenhanccinn enhancing strategies. SimpleVLA-RL reduces dependence on large-scald scale data, enables robust general- ization, and remarkably surpasses SFT in real-world tasks. Moreover, we identify a novel phenomenon "pushcut" during RL training, wherein the policy discovers unseen patterns beyond those seen in previous training process.
efficient RL framework tailored
INTRODUCTION
Bitoo-Language-Ace (VLA) models have emerged as a promising approach for general robotic manipulation by integrating visual perception, language understanding. and action generation unified framework (Firoozi et al. 2025 Kim et al.) 2024 Zhong al., 2025). Current VI.A training paradigm consists of two stages: large-scale pretraining on multimodal data (human manipulation videos (Sapkota et al. 2025). image-text pairs, and heterogeneous robot datasets (O'Neill et al. 2024)) Tollowed by supervised fine-tuning (SFT) on additional high-quality robot trajectories to enhance task-specific capabilities.
While imitation learning paradigm has achieved notable progress (Intelligence et al. 2025), its heavy dependence large-scale, high-quality data poses fundamental bottleneck that constrains fur- ther development of VLA models:the Generalization Bottleneck from Data Scarcity (Schulman et al. 2017b). 1) Data Scarcity: Scaling VLAA training through SFT requires massive amounts of high-quality robot trajectories, yet such data remains scarce and prohibitively expensive (Gao et al., 2024). Collecting expert demonstrations demands carefully designed scenarios, diverse manipula- tion objects, and skilled operators, which severely constrains both data scale and diversity (Bu et al.) 2025a Team et al. 2025a). 2) Poor Generalization: This data scarcity leads fundamental mismatch between training distributions and open-ended real-world tasks (Liu et al. (225a)). VLA
"Equal Contributions
CCorrepponding Authors

---

Published as a conference paper at ICLR 2026
Data Scarcity
Simulation & Real-World Tasks
SimpleVLARR
SFTayly
LIBERO
Robolwin2..
ReboTwin1.0
Full-Trjjettry 86.5
1 Succese
20
SimploVLA-RL
so (1.20.17%)
Surcies Rate
SHIOOROO Rate (%)
48.9
88.9
50
100
150
200
10
Training Steps
One
Fel
Terk
Tedl
Teski
Toolu
Avg
New Pattern Discovery: Pushcut Phenomenon
Generalization
SFT Model
After SimpleVLA-RL
Uneeen Took 3XS5
Spatial
Object
Gool
Grasp
Seen 5R/7))
Teen Task 18p%)
Sean 581%)
RL anobles the policy to acquire novel actiona never sean before.
RL provides strong, multidimensional generalization capabilites!
Figure 1: Overview of SimpleVLA-RIA An efficient RL framework for VLA that improves long- horizon planning under data scarcity, outperforms SFT in simulation and real-world tasks, reveals "pushcut" new-action phenomenon, and strengthens spatial/object/gaal generalization.
models trained limited, scene-specific data tend to memorize patterns rather than learn gener- alizable skills. Consequently, even minor distribution shifts,, unseen objects or environments, can cause compounding errors that severely limit generalization (Ross & Bagnell 2010). This problem becomes especially critical in compositional and long-horizon tasks (Gupta et al. 2019), revealing that simply scaling SFT data cannot build generalizable VLA models the open world.
To overcome this generalization bottleneck. VLA models need learning mechanism capable of interactive refinement beyond limited static datasets. Reinforcement Learning (RI.) offers this ca- pability through trial-and-error environmental interaction (Xu et al.) 2024). However, traditional robotics RL requires hand-crafted reward functions for each task, limiting scalability and general- ization novel scenarios where rewards undefined (Ibarz et al.) 2021 Kroemer et al.) 2021 Ma et al., 2023). This creates a dilemma: SFT is constrained by data, while traditional RL is con- strained by reward engineering. Recent breakthroughs in Large Reasoning Models (LRMss provide a crucial insight: using only sparse outcome rewards, RL can significantly enhance models' ability generate correct step-by-step reasoning chains (Guo al.. 2025a Yang al., 2025 Zeng al.) 2025 Team et al. 2025b). This raises a critical question for VLAs:: Can we leverage this outcome- driven RL paradigm to enhance VLA models' ability to generate step-by-step correct actions?
However, applying RL to VLA models presents unique technical challenges. First, unlike open-loop text generation in LLMs, VLAs require closed-loop environment interaction with continuous visual feedback, creating substantial computational overhead (Wang et al., 2025). Second, manipulation tasks face inefficient exploration due to high-dimensional action spaces and sparse rewards (Zhou al. 2025). Third, existing RL frameworks lack VLA-specific infrastructure for efficient inference and parallel environment interaction (Luo et al. 2025).
We introduce SimpleVLA-RI, an effective RI. framework for VIA models. Building upon Vol- cano Engine Reinforcement Learning for LLMs (veRL) (Sheng et al. 2024), we enable end-to-end rule-based online RL for VLA models through VLA-specific interactive trajectory sampling and loss computation. To support scalable RL training, we extend veRL with parallel multi-envirrnment rendering and adapt it into unified training inference inference-rendering framework. We also design three ipploration-enhaneee
strategies. Through SimpleVLA-RL. we significantly improve VLA per- formance under data-scarce conditions, enhance generalization capabilities, and achieve noticeable gains in real-world applications. Surprisingly, during RI. training, the policy discovers novel pat- terns beyond those in supervised data, phenomenon we term "pushcut". Our contributions include:
Efficient online RL framework for VLA: We develop efficient end-to-end VLA online RL framework based on veRL that enables stable, sample-effiche training, optimized for rendering parallelization and distributed training & inference.
2

---

Published as a conference paper at ICLR 2026
SFT
Limited Offline Trajectories
P Policy
T1
T3
Ey
SimpleVLA-RL.
Group Advantage Computation With Higher Clip Range
Exploration Enhanced-Rollle
Trajectories
Rewards
Policy
Higher
T
y
Temperature
action A
Dynamic
T2
72
1
Sampling
Environment
TN
FN
Figure 2: Overview of SimpleVLA-RL.
SOTA performance: We incorporate exploration-enhancing strategies, yielding consistent per- formance improvements of 10-15%. Moreover, SimpleVLA-RR. surpasses multiple SoTA base- lines on both LIBERO and RoboTwin 1.0 & 2.0.
Data efficiency and generalization: With only single demonstration per task, RL boosts LIBERO-Long success rates from 17.1% to 91.7%, and significantly outperforms SFT in spa- tial, object, and task generalization.
Real-world deployment capability: Simulation-trained policies transfer effectively to real- world, achieving strong sim-to-real improvements without requiring real robot data.
2 SIMPLEVLA-RL
DeepSeek-RI (uuo et al., 2025a) has achieved remarkable performance gains through online RL with the simple, scalable rule- basee reward, highlighting promising training paradigm. In this sec- tion, we introduce SimpleVLA--R
which extends this rule-based online RL framework to VLA models for embodied manipulation tasks shown in Figure Specifically,
training framework proceeds as follows: we begin by generating multiple trajectories for each input via random sam- pling (42.1). Each trajectory is then assigned a simple outcome reward for success, for failure) based on environment feedback (8 (.2). Leveraging these rewards together with the corresponding action token probabilities, we compute the GRPO loss update the policy model 2.4
2.1 INTERACTIVE VLA ROLLOUT
RL on VLA models differs fundamentally from LLMs in trajectory generation. enable online RL, policy models need to generate diverse trajectories from input for effective exploration. While LLMs naturally achieve this diversity through random sampling on text token distributions (Renze)
2024 De Rosa & Papa 2021). VLA models face unique challenge due their action decoding strategies. Current VLA models often employ three strategies: (1) generating action token distribu- tions similar LLMs (Black et 2024 Kim al. 2024), (2) diffusion-bases denoising on latent states (Liu et al. 2024 Cheang et al. 2025), and (3) deterministic regression via MLPs (Kim et al., 2025). Among these, the token-baset approach most compatible with PPO-like RL algorithms, as it naturally provides action distributions necessary for both random sampling and policy gradi- ent computation. Therefore, we adopt this approach, where the VI.A model outputs action token probability distributions and employs random sampling to generate diverse trajectories.
Furthermore, for given input query, LLM rollout proceeds by autoregressively generating tokens until reaching stop token or max output length. In contrast, VLA rollout requires continuous interaction with the environment to update the visual observation and robot state dynamically (as detailed in Appendix(B) This closed-loop interaction is necessary because each robotic action alters the environment, and subsequent actions must conditioned on real-time sensory feedback. We present the comparison of the rollout algorithm pseudo-code of LLMs and VLA in Listing

---

Published as a conference paper at ICLR 2026
2.2 OUTCOME REWARD MODELING
SimpleVLA-RL employs a straightforward binary reward function for RL training. Unlike tradi- tional RL approaches that require carefully crafted reward functions (HadiccldMMnnlll et al. [017]
Knox al. 2023 Booth al. 2023), follow DeepSeek-RII RI's approach by assigning trajectory- level rewards of either O or based solely on task completion. When the VLA model successfully completes a task, the entire trajectory is assigned a reward of I; otherwise, it receives reward of O. For gradient computation, these trajectory-level rewards are uniformly propagated to the individual action tokens. Consequently, all tokens within successful trajectories are assigned reward of 1, whereas those in unsuccessful trajectories are assigned reward of O. Our reward function is:
R(ai, st,z) 1, is successful(traa (a;, 8:)],
0, otherwise.
This simple outcome-level reward simple yet effective: scalable, broadly applicable across envi- ronments, and free from complex process-based design (Wu et al. (02T). By focusing solely on task completion, avoids the non-trans tra.sferaabbbitt issues typical of task task-specific rewards.
2.3 EXPLORATION ENHANCEMENTS
Previous works (Yu et al. 2025: Liu et al. 2025b di An et al. 2025) have demonstrated that encour- aging exploration during RL is critical. We observe that this factor becomes even more crucial in VLA RL. Manipulation tasks typically allow for a wide range of valid solutions. However, VLA models tend to converge on a narrow set of solution patterns, largely due to the homogeneity of their training trajectories, which limits the efficiency of RL. Promoting exploration encourages models to discover novel strategies and broaden the solution space, a property that is particularly advantageous in scenarios with low success rates. Building on this insight, we implement three key modifications to enhance the exploration of RL training: 1) employing dynamic sampling during trajectory rollout, 2) adjusting the clip range in the GRPO training objective, 3) and increasing the sampling tempera- ture during rollout.
Dynamic Sampling Critic-free RL algorithms suffer from vanishing gradients when trajectories are assigned the same rewards. For example, GRPO computes advantages using group-relative normalization, comparing each response's reward to the mean and standard deviation of rewards within its group of sampled outputs. When all trajectories share identical rewards, their advantage estimation becomes zero, resulting in null gradients and causing unstable training dynamics.
We address this challenge through Dynamic Sampling Yu et al. 2025 Cui et al. 2025a), a method that has been proven effective in LLM RL (Cui et al. 2025a Yu al. 2025 Team al. 2025b Shi et al. 2025). During rollout, we exclude groups in which all trajectories either succeed or fail. Sampling proceeds until the batch consists solely of groups with mixed outcomes, which can be formally expressed as:
0 < (traj,(a,, st) i wws.uucsffuftraajccccccc (a,, st)lH < G.
(2)
This ensures non-zero advantage estimates and stable gradient flow throughout training.
Clipping Higher PPO and GRPO employ clipping
the importance sampling ratio to restrict
the trust region (Schulman et al. (0113) and enhance RL stability (Schulman et al.) 2017a Shao et al.
2024). However, the upper clipping threshold restricts the probability increase of owwprobbbillity tokens, thereby potentially constraining exploration. Following DAPO (Yu et al. 2025). we modify
the clipping range in the GRPO training objective from [0.8, 1.2] to [0.8, 1.28).
Higher Rollout Temperature Recent works on LLM RL adjusting the rollout temperature to
promote exploration have been widely shown be effective, with sampling higher temperatures yielding particularly notable improvements (An et al. 2025 Liu et al. [025d]] Liao et al.) (025). To encourage the VLA model to generate more diverse trajectories during the rollout phase, we iecrasse the sampling temperature from 1.0 to 1.6. As shown in Figure 3 these modifications led to notable improvements.
2.4 TRAINING OBJECTIVE
We use the adopted GRPO algorithm (Shao et al. 2024) for online RL training on VLA models, with modifications as introduced in Section] 2.3 Moreover, we remove the KI. divergence regularization
4

---

Published as a conference paper at ICLR 2026
15%
15%
111 Long SR
HIRTRO Leeg SRt
118ER00 Long SR
Symgnin-Samginng
Clty Higher
we Temperature Higher
wis Dynumic Sampling
wio Clip Higher
we Temperature Higher
20
100
:50
202
253
ato
30
100
205 Training Stops
:50
253
ato
50
100
250
20:
250
30o
RI. Training Steps
RI. Training Steps
(a) Dynamic Sampling
(b) Clip Higher
(c) Higher Rollout Temperature
Figure 3: The effectiveness of three key enhancements: dynamic sampling, higher rollout tempera- ture, and clip higher.
following DAPO (Yu et al. 2025). This eliminates the need for reference model during training, reducing memory consumption and accelerating the training. Additionally, the KL penalty con- strains policy divergence from fixed reference, potentially limiting exploration of behaviors. Therefore, the policy is optimized via the following objective:
lad min (re)(0)A,, clip (r),,0),, 1-e,, 1 1+8n) A.)
[(0) (0))=BspDDaa sD -ees (1ss))
G
s.t. 0 < (traj, (a;, 8;): (access/trajj(ass <G
(3)
where
To St.e)
A, R: mean( {Ri
(ean(
(8))(0)
(4)
Tew (as,e si,e)
std (Ra)
Table 1: Main results of different VLA models on RoboTwinI.0.
Tobn10
Model
Hammer Beat
Block Handover
Blocks Stack
Shoe Place
Avg
DP
0.0
12.0
7.1
4,3
5.9
DP3
64.7
84.3
24.0
59.3
58.1
OLEFVLA-OFT
67.2
61.6
23.4
39.8
w/ ours
92.6
89.6
40.2
59.3
70.4
+25.4
+28.0
+33.1
+33.99
+3006
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Benchmarks We evaluate SimpleVLA-RI on three simulation benchmarks L.BERO (Liu et al,, 2023), RoboTwinl.0 (uu et al. 2025). and RoboTwin2.0 (Chen al.) 2025a), and conduct real- world experiments on RoboTwin2.0 tasks. LIBERO is a language-guided manipulation benchmark with five task suites: LIBERO-Goal, LIBERO-Spatial, LIBERO-Object, LIBERO-Long (10 tasks each with 50 demonstrations). and LIBERO-99 -00 (90 tasks for large-scale evaluation). Performance measured by average Success Rate (SR) across 50 held-out test scenarios per task. RoboTwin 1.0 provides 17 bimanual tasks, while RoboTwin22. extends to 50 tasks with 731 object instances and comprehensive domain randomization (clutter, lighting, background, tabletop height, language structions), enhancing task diversity and sim-to-real transfer. For Piper robotic arm and domain-randomize test scenarios. We select 12 tasks in RoboTwin2.0 and categorize them into 4 horizon levels based on average step counts, as detailed in Table
the Agilex task settings, with each task evaluated on 100 held-out
We apply Lim-leVLA-RL to OpenVLA-OFT (Kim et al. 2025), state-of-the- art regressegeesive VLA model with high performance and inference efficiency. Built on Open- VLA (Kim al. 2024), uses vision encoders and I.AMMA-7BB (Touvron al. 2023 as the
Backbones

---

Published as a conference paper at ICLR 2026
Table 2: Main results of different VLA models on RoboTwin2.0, organized by task horizon.
Short Horizon Tasks (100-130 Steps)
Model
Lift Pot
Beat Hammer Block
Pick Dual Bottles
Place Phone Stand
Avg
51.0
59.0
50.0
22.0
45.5
RDT
22.0
18.0
24.5
OLenVLA-OFT
28.1
29.7
17.1
21.3
w/ ours
64.1
87.5
68.3
39.6
64,9
+54.0
+59.4
+-38.6
+ +.22.5
+-3.6
Medium Horizon Tasks (150-230 Steps)
Model
Move Pot
Place A2B Left
Place Empty Cup
Handover Mic
Avg
T0
84.0
96.0
RDT
33.0
21.0
42.0
95.0
47.8
OpenVLA-OFT
28.1
37.5
77.3
45.3
47.1
w/ ours
61.2
45.3
94.2
89.2
72.5
+33.1
+7..8
+16.9
+-13.9
+25.
Long (280-320 Steps) & Extra Long Horizon Tasks (450-650 Steps)
Model
Handover Block
Stack Bowls Two
Blocks Rank Rgb
Put Bottles Dustbin
39.0
53.0
45.0
54.0
47.8
RDT
26.0
42.0
17.0
26.0
27.8
OpenVLA-OFT
33.1
40.6
70.2
42.2
46.5
57.8
75.8
81.3
60.9
+21.7
+-352
+11.1
188.7
22
Overall Avg
RDT: 33.3
mgg 52.7
Open 38.3 w/ ours: 68.8
+30.5
backbone with action chunk and parallel decoding, making suitable for online RL's frequent infer- ence requirements. Our implementation of the Open VLA-OFT differs from the official version (see
Appendix G.2 for modifications and Appendix G.3 for hyperparameters).
Baselines
We compare with advanced VLA models: UniVLA [Bu et al. 2025b), RDT-1B Liu
al.
2024), my (Black al.) 2024), Taat
(Pertsch et al.
2025), Nora (Hung al. 2025). Open-
VLA
Kim et al.
2024). Octo (Team et al.
2024). DP (Chi et al. 2024 and DP3 (Ze al. 2024
3.2
MAIN RESULTS
We evaluate SimpleVLA-RL. on LIBERO. RoboTwin1.0, and RoboTwin2.0 using two-stage paradigm: SFT followed by SimpleVLA-RL. on OpenVLA-OFT, while baselines use SFT only. For LIBERO's four task suites, we perform SFT with 500 demonstrations per task suite, then RL. on 500 simulation scenarios. For RoboTTwinl. .0, we use 50 demonstrations per task for single-task SFT, then 100 scenarios per task for RL. For RoboTwin2.0, we use 1,000 demonstrations per task for single-task SFT, then 1,000 scenarios per task for RL.
Tables
2 and present results on LIBERO, Twin1..n and
Table 3: Main results on LIBERO.
RoboTwin2.0 benchmarks. On LIBERO. SimpleVLA-RL improves OpenVLA- OFT from 91% to 99% average success rate, achieving SoTA performance and surpassing models like and UniVLA. For long-horizon tasks in LIBERO- Long, SimpleVLA-RL reaches 98.5% success rate, with a 12% improve- ment over baseline and 13.3% over To. On RoboTTwil 1.0's dual-arm tasks, SimpleVLA-RI. achieves 30.6% gains (39.8% to 70.4%). Across RoboTwin2.0's 12 tasks. SimpleVLA-RL delivers 80%
LIBERO
Model
Spatial
Object
Goal
Long
Avg
Octo
78.9
85.7
84.6
51.1
75.1
OLenVLA
84.7
88.4
79.2
53.7
76.5
Nora
92.2
95.4
89.4
74.6
87.9
No FAST
96.4
96.8
88.6
60.2
85.5
Tо
96.8
98.8
95.8
85.2
94.2
UniVLA
96.5
96.8
95.6
92.0
95.2
Open VLA-OFT
91.6
95.3
90.6
86.5
91.0
w/ ours
99.4
99.1
99.2
98.5
99.1
A
+7.8
+3.8
+8.6
12.0
+8.1
relative improvement (38.3% to 68.8%), outperforming SoTA methods including To (52.7%)
and RDT (33.3%). Even Extra-Long- Horizon tasks like "Blocks Rank Rgb" and "Put Bottles Dustbin", SimpleVLA-RL. achieves 11.1% and 18.7% point gains respectively, demonstrating effectiveness across all horizon levels. These results validate that SimpleVL.A-RI consistently

---

Published as a conference paper at ICLR 2026
improves model performance across diverse benchmarks without requiring additional demonstration data, proving the effectiivness of outcome- rewards even for complex oanghorizon tasks.
ANALYSIS
In this section, we analyze the role of SimpleVLA-RL in addressing three key challenges that hinder the further advancement and scaling of the VLA model: data, generalization, and real- world tasks. Below are several key takeaways:
Takeaways
1. Data: Simple LLA-RL can significantly reduce reliance on demonstration data, effec- tively alleviating the data scarcity bottleneck that constrains VLA scaling (8 4.1
2. Generalization: Compared to SFT, SimpleVLA-RL. demonstrates strong generalization in spatial configurations, object types, and task settings ($ 4.2)
3. Real-world Task: SimpleVLA-RI. exhibits strong sim-to-real transfer, with large-scale
simulation training remarkably improving real-world performance, indicating a promising
path for scaling up real-world policy ($ 4.3).
4.1 OVERCOMING DATA SCARCITY
Developing foundation VLA models for manipulation tasks requires large-scale demonstration data
for training (Liu et al. 2024 Black et al. 2024 Intelligence al. 2025). This data scaling paradigm
has been proven in the NLP area (Hoffmann et al.) [2022] Achiam et al. 2023 Touvron et al.) 2023).
However, acquiring high-quality trajectory data for embodied manipulation tasks remains expensive
and difficult, creating fundamental bottleneck for VLA model development (Bi et al.
2025 Zhong
al. 2025). Therefore, we investigate whether SimpleVLA--E can enhance VLA models even with extremely limited demonstration trajectories to overcome this limitation.
Settings To simulate scenarios with scarce demonstration data, we finetune VLA- using only one demonstration data per task, denoted as One- One-Trajectory SFT. Given that each of the four LIBERO task suites contains 10 distinct tasks, we utilize merely 10 demonstration data per task suite. For comparison, we also conduct an experiment using all available demonstration data for each task, 500 per task suite, denoted as Full-Trajectory SFT. Following both One- ene-Trujectory SFT and Full-Trajectory SFT, we apply SimpleVLA-RI. on the SFT model.
Results As shown in Table SFT per- formance degrades significantly with lim- ited data. Under One-Trajectory SFT, suc- cess rates drop below 63.6% for LIBERO- Spatial/Objeet/Goall and to only 17.3% for LiBERO-Long compared to 91.0% aver- age under Full-Trajectory SFT. Remarkably, applying SimpleVLA-RL One-Trajectory SFT models increases the average success rate from 48.9% 96.9%, surpassing even Full-Trajectory SFT's 91.0%. LIBERO- Long improves dramatically from 17.3% to 91.7%, while the other three task suites all exceed 98%. The performance gap between Tne-Trajectory SFT RL (96.9%) and Full-Trajectons SFT + RL (99.1%) 1%) is merely 2.2%. The results demonstrate that SimpleVLA-RI can substan- tially improve performance even in data-scarce scenarios, suggesting that online RL enables further scaling of VLA training through trial-and-erro exploration, even with minimal demonstration data.
Table 4: Comparisons between One Trajectoy and Trajecrayyy SFT on LIBERO.
LIBERO
Model
Spatial
Object
Goal
Long
Avg
One- Trujectory SFT
OpenVLA-OFT
63.6
54.9
59.6
17.3
48.9
w/ ourr
98.2
98.7
98.8
91.7
96.9
A
+34.6
+43.8
+39.2
-74.4
+48.0
Full Trajectory SIT
OL--VLA-OFT
91.6
95.3
90.6
86.5
91.0
w/ ours
99.4
99.1
99.2
98.5
99.1
+7.8
+3.8
+8.6
12.0
+8.1
4.2 GENERALIZATION ANALYSIS
The generalization ability of VLA models remains a key challenge (Intelligence et al. 2025 Zhong et al. 2025 Liu et al. 2025a). This subsection evaluates how SFT and online RL methods like SimpleV VLA- affect VLA generalization across three dimensions: spatial (LIBERO- (aRRO--paaial,...... ob- jects LLIEERO-Objec and tasks (LDBERO-Goal).

---

Published as a conference paper at ICLR 2026
Goal Unseen Task put the wine bottle on top oabinet
Goal Unseen Task 2
Goal Unseen Task put the wine bottle on
Unseon Tack SR (N)
40
put creams cheese the bowl
Unsoon Task SR (N)
Unseon Task SR
100
20
40
RL
RL
RL
SFT
20-
SFT
SFT
T6o
201 Seen Tasks SR (%)
40
Tin
60
76
40
T60
Seen Tasks SR (%)
Seen Tasks SR (%)
Object Unseen Task
Object Unseen Task 2
Object Unseen Task
Unseen Taok SR (%)
pack up the cream ohreae and piace ooster
Unseea Taak 190
ed
ploce up the sclec dreamms and place the bevtet
Unseea Taek SR (%)
80
uree the milk and siace it the baiee
60
RL
RL
23
SPT
40
SFT
RL
SFT
01
50
80
90
60
80 Seen Tasks SR (%)
100
100
Seen Tasks SR (%)
Seen Tasks SR (%)
Spatial Unseen Task
Spatial Unseen 2
Spatial Unseen Task 3 plck up Block bosl on the ronskiin
Unsoon Task SR (%)
6d
blrck bosel betwees the piate romelón
pick up Lle black bowl nest to the romsiii
Unseen Task SR (%)
50
RT.
Unseen Tack
RI.
50
RI
50
SFT
SPT
25
SFT
Seen Tasks SR (%)
Seen Tasks SR
Seen Tasks SR (%)
Figure 4: Generalization Analysis on LIBERO: Goal Unseen (Top), Object Unseen (Middle), Spa- tial Unseen (Bottom).
Settings We experiment three LIBERO task suites (Spatial, Object, Goal), each containing ten tasks. For each suite, we randomly select nine tasks as seen tasks for RL or SFT training, while reserving the remaining task as the unseen task for out-of-distribution evaluation. For both methods, we first fine-tune OpenVLA-OFF under the One-Trajectory SFT setting to obtain base model with non-zero success rates, since the original model achieves 0% on LIBERO and incapable of performing online RL. For SFT, we further fine-tune the One-Trajectory SFT base model ( using 450 demonstrations from seen tasks on each task suite. For RL, we perform SimpleVLA- RL on the One- Trajectjcy SFT base model using 450 scenarios from 9 seen tasks. We plot how unseen task performance evolves as training task success rates increase during training.
Results Figure 4 presents the results. While both SFT and RL achieve over 90% success rates on training tasks, their performance on unseen tasks diverges significantly. training progresses, SimpleVLA-RL. shows consistent improvement unseen tasks across all settings, whereas SFT suffers from severe overfitting, often experiencing catastrophic forgetting with success rates of un- seen tasks dropping to 0%. On LIBERO-Goal, SFT immediately drops 0% all three unseen tasks at training onset, likely because these tasks involve diverse objects and manipulation strate- gies with minimal transferable components. In contrast, SimpleVLA--R maintains performance and achieves 5%-15% improvements. On LIBERO-Object, SFT improves only on Unseen Task (57.8% to 74.6%) while failing the other two. SimpleVLA-RR. improves across all three tasks, with notable gains of 36.5% on Task 2 and 16.4% on Task 3. On LIBERO-Spatial, SFT degrades by 10% on Unseen Task 1 and completely fails on the remaining tasks, while SimpleVLA-RL. im- proves Task performance from 43.3% to 71.8% and achieves 7.1% % and 13.3% gains on the other tasks. These results demonstrate that RL training enables VLA models to retain previously acquired capabilities while learning generalizable skills from diverse tasks.
4.3 REAL-WORLD EXPERIMENTS
Table 5: Real-world experiment (sim2real) results.
Stack Bowls
Place Empty Cup
Pick Bottle
Click Bell
Avg
RDT
60.0
4.0
10.0
20.0
23.5
Open VLA-OFT
38.0
2.0
0.0
30.0
17.5
w/ ours
70.0
10.0
14.0
60.0
38.5
A
+32.0
+8.0
+14.0
+30.0
+21.0
8

---

Published as a conference paper at ICLR 2026
To evaluate the real-world effectivenes. of SimpleVLA-RL. conduct sim-to-real experiments four RoboTwin2.0 tasks (detailed in Appendix Stack Bowls, Handover Block, Pick Bottle, and Click Bell. We employ Open OF as the policy model, RDT as the baseline model, and execute on two AgileX Piper robotic arms. For each task, we first use 1000 simulation trajectories for SFT. Then we apply VimpleVLA--B model. The entire training process uses only simulation data without any real-world demonstrations. We evaluate both the SFT and RL models on clean tabletops with unseen backgrounds in the real world. Each task is tested with 50 trials. The RDT baseline model only undergoes the SFT stage.
the SFT model using 1000 simulation scenarios obtain an RI.
The sim2real results in Table 5 demonstrate that SimpleV world success rates of VLA models, with an average improvement from 17.5% to 38.5%, surpassing RDT's 23.5%. For instance, in the Stack Bowls task, SimpleVL.ARI. achieves a 84% relative im- provement, lifting performance from 38% to 70%_On the Pick Bottle task, which demands higher action precision, as the bottle will fall if the robotic arm not perfectly aligned on the first tempt, the SFT model fails completely while VLA-RL achieves a 14% success rate. This demonstrates RL's effectiveness in improving action precision. Using SimpleVLA-RI for low-cost, large-scale, and highly parallel RL training in simulation, significantly improve the real-world performance of simulation-trained VLA models. This demonstrates a promising path for scaling real-world policies: using rich simulation assets and high-fidelity simulators for cost-effective RL. training to achieve superior performance in real-world deployment.
significantly improves the real-
5 DISCUSSIONS
5.1 "PUSHCUT": EMERGENCE OF NEW PATTERNS THROUGH RL
Task: Move Can Pot
Task: Place A2B Right
SFT
Grasp
SFT
Grasp
RL
RL
sh
(a) "move can pot" task: Model learned to push the can to the pot (bottom) instead of grasp-move-place in the demonstration data (top).
(b) "place a2h right" task: Model learned to push to B's right (bottom) instead of demonstrated grasp- move-place (top).
Figure 5: Illustration of "pushcut". Emergent pushing behaviors through RL in RoboTwin2.0 tasks.
During RL training with SimpleVLA-RRE we observe an emergent behavior we call "pushcut" (a push-driven shorteut), where the VLA model discovers novel strategies absent from the demonstra- tion data. In the move can pot task of Robo.win2.0,, all demonstrations follow a grasp-move-place strategy (Figure 5al top). However, after RL training, the model autonomously discovers a more efficient solution: directly pushing the can to the target location instead of grasping it (Figure Ba bottom). Similar behaviors emerge the place a2b left/right task. where the RL-trained model learns to push Object A into position rather than following the demonstrated grasp-move-place ap- proach (Figure 5b).
This "pushcut" phenomenon parallels the "Aha Moment" in DeepSeek-R1 (Guo et al. 2025a). as both emerge through RL-drrvee exploration. This phenomenon highlights the fundamental distinc- tion between SFT and RL. While SFT merely replicates patterns from demonstrations, RL enables the discovery of novel strategies through reward-driven exploration. The outcome-leval reward de- sign is crucial here: since both grasping and pushing receive equal rewards upon task completion, the sparse reward structure avoids procedural constraints, allowing the agent to explore freely and discover unexpected yet effective solutions.

---

Published as a conference paper at ICLR 2026
5.2 FAILURE MODES OF SIMPLEVLA-RL
We conduct ablation studies to identify failure conditions and key influencing factors of SimpleVLA-RL (see Appendix
C for full results and analysis). Our experiments reveal that model priors the critical factor determining effectiivessss RL fails completely when the base model lacks initial task capability (0% success rate). Furthermore, we find a performance threshold: when initial success rates too low (< 5%), RL improvements remain negligible, while stronger initial models achieve substantially better final performance after RL training.
CONCLUSION
In this work, we present SimpleVLA-RL, RL framework tailored for VLA models that extends veRL with VLA-specific trajectory sampling and parallelized ieannnggiifeeenee-eeeee bilities. SimpleVLA-RL. demonstrates significant improvements in data efficiency, generalization, and sim-to-real transfer. These results across LIBERO and RoboTwin benchmarks highlight RL's potential to both alleviate data scarcity and substantially enhance VLA generalization, paving the way for more autonomous and adaptable robotic models.
capa-
ACKNOWLEDGEMENTS
This work supported by National Science and Technology Major Project (2023/DD012 1403), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001) National Natural Science Foundation of China (No. 62406165), and Shanghai Municipal Science and Technology Major Project. We thank anonymous reviewers for their insightful comments and suggestions.
ETHICS STATEMENT
This work presents SimpleVLA-RL, reinforcement learning framework for improving Vision- Language-Action models in robotic manipulation. Our research aims to advance autonomous robotics for beneficial applications in manufacturing, healthcare, and assistive technologies. We conduct experiments exclusively in simulation environments and controlled laboratory settings with standard manipulation tasks, ensuring safe development practices. Our approach reduces reliance on large-scale human-operated data collection, minimizing both human labor costs and potential safety risks associated with extensive teleoperation. By enabling more sample-efiiciet training and bet- ter generalization, VLA-Re promotes environmentally conscious research through reduced computational requirements compared to scaling supervised learning alone, All experiments use publicly available benchmarks and models to ensure transparent, reproducible research.
REPRODUCIBILITT STATEMENT
We provide comprehensive details to ensure reproducibility of our work. The complete algorith- mic formulation of SimpleVLA-RL. and training procedures described in Section 2 including trajectory sampling, loss computation, and exploration strategies. All experimental configurations, model hyperparameters, hardware specifications, and robotic arm setups are detailed in Appendix G We provide implementation specifics built upon the open-source veRL framework, evaluation pro- tocols for LIBERO and RoboTwin benchmarks, and baseline comparisons. Additionally, we include ablation study configurations and real- wol-w deployment settings. All mathematical formulations, implementation details, and experimental configurations necessary for reproducing our results are included in the paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint S3X3:233330877222 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn ing from human feedback in lms. arXiv preprint arXiv:2402 /4740, 2024.
10

---

Published as a conference paper at ICLR 2026
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: post-training recipe for scal- ing reinforcement learning advanced reasoning models, 2025. URL httts://hkuun gi github. i httpl///oooo////ooleeeee
Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, and Jun Zhu. H-rdt: Human manipulation enhanced bimanual robotic manipulation. arXiv preprint 2rXiv:250..23523, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi.0: A sision-languge--enn flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.
Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task spec- ifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5920-5922 2023.
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06665 2025a.
Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint 20Xx::2505.06111, 20255..
Chilam Cheang, Sijin Chen, Zhongren Cui, Yingdong Hu, Liqun Huang, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Xiao Ma, al. Gr-3 technical report. arXiv preprint arXiv:2507. /5493, 2025.
Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Qiwei Liang, Zixuan Li, Xi- anliang Lin, Yiheng Ge, Zhenyu Gu, et al. Robotwin 2.0: A scalable data generator and bench- mark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506. 180888 2025a.
William Chen, Suneel Belkhale, Suvir Mirchandani, Oier Mees, Danny Driess, Karl Pertsch, and Sergey Levine. Training strategies for efficient embodied reasoning. arXiv preprint arXiv:2505.08243, 2025b.
Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Con- rft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.054450. 2025c.
Zengjue Chen, Runliang Niu, He Kong, and Qi Wang. Tgrpo: Fine-tuning vision-language-action model via trajectory-wiss wise group relative policy optimization. arXiv preprint arXiv:2506.08440, 2025d.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22611 2025b.
Gustavo De Rosa and Joao P Papa. A survey on text generation using generative adversarial networks. Pattern Recognition, 119:108098, 2021.

---

Published as a conference paper at ICLR 2026
Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models robotics: Ap- plications, challenges, and the future. The International Journal of Robotics Research, 44(5): 701-739, 2025.
Jensen Gao. Annie Xie, Ted Xiao, Chelsea Finn, and Dorsa Sadigh. Efficient data collection for robotic manipulation via compositional generalization. arXiv preprint arXiv:2403.05 10, 2024.
Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, et al. Roboverse: Towards a unified plat- form, dataset and benchmark for scalable and generalizable robot learning. arXiv preprini arXiv:2504. 18904, 2025.
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025a.
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving vision-language-action model with online reinforcement learning. arXiv preprint arXiv:2501. 2025b.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 11X11:1910.11956, 2019.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Jordan Hoffmannn, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang. Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv: 148. 2024.
Chia-Yo Yu Hung, Qi Sun, Pengfei Hong. Amir Zadeh, Chuan Li, Tan, Navonil Majumder, Soujanya Poria, al. Nora: A small open-sourced generalist vision language action model for embodied tasks. arXiv preprint arXiv:2504. 19854, 2025.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 4064-5):698-721, 2021.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, al. pi {0.5): lasion-language- action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 24Xiv:2412.16777 2024.
Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. arXiv preprint 2rXiv:2410.241166 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint (cKv::2006.09246, 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning language- models: Opti- mizing speed and success. arXiv preprint arXiv:2502. 2025.
12

---

Published as a conference paper at ICLR 2026
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829, 2023.
Oliver Kroemer, Scott Nickum, and George Konidaris. review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of machine learning research, 22(30): 1-:00)--82.. 2021.
Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, and Huaiyu Wan. Enhancing efficiency and exploration in reinforcement learning for llms. arXiv preprint arXiv:2505. 18573, 2025.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang. What can rl bring vla generalization? empirical study. arXiv preprint arXiv:2505. 19789, 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprini 25X5::2505.24844 2025b.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-lb; lb: diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.0786/ 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078: 2025c.
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee. Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Accreason-nemottron 1.1: Advancing math and code reasoning through sft and synergy. arXiv preprint arXiv:2506. 13284, 2025d.
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2.005. 18719, 2025.
Jianlan Luo. Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine. Serl: software suite for sample-efficient robotic reinforcement learning. arXiv preprint arXiv:2401.16011 2025.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via malue-implicit pre-training. 2022.
arXiv preprint arXiv:22
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
Yao Mu, Tianxing Chen, Shijia Peng. Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (early version). In European Conference on Computer Vision, pp. 264-273. Springer, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Abby O'illl,, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 JEEE International Conference Roboties and Automation (ICRA), pp. 6892-6903. IEEE, 2024.
13

---

Published as a conference paper at ICLR 2026
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- ssioo-language-actioo models. arXiv preprint arX1v:2501.09747, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Findings of the association for computational linguistics: EMNLP 2024, pp. 7346-7356, 2024.
Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 661-668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https:
//proceedings.. mw.. wwt.:isssss.ppee./911 http://////ooso.......... html
Ranjan Sapkota, Yang Cao, Konstantinos Roumeliotis, and Manoj Karkee. Vision-language-action models: Concepts, progress, applications and challenges. arXiv preprint arXiv:2505. 04769, 2025.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017b.
17377
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
Junyang Shu, Zhiwei Lin, and Yongtao Wang. Rftf: Reinforcement fine-tuning for embodied agents with temporal feedback. arXiv preprint arXiv:2505. 2025.
Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl. Interactive post-training for vision- language-action models. arXiv preprint arXiv:2505. 2025.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing into the physical world. arXiv preprint arXiv:2503..0020, 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501.12599 2025b.
Octo Model Team, Dibya Ghosh. Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 240..2213... 2024.
RLinf Team. Rlinf: Reinforcement learning infrastructure for agentic ai. htttss://iithe
com/RLinf /R.. R 2025. GitHub repository.
14

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307700228. 2023.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning- In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 7601-7614, 2024.
Yihao Wang, Pengxiang Ding, Lingxiao Li, Can Cui, Zirui Ge, Xinyang Tong, Wenxuan Song, Han Zhao, Wei Zhao, Pengxu Hou, Siteng Huang. Yifan Tang, Wenhui Wang, Ru Zhang, Jianyi Liu, and Donglin Wang. Vla-adapter: An effective paradigm for tiny-scale vision-language-action model. arXiv preprint arXiv:2509.09372 2025.
Zheng Wu, Wenzhao Lian, Vaibhav Unhelkar, Masayoshi Tomizuka, and Stefan Schaal. Learning dense rewards for contact-rich manipulation tasks. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 6214-6221. IEEE, 2021.
Charles Xu, Qiyang Li, Jianlan Luo, and Sergey Levine. Rldg: Robotic generalist policy distillation abs via reinforcement learning, 2024. URL httpp:://axxi.... wttp://gggaaaa.......o/ 000pa//411.999
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505. 200000080.9888 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open- IIm reinforcement learning system at scale. arXiv preprint 2rXiv:2503.144:1 2025.
Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learning. arXiv preprint arXiv:24077 15815, 2024.
Michat Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of tf-toughtt reasoning. arXiv preprint urXiv:2407.08693 2024.
Yanjie Ze, Gu Zhang. Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint MrXi::2403.03954. 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simpleri- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Hongyin Zhang, Zifeng Zhuang, Han Zhao, Pengxiang Ding, Hongchao Lu, and Donglin Wang- Reinbot: Amplifying robot visual-language manipulation with reinforcement learning. arXiv preprint 1iXiv:2505507395 2025.
Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang. Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19309 2024.
Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, et al. A survey labuullanguag--atttin models: An action tokenization perspective. arXiv preprint arXiv:2507.01925. 2025.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762. 2025.
Yuxin Zuo Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
15

---

Published as a conference paper at ICLR 2026
A
RELATED WORKS
A.1 REINFORCEMENT LEARNING FOR LARGE LANGUAGE MODELS
Reinforcement Learning (RL) for Large Language Models (LLMs) has achieved remarkable suc- cess, demonstrating its ability to induce complex reasoning behaviors such as selffverification and iterative optimization, thereby significantly enhancing model performance on reasoning tasks (Guo
et al.
2025a Jaech et al., 2024; Liu et al., 2025c Cui et al. 2025a Zeng et al. 2025 Zuo et al. Recent advancements in Large Reasoning Models (LRMs), such as DeepSeek-Rl RI (Guo et al,,
2025
2025a). highlight the effectiveness of RL in boosting reasoning capabilities even with simple rule- based rewards, as exemplified by GRPO (Shao et al., 2024). This approach differs substantially from Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), which aligns base S
models with human preferences using algorithms like Proximal Policy Optimization (PPO) (Schul-
man et al. 2017a) and heavily relies preference modeling.
Recent studies have increasingly focused on enhancing exploration in reinforcement learning to en- able longer training horizons and improved performance. DAPO (Yu et al., 2025) introduces Clip- Higher, a decoupled variant of PPO clipping, which sets a higher upper bound relative to the lower one (e.g.. 8r. 0.2, Eн 0.28). This adjustment allows low-likelihood but potentially valuable tokens to increase in probability, thereby encouraging exploration. Building on this, POLARIS (An) al.] 2025) employs a staged curriculum of temperature increases (c.g.. 0.7 + 1.0 - 1.1 for 7B model) to gradually expand trajectory diversity and facilitate more robust policy discovery. In parallel, Entropy Mechanism (Cui et al. 2025b) addresses entropy collapse, a persistent issue in extended training, through methods such as Clip-Cov and KL-Cov, which selectively clip probabili- ties or penalize high-covarian covariiaee tokens to sustain effective exploration. Similarly, ProRL (Liu al.) 2025b) combines KL control with reference policy resetting to preserve stability and extend training without degrading performance. A complementary line of work regulates entropy via temperature tuning. Acereason-nemotuue 1.1 (Liu et al. 2025d) advocates adjusting temperatures stabilize post-scaling entropy around a target (c.g., 0.3), balancing exploration and exploitation. [iao et al.. (2025) further proposes a dynamic scheduler that adapts temperature over time maintain stable entropy, thereby supporting sustained performance gains.
A.2 VISION LANGUAGE ACTION MODELS
In the field of robotic manipulation tasks, VLA models (Kim et al., 2024 2025 Liu et al. 2024 Bu et al. 2025b: Hung et al.] 2025; Black et al. 22224] Pertsch et al., [025 Intelligence et al. 2025) have shown better performance and task generalization compared to traditional policy-based approaches (Ma et al. [022]] Yuan et al. 2024). These models integrate the VLM or LLM back- bone with action modules through unified end-to-end training (Zhong et al.] 2025). This approach enables comprehensive multimodal understanding and fine-grained motor control (Firozzi et al.) 2025). Currently, many studies are focused on enhancing the effectiveness VLA models. For ex- ample, E-COT (Zawalski et al., 2024 Chen et al.) 2025b) introduced Embedded Chain of Thought (ECoT) to improve the spatial reasoming ability of VLA models. RDT-IB and VPP (Liu et al. [024]] Hu al. (204)) proposed diffusion-based frameworks for VIA models. Agibot world and Robo- verse (Geng et al. 2025 Bu et al.) 2025a) aim to build larger-scale simulation environments and tra- jectory datasets improve the ti-rel transfer and generalization capabilities of VLA models. Additionally, Dexmimicgen (Jiang et al. 2024) explores automated methods to generate high-quality trajectory data address the issue of data scarcity in robotics. Despite the rapid advancements in the VLA domain, imitation learning remains the dominant training paradigm for VLA models (Sap- kota et al. (2025) Kim al.) 2024 2025 Liu et al.) 20244) Bu al., 20256 Hung et al. 2025 Black et al. 20244] Pertsch et al. 2025 Intelligence et al. 2005). Current VIAA models typically follow
a two-stage paradigm: pretraining on multimodal data (e.g., Open X-Embodiment (O Neill et al.,
2024h) followed by SFT on collected robot trajectories. However, imitation learning is limited by its dependence on expensive, high-quality trajectory data and poor generalization to unseen scenarios.
VLA RL Methods Recently, some efforts have attempted to apply RL to VLA training. GRAPE (Zhang et al., 2024) utilized Direct Preference Optimization (DPO) (Rafailov et al. 2023) to train VLA models by integrating human preferences. ConRFT (Chen al. (025c) introduced Reinforced Fine-Tunie Tuning (Trung et al. 2024) to train VLA models in woald environments, it- eratively training VLAs through alternating RL and SFT rounds. ReinboT (Zhang al. 2025) fo-
16

---

Published as a conference paper at ICLR 2026
cused on dense reward design and optimized VLA models through reward maximization. Guo al. (2025b) proposed an iterative training framework that combines Supervised Fine-Tuning (SFT) and RL stages address training instability and computational overhead. More recent works have fur- ther advanced VLA RL methods. Concurrently, RIPT-VLA (Tan et al. 2025) investigates a closely related problem, employing RLOO (Ahmadian et al.) 2024) for VLA RL. training. Moreover, Liu iet al. (2025a) investigates RL's impact on VLA generalization capabilities, demonstrating signifi- cant improvements over SFT in unseen environments, objects, and textures. RLinf (Team) (025)
designed flexible, scalable framework for VLA RL that unifies rendering, inference, and training, improving both VLA training efficiency and performance. VLA-RL (Lu al., 2025) applies the PPO algorithm to the VLA model. TGRPO (Chen et al. 2025d) uses Claude3.7 to evaluate trajec- tories and optimizes VLA with GRPO. RFTF (Shu et al. 2025) uses value models to generate dense rewards in embodied scenarios for VIA online RI.. Compared to the above works, our paper further explores the effectiveness of VLA RL on real-world robotic tasks. We also conduct comprehensive analyses on how VLA RL addresses data scarcity challenges and improves policy generalization.
B PRELIMINARIES
To provide an intuitive illustration of the existing gap when extending RL methodologies from LLMs the VLA domain, we formalize RL for both LLMs and VLA models, presenting their state repre- sentations, action spaces, reward functions, and environments in this section.
B.1
RL FORMULATION FOR LLMS
State (st): At step f, the state s comprises the input prompt and previously generated tokens:
(Trrompt V1, ./.....> 11-1),
(5)
where prompp denotes the initial prompt and y/ denotes the t-th generated token.
Action (az): An action corresponds to selecting the next token from the vocabulary V. At each step, the policy outputs probability distribution over tokens, and the action token is selected via random sampling. Formally, the action is defined as:
= Vf E V, where Y/ > mo(- = softmax (Jo(s))/T)
where fo(st) E R VI represents the LLM logit outputs and is the temperature parameter control- ling the randomness of sampling.
The environment provides reward signals upon sequence completion. In rule-based settings, binary rewards are assigned based on the correctness. Alternatively, learned reward mod- els or human feedback systems provide continuous rewards based on criteria such as helpfulness, harmlessness, or task alignment. The reward computed as follows:
Environment:
1,
if 7 satisfies correctness criteria
r(r)
r(7) RT(T) [0, 1].
0, otherwise
where Rs is learned reward model and T (Iprompts V1, 1/2 generated sequence of length Taqg
VT represents the complete
Given an input prompt Trrompt: the L.M auto-regressively generates sequence by sam- pling tokens from Ty (3els:) until termination, without intermediate environmental feedback. With a non-zero temperature T, the policy can produce diverse rollouts that explore different solution paths.
Rollout:
B.2 RL FORMULATION FOR VLAS
State (st): The state consists of multimodal observations including visual input (RGB images, depth maps, point clouds). proprioceptive information (joint angles, end-effector pose), and language instructions of the tasks. Formally, the state is defined as:
St (o of prop Iusk);
where of multimodal observations, of is proprioceptive information, and Lank is language in- structions.
17

---

Published as a conference paper at ICLR 2026
Action (at): Actions control commands in the robot action space, typically end-effector deltas or joint angle targets, where a, E IR" (e.g., d d 7 for 6-DoF pose plus gripper position). Most VLA policies generate actions through either Iettusioonbbseee action expert discrete action tokenizer. The action is defined as follows:
dr = Decoder (h))s)))
Decoder (Diffusion Expert, Action Tokenizer), at R
where ho(8t) represents the hidden state of 8, in the VLA model, and Decoder the action decoder.
Environment: The environment represents the physical world or simulation where the robot oper- ates. provides state transitions st+1 1 Env(st. (e: 0e) and reward signals:
{1,
if task success (10)
= a a Inccess + (1- Σ wi (s at). E 1], Iaccesss
00 otherwise
where (5, at) represents process rewards (e.g. distance goal), wa weights, and balances outcome and process rewards.
Rollout: VLA models generate trajectories through iterative interaction with the environment. At each timestep, the policy Te takes the current state 8, as input and outputs an action chunk (a,, d1+1y..., 014 of length k. The robot executes these actions sequentially and the environ- ment produces updated states based on physical dynamics. After execution, the model takes the new state St-k as input and generates the next action chunk. This process continues until task completion or maximum episode length, producing a complete trajectory T = ((so, oo), (31, (1),....................... (ST, QT)) through interactive sampling.
B.3 GROUP RELATIVE POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO) (Shao al. 2024 is RL method that eliminates the value function by computing advantages through group-relative normalization. Given an initial state so, the behavior policy Tés generates G trajectories bn) The GRPO objective employs PPO-style clipping with KL regularization to constrain policy updates:
JGRPO Eeg DDrri Arihns
min (1n
T
(11)
3 ( (TolI
where the importance sampling ratio ri,)(0) and the normalized advantage A, are defined as:
To TT((n,e(ss.)) To (0t, (81.))
A R4 - mean me (
(0) =
(12)
acd((R,)
Here R2 denotes the total reward of the i-th trajectory, > 0 is the PPO clipping parameter that limits the policy ratio, and 3 0 is the coefficient controlling the strength of KL regularization with respect to the reference policy Tee--
C FAILURE MODES OF SIMPLEVLA-RL
This subsection investigates the failure conditions of SimpleVLA-RI, and key influencing factors. Through experiments on five RoboTwin2.0 tasks, we find that the model priors are the critical factor determining RL effectiveness.
Settings Each task trained under domain randomization with a single-task setting. We compare three model variants: (1) the OpenVLA-OFT base model without trajectory fine-tuning (0 trajec- tories SFT); (2) the model fine-tuned with 100 demonstration trajectories per task (100 trajectories SFT); and (3) the model fine-tuned with 1000 demonstration trajectories per task( 1000 trajectories SFT). All models undergo Simple VLA-RL training on 1000 training scenarios and are evaluated on 100 held-out test scenarios.
18

---

Published as a conference paper at ICLR 2026
Table 6: Impact of initial model capability on SimpleVLA-Re performance.
RoboTwin2.0
Move +RL
A2B Right
Phone Stand
Bottles
Avg
0
0
0
100 trajs SFT
9.4
7.8
7.8
10.1
1.2
7.3
+RL
51.6
25.0
27.2
18.8
4.3
25.4
+42.2
+17.2
+19.4
+3.1
-881
1000 trajs SFT
28.1
37.5
28.7
17.1
29.7
28.2
+L
61.2
45.3
37.5
39.6
68.3
50.4
+33.1
+7.8
+8.8
+22.5
+-8.6
-22.2
RL fails completely when the base model has no initial task ability. Table reports the results. The base model (O-trajectory SFT) achieves 0% success rate across all tasks, exhibiting task- relevant behaviors. Despite extensive pretraining, OpenVLA shows extremely limited zero-shot generalization, consistent with findings Kim et al. (2025). Because no successful trajectories generated during sampling and only outcome rewards (without process rewards) are employed, every trajectory receives zero reward. As result, RL. is unable to improve performance, which remains at 0%.
The model prior has a significant impact on the effectiveness of RL. Initial capability is strongly correlated with post-RL performance. The 100-trajectory SFT model improves from 7.3% to 25, 25,4% (an 18.1% gain), while the tojo-trajyctory SFT model improves from 28.2% to 50.4% (a 22.2% gain) in average success rate. This trend is consistent across tasks. For instance, in the por task, the 100-trajectory SFT model improves from 9.4% to 51.6%, whereas the 1000-trajectory SFT model improves from 28. 1.19 %% to 61.2%. These results highlight that stronger initial capabili- ties provide more effective starting points for exploration, thereby facilitating greater performance improvements.
can
RL effectiveness has a threshold: when initial ability is too low, improvements remain negligi- ble. Our findings further reveal that the effectiveness of RL subject performance threshold. When initial success rates are very low, online RL with outcome rewards yields only marginal im- provements. For example, in the pick dual bottles task, the 100-trajectory SFT model improves from 1.2% to 4.3%, while the 1000-trajectory SFT model improves from 29.7% to 68.3%. Similarly, in the place phone task, the 100-trajectory SFT model gains 8.7%, compared 1000-trajectory SFT model. The results indicate that minimal level of task competence essential for effective RL. Below this threshold, exploration is ineffective and RL fails to produce meaningful gains.
22.5% gain for the
D ADDITIONAL ABLATION AND ANALYSIS EXPERIMENTS
D.1 ABLATION STUDY ON KL REGULARIZATION
In this section, we present the ablation study on KL regularization and analyze the training stability after removing the KL constraint. The experimental settings are consistent with Figure 3 We apply SimpleVLA-RR RL with OpenVLA-OO
on LIBERO-Long under the one-trajectory setting. compar- ing two configurations: with and without the KI constraint. The results are shown in Figure 6 To further verify the stability of model updates, we visualize the gradient norm, policy gradient loss, and sampling accuracy curves throughout training with and without the KL constraint in Figure 7
The experimental results demonstrate that removing the KL regularization leads to a slight improve- ment in model performance while maintaining comparable training stability. The loss curves, gradi- ent norms, and sampling accuracy all exhibit smooth convergence patterns. Furthermore, removing the KL constraint simplifies the training framework by eliminating the need to compute reference model sampling probabilities and load additional reference models, resulting in approximately 10% reduction in training time.
19

---

Published as a conference paper at ICLR 2026
LIBERO-LON Long SR (%)
70-
60
30-
20
Constraint
100
150
200
250
300
RL Training Steps
Figure 6: Ablation study on KL constraint on LIBERO-Long, Removing the KL constraint achieves comparable performance.
XL Cosstruint wi Kineetntrintt
Palley Gradlent Lone
Gradient Norm 13
0m0
Kl-Coestment
Sauallag Accurasy
0.002
0.002
0.001
RL Training Steps
RLL Training
Truining Steps
(a) Gradient Norm
(b) Policy Gradient Loss
(c) Sampling Accuracy
Figure 7: Training stability analysis with and without KL constraint LIBERO-Long. Training
without KL constraint remains equally stable.
D.2 TRAINING STABILITY
We provide training curves for Open VLA-OFFT and PiO.5 on the RoboTwiin.0 benchmark without KL regularization, including policy gradient loss, gradient norm, and sampling accuracy throughout training. Specifically, Open VLA-OFT is trained on the Lift Pot task from RoboTTwin2.0, as shown in Figure 8 PiO.5 trained on a mixture
tasks from RoboTwin2.00 Lift Pot, Beat Hammer Block, Pick Dual Bottles, Place Phone Stand, Move Can Pot, Place A2B Left, Place Empty Cup, and Han- dover Mic, as shown Figure 9 The results demonstrate stable training dynamics throughout the optimization process, with no significant fluctuations instabilities observed in any of the metrics.
Pollyy Gradient Lees
a.07
Sampting Accunccy (%)
Gradient Norm
0.05
80
40.
-6
000
0.
RL. Training Steps
RL. Training Steps
RL. Training Staps
(a) Gradient Norm
(b) Policy Gradient Loss
(c) Sampling Accuracy
Figure 8: Training stability of VLenVLA-OFT on RoboTwin2.0 Lift Pot task. The gradient norm and policy gradient loss remain stable throughout training.
E MULTI-TASK RL TRAINING ON LIBERO-90
To evaluate the effectiveness of SimpleVLA-RL in multi-task mixed training settings, we conduct RL training the LIBERO-90 task suite, which contains 90 different tasks. Each task consists of 50 scenarios, resulting total of 4,500 scenarios across all 90 tasks that serve as the training set for SimpleVLA-Re Figure 10 shows the results of different models LIBERO-90 as the performance of OpenVLA-OE SET and Open VLA-OFF RL on LIBERO-99 90 under the one- trajectory SFT setting.
well
20

---

Published as a conference paper at ICLR 2026
Policy Gradlent Loss (x1e-3
Norn
0.0775
0.00/0
Pio.s RL Tosks
2,000
M M
P10.5 Tasks
Sumpling Accuracy (%a
10
mm A/p
1
W
M
0.0000
0.150
70
2,088
150.5 Tanks
60
RI. Training Steps
RE. Training Steps
RT. Training Steps
(a) Gradient Norm
(b) Policy Gradient Loss
(c) Sampling Accuracy
Figure 9: Training stability of PiO.5 on RoboTwin2.0 with tasks. The training metrics show stable optimization.
Results of Different Models on LIBERO-90
88.3
84.4
81.3
80
75.4
LIBERO-30 Success Rate (%)
54.4
40
ACT
PRISE
vO-8eT
ResNett
OpenVLA-OFT
OpenVLA A-OF
Figure 10: Results of different models on LIBERO-90.
21

---

Published as a conference paper at ICLR 2026
F COMPARISON OF LLM AND VLA ROLLOUT ALGORITHMS
To better illustrate the concrete rollout process in SimpleVLA-RL and highlight the key differences between VLA rollout SimpleVLA-RL and LLM rollout the veRL framework, we present comparative pseudo-code implementation in Listing 1.
def rollout (policy, dataset, number Samber_sampee-, steps=None): rollout_dataset. for batch in dataset:
batch = batch.rrpodd (nunber_sampler
# LIM generates diverse outputs using random sampling
outputs policy.generate (batch, temperature-1.0) rollout_dattsst.dppend. (batch, outputs))
& Parallel process__poo..... pool.submit abatch.initiale. envs
initialization and interaction
process states env process pool. poo.. submit (envs.
for in range (max_steps):
$ VLA generates diverse trajectories using temperature sampling action tekens
actions cailout_dataset.eppe
(states, temperature-1.00
(f" (s,a) for 8,
in zip stanvssstttessccttiiei states, denes = anv_process_poll submit (envs. actions) $ Remove completed tasks
active (ee for s, in (enve, states, dones) if not dl if not active:
break envs, states zip return rellout sollout_dataset dataset
Listing 1:
Pseudo-code for the adopted veRL rollout function: from LLM-based generation inter- active VLA sampling with synchronous environment parallelism.
G EXPERIMENTAL CONFIGURATION AND IMPLEMENTATION
G.1 ROBOTWIN2.06 TASK CLASSIFICATION AND DETAILS
We classified the 12 tasks in RoboTwin2.0 based their average number of steps, categorizing them into Short Horizon Tasks, Medium Horizon Tasks, Long Horizon Tasks, and Extra Long Horizon Tasks. The shows the specific number of steps and classification for each task.
Detailed descriptions of the 4 real-world tasks (Stack Bowls, Handover Block, Pick Bottle, and Click Bell) and 12 RoboTwin2.0 tasks can be found /ttppes/rrooott.--ppaaaaooo hhuu.. 1c
/ndex html
G.2 BACKBONE MODIFICATION DETAILS
Our implementation of the OpenVLA-OFF model differs from the official version. To achieve im- proved training and inference efficiency, we utilize only single-view images, language instructions, and robot proprioceptive states as model inputs, whereas the official model additionally incorpo- rates wrist camera images. Additionally, in the LIBERO, we don't 't use robot proprioceptive states model inputs. Regarding the model architecture, we employ only parallel decoding and action chunking designs. We use the LLaMA2 output head to generate action tokens and the cross-entropy loss. whereas the official model uses an MLP to generate continuous actions and LI regression. Due to the differences in model inputs and architecture, cannot the official checkpoints. We modify the official codebase and performed SFT from scratch using the same datasets and hyperpa- rameters the official implementation.
22

---

Published as a conference paper at ICLR 2026
Table 7: RoboTwin 2.0 task classification based on planning horizon and required steps.
Task Name
Steps
Horizon
Horizon Group
Short Horizon
Tasks (112-130 steps)
lift_pot
112
Short
heameer hammer
113
Short
Average: 121 steps
pick dual bottles
127
Short
Count: tasks
place.phone stand
130
Short
Medium Horizon Tasks (151-223 steps)
move_can_po
151
Medium
place_a2b.let place.empty cup
155
Medium
Average: 176 steps Count: tasks
174
Medium
handover mic
223
Medium
Long Horizon Tasks (283-313 steps)
handover black
283
Long
Average: 298 steps Count: tasks
stac_hoolssttoo
313
Long
Extra Long Horizon Tasks (466-637 steps)
rrnkks_
466
Extra-Long
Average: 552 steps Count: tasks
put. bottle. dustbin
637
Extra-Long
Overall Statistics
Total: 12 tasks, Average: 256 steps
G.3 IMPLEMENTATION DETAILS AND HYPERPARAMETERS
For training infrastructure, we employ 8XNVIDIA A800 80GB for full- full-parameter training. The training hyperparameters are configured as follows: learning rate Ir = 5 x 10 training batch size of 64, sampling count of 8, mini-batch size of 128, clip ratio εT. = 0.2, εи 0.28, and temperature 1.6. The number of action chunks is the LIBERO and 25 in the Robo 1.0m1.0... The model is configured with total of 256 action tokens. The maximum interaction step is set to 512 in the LIBERO and 200, 400, or 800 in the RoboTwinI 1.02.00, depending on different tasks.
Regarding training time, the wall-clock training time varies by benchmark:
RoboTTin benchmark: Using A800-80GB GPUs, the wall-clock training time for single-task RL using OpenVLA-OO is approximately 12-24 hours, depending on the ini- tial success rate and maximum episode length. Training time increases when the initial success rate is lower and tasks require more steps.
LIBERO benchmark: Under the One-Trajectory SFT setting, RL training takes approxi- mately or days; under the Full-Trajectory SFT setting, training time is shorter, around day. Tasks with lower initial success rates require longer training time.
RoboTwin benchmark: Using 8 A800 GPUs, the wall-clock training time for VLenVLA-OPF single- task RI. is approximately 12-24 hours, depending on the task's initial success rate and maximum episode length. Training time increases when the initial success rate is lower or when tasks require more steps.
LIBERO benchmark: Under the one-traj setting, RL training takes longer, approximately 1-2 days; under the traj-all setting, training time relatively shorter, around day. Training time increases correspondingly when the initial success rate is lower.
During the rollout phase of RI. training, we employ random sampling. For evaluation, we utilize greedy sampling, with each benchmark tested three times for reproducibility.
G.4 ROBOT HARDWARE DETAILS
For real-world experiments, employ AgileX Cobot Magic, which is a mobile platform with an Aloha configuration consisting of four robotic arms. Each arm an AgileX Piper with six degrees of freedom, equipped with one-DoF parallel gripper. A RealSense D435 RGB-D camera is mounted
23

---

Published as a conference paper at ICLR 2026
on the platform, capturing RGB images in real time at a resolution of 640 x 480 with frame rate of approximately 30 Hz.
H
THE USE OF LARGE LANGUAGE MODELS
We utilized LLMs for grammatical refinement and clarity improvements in our manuscript. Specif- ically, used ChatGPT (GPT-S-Thinking) help polish the language and correct grammatical emross in our draft. The assistance was limited to improving readability and ensuring adherence to academic writing conventions, while all technical content, experimental design, and scientific contributions remain entirely our own work.
24


Reviews:
[{"review_id": "cfJ7FItlWP", "reviewer": "Reviewer_qg2P", "summary": "This paper introduces SimpleVLA-RL, an RL framework built on top of VeRL to train Vision-Language-Action (VLA) models for robotic manipulation tasks. The work addresses two fundamental challenges in current VLA training: (1) the scarcity and high cost of large-scale robotic trajectory data required for supervised fine-tuning (SFT), and (2) limited generalization to tasks under distribution shift. SimpleVLA-RL leverages all recent findings from LLM RL finetuning literature such as GRPO with dynamic sampling, variable clipping threshold, high temperature rollout sampling and no KL penalty to build a stable RL finetuning setup for VLA training. The paper itself does not introduce any new ideas or algorithms to enable RL training for VLA but shows that findings from LLM community generalize to robotics without any modifications.", "questions": "On L177 authors claim that critic free RL algorithms suffer from vanishing gradients problem. Can authors add a supporting citation for this? I am unsure how true this claim is. Critic free RL algorithms suffer from high-variance but there is no inherent issue in these algorithms that causes vanishing gradients. If there are no supporting citations for this statement I would suggest the authors to remove this line.\n\nThis paper does not propose any novel ideas to enable RL for VLAs. It can be seen as more of application of recent techniques used in RL finetuning for LLMs applied directly to VLA finetuning. However, it does have valuable experiments and insights that most of the techniques used in making RL training stable and effective for LLMs generalize to VLA finetuning in the current setup. That is why I recommend the paper to be accepted conditioned that authors address the above mentioned weaknesses and minor issues in some claims made.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Application of RL for training low-level control policies for manipulation to scale VLA training without relying on  large amounts of demonstration data.\n2. The experimental results clearly show performance improvements achievable through RL training are significant\n3. The analysis section is interesting and studies valuable questions with well thought out experiments", "weaknesses": "1. The experiments section doesn’t mention details about the training datasets and benchmarks. From the current description it is a bit unclear to me on what task, episodes, and bechmarks was the RL finetuning for the VLA policies done? My understanding is OpenVLA-OFT was first trained on real world robotics datasets and then for benchmarking on libero it was finetuned with small amounts of simulation only trajectories. I would like to understand for experiments presented in tables 1, 2 and 3 did the authors use the SFT finetuned OpenVLA-OFT model as initial checkpoint for RL or RL is performed directly on base model (aka not finetuned on sim data with SFT).\n2. Another concern I have is the SFT finetuned results for full libero dataset experiment used by OpenVLA-OFT in the paper does not match the results presented in the original paper table 2 here https://arxiv.org/pdf/2502.19645. I would like to ask authors to clarify why there is such a big gap (ex: Spatial SR is 96.2 in original paper vs 91.6 in table 3 and 4). \n3. The experiments presented in analysis section 4.2 are definitely interesting and valuable. However, the details about how the amount of compute/data used for both SFT and RL is a confounding variable that is unaccounted for. Can authors please describe the exact amount of model updates, trajectories used for this comparison? I believe such a comparison only makes sense when done under a fixed budget of training updates to get a clear idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6j65nF72L0", "reviewer": "Reviewer_EvC5", "summary": "The paper demonstrates how advances in LLM post-training can be ported over to robotics domains by finetuning VLA models in simulation with simple modifications to GRPO. Specifically, the paper introduces best practices to encourage exploration and ensure stable training, such as dynamic sampling of batches to ensure only informative gradients are propagated, best practices for clipping policy updates, and high-temperature sampling. The approach only uses sparse rewards. The paper demonstrates impressive improvements over SFT, and then demonstrates these improvements hold when transferring back to the real world.", "questions": "Can the authors provide some intuition for why they were able to effectively optimize through diffusion-based policies with such a simple approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "$\\textbf{Simplicity + Scalability}$: The approach is extremely simple and scalable, and I expect the community to adopt and build on the work. Specifically, the approach is much simpler than recently proposed techniques for training diffusion-based policy heads, and opens the door for porting over new insights from LLM finetuning to robotics. \n\n$\\textbf{Strong Results}$:  To my knowledge, this is the first work to demonstrate reliable improvement with RL for VLAs in simulated domains. The paper demonstrates the ability of the method to achieve substantial improvements in success rates across a range of tasks, demonstrates some ability to generalize to out of distribution tasks, and demonstrates these improvements actually transfer to the real world.\n\n$\\textbf{Thorough Evaluation}$: The paper has a strong evaluation and clearly conveys the importance of each of the design decisions. \n\n$\\textbf{Timliness:}$ In short, this is a solid, timley submission and an easy acceptance for me.", "weaknesses": "$\\textbf{Additional Related Work}$: The discussion of related work focuses primarily on other approaches for post-training VLAs. However, the paper would be strengthened by additional discussion one recent works which finetune diffusion policy heads, such as Diffusion Policy Policy Optimization. We know that modifications to GRPO are effective at post-training LLMs, but to me the most surprising thing about this paper is that it was able to stably optimize through a diffusion head. Some additional intuition on why this simple approach works so well would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhpwjXWX9L", "reviewer": "Reviewer_ogXC", "summary": "This paper proposes **SimpleVLA-RL**, a framework designed to overcome the dependence of Vision-Language-Action (VLA) models on large-scale demonstration data during supervised fine-tuning. Building upon the veRL framework, the authors introduce tailored optimizations for reinforcement learning on autoregressive VLA models and validate their approach through both simulation benchmarks and real-world robotic experiments.", "questions": "1. The current RL framework is validated primarily on autoregressive discrete-action VLAs. Have the authors considered extending SimpleVLA-RL to continuous-action VLA models?\n2. Since the proposed method primarily relies on training in simulation and transferring to the real world, have the authors examined the potential sim-to-real gap? \n3. The current generalization experiments focus primarily on task and scene transfer. Have the authors considered evaluating language generalization—for example, testing the model’s robustness to paraphrased or diverse natural-language commands within the same scene?\n4. The authors remove the KL regularization term and adopt a sparse outcome-based reward. While this design may encourage greater exploration, it could also weaken policy constraints and optimization signals. Have the authors observed any instability or performance oscillation during training as a result?\n5. The current RL training primarily focuses on in-domain tasks that share similar distributions with the supervised fine-tuning data. Have the authors considered performing RL training on out-of-domain tasks to further evaluate the model’s generalization ability and policy stability under distributional shifts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, clearly structured, and presents a concise yet precise problem formulation. The experimental design is robust and the results are solid, demonstrating substantial improvements over existing state-of-the-art models across multiple benchmarks. Notably, the experiment using only a single demonstration during the RL phase provides compelling evidence that reinforcement learning can effectively mitigate the challenge of demonstration data scarcity.", "weaknesses": "Although the experimental results are comprehensive and detailed, the methodological innovation appears somewhat limited. The proposed parameter adjustments effectively improve performance, yet the paper does not sufficiently justify the suitability of directly transferring RL algorithms from large language models to VLA settings. \n\nMoreover, the removal of the KL regularization term lacks supporting ablation studies, and the sparsity of the outcome-based reward may weaken the gradient signal, potentially affecting the stability of policy optimization. Furthermore, while the “Pushcut” phenomenon is intriguing and conceptually inspiring, it is demonstrated only in simulation, without quantitative validation in real-world settings—thus its generality and robustness remain uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S1HBMsHLfO", "reviewer": "Reviewer_urcu", "summary": "This paper proposes SimpleVLA-RL, a practical online RL framework for Vision-Language-Action (VLA) models that scales beyond limited demonstration data. Building on veRL, the authors implement VLA-specific interactive rollout (closed-loop env stepping with token-based action distributions), simple outcome-only (0/1) trajectory rewards, and GRPO training with three exploration enhancers: dynamic sampling (discard all-success/all-failure groups), higher clipping range (0.8–1.28), and higher rollout temperature (1.6). They also drop the KL penalty (no reference model) to reduce memory and encourage exploration. Experiments fine-tune OpenVLA-OFT and then apply online RL across LIBERO, RoboTwin-1.0, RoboTwin-2.0, plus real-robot sim2real tests. Results show ~99% avg SR on LIBERO, large gains on RoboTwin (e.g., 38.3→68.8% overall on RT-2.0; 39.8→70.4% on RT-1.0), strong data-efficiency (with one demonstration per task: 48.9→96.9% on LIBERO), improved generalization to unseen spatial/object/goal tasks, and notable sim2real improvements.", "questions": "please see the weakness and try to answer them, especially weakness 3. I am curious about if you explored how to deal with it.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Clear, simple RL recipe for VLAs (outcome-only rewards + GRPO) that removes reward-engineering and reference-model overhead; the method details (interactive rollout, reward assignment, GRPO objective) are explicit and easy to reproduce.\n\n2. Exploration enhancements are well-motivated and ablated (dynamic sampling / higher clip / higher temperature), each contributing sizable gains. \n\n3. Strong, broad empirical gains: SOTA-level LIBERO (~99% avg), large improvements across short/medium/long/extra-long horizons on RoboTwin-2.0, and substantial jumps on RoboTwin-1.0. \n\n4. Data efficiency: with one traj per task, RL lifts LIBERO-Long from 17.3%→91.7% and overall 48.9%→96.9%, nearly closing the gap to full-data training. \n\n5. Generalization: RL improves OOD tasks while SFT overfits/forgets—documented across spatial/object/goal suites with learning-progress plots.", "weaknesses": "(1) pic 5.1 is a bit confusing for me. I am not sure why the upper half has the different background to the bottom half. I think the best way to show emergent behavior is comparing behaviors under same conditions.\n\n(2) Compute/throughput transparency: The paper lacks precise reporting of wall-clock training time, GPU type.\n\n(3) Ablations indicate RL fails when the base success rate is ~0% and remains weak <5%; this limits plug-and-play use in very hard tasks without warm-starts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)