SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
Authors: name: Haozhan Li, name: Yuxin Zuo, name: Jiale Yu, name: Yuhao Zhang, name: Zhaohui Yang, name: Kaiyan Zhang, name: Xuekai Zhu, name: Yuchen Zhang, name: Tianxing Chen, name: Ganqu Cui, name: Dehui Wang, name: Dingxiang Luo, name: Yuchen Fan, name: Youbang Sun, name: Jia Zeng, name: Jiangmiao Pang, name: Shanghang Zhang, name: Yu Wang, name: Yao Mu, name: Bowen Zhou, name: Ning Ding

Vision-Language-Action (VLA) models have recently emerged as a powerful
paradigm for robotic manipulation. Despite substantial progress enabled by
large-scale pretraining and supervised fine-tuning (SFT), these models face two
fundamental challenges: (i) the scarcity and high cost of large-scale
human-operated robotic trajectories required for SFT scaling, and (ii) limited
generalization to tasks involving distribution shift. Recent breakthroughs in
Large Reasoning Models (LRMs) demonstrate that reinforcement learning (RL) can
dramatically enhance step-by-step reasoning capabilities, raising a natural
question: Can RL similarly improve the long-horizon step-by-step action
planning of VLA? In this work, we introduce SimpleVLA-RL, an efficient RL
framework tailored for VLA models. Building upon veRL, we introduce
VLA-specific trajectory sampling, scalable parallelization, multi-environment
rendering, and optimized loss computation. When applied to OpenVLA-OFT,
SimpleVLA-RL achieves SoTA performance on LIBERO and even outperforms pi_0
on RoboTwin 1.0\&2.0 with the exploration-enhancing strategies we introduce.
SimpleVLA-RL not only reduces dependence on large-scale data and enables robust
generalization, but also remarkably surpasses SFT in real-world tasks.
Moreover, we identify a novel phenomenon ``pushcut'' during RL training,
wherein the policy discovers previously unseen patterns beyond those seen in
the previous training process. Github: https://github.com/PRIME-RL/SimpleVLA-RL