RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
Authors: name: Hongzhi Zang; name: Mingjie Wei; name: Si Xu; name: Yongji Wu; name: Zhen Guo; name: Yuanqing Wang; name: Hao Lin; name: Liangzhi Shi; name: Yuqing Xie; name: Zhexuan Xu; name: Zhihao Liu; name: Kang Chen; name: Wenhao Tang; name: Quanlu Zhang; name: Weinan Zhang; name: Chao Yu; name: Yu Wang
Published: 2025-10-08 07:05:13

Abstract: Recent progress in vision and language foundation models has significantly
advanced multimodal understanding, reasoning, and generation, inspiring a surge
of interest in extending such capabilities to embodied settings through
vision-language-action (VLA) models. Yet, most VLA models are still trained
with supervised fine-tuning (SFT), which struggles to generalize under
distribution shifts due to error accumulation. Reinforcement learning (RL)
offers a promising alternative by directly optimizing task performance through
interaction, but existing attempts remain fragmented and lack a unified
platform for fair and systematic comparison across model architectures and
algorithmic designs. To address this gap, we introduce RLinf-VLA, a unified and
efficient framework for scalable RL training of VLA models. The system adopts a
highly flexible resource allocation design that addresses the challenge of
integrating rendering, training, and inference in RL+VLA training. In
particular, for GPU-parallelized simulators, RLinf-VLA implements a novel
hybrid fine-grained pipeline allocation mode, achieving a 1.61x-1.88x speedup
in training. Through a unified interface, RLinf-VLA seamlessly supports diverse
VLA architectures (e.g., OpenVLA, OpenVLA-OFT), multiple RL algorithms (e.g.,
PPO, GRPO), and various simulators (e.g., ManiSkill, LIBERO). In simulation, a
unified model achieves 98.11\% across 130 LIBERO tasks and 97.66\% across 25
ManiSkill tasks. Beyond empirical performance, our study distills a set of best
practices for applying RL to VLA training and sheds light on emerging patterns
in this integration. Furthermore, we present preliminary deployment on a
real-world Franka robot, where RL-trained policies exhibit stronger
generalization than those trained with SFT. We envision RLinf-VLA as a
foundation to accelerate and standardize research on embodied intelligence.