What Can RL Bring to VLA Generalization? An Empirical Study
Authors: Jijia Liu
Feng Gao
Bingwen Wei
Xinlei Chen
Qingmin Liao
Yi Wu
Chao Yu
Yu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Vision-Language-Action Model
Policy Generalization
Primary area: reinforcement_learning
TL;DR: We conduct an empirical study to evaluate the generalization benefits of reinforcement learning fine-tuning versus supervised fine-tuning for vision-language-action models and provide some findings and analyses.

Abstract: Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. 
However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. 
To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io

What Can RL Bring to VLA Generalization? An Empirical Study
Jijia Liu
Feng Gao
Bingwen Wei
Xinlei Chen
Qingmin Liao" Yi Wu
Chao Yu Yu Wang
Shenzhen International Graduate School, Tsinghua University
Institute for Interdisciplinary Information Sciences, Tsinghua University
Department of Electronic Engineering, Tsinghua University
Vision
Semantics
Training
Training
Objects
Unseen Receptacles
Unseen Instruction Phrasings
Table
Multi-Object (both seen)
Multi-Objee (both unseen)
Distractive Receptacle
(both unseen)
Dynanic Texture (wook)
Dynamic (strong)
Execution
RL Fine-tuning for VLA Generalization
Position (object Receptacle)
Dynamic se (weak)
Training
Unseen Robot Init
Dynanic Noise (strong)
Mid-Episode Object Reposition
SFT
SFT
SFT
Avg.
Avg. Perf Drop
Vision:
Serantics: RL S
Execution: RL SFT
Objects
Receptacles
Tables
Textures
Figure 1: Overview of our study for evaluating how RL enhances VLA generalization in terms of Vision, Semantics, and Execution: in moderate improvements in Semantics, and performance par with SFT for Vision.
tests, RL yields substantial gains Execution,
Abstract
Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. However, their predominant training via supervised fine-tuning (SFT) limits generalization due susceptibility to compounding distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. To address this, our study introduces comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of
under
"Equal contribution.
Corresponding authors: (yuchao, yu-wang| @ mail..sinngua.cdum
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tume fine-tuning, particularly with PPO, signifi- cantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at htttss//rlwlls.. github. io.
Introduction
Vision-Language-Actie (VI.A) models represent an emerging class of foundation models [Ma et al., 2024, Firoozi et al. 2023] that unify perception, language understanding, and embodied control. By leveraging vision- vision-langual language models pretrained on internet-scale data and further training on large,
heterogeneous robot demonstration datasets [Collaboration et al. 2023 Khazatsky et al. 2024]. VLAs can interpret sensor observations and natural language instructions to directly map them to robot actions. This paradigm has demonstrated promising generalization across diverse tasks includdig
single-arm, bimanual, and mobile manipulation [Tea et al. 2024 Kim et al. [024 Liu et al. [2024]
Wen et al. 2025]. navigation [Shah et al. 2022]. and even complex long-horizon activities like kitchen
or bedroom cleaning in unseen scenarios [Black al. 2024 Intelligence et al.) 2025
Despite this promise, VIAA model training predominantly relies on supervised fine-tuning (SFT) through behavioral cloning of demonstration labels (Kim al. (025)] This approach, whether in pretraining or few-shot adaptation, is inherently susceptible to compounding errors under distribution shift: minor deviations from expert trajectories accumulate, steering the policy into unfamiliar
states and undermining robust performance [Ross and Bagnell| 2010 De Haan et al., 2019 Belkhale let al., 2023 Foster et al. (0024). This mismatch between training and testing distributions fundamen- tally limits robustness, with research highlighting issues like quadratic regret growth relative to the task horizon under such conditions [Ross and Bagnell 2010).
In contrast, reinforcement learning (RI.) offers a paradigm that directly optimizes cumulative task rewards through trial-and-error, enabling policies to explore beyond narrow expert data and learn corrective behaviors. Crucially, in the broader foundation model landscape, particularly for Large Language Models (LLMs) and Vision-Language Models (VLMs), recent studies have underscored RL's advantages for generalization [Ouyang et al. 2022 Zhai et al., 2024 Huang et al.) 2025). Compelling evidence suggests that while SFT tends to memorize training data, RL fine-tuning can lead to substantially better out-of titoo--distribution performance and unlock greater reasoning capabilities [Chu et al.) 2025| Huang et al.] 2025 Ma et al.) (025). Drawing from RL's established success in robotics and these encouraging results from other large-scale models, RL fine- toning is increasingly
being applied to VLAs [Collaboration et al., 2023 Walke et al.) [003 Khazatsky et al.] [0024),
with approaches ranging from incorporating human feedback [Chen et al. 2022) to using offline
RI updates (Zhang al. [024bb algorithms like PPO (Schulman et al. 2017 sometimes in
multi-stage processes with imitation learning [Guo et al. 2025bb.
However, despite these pioneering efforts, systematic understanding of what specific generalization benefits RL fine-tuning confers upon VLAs, especially in direct comparison to SFT baselines. and how their respective strengths differ, remains insufficiently developed (Hu et al.) 2024 Mark et al. 2024]. For instance, while recent work like FLaRe [Hu et al. [2024] demonstrated PPO utility for fine-tuning VLAs, comprehensive analysis of the resulting model's generalization capabilities was not its primary focus.
This paper aims to address this critical gap. We undertake a systematic study to dissect the general- ization properties of VIAAs fine-tuned with RL versus SFT.
Specifically, we empirically investigate: What unique benefits can RI. bring to VLA generalization compared to supervised fine-tuning?
To answer this, we center our investigations on the representative pick-and-place task. On this task, we first evaluate mainstream RI. algorithms for large-scale models (PPO [Schulman et al. 2017 Ouyang et al.] [002), DPO [Rafailov al. 2023, /hhng et al.) [024b), GRPO [Shao et al. 2024
Guo et al. 2025a) to identify effective VLA line-tuning strategies. We then conduct broad

---

comparative evaluation, also on pick-and-placee pinpointing where RL fine-tuning outperforms SFT. As previewed in Fig. [ this comprehensive evaluation rigorously examines generalization across three key dimensions: 1) Vision: challenging generalization with novel backgrounds (unseen tables) and by overlaying unseen textures on the foreground or the entire image. 2) Semanties: testing understanding via unseen objects, novel receptacles, and varied instruction phrasings to probe language sensitivity and object recognition. 3) Execution: probing robustness by varying initial robot states, object/receptacle positions, and introducing dynamic disturbances like random object reposition within episodes.
Through extensive experiments and deep analyses, we:
Establish a rigorous and challenging benchmark to evaluate how VLA fine-tuning methods affect generalization across diverse visual, semantic, and execution dimensions.
Identify PPO as the preferred RL algorithm for VLA fine-tuning over GRPO and DPO, while also discuss key challenges in adapting these RL algorithms from LLM/VLM paradigms to the distinct requirements of VLAs.
Develop an efficient PPO-based VLA fine-tuning recipe, enabled by a shared actor-critic backbone, VLA model warm-up, up, and minimal PPO training epochs.
Demonstrate RL's superior generalization over SFT in semantic understanding and embodied execution for VLAs, while maintaining comparable visual robustness.
2 Related works
2.1 Vision-Language-Acciion (VLA) models
In robotic tasks that utilize both vision and language inputs, recent research demonstrates that leveraging knowledge from pre-trained L.MMs VLMs (Chen et al. 2023 Driess et al.] 2023
Karamcheti et al., 2024 Beyer et al. 2024 leads to better policy generalization compared to smaller
models [Zhao et al. 2023 Chi al. 2023]. By harnessing the capabilities of these foundation models
and utilizing scage robotic datasets, VLA models [Brohan et al. 2023b a Team et al. 2024
Kim et al. 2024 Liu al.
2024 Wu al. 2023 Cheang al.) 2024 Black et al. 2024 Intelligence
et al. [2025] provide a strong basis for embodied agent. Our work builds upon VLA and seeks to
further improve its generalization abilities.
2.2 RL fine-tuning for large-scale (Vision-)Language Models
Fine-tuning large language models has shown promising results (Stiennon al. 2020 Ouyang et al.
2022 Achiam et al., [0023, especially when using high-quality supervised data (Zhang et al. 2024a
Zhu et al. 2023 Zhang et al. [2023b]. Some work aims align LLMs with human preferences
for helpfulness and safety |Kaulmann et al. 2023 Bai et al., 2022]. using online RL algorithms
[Schulman et al. 2017 Ramamurthy al. 2023 Zheng al. [023] or offline updates (Rafailvv et al.
Ivison et al. [2024, et al. 2024]. Other studies have shown that online RL fine-tuning on
math and coding tasks [Shao al. 2024 Guo al. 2025 can significantly improve the reasoning
abilities of LLMs [Guo et al. [025a Jaachh et al. 2024 Team et al. 2025). Although RL has been
extensively explored in LLMs [Chu et al. 2025 its generalization potential for similar-scale VLA
models remains underexplored.
2.3
Fine-tuning policies for robotic tasks
VLA fine-tuning largely relies on supervised data (Kmm et al. 2025 Liu et al. 2025 Zhho et al.
2025 Zheng et al., 2024). However, the scarcity of extensive and high-quality data often hampers the
ability of VLAs to generalize to unseen scenarios or perturbations [Collaboration et al. 2023 Walke
al. 2023 Khazatsky et al. 2024]. To move beyond imitation and enable reward-driven learning,
RL fine-tuning has been explored in various settings, from small-scale models (Ren et al. 2024 to
large models with human interventions [Chen al.) [205). or through offline updates [Zhang et al.) 2024b). Guo et al. (0255) proposed using PPO [Schulman et al., 2017] to fine-tune VLA models, incorporating imitation learning in a two-stage process. Despite these efforts, the generalization ability of RL fine-tuning in VLA models remains insufficiently studied (Hu et 2024, Mark et al. 2024]. Moreover, we find that VI.As can be directly fine-tuned with online PPO without additional
training stages imitation learning, underscoring RL's scalability for robot tasks.
3

---

3 Preliminaries
3.1 Problem formulation
We model each language-con robotic task T E T as a Partially Observable Markov Decision Process (POMDP), defined as tuple M AS P. R. O, C, P(so), Here, the state space includes robot and environment states, A the action space of control commands, and O the observation space of sensor outputs. Transitions follow 8t41 ~ P( St. at), and is the discount factor. Each task has natural language instructions Lr C L and a reward R(s, I) E R when state s completes task stages satisfies instruction 1. Episodes start with an instruction ~ Cr and initial state 8q ~ P(sg), which defines the robot's ' initial pose and environment. The policy Ty uses the last H. observations +1 and the language instruction / to output actions 0r mp(a Oz- 14 1). constructing trajectory T = (0o; .o....).
Supervised fine-tuning (SFT). SFT learns from expert-collected demonstration dataset Dr {(7(9, ) where each trajectory is 7 (o (8) (i)
(6)
(4)
an - -1 and N the total
number of trajectories. The VLA policy To minimizes:
K;-1
(SFT
Σ Σ
(SFT ())), a!0 = no (o 10))
(F0,,10) t=0
where fsFr is a loss function (e.g., next-token prediction, L, regression, diffusion) based on VLA architecture and action representation [Kim et al. 2025).
RL fine-tuning, RL fine-tuning maximizes scalar rewards or preference signals through direct interactions with environments. With rewards, it minimizes the negative discounted return episode of length M:
M M-1
CRL( (0) - ET To
Σ
7 H'((s,))) 1).
often via a policy- sallcytgradiet surrogate:
M M-1
Cra(0) -
Σ
AT log Tela: O- 0):
using an advantage estimator AÇ. With preferences, methods like Direct Preference Optimisation
(DPO) [Rafailov et al. 2023 use pairwise/ranked feedback optimize Ty for preferred trajectories.
3.2 Vision-Language-Actiin models
We base our study on OpenVLA [im eet al. [2024] (Fig. [2) an open-source model that achieves state-of-the-art per- formance by pairing a fused visual en- coder of SigLIP /hai al. 2023] and DI- NOv2 [Oquab et al.) 2023 with a Llama- 2 7B language backbone Touvron et al. 2023), built on the Prismatic VLM Karam- cheti al. 2024). each time step the pol- icy receives a single RGB image o; and an instruction i.c., the history length H = 1. The image is embedded into visual tokens, the instruction is tokenised with Llama 2's tokenizer, and the resulting token sequence is fed to the causal transformer decoder.
OpenVLA
5d
Liama 7B
Fiuut Sragg
boui"
DinoV2
Skgl
Language Vntruction
Figure 2: Architecture of the Open VLA model jet al., 2024], reproduced from the official open-source code and checkpoints.
Kim
OLe follows the RT-2 discretisation recipe [Brohan al. 2023a]: each scalar in the continuous
command E Rd is mapped to one of 256 bins that uniformly split the range between its 1st and 99th percentiles in the training set, producing an action-token vector u, E {0, ....... 255 255)d.. dd These action tokens overwrite the 256 least-used tokens in the Ilama-2 vocabulary, so the language model can emit them directly. The network is then trained with the usual next-token cross-entropy objective, and the cross-entropy loss is computed solely on the predicted action tokens.

---

Colloor
Susly
NO 1ORD
Polic
DPD
250c
50.b
75
(a) Overview of different fine-tuning methods
(b) Performance comparison
Figure 3: Overview of VLA fine-tuning methods: SFT learns from offline demonstrations, whereas DPO, GRPO, and PPO use RL-based updates-eemploinnn preference alignment, group-relative advantage estimation, and standard actor-critic PPO with generalized advantage estimation (GAE), respectively; and performance comparison between different RL fine-tuning algorithms.
4 Effective RL fine-tuning of VLA models
We begin our study by exploring how fine-tune VLA models with reinforcement learning. As these models inherit a similar scale and structure from their pre-trained LLM/VLM backbones, we first test whether common RL algorithms for large-scale models like LLMs/VLMs transfer effectively to VLAs. In Sec. 4.1 we will show that Proximal Policy Optimization (PPO) (Schulman et al. 2017 consistently improves performance, whereas Direct Preference Optimization (DPO) Rafailov et al. 2023 Zhang et al. 2024b) and Group Relative Policy Optimization (GRPO) [Shao et al. [024]] struggle to learn something for POMDP robotic tasks. Guided by this finding, we then ablate key design choices within PPO to distill an effective fine-tuning recipe for VLA models in Sec. 4.2
4.1 RL algorithms: PPO, GRPO, DPO
Here, we consider three representative RL algorithms: PPO [Schulman et al., 2017 GRPO (Sha et al. 2024] and DPO [Rafailov et al. 2023 Zhang et al. [2024b]. All models are fine-tuned using
Low-Rank Adaptation (LoRA) Hu et al. 2022] with rank 32.
PPO performs on-policy updates, computing policy gradients with clipped importance ratios and Generalised Advantage Estimation (GAE) from freshly collected rollouts. Recent work Open-Reasonee Zero (ORZ) by [Hu et al. 2025] shows that disabling GAE (setting y 1 and À = 1) can boost performance when fine-tuning large models. Accordingly, we evaluate both the standard algorithm (PPO) and this variant (PPO-ORZ) in experiments.
GRPO estimates the baseline using group of samples, allowing for direct computation of advantages without explicit value estimation. While natural language generation and robotic multi-step MDP tasks inherently different, we follow the GRPO setup [Shao al. 2024] by sampling trajectories from common initial state, with each group containing trajectories.
DPO exploits offline datasets annotated with pairwise preferences. In robotic tasks, however, obtaining contrastive trajectories from the same initial state is difficult. Following [Zhang et al. 2024b), we infer trajectory-level preferences from reward signals. To ensure fair comparison, we employ only the baaee sparse reward described in Sec. 5.1 rather than the richer reward scheme used in the original implementation by [Zhang al. 2024b]
These algorithms are illustrated in Fig. 3a with further details and implementation specifics provided in Sec. A.I We evaluate the performance on the lil--and--aeee task described in Sec. 5.1
The results are presented in Fig. 3b with each experiment conducted using two different random seeds. Our findings indicate that PPO consistently outperforms GRPO, We attribute this to fundamental differences in environment dynamics between natural language tasks and robotic tasks [Li et al. 2024b]. We hypothesis that, in robotic tasks' POMDP, each action sequentially alters the environment state in a non-stationary manner, which may destabilizes GRPO's advantage estimates. Additionally, PPO surpasses DPO performance. We hypothesize that this due to sparse reward structure, which makes it challenging to distinguish between the quality of different trajectories, as well as significant distribution shifts between the offline dataset and interactive execution Prudeecio et
2023
5

---

V(*)
Action
Value Head
aa
at
0.75
Transformer
A=
OpenVLA
hall
Sarp. AC
Image Tokens
Language Tokens
200c
So6k
(a) Model architecture actor-critic OpenVLA
(b) Performance comparison
Figure 4: PPO with shared actor-critic backbone, where (ss) is predicted by three-layer MLP. We
compare the performance different critic designs, as well as a separate actor-critic architecture.
1.0
1.0
1.0
Success Rate
0.75
Spccess Rate
success
Ep.
Ep.=1
0.25
warm-up w/o wamn-up
PPO Ep.-2
0.25
PPO Ep.-2
Ep.
Ep.
0
0.5M
1.5M
0
300k Env. Stops
450k
600k
Env. Steps
Wall Clock Time
(a) VLA warm-up
(b) PPO epoch (wall-clock time)
(c) PPO epoch (env. steps)
Figure 5: Performance comparison to verify our efficient training designs.
4.2 Design factors of PPO
We employ several key design choices to enable PPO to work effectively with VLe. VLA. With these designs, our main experiments require about 42 hours on single NVIDIA A A100 GPU to converge. In the following, we introduce these design choices and present ablation studies assess their impact. We conduct each experiment with two different random seeds.
Shared actor-critic backbone. Since apO most modern RL methods-use an actor-critic formulation (Konda and Tsitsiklis, 1999] we treat the pretrained VLA policy as the actor and attach lightweight critic that estimates the state value V(s). To keep the architecture compact, the actor and critic share the entire Transformer backbone; a three-layer MLP value head (see Fig. 4at takes the hidden vector h" produced by the final Transformer block (Vaswani et al. 2017 at the first action-token position and regresses
a scalar value. To verify its efficacy, we evaluated several critic designs. Feeding the value head with the first action-token embedding ht" produced the highest and most stable returns, outperforming the last-token input h² and the concatenation [ /ho..... hr] (see Fig. (b)). A critic that uses its own Transformer backbone achieved comparable rewards but trained 35 % slower and consumed 83 more VRAM (81.3 GB vs. 44.4 GB). These results confirm that shared backbone with ht" is the most efficient choice.
VLA warm-up. We run all experiments with the official OpenVLA checkpoint [Kim et al. 2024 pretrained the OXE dataset [Collaboration al. 2023]. Due to the poor performance of this check- point on our benchmark, we warm it up with 140 demonstration trajectories gathered by Octo-Small TTam et al. 2024] and motion planner (see See. (.2).. As shown in Fig. [3a] the warm-up model reaches convergence with roughly 50 fewer environment steps, while both initialisations attain comparable asymptotic returns given sufficient interaction. Unless stated otherwise, all subsequent experiments use the warmed-up Open VLA model for initialization.
Minimal PPO epoch. We show that the update- to data ratio a key factor for efficient fine-tuning. In PPO this ratio is set by the epoch hyper- parameter, which specifies how many gradient passes each batch receives. Fig. 5bg and Fig. (5c that increasing the PPO epoch beyond one yields no gains in return sample efficiency, yet lengthens wall-clock time almost linearly. Accordingly, we fix epoch in all remaining experiments, achieving the fastest training without sacrificing performance.

---

1.0
1.0
Narraa and Scane
0.73
8.75
succoss halg
6,75
Success Bate
0.5
0.5
05
0.3
0.2
0.25
6.2.5
C23
0.
dk de 16c 16
0.0
0..0
Namber
kumber
(a) SFT@ Training
(b) SFT@ Eval
(c) RL @Trainin
(d) R Eval
Figure 6: Effect of data scale on SFT performance. The reported metrics include grasp accuracy,
continuous grasp accuracy, and task success rate.
Evaluating fine-tuning methods on VLA generalization
5.1 Environments and datasets
To thoroughly investigate the generalization capabilities of VLA, we focus typical pick-and-place task [Kim et al. 2025 Li et al. 2024a], where the agent is instructed to place an object from the table into receptacle. Inspired by prior works [Fan et al. 2025 Stone et al. 2023 and the concept of Vision-Languuge-AAee Action models, we define three dimensions of generalization: Vision, Semantics (Language), and Execution (Action), illustrated in Fig.
Vision: We include both foreground (Dynamic Textures) and background (Unseen Table) changes, as well as image-level Dynamic Noise, applied with either weak strong intensity,
Semantics: We consider previously unseen variations in Objects, Receptacles, and Instruc- tion Phrasings. Additionally, we design new tasks where one of two objects must be picked (Multi-Objechi using either seen unseen object sets. We also include task with Distractive Receplacle, and a task requiring the object to be placed in one of two unseen receptacles Muulfi-Reepptclle
Execution: We investigate changes in the initial positions of both the object and the recepta- cle, as well as the robot initial pose. Furthermore, we introduce a new scenario in which the object'' 's position changes during the task (Mid-ppisode Object Reposition).
To probe generalisation, we randomise each task along three axes during training: Vision (16 tables), Semantics (16 objects), and Execution (perturbations of object and receptacle poses). At test time we hold at least one of these factors out of distribution, introducing nine novel objects, sixteen unseen receptacles, five new table surroudings, and sixteen distractor textures. Assets are drawn from Objaverse (Deitke et al. 2023 and other public sources; additional table appearances are synthesised with Stable Diffusion [Rombach et al. 2022] and ControlNet (Zhang al.) [023a]], ensuring surface variation without changing the table's global pose. More details of asset acquisition and task specification are provided in Secs. A.3 A.44 ManiSkill (aoo al. 2024) with an 8-DoF FiddowX-250s arm. At every step the agent observes a 640 x 480 RGB frame and a natural-language instruction, and outputs Cartesian end-effector delta plus binary gripper signal. Rewards are sparse: 0.1 for grasping and continuously holding the correct object, and 1.0 for placing successfully. For supervised fine-tuning we collect demonstration trajectories using the MPLib motion planner [Guo et al. 2024] and fine-tuned using LoRA [Hu et al.
A.4 Built upon these assets, our tasks run in
2022]; further dataset details can be found in Sec. A.2
5.2 Effect of data scale on SFT performance
Following Lin et al. 2025 we study how supervised fine-tuning (SFT) scales with demonstration count. We train OpenVL. to convergence on datasets ranging from few hundred 64k expert trajectories (~~ 1.26M transitions) and report average scores over three random seeds in-distribution and on unseen objects/tables (Fig. 6a and [bb. We observe that performance plateaus at roughly 16k trajectories in both settings, so we adopt the 16k-trajectory SFT checkpoint as the baseline for comparing RL fine-tuning methods.
5.3 Performance comparison between RL and SFT
Building on preceding analyses, we plot success rates during RI. training for both the training distribution and the OOD object/table split in Fig, 6c and 6d. alongside SFT scores at various data
7

---

Succeen take
oss 208
Solro
0.724
PM
0.77
CAte 333
o. 5.720
Textrred
Taduren
Insrrcc
Tabl
IND
ON
Oni (MD)
M.ORi (000))
Didtura Reces
Vi. Recep
Semaniics
Pettmare Drey
130%
25.0
5.3%
MON (O0t)
Wotot Pedo
0% Rip
Statubb Races.
Avarage 796
Evai Execucion
Average
Saccenn Rane
3444
0.75 ara
cu A30
ASA
2000
9.281
2.34
Textures
Petfmmacc Drng 290
Tabe
42%
Oty Rop
-32
M-001 (OC0)
46.995
26trth seces.
V-foces
Figure 7: (a) Performance comparison between SFT and RL across different tasks. Both success rate and relative performance drop are reported. (b) Radar chart of success rate. (c) Radar chart of the performance ratio of out-of-distribution (OOD) in-distribution (IND).
scales. RL overtakes the strongest SFT checkpoint (SFT-16k) after roughly 0.4 M environment steps on OOD task. At convergence, it performs comparably to SFT-16k in the training setting and 42.6 6 % better on unseen objects and tables, showing that reinforcement learning not only improves policy performance under the training distribution but also provides markedly stronger generalization.
We further evaluate the generalization performance of the converged RL policy and the best- performing SFT policy (SFT-16K, shortly denoted as SFT) on OOD tasks, as shown in Fig. 7 Here we report average success rates and average performance drop over three random seeds while full detailed results can be found in Sec. B.3 The relative performance drop is calculated as P P 0OD-IND
IND The
results indicate that RL performs comparably to SFT Vision tasks, noticeably better in Semantics, and significantly better in Execution.
For Vision, hypothesize that neither RL nor SFT training induces visual robustness beyond the imposed visual randomness (i.e., random training tables), resulting in similar performance for both methods. In Semantics tasks, RL notably outperforms SFT when faced with OOD objects, both in single and multiple object scenarios. We hypothesize that, through trial-and-error, RL is able to learn the skill of "grasp"" in manner that is less dependent object type, leading improved generalization. For Execution tasks, RL surpasses SFT across all three evaluated scenarios: OOD object and receptacle positions, OOD robot initial positions, and mid-episode object repositioning.
5.4 Dissecting RL's contributions to generalization
To probe qualitative differences between the two ap- proaches, we visualise policy roll-outs on four rep- resentative tasks (Fig. 99. In Vision/Dynamic-Nouis (strong), the SFT agent repeatedly drops the object immediately after grasping, indicating that heavy vi- sual perturbations prevent from localising the recep- tacle, whereas the RL agent completes the placement. With an Unseen Object in Semantics, SFT keeps at- tempting to grasp the item already holds and stalls, while RL lifts it and finishes the task. This may be attributed RL's extensive trial-and anre-or experience during training, enabling it to make finer-gran anainee con- trol when interacting with unseen objects. RL also learned to recover from failed grasps and mid-episode object shifts two Execution tests, whereas SFT marches on in spite of position errors, likely because such cases never appear in the demonstration data. The trajectory distributions encountered during
(a) SFT
(b) RL
Figure 8: Training trajectories of SFT (left) and RL (right); color encodes gripper Z-axis rotation.

---

Vision
SFT
Keep draplnn!!
Dynanic (strong)
RL
gresp
Semantics
SFT
Approsch
graup
RL
Objects
Approach
Successful grasp
Execution
SFT
Fottel affest
grasp
Unseen Robot Init Pose
RL
Adjitt lecation
Brong location
Fatted grase
SFT
Seepp off taste!
Mid-Episode Object Reposition
RL
Approach
Re-torget
Figure 9; Visualization of some concrete examples.
training (Fig. 8) echo this gap: RL trajectories span broader workspace and richer range of end-effector orientations, whereas SFT roll-outs cluster along the motion motion-planner paths present the dataset. This wider coverage appears key to RL's superior generalisation on Execution tasks.
6 Conclusion
This study puts forward demanding benchmark that probes how RL fine-tuning shapes the out-of- distribution generalization of VLA models. After adapting several RL methods to VLAs, we find that a streamlined PPO cerraant-eeturie shared actor-critic backbone, simple warm-up, and only minimal PPO srochs-yielde the strongest and most efficient gains. Relative to supervised fine-tuning, this RI. approach substantially improves semantic grounding and action execution under distribution shifts, while matching its resilience to visual perturbations. These results underscore the promise of reinforcement learning as route towards more generalizable embodied agents.
Limitations. Despite its contributions, our study has several limitations. First, we rely solely on planner-generated demonstrations for supervised fine-tuning, which may not fully capture the variability present in human collan-collected data. Second, our evaluation is confined to aad---nd-pcc tasks; scaling to a broader, more complex, multi-task setting remains an important direction for future work. Third, all experiments are conducted in simulation, so integrating RI. fine-tuning with sim-to-real transfer to validate VI.A generalization on physical robots crucial next step. We hope findings will inform and accelerate these future efforts.
motion- planner---ee------ee generated

---

Acknowledgements
This research was supported by National Natural Science Foundation of China (23325505,,220611555 Tsinghua University Initiative Scientific Research Program, Tsinghua- Efort Joint Research Center for EAI Computation and Perception, Beijing National Research Center for Information Science, Technology (BNRist), Beijing Innovation Center for Future Chips, State Key laboratory of Space Network and Communications, and Beijing Zhongguancun Academy Project C20250301.
We would like to express our sincere gratitude Liangzhi Shi, who carried out the deployment on the Franka Panda robotic arm and provided valuable sim-to-real evaluation results. We are also grateful to Hongzhi Zang for her insightful advice on GRPO performance tuning, which significantly improved the quality of this work.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0 225.2222,. 2022.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in neural information processing systems, 36:80375-80335, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bosnjak, Xi Chen, Matthis Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier J. Hénaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer. CoRR, 0os/2240.007222 2024. URL httpp:://oi..rrreeeee org/ 48550/ arXiv. 2407 077266
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak. Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. mn A vision-language- org/ h/gg//b///// 2410. 24164
action flow model for general robot control, 2024. URL
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision-languager models transfer web knowledge to robotic control. arXiv preprint 1xX.:23377.18188 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huel Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu. Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023b. URL https://arxiv.or org. wttpg/////.///aa/////////// 06817
Chi-Lam Cheang, Guangzeng Chen, Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, and Minzhao Zhu. Gr-2: A generative vadeo-language- action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.0615 2024.
10

---

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang. Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up a multilingual vision and language model, 2023. URL https://arxiv org abs/ a305 2305 18565
Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Conrft: A rein- forced fine-tuning method for vla models via consistency policy, arXiv preprint arXiv:2502.05450, 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training, arXiv preprint arXiv:2501. 2025.
Open X.-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Raj. Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian lchter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruy Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang, Haochen Shi, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x-embodiment: Robotic learning datasets and rt-x models. CoRR, abs/2310.08864, 2023. URL http://dblp www.ttiieerrr....iii...eeeieeiet wweeerreeeeeeeee journals/ wttpeaaaaacoccooarreeeeeeeoeeeee 2311 htms#abs---a-........maeemmmmmme 2310-08864
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning.
Advances in neural information processing systems, 32, 2019.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osear Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 13142-13153, 2023.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023. URL /https://aaxiiii http://wwii..a./..a///// aa////baa///////// 23t.//33333333383
Cunxin Fan, Xiaosong Jia, Yihang Sun, Yixiao Wang, Jianglan Wei, Ziyang Gong, Xiangyu Zhao, Masayoshi Tomizuka, Xue Yang. Junchi Yan, and Mingyu Ding. Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions, 2025.
peattps://arce //arxiv....i.////a org/ /bs/
2505. 02152
Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Ap- plications, challenges, and the future. The International Journal of Robotics Research, page 02783649241221558, 2023.
Dylan Foster, Adam Block, and Dipendra Misra. behavior cloning all you need? understanding horizon in imitation learning. arXiv preprint arXiv:2407.15007, 2024.
11

---

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini
20::::2211.999. 2025a.
Runlin (Kolin) Guo, Xinsong Lin, Minghua Liu, Jiayuan Gu, and Hao Su. Mplib: a lightweight motion planning library, 2024. URL lattps:/ //t:: thub. com/ http:haalusaalbbbbbbbba/ Accessed: 2024-05-13. 13.
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving vision-language-actian model with online reinforcement learning, arXiv preprint arXiv:2501. 16664, 2025b.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL nZeVKeeFYfg
net/ met/forum?id
Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martin-Martin, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. Flare: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning. arXiv preprint arXiv:2409 2024.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zzrro: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint or arXiv:2503.2429/ 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint wwXXv:2503.06749 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowiez, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. To.5: vision-language-attioo model with open-world generalization, 2025. URL httttttss///aaxtt org/ abs/2504. 16054
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37: 36602-36633, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML), 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925 10, 2023.
Alexander Khazatsky, Karl Pertsch. Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang. Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson,
12

---

Charlotte Le, Yunshuang Li, Xinyu Lin, Zehan Ma. Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Khuong Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang. Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín- Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song. Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. In RSS 2024 Workshop: Data Generation for Robotics, 2024. URL https:://ponreriee..ntttt http://wwwwweeetootoneeoo wttp:ee/eooooommddn1dd21----// M12pTYLLLI
Moo Jin Kim. Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-tanguage-action model. arXiv preprini arXiv:2406.0922 2024.
Moo Jin Kim. Chelsea Finn, and Percy Liang. Fine-tuming vision Langun--nnagge-eee action models: Optimizing speed and success. arXiv preprint 2rXiv:2502. 2025.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint S.Kiv:2400505944 2024a.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi ShiiQuan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2024b. URL https //openreview.nete www.://utteeeoouumeeeeam d=-rrnmkkedS
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation, 2025. URL URL./ttpss//axxx.ooor//bb/////// 18647
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su. and Jun Zhu. Rdt-1b: lb: diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.0786. 2024.
Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thoupe for embodied task planning. arXiv preprint arXiv:2501.10074 2025.
Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme. arXiv preprint arXiv:2504.02587, 2025.
Yueen Ma, Zixing Song, Yuzheng Zhuang. Jianye Hao, and Irwin King. A survey on vision-language action models for embodied ai. arXiv preprint arXiv:2405. 0t..24445..444,,. 2024.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline and online rl fine-tuning of any class and backbone. arXiv preprint arX1v:2412.06685, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXhn:2304.07193, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
13

---

Hung Pham and Quang Cuong Pham. A new approach to time-optimal path parameterization based on reachability analysis. IEEE Transactions on Robotics, 34(3):645-66 659, 2018. doi: 20009//RR.2008..200
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions Neural Networks and Learning Systems, 2023.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741,, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization, 2023. URL htttp:: http://wawivvorar.aaaa/ abs 2210. 012441
Allen Z Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.005: 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 661- 668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL haattps::/proceeeinngg mlr.
html
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 150.::040300 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. LM-nav: Robotic navigation with large pre-trained models of language, vision, and action. In 6th Annual Conference on Robot Learning, 2022. htttttttss:///peeeeeet
meet/forrumidd-WWAASS
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033 440.0030000, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-300 3001,,01,12200000 2020.
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained languael models. arXiv preprint arXiv:2303.00990 2023.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with 1lss. arXiv preprint 2rX1v:2501.12599 2025.
14

---

Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /2213, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng. Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang. Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning scale. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.reer wtwp:rreewiieeeeefffrrrt id= F55MIATILU
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05885, 2025.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint a404v:2404. 10719, 2024.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances neural information processing systems, 37:1 11095, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986, 2023.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets 11m finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv.2402.17199, 2024a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847, 2023a.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint 2r0iv:2308. 10792, 2023b.
Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint 1:XX::2411119009 2024b.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705,, 2023.
Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao, and Donglin Wang. Vlas: Vision-language-action model with speech instructions for customized robot manipulation. arXiv preprint arXiv: 25..::3502..3. 2025.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf large language models part Ppo. arXiv preprint arXiv:2307.0496, 2023.
15

---

Rujjie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprini arXiv:2412.1 10345, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing rision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitation of our work in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
No theoretical assumptions or proofs are included in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code provided in the supplementary materials contains details to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We release the code for this paper in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwtsSubbmiiiioooiiooooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testing details are provided in both the experimental section and the appendix of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We clarify the statistical significance of the experiments in the experimental section of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce the computational resources used to carry out the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the appendix of this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose significant risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We state all assets used in this work in the appendix of this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper contains no crowdsourcing or research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper contains no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

A Implementation Details
A.1 Implementation of different fine-tuning algorithm
LoRA finetuning Instead of fine-tuning the entire Open model, we use Low-Rank Adaptation (LoRA) [Hu et al.) 2022 to reduce computational costs for all fine-tuning methods in this paper, including both RL and SFT. Specifically, we apply LoRA modules with rank r = 32 to all linear layers in the Open VLA model, while fully fine-tuning the value head.
PPO update
The policy is updated by maximizing the following objective (Schulman et al. 2017
To(aa 8t) Ac, clip (T (at (8))
To(aa se) (ar sx)
LCLIP (0) = EE
min
-, 1 + A.
where Ta(a; st) is calculated as the product of the probabilities of each action token, and the advantage A, is given by Generalized Advantage Estimator (GAE) [Schulman et al. 2015 where
T T-t-1
A,
frees (8(444+1) ((s+4))
1=0
GRPO implementation We follow the outcome supervision RL paradigm in GRPO (Shao et al. where the advantage is computed the normalized outcome rewards within each group:
2024
r mean(r)
A
(3)
(rd
,r
(4)
where r¹ is the outcome reward for trajectory i, and G denotes the number of trajectories in a group.
To stabilize training, use total of 32 groups, each containing trajectories. For successful placement episodes, the trajectory truncated immediately after the first successful step. In contrast, unsuccessful episodes remain untruncated. The outcome reward is given only at the final step of each episode and is defined as the sum the grasping, holding, and placement rewards.
DPO implementation We employ improved version of DPO as proposed by (Zhang al. 2024b), namely Trajectory-wise Preference Optimization (TPO), which fine-tunes the vision-language agent (VLA) by aligning its policy with human preferences over entire trajectories, rather than individual actions. This is achieved by comparing pairs of trajectories (one preferred, one rejected) and optimizing the following loss:
LIPO EEg log a (8 (8(1 (1(188 To(Gw) log
To (C(Ci)
(5)
Teet (Sw) Trer
where So and G are the preferred and rejected trajectories, respectively.
In the original TPO paper [Zhang al. 2024b) the total reward consists of task success reward, a self-evaluated score, and an LLM-generated dense reward. In our task, only sparse reward is available, as described in Sec. 5.1 Therefore, we use only this sparse reward for simplicity.
A.2 Details of motion planner and SFT dataset
We use the "plan_screw" screw" function provided by [Guo et al. 2024]. In detail, we employ task-space guided iterative inverse kinematics method to generate locally feasible path that approximates screw (helical) motion, using the robot's kinematic model (from URDF) and the current joint angles. This path is then post-processed using Time-Optimal Path Parameterization algorithm (TOPP) [Pham and Pham 2018). which assigns timing characteristics to the geometric path, resulting in an efficient and executable trajectory (in joint space) that satisfies the robot's 's kinematic constraints. We perform forward kinematics on the output joint positions and robot current joint positions to obtain the corresponding end-effector poses, and then compute their differences obtain "delta_ee_ppse"
However, data collected by the motion planner contains a fraction of idling actions. When trained with SFT on such data, the fine-tuned OpenVLA model often gets stuck during task execution. To

---

address this, we employ an action filtering technique: actions with a delta position norm less than 0.01 and delta Euler angle norm less than 0.06 are discarded. Using this filter, approximately one-third of the actions are removed, which significantly alleviates the issue of the model getting stuck. We apply action filtering to all data collected by the motion planner.
A.3 Details of assets usage
As described in Sec. 5.1 acquire digital assets from multiple sources. The 3D assets of objects and receptacles are sourced from the ManiSkill simulator [Tao et al. 2024). Objaverse [Deitke et al.) synthesized using Stable Diffusion [Rombach et al. 2022] and ControlNet [Zhang et al. 2023a]; ; specifically, we use the online service provided by LibL LibA Distractor textures are downloaded from ambientCC We comply
2023), or downloaded from Sketchfatt Additional table appearances
with the licenses of these assets accordingly.
A.4 Details of tasks for generalization evaluation
As described in Sec. 3.I we build multiple tasks test the generalization of VLA modes. The details of each task is detailed as below:
Training: At the start of each episode, one object from the 16 training objects and one of the 16 training table appearances is selected. The object and the receptacle (yellow plate) are placed on the table, with their positions randomized within a rectangular area on the table. The language instruction "put SO$ on SR$", where $O$ and $R$ denote the names the object and receptacle, respectively.
Unseen Table: The table appearance is selected from 5 unseen appearances. Other settings are the same as the Training setting.
Dynamic Texture (weak): In addition to selecting an object and a table appearance, a texture from the 16 textures is chosen at the start of each episode. This texture is cropped and resized differently at each step in the episode, and overlaid on top of the object, receptacle, and robot arm in the image, with image mixing transparency of 0.3.
Dynamic Texture (strong): The settings are the same as in the Dynamic Texture (weak) setting, except with an image mixing transparency of 0.5.
Dynamic Noise (weak): addition to selecting an object and table appearance, a texture from the 16 textures is chosen at the start of each episode. This texture is cropped and resized differently at each step in the episode, and is overlaid on whole image, with an image mixing transparency of 0.3.
Dynamic Noise (strong): The settings are the same as in the Dynamic Noise (weak) setting, except with an image mixing transparency of 0.5.
Unseen Objects: The object is selected from the 9 unseen objects. Other settings are the same in the Training setting.
Unseen Receptacles: In addition to selecting object and table appearance, receptacle from the 16 unseen receptacles is chosen at the start of each episode, replacing the default training receptacle (yellow plate). Other settings are the same in the Training setting.
Unseen Instruction Phrasing: In addition to selecting object and table appearance, language instruction template from the 16 unseen language templates is chosen at the start of each episode, replacing the default language template ("put $OS on SRS"). Other settings are the same as in the Training setting. The unseen language templates are:
Place the SO$ on the SR$
set $O$ on SRS
move the $OS the SRS
Take the $O$ and put on the $R$
https://sketchhall bttpe://iblib.art/)
httpe://ambientog.ccom
25

---

pick $OS and set it down on SR$
please put the SO$ on the $R$
Put $OS onto SRS.
place the $OS onto the SRS surface
Make sure $O$ is on SRS.
on the SRS, put the $O$
put the $O$ where the $R$ is
Move the SO$ from the table to the SRS
Move SO$ so it's on SRS.
Can you put SOS on SRS?
SOS on the $RS, please.
the SOS should be placed on the SRS.
Note that there are variations punetuation and capitalization these instruction templates.
Multi-Objert (both seen) At the start of each episode, two different objects from the 16 training objects and one of the training table appearances is selected. These two objects and the receptacle (yellow plate)
placed the table, with their positions randomized within rectangular area on the table. The language instruction is "put SO$ on $RS", where $OS and $R$ denote the names of the fist object and receptacle, respectively.
Multi- Object (both unseen) The two objects are selected from the 9 unseen objects, and they are different from each other. Other settings are the same as in the Multi- Object (both seen) setting.
Distructive Receptacle: addition to selecting an object and table appearance, a distractive receptacle from the 16 unseen receptacles is chosen at the start of each episode, and placed on the table without being used. Other settings are the same as the Training setting.
Multi- Reciiialee (both unseen): At the start of each episode, an objects from the 16 training objects, two different receptacles from the 16 unseen receptacles, and one of the 16 training table appearances is selected. The object and the receptacles their positions randomized within a rectangular area on the table. The language instruction is "put $O$ on $R$". where SO$ and $RS denote the names of the object and the first receptacle, respectively.
placed the table, with
Unseen Position (Ohject & Receptacle): The object and the receptacle (yellow plate) are placed on the table, with their positions randomized within surrounding square frame, larger than the rectangular area, on the table. Other settings are the same as in the Training setting.
Unseen Robot Init Pose: The initial pose of each articulation of the robot is randomized at the start of each episode, rather than being set to a fixed pose as in the Training setting. Other settings are the same as in the Training setting.
Mid- Miidode Object Reposition: The object teleported new random position on the table at the 5-th timestep in episode. Other settings are the same as in the Training setting.
B Additional Experiment Results
B.1 PPO performance generation temperature
We evaluate the performance of PPO with different generation temperatures during training, as shown in Fig. TO. The results show that very large temperature hinders the training process, while moderate or small temperature leads to good performance. Therefore, we use a temperature of 1.0 in PPO.
B.2 PPO performance on LoRA rank
evaluate the performance of PPO with different LoR ranks, as shown Fig. II The results show that a lower LoRA rank leads to slightly more efficient training. In all of our fine-tuning experiments, we use LoRA rank 32 to ensure sufficient model capacity. A more detailed study of different LoRA ranks is left for future work.
26

---

1.0
0.75
0.25
temp 1.0
temp-1.3 1.5
temp 0.5
250k
500k
750k
1M
Env. Steps
Figure 10: PPO performance on generation temperature during training.
0.75
Rate
0.5
0.25
LoRA rank=32
LoRA rank-8
LoRA rank-12 128
250k
500k
750k
1M
Env. Steps
Figure 11: PPO performance different LoRA rank.
B.3 Full results of OOD tasks
Detailed results of Fig. 7 are listed in Tab. where grasp accuracy, continuous grasp accuracy and task success rate are reported.
B.4 Performance comparison on sim-to-real generalization
As discussed, all of our previous experiments are conducted in simulation for scalability and repro- ducibility when comparing RL and SFT. Nonetheless, real-world performance critical factor to validate. We therefore include a preliminary real-world experiment to assess sim-to-real generaliz- ability.
Specifically, we replaced the WidowX arm with a Franka Panda and aligned the camera views. Using the same pipeline in our main experiments, retrained VLA models with both RL and SFT under this configuration, and deployed them zero-shot on the real robot. This naturally introduces an OOD setting involving kinematic differences, varied backgrounds, and different real- worldworld objects and receptacles.
In simulation, the training environment contains 16 objects (as described in Sec. 5. 7 backgrounds, and a plate as the receptacle. The real-world evaluation involves 6 new objects (pepper, cucumber, banana, mangosteen, kiwi, sponge), table as the background, and green bowl the receptacle. In both simulation and real-world setups, the initial positions objects and receptacles are randomized. For real-world evaluation, the initial conditions of each trail are aligned across SFT and RL to ensure a fair comparison. The training and evaluation environments are illustrated in Fig.
We report the average performance over 30 real-world trials in Tab. 2 The results show that RL out- performs SFT both grasp and pick-and-place success rates, demonstrating stronger generalization ability. Qualitatively, we observed that SFT often exhibits excessive movement and overshooting, leading to suboptimal grasp poses and failures. contrast, RI. shows some jitter but successfully iteratively adjusts the end-effector pose, leading to higher grasp success.
27

---

Table 1:
Detailed results across different tasks, with the best results across SFT and RL highlighted in bold. Standard deviations are reported in parentheses.
Acc.
Cont. Acc.
Suc.
SFT
RL
SFT
RL
SFT
RL
IND
0.922 (.113)
0.974 (007)
0.906 (.026)
0.948 (.007)
0.781 (013)
0.938 (.013)
Table
0.880 (.022)
0.979 (007)
0.828 (.022)
0.927 (019)
0.719 (051)
0.844 (.034)
Texture-w
0.896 (.119)
0.948 (019)
0.870 (.039)
0.906 (.046)
0.719 (.078)
0.833 (.048)
Texture-s
0.839 (.019)
0.802 (039)
0.750 (.034)
0.703 (0666)
0.557 (041)
0.630 (.039)
Noise-w
0,901 (0119)
0.948 (.027)
0.854 (041))
0.911 (.027)
0.708 (.032)
0.854 (.027)
Noise-s
0.781 (.013)
0.823 (.041)
0.688 (.034)
0.750 (0038)
0.505 (.027)
0.667 (.048)
Obj.
0,818 (,027)
0.875 (.000)
0.724 (.032)
0.833 (.039)
0.453 (.044)
0.714 (011))
Recep.
0.844 (.013)
0.911 (039)
0.802 (.032)
0.833 (.019)
0.615 (.019)
0.750 (.022)
Instruct
0.885 (,045)
0.969 (013)
0.859 (.044)
0.948 (.027)
0.672 (.034)
0.891 (.034)
M-Obj. (IND)
0.797 (U13)
0.823 (.027)
0.771 (.019)
0.797 (001))
0.615 (.052)
0.750 (046)
M-Obj. (OOD)
0,656 (U34))
0.750 (.046)
0,583 (.019)
0.688 (0038
0.297 (.013)
0.578 (034)
Disturb Recep.
0,880 (U32)
0.948 (.007)
0.844 (.044)
0.896
(.027)
0.672 (.056)
0.812 (.04))
M-Recep.
0.865 (.019)
0.885 (.007)
0.818 (.04))
0.797 (.034)
0.458 (.065)
0.599 (.063)
Obj. Pos.
0.865 (.019)
0,943 (.007)
0.802 (.00))
0.917 (011))
0.568 (.070)
0.807 (019)
Robot Pose
0.646 (.027)
0.917 (.063)
0.568 (.019)
0.844 (.071)
0.339 (.052)
0.797 (.044)
Obj. Rep.
0.573 (.039)
0.891 (.034)
0.474 (.060)
0.807 (.045)
0.286 (.041)
0.745 (.048)
(b)
(c)
(d)
Figure 12: Visualization of the training and evaluation environments for the preliminary real-world evaluation: (a)-(b) examples of simulation training environments; (c)-(d) examples of real-world evaluation settings.
B.5 Performance comparison with action chunk
To evaluate the robustness of our conclusions with stronger architectures, we additionally test with
Open VLA-OFFF [Kim et al. 2025). which enhance OpenVLA with action chunking.
We adopted an action chunking strategy with chunk size of 4, enabling the model to predict sequences of actions rather than single steps. To initialize training, we first collected 400 trajectories through motion planning without applying any action filtering, and then trained a warm-up model using a checkpoint from Huggingface
Building on this, conducted RL fine-tuning in which each observation corresponds to a se- quence of 4 actions and is optimized based on cumulative rewards on the following 4 steps. Given the increased dimensionality of the action space, we adjusted the PPO algorithm by introducing cirion-dimension. -esiessoo-iiiieeeee clipping, reducing the clipping ratio from 0.2 to 0. and tuning the hyperpa- rameters with discount factor 7 0.96 and GAE 0.85 to improve training stability.
Results are summarized in Fig. 13 and Tab. 3 showing that RI. maintains its advantage over SFT even under this more advanced architecture.
'Haozhan72/Openvlaava Stirchanrz/ppeaaaaoo-S-illeeee
28

---

Table Preliminary real-worke evaluation results. RL fine-tuning in simulation helps sim-to-real transfer compared to SFT.
SFT
RL
Grasp Success Rate
0.10
0.43
Fick-and-Place Success Rate
0.00
0.27
542
0.597
6.69
5,52
A41
0:733
3.39
IND
ON
N-ON, (MB)
M:0 (o0N)
Somantics
23. E%
36
22.4
25 6%
3.547
0.578
3.625
15-1
BA50
Texturo
24tures
hokse
Visien
ON. PLS.
PUIKE Pa52
0b: Nel.
Execution
3%
26
Figure 13:
Performance comparison between SFT and RL across different tasks with action chunking. Both success rate and relative performance drop are reported.
B.6 Extended performance comparison on opening articulated objects
To move beyond simple pick-and-place and introduce richer manipulation dexterity, we design an open articulated objects task, and replace the WidowX robot arm with Franka Panda arm to provide larger working space. The agent must grasp object's handle and open the door past 20". The reward function is shaped as follows: a reward of 0.1 is given when the handle first grasped, an additional 0.1 is granted if the grasp is sustained for five consecutive steps, and final 1.0 is provided upon successful door opening.
During training, we introduce randomization across vision, semantics, and execution to improve generalization. For vision, we randomize over 16 background assets (as described Sec. (.1). For semantics, employ 8 articulated objects that vary in size, name, handle type (bar or boxed), and handle position (up, down, left, or right). For execution, we perturb the object frame with random translations up to 8 cm and rotations up to = The training and evaluation environments are illustrated in Fig. 14
We then evaluate generalization on six cuttof-distributiin (OOD) splits:
Vision: 5 novel background scenes,
Vision: whole-scene dynamic noise with an image mixing transparency of 0.5,
Semantics: 4 unseen articulated objects (microwave, dishwasher, mini-fridge, oven),
Semantics: 16 re- re-phresed language instructions,
Execution: translations up to and rotations up to + /24,
Execution: random initial offsets on joints 0/1/2/4 the robot arm.
As shown in Tab. RL continues to outperform SFT on semantic and execution splits, consistent with our main pick-and results.
29

---

Table Detailed results across different tasks with action chunking, with the best results across SFT and RL highlighted in bold. Standard deviations are reported in parentheses.
Acc.
Cont. Acc.
Suc.
SFT
RL
SFT
RL
SFT
RL
IND
0.984 (.13))
0.969 (022)
0.953 (.026)
0.953 (.022)
0.776 (0229)
0.891 (.44))
Table
0.964 (.007)
0.948 (019)
0.953 (.013)
0.948 (019)
0.812 (046)
0.885 (.041)
Texture-w
0.844 (,046)
0.885 (015)
0.807 (.022)
0.870 (.077)
0.547 (.022)
0.792 (.029)
Texture-s
0.719 (.034)
0.781 (0566
0.661 (.015)
0.734 (.066
0.401 (019)
0.578 (.07)
Noise-w
0,932 (0115)
0.974 (.007)
0.880 (.019)
0.958 (.027)
0.703 (.046)
0.870 (003)
Noise-s
0.755 (.039)
0.839 (.041)
0.724 (.019)
0.786 (.045)
0.401 (.058)
0.625 (.022)
Obj.
0,807 (,007)
0.901 (.01))
0.740 (.007)
0.875 (.022)
0.542 (.037)
0.693 (.037)
Recep.
0,927 (.027)
0.932 (004)
0.885 (.019)
0.911 (,053)
0.609 (011)
0.682 (.032)
Instruct
0.620 (,007)
0.641 (0113
0.604 (001))
0.630 (.019)
0.479 (.029)
0.562 (.026)
M-Obj. (IND)
0,661 (U75)
0.745 (.019)
0.641 (.080)
0.714 (032)
0.542 (.059)
0.641 (.066)
M-Obj. (OOD)
0.589 (008)
0.661 (.019)
0.552 (.011)
0.620 (.019)
0.365 (.032)
0.495 (048)
Disturb Recep.
0.901 (.22))
0.870 (002)
0.891 (.022)
0.849 (048)
0.599 (.039)
0.719 (.U6)
M-Recep.
0.891 (.034)
0.859 (038)
0.849 (.060)
0.828 (.058)
0.365 (.032)
0.297 (.056)
Obj. Pos.
0,807 (0155)
0.917 (.02))
0.771 (015)
0.896 (002))
0.453 (.022)
0.682 (.045)
Robot Pose
0.557 (.019)
0.797 (.034)
0.484 (.034)
0.734 (0046)
0.281 (.013)
0.589 (.029)
Obj. Rep.
0.219 (.038)
0.510 (015)
0.177 (.041)
0.453 (.013)
0.068 (055)
0.318 (.007)
(b)
(c)
Figure 14: Illustration of the open articulated objects task. (a)-(b) Two representative training scenarios. (c) Set of articulated objects used in training (left two rows) and in OOD) testing (the most right row).
C Broader impacts
Our work advances the understanding and generalization capabilities of Vision-Language Action (VLA) models embodied AI, which may accelerate the deployment of more robust and adaptive agents in real-world environments. Improved generalization can enhance the reliability of AI systems applications such as assistive robotics, autonomous navigation, and household automation, potentially benefiting society by making these technologies safer and more accessible.
However, VLAs become more capable and generalizable, there is also risk of unintended consequences. such misuse in surveillance, privacy violations, or deployment in safety-critica scenarios without adequate oversight. We encourage responsible use of our findings and recommend that future deployments of VLA-based systems consider safety, ethical, and privacy implications.
30

---

Table 4: Performance comparison the open articulated objects task, with results averaged over
random seeds. Standard deviations are reported parentheses.
SFT
RL
Success Rate
Degradation Ratio
Success Rate
Degradation Ratio
IND
0.745 (011)
0.724 (.045)
New Backgrounds
0.766 (.71)
+0.028
0.734 (058)
+0,014
Dynamic Noise
0.526 (.015
-0.294
0.448 (.037)
-0,381
Vision Avg.
0.646 (.043)
-0.130
0.591 (.048)
-0.184
Articulated Obj.
0.042 (.015)
-0.944
0.151 (.019)
-0.791
Re-phrased Inst.
0.703 (.077)
-0.056
0.620 (041)
-0.144
Semantic Avg.
0.373 (.046)
-0.500
0.386 (0000
-0.468
Obj. Pose
0.448 (0022
-0.399
0.484 (0899
-0.331
EE Pose
0.151 (.27)
-0.797
0.312 (013)
-0.568
Execution Avg.
0.300 (.030)
-0.600
0.398 (.051)
-0.450
31


Reviews:
[{"review_id": "rhJcL5FXQN", "reviewer": "Reviewer_QKnh", "summary": "This paper presents an empirical evaluation of the effect of RL training of VLA models on their subsequent generalization performance.  Generalization is defined along Visual, Semantic, and Execution dimensions.  The paper also provides experiments and discussion to help those wishing to employ RL for fine-tuning choose the specific method to use.", "questions": "Some small points, discussion of which may improve the paper:\n\n- In section 4.2, can you please expand on the choice of $h^0$?  What does the superscript there signify and what might the improved performance of that choice suggest about what information the critic is using.\n- In line 280, the paper suggests that trial-and-error RL is able to generate a general \"grasp\" skill.  Why would this not work for vision to generate a general \"perception\" of graspable objects, as discussed immediately prior?  Can you clarify why those situations might be different?\n- Could you address the motion-planning limitation in simple ways, by introducing noise, for example?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper is a very clear empirical study of a particular method and its effects on well-defined dimensions.  The problem tackled is well-motivated and timely, providing insight into tradeoffs for a difficult decision about how to fine-tune VLAs.  The scope of the paper is well defined, and the paper does not claim more than it shows.  The primary weaknesses of the paper are already identified by the authors:  motion-planner demonstrations on simple, single-task benchmarks.  Despite these weaknesses, the paper will be interesting to many who are interested in VLAs.", "was_revised": true, "final_justification": "Thank you for your thorough rebuttal, which has brought some more clarity around the issues I raised.  I think this is a solid paper that should be accepted, and so will maintain my score.", "rebuttal": "We sincerely appreciate the reviewer’s supportive comments on our contributions and the potential impact on the community. Below, we address your concerns clearly and thoroughly.\n\n> ### Q1: Clarification on $h^0$ and analysis of critic information\n\nThank you for pointing out the ambiguity regarding $h^0$. We will include an expanded and clarified explanation in the revised version.\n- **Detailed Explanation of $h^0$**: We adopt the commonly used discrete action tokenization approach from prior work [1,2], where each action dimension is discretized into 256 bins and subsequently tokenized, projecting actions into a token space. Specifically, we denote the final-layer embedding corresponding to each action token as $h^i$, where the index $i$ refers to the $i$-th action dimension. Consequently, $h^0$ specifically represents the embedding associated with the first action token.\n- **Rationale for Selecting $h^0$ as Critic Input**: In PPO, the critic is defined as the state-value function $V(s)$, which theoretically necessitates state-only information as input. In VLA models, observations encompass both visual inputs (images) and textual instructions, from which the VLM backbone extracts meaningful state representations. Then, a critical question arises: **Why specifically choose $h^0$ over other embeddings for approximating $V(s)$, i.e., $V(s)\\approx V(h^0)$?**\n\n  Our hypothesis is that the embedding serving as input to the critic should predominantly capture state information with minimal interference from action-related information. Empirically, $h^0$ sits precisely at the boundary between state and action embeddings in the auto-regressive VLA. To substantiate this choice, we have conducted experiments (Fig. 4b) demonstrating significant performance degradation when using embeddings that encapsulate more action-oriented information—such as $h^n$, corresponding to the last action token, or $h^{all}$, representing the concatenation of all action token embeddings. We attribute this performance drop to action embeddings introducing unstable, noisy signals during training, inherently conflicting with PPO's requirement that critic inputs strictly represent state information.\n\n[1] OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024.\n\n[2] RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023.\n\n> ### Q2: Why RL works for \"grasp\" skills but not general \"perception\"\n\nWe appreciate the reviewer’s thoughtful question. The distinction arises from which component of the VLA the generalization challenge primarily stresses. Our evaluation framework is explicitly designed to probe *Vision*, *Semantics*, and *Execution* generalization along separate axes, each of which targets different stages of the perception-to-action process.\n\nIn **Semantics**, the challenge lies in generalizing across object identities, instructions, and receptacle configurations. RL benefits from trial-and-error by refining its policy through interaction—e.g., learning how to retry or adapt grasping behaviors in response to failures. As shown in Fig. 7 and Fig. 9, this enables RL to outperform SFT in unseen object scenarios by learning robust strategies.\n\nIn contrast, **Vision** generalization introduces perturbations (e.g., novel textures, noise, unseen tables) that mostly challenge the perception module. While we do jointly train the vision encoder with RL, the signal from policy gradients remains sparse and task-driven, making it harder to recover from upstream perception errors such as mislocalization or occlusion. These errors degrade input quality before the agent can reason or act effectively. As a result, RL struggles to improve perception robustness beyond what is already achieved during pretraining, leading to comparable performance drops to SFT. \n\nWe will clarify this in the revision to emphasize that all three axes—*Vision*, *Semantics*, and *Execution*—are essential and complementary in evaluating VLA generalization. \n\n> ### Q3: Limited motion-planning-based data collection.\n\nWe appreciate the reviewer’s insightful comment regarding the use of motion-planner demonstrations. We acknowledge this as a potential limitation and have discussed it in the limitations section. In our experiments, we have tuned our motion planner to explore a larger space. Besides, we also tried to combine motion-planned data with demonstrations collected using a pretrained policy (e.g., Octo), as noted in Sec. 4.2 (lines 213–214). However, we found that the performance difference between motion-planned and Octo-collected data was minimal for SFT. Consequently, we scaled up motion-planning-based demonstrations (up to 64k) for our main SFT experiments.\n\nRegarding the suggestion of injecting noise into the planner, we note that while simple noise can increase variability, it also risks degrading demonstration quality for the whole dataset. Designing noise that balances meaningful exploration with high-quality trajectories is non-trivial. Alternatively, more principled approaches—such as integrating neural motion planners [3] or further employing hierarchical pipelines like ManipGen [4]—could offer more robust data diversity. These are promising directions we plan to explore, but given the time constraints of the review process, we leave them as future work.\n\n[3] Neural MP: A Generalist Neural Motion Planner. IROS 2025.\n\n[4] Local Policies Enable Zero-shot Long-Horizon Manipulation. ICRA 2025."}, {"review_id": "HtnkmLvBdY", "reviewer": "Reviewer_ePmx", "summary": "This paper introduces a new benchmark to test the generalization capabilities of VLAs on a pick-and-place task, focusing on three factors: 1) visual appearance generalization, 2) semantic-level generalization, and 3) execution level generalization. It picks OpenVLA as the backbone model, and performed a thorough empirical study over several RL algorithms, including PPO, GRPO, and DPO, compared with SFT. The paper draws the conclusions that 1) RL works better in semantics and execution, but performs equally well in vision; 2) PPO works the best. It also gives insights about the design choices of PPOs for better performance in RL.", "questions": "1. Is it possible to verify the conclusions on a real-robot? Even a simple task would be helpful.\n\n2. Have the authors tried a more advanced policy, e.g., pi0?\n\n3. Have the authors tried a standard off-policy RL algorithm, e.g. SAC / MPO, instead of the methods from RLHF literature?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThis paper studies an interesting and important question: what can RL bring to VLA generalization, which from my point of view, is an important topic the community will have to work on in the future. It has also designed a few meaningful tasks and performed a thorough study over the algorithms.\n\nWeaknesses\n\nThere are a few weaknesses of the work and I’m very concerned about whether the results and insights can be transferred to real-world settings.\n\n1. The biggest issue is that the findings are drawn from an on-policy RL algorithm, PPO. However, it is extremely challenging to deploy any on-policy RL algorithms on a real robot due to the poor sample efficiency. In Fig. 6, all plots need 2m steps, which is merely impossible for a real robot. In addition, the only applicable algorithm, DPO, doesn’t seem to give any performance gain.\n2. The other issue is the choice of the base model, OpenVLA. OpenVLA predicts a single-step action per time-step. In contrast, latest VLA policies often predicts an “action chunk” which consists of multi-step predictions. Given an action-chunk policy, it is possible that modifications to the RL algorithms are required to make it work. Besides, these VLA models often rely on diffusion policies / flow matching policies, which might require extra efforts during RL.\n3. The benchmark is overly simplified as a pick-place task. The conclusions can be hard to transfer to tasks that require more dexterous skills, e.g., folding clothes.", "was_revised": true, "final_justification": "I think the paper do have some merit, as also agreed by other reviewers. However, I'm still concerned about its way of running experiments. On-policy RL algorithms have natural limitations when applied to robotics, and I think from the very beginning, choosing this family of algorithms is a less reasonable decision choice. However, I also feel this work has certain merits and I'm not opposed to an acceptance.", "rebuttal": "Thank you for your insightful comments and for highlighting these important considerations. We appreciate the opportunity to further clarify the goals and contributions of our paper.\n\nWe acknowledge that real-world RL for VLA and advanced versions with flow matching are still in early development stages. Our focus, therefore, is not on achieving SOTA performance, but on providing a thorough comparative analysis of RL versus SFT that offers insights applicable across different methods and models.\n\nTo this end, we **strategically chose a simulation-based evaluation** for two compelling reasons:\n\n- **Rigorous benchmarking**: Large-scale controlled simulations are essential for comprehensive ablation studies, reproducibility, and broad scenario coverage—critical requirements for establishing a benchmark with lasting research impact. Real-world trials alone are hard to provide this level of systematic analysis.\n- **Intrinsic merit**: Simulation-based RL training has its own value, especially given ongoing debates regarding whether simulation-based training can enhance VLA's abilities. Concurrent works [1,2] have also demonstrated the potential advantages of on-policy RL trained in simulated environments. To address further concerns, **we have included preliminary real-world results validating sim-to-real transfer** during this rebuttal period (detailed in our Q1 response).\n\nTo support the generalizability of our findings, we have conducted extensive additional experiments, including tests with alternative VLA architectures, more complex tasks, and off-policy RL algorithms. These results consistently validate our core findings about the distinct effects of RL in comparison to SFT.\n\n[1] Interactive Post-Training for Vision-Language-Action Models. arXiv:2505.17016\n\n[2] VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. arXiv:2505.18719\n\nBelow, we will address your specific questions.\n\n> ### Q1: Additional experiments to verify the conclusions on a real-robot\n\nAs discussed, our experiments are intentionally conducted in simulation for scalability and reproducibility when comparing RL and SFT. Nonetheless, we fully agree that real-world performance is a critical aspect to validate and have therefore included a preliminary real-world experiment to assess sim-to-real generalizability—adding an additional generalization dimension to our evaluation.\n\nAccording to our real-world setups, we adapted the simulation by replacing the WidowX arm with a Franka Panda and aligning the camera views. Using the same pipeline as in our main experiments, we re-trained VLA models with both RL and SFT under this new configuration and deployed them zero-shot on the real robot. This setting introduces a challenging OOD setting involving kinematic differences between the simulated and real arms, varied backgrounds, and different objects (including pepper, cucumber, banana, mangosteen, kiwi, and sponge) and receptacles (in specific, a green bowl). We report the average performance over 30 real-world trials in this table:\n\n||Success Rate for Grasping|Success Rate for full Pick-and-Place|\n|-|-|-|\n|RL|**0.43**|**0.27**|\n|SFT|0.1|0.0|\n\nAs shown in the table, RL fine-tuning in simulation can also help the sim-to-real generalization compared to SFT. We observed that when approaching the object, SFT often exhibits excessive movement and tends to overshoot, resulting in a suboptimal grasping pose and subsequent grasp failures. In contrast, RL, despite showing some jitter in its actions, is able to iteratively adjust its grasping pose around the object. This leads to a higher grasp success rate.\n\n> ### Q2: Additional experiments with a standard off-policy RL algorithm (e.g. SAC)\n\nWe sincerely thank the reviewer for this suggestion. We agree that evaluating our approach with a standard off-policy algorithm like SAC would further strengthen our work.\n\nHowever, applying SAC to OpenVLA is non-trivial, especially within the limited rebuttal period. Our VLA model features a discrete, high-dimensional action space (7 dimensions, each with 256 bins), requiring techniques like the Gumbel-Softmax trick for differentiability. This results in a very high-dimensional (256 × 7) action input for the Q-network, making stable learning particularly challenging.\n\nDespite these technical hurdles, we are trying our best to conduct these experiments and hope to provide these additional results before the discussion period concludes. **We are fully committed to incorporating the complete results and a detailed analysis of the SAC experiments into a revised version of our paper.** We appreciate the reviewer's valuable feedback to further complement our study.\n\n> ### Q3: Additional experiments with a more advanced VLA model\n\nThank you for raising this insightful question regarding the choice of our base model. We agree that advanced VLA architectures, such as Pi-0, which incorporate diffusion or flow-matching policies, indeed pose unique challenges for RL fine-tuning. To date, end-to-end RL methods for models like Pi-0 have not been convincingly demonstrated in the literature, and developing such methods would extend beyond the scope of our systematic comparative study.\n\nNevertheless, we acknowledge the importance of evaluating our conclusions with a more advanced VLA architecture. To address your concern, we have introduced an additional model, OpenVLA-OFT [3], which enhances the original OpenVLA model by integrating action chunking. To adapt our RL pipeline for OpenVLA-OFT, we made the following modifications:\n1. **Action Chunking:** We utilized an action chunk size of 4.\n2. **Warm-up Model Training:**\n   - Collected 400 trajectories via motion planning without action filtering.\n   - Trained a warm-up model from `Haozhan72/Openvla-oft-SFT-libero10-trajall` (a checkpoint from Huggingface).\n3. **RL Fine-tuning Adjustments:**\n   - Each observation predicts a sequence of 4 actions, with cumulative rewards.\n   - To accommodate the higher dimensional action space, we modified the PPO clipping mechanism to perform action-dimension-wise clipping and reduced the clip ratio from 0.2 to 0.1, and set gamma to 0.96 and lambda to 0.85 for stability.\n   \nResults show that RL maintains its advantage over SFT even with this more advanced architecture:\n\n|Tasks|SFT Success Rate (3 seeds)|SFT Decline Ratio|RL Success Rate (3 seeds)|RL Decline Ratio|\n|-|-|-|-|-|\n|In Distribution|0.536||0.891||\n||||||\n|Table|0.568|**0.058**|0.885|-0.006|\n|Texture-w|0.438|-0.184|0.792|**-0.111**|\n|Texture-s|0.328|-0.388|0.578|**-0.351**|\n|Noise-w|0.464|-0.136|0.870|**-0.023**|\n|Noise-s|0.349|-0.35|0.625|**-0.298**|\n|**Vision Avg.**|0.429|-0.2|0.75|**-0.158**|\n||||||\n|Obj.|0.438|**-0.184**|0.693|-0.222|\n|Recep.|0.349|-0.35|0.682|**-0.234**|\n|Instruct|0.318|-0.408|0.562|**-0.368**|\n|M-Obj. (IND)|0.438|**-0.184**|0.641|-0.281|\n|M-Obj. (OOD)|0.234|-0.563|0.495|**-0.444**|\n|Disturb Recep.|0.432|-0.194|0.719|**-0.193**|\n|M-Recep.|0.182|**-0.66**|0.297|-0.667|\n|**Semantic Avg.**|0.342|-0.363|0.584|**-0.344**|\n||||||\n|Obj. Pos.|0.266|-0.505|0.682|**-0.234**|\n|Robot Pose|0.188|-0.65|0.589|**-0.339**|\n|Obj. Rep.|0.057|-0.893|0.318|**-0.643**|\n|**Execution Avg.**|0.17|-0.683|0.53|**-0.405**|\n\n[3] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. RSS 2025.\n\n> ### Q4: Additional experiments on more complex tasks\n\nThank you for highlighting the need for a task that demands richer dexterity than pick‑and‑place. To address this, we extended the benchmark with an **open‑cabinet task** that requires grasping a handle and opening the door past 20 °. The reward function assigns 0.1 when the handle is first grasped, 0.1 when the grasp is sustained for five consecutive steps, and 1.0 upon successful door opening. During training, we randomized:\n- Vision: 16 backgrounds from the original assets.\n- Semantics: 8 cabinets of varying sizes, names, handle types (bar, boxed) and handle positions (up, down, left, right).\n- Execution: Cabinet frame translations (±8 cm) and rotations (±π/48).\n\nWe then test on **six OOD splits**: (i) 5 new backgrounds, (ii) whole‑scene dynamic noise, (iii) 4 novel articulated objects (e.g., microwave, mini‑fridge), (iv) 20 re‑phrased language instructions, (v) cabinet translations up to 12 cm with rotations up to ±$\\pi$/24, and (vi) random initial offsets on joints 0/1/2/4 of the end‑effector.\n\nAmong all splits, RL fine-tuning outperforms SFT on execution and semantic OOD tests, and performs comparably on vision, consistent with our findings on the pick-and-place task.\n\n\n|splits|SFT Success Rate (3 seeds)|SFT Decline Ratio|RL Success Rate (3 seeds)|RL Decline Ratio|\n|-|-|-|-|-|\n|In Distribution|0.745||0.724|-|\n||||||\n|New Backgrounds|0.766|**0.028**|0.734|0.014|\n|Dynamic Noise|0.526|**-0.294**|0.448|-0.381|\n|**Vision Avg.**|0.646|**-0.13**|0.591|-0.1835|\n||||||\n|Articulated Obj.|0.042|-0.944|0.151|**-0.791**|\n|Re‑phrased Inst.|0.703|**-0.056**|0.620|-0.144|\n|**Semantic Avg.**|0.373|-0.5|0.386|**-0.4675**|\n||||||\n|Obj. Pose|0.448|-0.399|0.484|**-0.331**|\n|EE Pose|0.151|-0.797|0.312|**-0.568**|\n|**Execution Avg.**|0.3|-0.6|0.398|**-0.4495**|\n\n---\n---\nWe deeply appreciate your thorough review and constructive feedback, which has enabled us to strengthen our work substantially. We are truly encouraged by your recognition of our study as “thorough”, as our primary goal was to provide the community with **a systematic and reproducible empirical comparison of RL and SFT**, rather than to propose a new state-of-the-art algorithm.\n\nIn response to your concerns, we have conducted a wide range of new experiments within the limited rebuttal period. We believe that these additional results further enhance the integrity of our study. **We kindly ask you to reconsider your evaluation in light of our clarifications and the new experiments.**"}, {"review_id": "P1Ze1eePnj", "reviewer": "Reviewer_84hZ", "summary": "The paper can be split into two parts. The first part shows that for RL fine-tuning VLAs, PPO is more effective than LLM-derived methods like DPO, GRPO, and explores design choices of PPO such as critic design, warm-up, and UTD tuning. The second shows that RL tuning (with PPO) has superior generalization capabilities compared to supervised fine-tuning (SFT). For this, the authors design a novel benchmark (based on Maniskill) to systematically evaluate generalizability on novel vision, novel semantics (e.g., unseen objects), and novel execution (e.g., random initial state).", "questions": "1. In Figure 6a and 6b, the performance does not plateau after 16k but visibly drops on success rate. Why do you think this happened?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Straightforward and concrete experimental setup, and strong results.\n- Extensive investigation on VLA's generalization in pick-and-place with diverse out-of-domain scenarios, accompanied with interesting analyses on the results (figure 7,8,9).\n\nWeaknesses\n- The paper spends a great amount of writing space to PPO itself, which seems redundant to the main topic (generalization). I think the paper would have been more coherent if the design choices were investigated on their effect to generalization capabilities (although I'm not sure about the connection between generalization and the components in Section 4.2).\n- The dynamic texture and dynamic noise (applying flat texture noise to the robot/object or to the entire view) design seems a bit unrealistic; I doubt that those visual noises will actually occur in practice.\n- As mentioned in the limitation, the quality of the SFT dataset is far from satisfactory (figure 8a), which raises questions on the results (especially that of figure6a,b).", "was_revised": true, "final_justification": "I think the paper has enough contribution to the community to be accepted as a paper (dataset and pipeline for evaluating generalization of VLAs). However, the overall structure and quality of the paper holds me from giving a higher score. Although I came to agree that successfully training VLA with RL can be a contribution, I still believe it is not a significant one.", "rebuttal": "We sincerely thank the reviewers for their valuable feedback and positive evaluation of our research. Below, we provide detailed responses to each of your comments.\n\n> ### Q1: Meaning of our PPO recipe and experiments\n\nThank you for your attention to the desciptions devoted to PPO and Section 4.2. We understand the concern and appreciate the opportunity to clarify its role in the paper.\n\nWhen we conducted this study, applying PPO to finetune a VLA model remained a non-trivial and open problem. Although there have been some emerging concurrent work [1,2] on PPO+VLA since our submission, our work is one of initial studies in this direction. There was no established or off-the-shelf RL method that could directly be used to study generalization in our study. As such, developing an effective PPO training recipe was necessary to enable a systematic and fair comparison of generalization between RL and SFT. Without this foundation, it would be difficult to make credible claims about RL’s potential contributions to generalization.\n\nMoreover, the training decisions described in Section 4.2 are not standard or trivial; through empirical tuning, we identified several key design choices that significantly impact performance and training efficiency. These insights are valuable in their own right and can guide future work on applying RL to VLA models.\n\nBesides, the effects of these design choices are not isolated to in-distribution performance but show consistent relative performance for out-of-distribution tests. Thus, we base the rest of our comparative experiments on this PPO recipe.\n\nWe will make these motivations and the connection to generalization more explicit in the revised version to improve clarity and coherence.\n\n[1] Interactive Post-Training for Vision-Language-Action Models. arXiv:2505.17016\n\n[2] VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. arXiv:2505.18719\n\n> ### Q2: Is it realistic and practical to apply visual noise as done in our experiments?\n\nThank you for raising this concern. Stress-testing vision with deliberately extreme textures or lighting is already standard practice for gauging a manipulation policy’s perceptual invariance.\n- **Purpose-built stress tests are widely accepted**. The DMControl generalization benchmark [3] suite overlays randomly sampled textures and colours on every frame to create out-of-distribution (OOD) views; many works like [4] evaluate policy performance under these texture-shifted test sets, treating them as a proxy for unexpected backgrounds and camera disturbances.\n- **Real-robot studies adopt the similar idea**. Prior work on manipulation like [5] evaluates the same policy after changing table, background colours, camera viewpoints, and dynamic Disco lighting conditions, demonstrating that such appearance swaps are considered a practical robustness check.\n- **Community benchmarks adopt similar disturbed evaluation**. The Colosseum benchmark [6] includes “Texture Swap” and “Lighting Perturbation” splits in which object and table surfaces, as well as global illumination hues, are randomised at test time; leading methods are ranked by how gracefully they handle these overlays.\n\nOur Dynamic-Texture and Dynamic-Noise splits follow the same philosophy: although the specific patterns may be unlikely to appear in daily operation, they create a controlled yet harsh visual perturbation that reveals whether a policy has learned stable, object-centric features rather than over-fitting to background colour or illumination. Therefore, the evaluation protocol is both practical and comparable to widely adopted benchmarks.\n\n[3] Generalization in reinforcement learning by soft data augmentation. ICRA 2021\n\n[4] Generalizable Visual Reinforcement Learning with Segment Anything Model. arXiv:2312.17116\n\n[5] Learning to Manipulate Anywhere: A Visual-Generalizable Framework for Reinforcement Learning. CoRL 2024\n\n[6] The Colosseum: A Benchmark for Evaluating Generalization for Robotic Manipulation. RSS 2024\n\n\n> ### Q3: The quality of the SFT dataset\n\nThank you for your insightful question. We acknowledge the quality limitation of our SFT dataset as mentioned in the paper. To address this concern, we have taken steps to improve data diversity by tuning our motion planner to explore a larger state space. Additionally, as noted in Section 4.2 (lines 213-214), we experimented with combining motion-planned data with demonstrations collected using a pretrained policy (e.g., Octo). However, our experiments revealed that the performance difference between motion-planned and Octo-collected data was minimal for SFT. Therefore, considering scalability advantages, we adopted the motion planner approach to collect up to 64k trajectories.\n\nRegarding the performance drop after 16k trajectories in Figures 6a and 6b, we hypothesize that with larger datasets, it becomes increasingly challenging for the policy to effectively cover all training samples, potentially leading to the observed performance degradation. It's worth noting that while the success rate shows a decline, the grasp accuracy and continuous grasp accuracy metrics appear to plateau, suggesting that the policy maintains certain capabilities despite the overall performance drop. This phenomenon warrants further investigation, which we plan to investigate further in future work."}, {"review_id": "8AbfLJ7qtg", "reviewer": "Reviewer_vvPb", "summary": "The authors propose a novel benchmark to text the generalization capabilities of Vision-Language Action (VLA) models. They are able to text the generalization capabilities of the VLA models along the vision, semantics and execution axis. The authors then test the performance of SFT and a few RL algorithms on different OOD dimensions. They found that PPO generalizes better than SFT and some RL baselines for semantic understanding and execution robustness, while it matches the baselines for visual tasks. The authors also provide an insightful analysis of the exploration performed by RL in Figure 8.", "questions": "* For PPO epoch >1, do you recompute the advantages using the updated value function?\n* Can you compare your study to the ones reported in Figure 4 of RT2?", "limitations": "* The authors mostly compared PPO against algorithms designed for single step LLM training. It is not surprising that they do not perform very well on multi-step robotic tasks. A more interesting comparison would be to compare against RL algorithms such as A3C, TD3, VMPO, etc.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths \n\n## Quality\n* Well designed benchmark to test OOD performance along controllable axis: vision, semantics, and execution. The field needs more carefully designed empirical work like this.\n* Multiple seeds (2 and 3) are used. While more seeds could be used, it is a big improvement over a single seed.\n\n## Novelty\n* Surprisingly most VLA are not trained via RL e.g. RT2, \\pi_0, Octo, etc. By showing the advantages of PPO over the SFT baseline, this work motivates further investigation for training VLA with RL methods.\n\n## Impact\n* The authors provide an efficient training recipe (42h on a A100 gpu) and several ablation (shared backbone, val warm up and number of ppo epochs) for the PPO design choices. This simple and efficient recipe is impactful will allow other academic lab to build on this work.\n\n## Visualization\n* The visualization of the training trajectories in Figure 8 is interesting. \n\n# Weaknesses\n\n## Comprehensiveness\n* The authors test a single model (OpenVLA) and a single task (pick and place).\n\n## Significance\n* The main takeaway is not surprising: RL algorithms designed for multi-step tasks (PPO) perform better on multi-step tasks than algorithms designed for single-step tasks (GRPO and DPO). I think the paper would be more interesting if the RL algorithms used for comparison were carefully chosen, e.g. soft actor critic, etc.\n\n\n## Clarity\n* In Figure 1, I am not sure that the emojis are very useful nor what they imply. It seems like the OOD RL performance drops similarly across vision, semantics and execution, but 3 different emojis are used.", "was_revised": true, "final_justification": "This paper introduced an interesting benchmark to evaluate Vision-Language Action models on out of distribution tasks. While doing a literature review, I was surprised that RL has not been widely explored by previous work. Therefore, I think their contribution is worth accepting to the conference.", "rebuttal": "We sincerely appreciate the reviewer’s supportive comments on our strengths across the quality, novelty, impact, and visualization. Below, we address your remaining concerns clearly and thoroughly.\n\n> ### Q1: Extend to more VLA models and more complex tasks\n\nThank you for raising this important concern. While our original submission focused on a single model (OpenVLA) and task (pick-and-place) to establish a controlled foundation for comparison, we have since expanded our study in two complementary directions to reinforce the generality of our findings.\n\n**First, to evaluate broader VLA architectures**, we conducted additional experiments on **OpenVLA-OFT** [3], a stronger variant of OpenVLA that incorporates **action chunking**. This model poses increased challenges for RL training due to its higher-dimensional, temporally extended action space. We adapted our RL pipeline accordingly—modifying reward computation, clip ratio, and advantage estimation—and observed consistent improvements from RL fine-tuning over SFT. These results indicate that the benefits of RL are not confined to the original model and extend to more complex action representations.\n\n|Tasks|SFT Success Rate (3 seeds)|SFT Decline Ratio|RL Success Rate (3 seeds)|RL Decline Ratio|\n|-|-|-|-|-|\n|In Distribution|0.536||0.891||\n||||||\n|Table|0.568|**0.058**|0.885|-0.006|\n|Texture-w|0.438|-0.184|0.792|**-0.111**|\n|Texture-s|0.328|-0.388|0.578|**-0.351**|\n|Noise-w|0.464|-0.136|0.870|**-0.023**|\n|Noise-s|0.349|-0.35|0.625|**-0.298**|\n|**Vision Avg.**|0.429|-0.2|0.75|**-0.158**|\n||||||\n|Obj.|0.438|**-0.184**|0.693|-0.222|\n|Recep.|0.349|-0.35|0.682|**-0.234**|\n|Instruct|0.318|-0.408|0.562|**-0.368**|\n|M-Obj. (IND)|0.438|**-0.184**|0.641|-0.281|\n|M-Obj. (OOD)|0.234|-0.563|0.495|**-0.444**|\n|Disturb Recep.|0.432|-0.194|0.719|**-0.193**|\n|M-Recep.|0.182|**-0.66**|0.297|-0.667|\n|**Semantic Avg.**|0.342|-0.363|0.584|**-0.344**|\n||||||\n|Obj. Pos.|0.266|-0.505|0.682|**-0.234**|\n|Robot Pose|0.188|-0.65|0.589|**-0.339**|\n|Obj. Rep.|0.057|-0.893|0.318|**-0.643**|\n|**Execution Avg.**|0.17|-0.683|0.53|**-0.405**|\n\n**Second, to evaluate more complex tasks**, we introduced a more challenging **open-cabinet manipulation** benchmark. This task requires more dexterous manipulation with a articulated object, including handle grasping and door opening under significant semantic and geometric variability. Across six out-of-distribution splits—including novel objects, diverse cabinet configurations, and perturbed initial states—RL-fine-tuned policies again consistently outperformed SFT, especially on **execution** and **semantic generalization**. This echoes the trend observed in the pick-and-place setting and provides further evidence that RL enhances robustness across manipulation tasks.\n\n|splits|SFT Success Rate (3 seeds)|SFT Decline Ratio|RL Success Rate (3 seeds)|RL Decline Ratio|\n|-|-|-|-|-|\n|In Distribution|0.745||0.724|-|\n||||||\n|New Backgrounds|0.766|**0.028**|0.734|0.014|\n|Dynamic Noise|0.526|**-0.294**|0.448|-0.381|\n|**Vision Avg.**|0.646|**-0.13**|0.591|-0.1835|\n||||||\n|Articulated Obj.|0.042|-0.944|0.151|**-0.791**|\n|Re‑phrased Inst.|0.703|**-0.056**|0.620|-0.144|\n|**Semantic Avg.**|0.373|-0.5|0.386|**-0.4675**|\n||||||\n|Obj. Pose|0.448|-0.399|0.484|**-0.331**|\n|EE Pose|0.151|-0.797|0.312|**-0.568**|\n|**Execution Avg.**|0.3|-0.6|0.398|**-0.4495**|\n\nWhile each extension targets a different axis (model or task complexity), both strengthen our core conclusion: **RL fine-tuning offers more robust generalization than SFT, particularly in execution and semantic variation**, and this advantage holds across models and tasks of increasing complexity.\n\n> ### Q2: Extended to more RL algorithms like SAC\n\nWe sincerely thank the reviewer for this thoughtful suggestion. We agree that evaluating our approach with a standard off-policy algorithm such as SAC would be a valuable addition to the study and could further enrich our comparative analysis.\n\nThat said, we would like to emphasize that our current comparisons among PPO, GRPO, and DPO still offer meaningful contributions. GRPO and DPO represent recent advances specifically designed to fine-tune policies from demonstrations and preferences—both widely discussed in the context of language and vision-language models. Our finding that PPO, an on-policy RL algorithm, outperforms GRPO and DPO on embodied multi-step tasks is not merely expected—it provides a concrete empirical validation that these methods may fall short when applied to sequential VLA control tasks. To our knowledge, this is the first systematic comparison of these algorithms on such models, and the insights gained are directly actionable for researchers adapting RL methods to VLA domains.\n\nRegarding the inclusion of SAC, we acknowledge its relevance and have actively considered it. However, applying SAC to our setting is non-trivial, as OpenVLA operates in a discrete, high-dimensional action space: 7 dimensions with 256 bins each. This results in an effective action space of 256⁷ combinations, making traditional continuous-action SAC inapplicable without substantial adaptation. One workaround involves Gumbel-Softmax-based relaxation to approximate gradients for discrete actions, but this results in a 1792-dimensional action input to the Q-network, posing severe challenges for learning a stable and expressive critic.\n\nDespite these challenges, we are working diligently to explore SAC-based experiments within the remaining rebuttal/discussion period and will incorporate complete results in a revised version of the manuscript. We appreciate the reviewer’s suggestion and view it as a valuable direction for future iterations of this work.\n\n> ### Q3: Implication of emojis\n\nThank you for pointing this out, and we apologize for any confusion caused by the use of emojis in Figure 1. Our intent was not to suggest the absolute performance drop of RL in each OOD setting, but rather to highlight the relative performance of RL compared to SFT across the three axes. Specifically, the emojis were chosen to reflect: *in out-of-distribution tests, RL yields substantial gains in Execution, moderate improvements in Semantics, and performance on par with SFT for Vision.*\n\nWe used distinct emojis to visually summarize this pattern of relative advantage, not to indicate differences in absolute performance degradation. We appreciate the reviewer’s feedback and will revise the figure caption and visual elements in the revised version to make this intent clearer.\n\n> ### Q4: Advantage computation in PPO\n\nWe **did NOT** recompute advantages after the first pass. In PPO, a rollout is collected with the current policy, GAE is computed once with the value-function of that policy, and the resulting advantages are kept fixed while the same data are reused for several mini-batch updates across later epochs. This is also how many well-known RL codebases, like Stable-Baselines3, MAPPO and \"37 Implementation Details of PPO\" blog, handle PPO updates, so our implementation follows common practice.\n\n> ### Q5: Comparison between ours and RT2's Fig4 experiments\n\nAlthough both our study and the RT-2 experiments in Fig. 4 evaluate policies under several OOD settings, the research questions diverge:\n\n- **Our focus** is to ask *\"What unique benefits can RL bring to VLA generalization compared to SFT?\"*. Hence, we start from the **same pretrained VLA backbone** and compare an RL-fine-tuned policy with its SFT counterpart.\n- **RT-2’s focus** is to show that *web-scale pre-training endows VLA models with broadly transferable visual and semantic concepts*. Their evaluation therefore contrasts **different model architectures, data sources, and training pipelines**.\n\nBecause of these distinct aims and setups, the two studies are complementary rather than directly comparable. Moreover, our benchmark spans three orthogonal generalization dimensions, offering a more systematic evaluation suite than RT-2's three unseen tests."}]

Author rebuttal:

Meta-review: This paper studies the generalization effects of VLAs trained with RL in comparison to SFT. They find that PPO outperforms both SFT and other RL algorithms (GRPO). The paper is practical with sufficient empirical analysis, is clearly written and has a well-supported story, and is generally useful to researchers wishing to fine-tune VLAs in robotic environments. The remaining concern of being limited to PPO / on-policy remains, but I disagree: on-policy algorithms are common in robotic settings and have been in use for RL with VLAs, e.g., https://arxiv.org/abs/2412.08442. It would be a nice bonus to see off-policy algorithm results, but I agree with the challenges and the limited impact on the story, so it isn't required.

I therefore recommend the paper is accepted as a poster.
Decision: Accept (poster)