Title: What Can RL Bring to VLA Generalization? An Empirical Study
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Vision-Language-Action Model

Policy Generalization
Abstract: Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. 
However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. 
To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel benchmark to text the generalization capabilities of Vision-Language Action (VLA) models. They are able to text the generalization capabilities of the VLA models along the vision, semantics and execution axis. The authors then test the performance of SFT and a few RL algorithms on different OOD dimensions. They found that PPO generalizes better than SFT and some RL baselines for semantic understanding and execution robustness, while it matches the baselines for visual tasks. The authors also provide an insightful analysis of the exploration performed by RL in Figure 8.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n\n## Quality\n* Well designed benchmark to test OOD performance along controllable axis: vision, semantics, and execution. The field needs more carefully designed empirical work like this.\n* Multiple seeds (2 and 3) are used. While more seeds could be used, it is a big improvement over a single seed.\n\n## Novelty\n* Surprisingly most VLA are not trained via RL e.g. RT2, \\pi_0, Octo, etc. By showing the advantages of PPO over the SFT baseline, this work motivates further investigation for training VLA with RL methods.\n\n## Impact\n* The authors provide an efficient training recipe (42h on a A100 gpu) and several ablation (shared backbone, val warm up and number of ppo epochs) for the PPO design choices. This simple and efficient recipe is impactful will allow other academic lab to build on this work.\n\n## Visualization\n* The visualization of the training trajectories in Figure 8 is interesting. \n\n# Weaknesses\n\n## Comprehensiveness\n* The authors test a single model (OpenVLA) and a single task (pick and place).\n\n## Significance\n* The main takeaway is not surprising: RL algorithms designed for multi-step tasks (PPO) perform better on multi-step tasks than algorithms designed for single-step tasks (GRPO and DPO). I think the paper would be more interesting if the RL algorithms used for comparison were carefully chosen, e.g. soft actor critic, etc.\n\n\n## Clarity\n* In Figure 1, I am not sure that the emojis are very useful nor what they imply. It seems like the OOD RL performance drops similarly across vision, semantics and execution, but 3 different emojis are used.\n\nQUESTIONS:\n* For PPO epoch >1, do you recompute the advantages using the updated value function?\n* Can you compare your study to the ones reported in Figure 4 of RT2?\n\nLIMITATIONS:\n* The authors mostly compared PPO against algorithms designed for single step LLM training. It is not surprising that they do not perform very well on multi-step robotic tasks. A more interesting comparison would be to compare against RL algorithms such as A3C, TD3, VMPO, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-10"}

comments: SUMMARY:
The paper can be split into two parts. The first part shows that for RL fine-tuning VLAs, PPO is more effective than LLM-derived methods like DPO, GRPO, and explores design choices of PPO such as critic design, warm-up, and UTD tuning. The second shows that RL tuning (with PPO) has superior generalization capabilities compared to supervised fine-tuning (SFT). For this, the authors design a novel benchmark (based on Maniskill) to systematically evaluate generalizability on novel vision, novel semantics (e.g., unseen objects), and novel execution (e.g., random initial state).

STRENGTHS AND WEAKNESSES:
Strengths
- Straightforward and concrete experimental setup, and strong results.
- Extensive investigation on VLA's generalization in pick-and-place with diverse out-of-domain scenarios, accompanied with interesting analyses on the results (figure 7,8,9).

Weaknesses
- The paper spends a great amount of writing space to PPO itself, which seems redundant to the main topic (generalization). I think the paper would have been more coherent if the design choices were investigated on their effect to generalization capabilities (although I'm not sure about the connection between generalization and the components in Section 4.2).
- The dynamic texture and dynamic noise (applying flat texture noise to the robot/object or to the entire view) design seems a bit unrealistic; I doubt that those visual noises will actually occur in practice.
- As mentioned in the limitation, the quality of the SFT dataset is far from satisfactory (figure 8a), which raises questions on the results (especially that of figure6a,b).

QUESTIONS:
1. In Figure 6a and 6b, the performance does not plateau after 16k but visibly drops on success rate. Why do you think this happened?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new benchmark to test the generalization capabilities of VLAs on a pick-and-place task, focusing on three factors: 1) visual appearance generalization, 2) semantic-level generalization, and 3) execution level generalization. It picks OpenVLA as the backbone model, and performed a thorough empirical study over several RL algorithms, including PPO, GRPO, and DPO, compared with SFT. The paper draws the conclusions that 1) RL works better in semantics and execution, but performs equally well in vision; 2) PPO works the best. It also gives insights about the design choices of PPOs for better performance in RL.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThis paper studies an interesting and important question: what can RL bring to VLA generalization, which from my point of view, is an important topic the community will have to work on in the future. It has also designed a few meaningful tasks and performed a thorough study over the algorithms.\n\nWeaknesses\n\nThere are a few weaknesses of the work and I’m very concerned about whether the results and insights can be transferred to real-world settings.\n\n1. The biggest issue is that the findings are drawn from an on-policy RL algorithm, PPO. However, it is extremely challenging to deploy any on-policy RL algorithms on a real robot due to the poor sample efficiency. In Fig. 6, all plots need 2m steps, which is merely impossible for a real robot. In addition, the only applicable algorithm, DPO, doesn’t seem to give any performance gain.\n2. The other issue is the choice of the base model, OpenVLA. OpenVLA predicts a single-step action per time-step. In contrast, latest VLA policies often predicts an “action chunk” which consists of multi-step predictions. Given an action-chunk policy, it is possible that modifications to the RL algorithms are required to make it work. Besides, these VLA models often rely on diffusion policies / flow matching policies, which might require extra efforts during RL.\n3. The benchmark is overly simplified as a pick-place task. The conclusions can be hard to transfer to tasks that require more dexterous skills, e.g., folding clothes.\n\nQUESTIONS:\n1. Is it possible to verify the conclusions on a real-robot? Even a simple task would be helpful.\n\n2. Have the authors tried a more advanced policy, e.g., pi0?\n\n3. Have the authors tried a standard off-policy RL algorithm, e.g. SAC / MPO, instead of the methods from RLHF literature?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper presents an empirical evaluation of the effect of RL training of VLA models on their subsequent generalization performance.  Generalization is defined along Visual, Semantic, and Execution dimensions.  The paper also provides experiments and discussion to help those wishing to employ RL for fine-tuning choose the specific method to use.

STRENGTHS AND WEAKNESSES:
This paper is a very clear empirical study of a particular method and its effects on well-defined dimensions.  The problem tackled is well-motivated and timely, providing insight into tradeoffs for a difficult decision about how to fine-tune VLAs.  The scope of the paper is well defined, and the paper does not claim more than it shows.  The primary weaknesses of the paper are already identified by the authors:  motion-planner demonstrations on simple, single-task benchmarks.  Despite these weaknesses, the paper will be interesting to many who are interested in VLAs.

QUESTIONS:
Some small points, discussion of which may improve the paper:

- In section 4.2, can you please expand on the choice of $h^0$?  What does the superscript there signify and what might the improved performance of that choice suggest about what information the critic is using.
- In line 280, the paper suggests that trial-and-error RL is able to generate a general "grasp" skill.  Why would this not work for vision to generate a general "perception" of graspable objects, as discussed immediately prior?  Can you clarify why those situations might be different?
- Could you address the motion-planning limitation in simple ways, by introducing noise, for example?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02