DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Authors: Junjie Wen
Yichen Zhu
Minjie Zhu
Zhibin Tang
Jinming Li
Zhongyi Zhou
Xiaoyu Liu
Chaomin Shen
Yaxin Peng
Feifei Feng
Venue: ICML 2025 poster (2025)
Keywords: vision-language-action models
reasoning
Primary area: applications->robotics

Abstract: In this paper, we present DiffusionVLA, a novel framework that integrates autoregressive reasoning with diffusion policies to address the limitations of existing methods: while autoregressive Vision-Language-Action (VLA) models lack precise and robust action generation, diffusion-based policies inherently lack reasoning capabilities. Central to our approach is autoregressive reasoning — a task decomposition and explanation process enabled by a pre-trained VLM — to guide diffusion-based action policies. To tightly couple reasoning with action generation, we introduce a reasoning injection module that directly embeds self-generated reasoning phrases into the policy learning process. The framework is simple, flexible, and efficient, enabling seamless deployment across diverse robotic platforms.

We conduct extensive experiments using multiple real robots to validate the effectiveness of DiVLA. Our tests include a challenging factory sorting task, where DiVLA successfully categorizes objects, including those not seen during training. The reasoning injection module enhances interpretability, enabling explicit failure diagnosis by visualizing the model’s decision process. Additionally, we test DiVLA on a zero-shot bin-picking task, achieving \textbf{63.7\% accuracy on 102 previously unseen objects}. Our method demonstrates robustness to visual changes, such as distractors and new backgrounds, and easily adapts to new embodiments. Furthermore, DiVLA can follow novel instructions and retain conversational ability. Notably, DiVLA is data-efficient and fast at inference; our smallest DiVLA-2B runs 82Hz on a single A6000 GPU. Finally, we scale the model from 2B to 72B parameters, showcasing improved generalization capabilities with increased model size.

Language Instruction Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Junjie Wen 12*
Yichen Zhu 1+
Minjie Zhu 12*
Zhibin Tang
Jinming Li
Zhongyi Zhou
Xiaoyu Liu
Chaomin Shen 2
Yaxin Peng
Feifei Feng'
diltusionn-llagithub.io
Internet Scale Vision- Language
Generalize Visual Changes
Encoder
Distractor
Large Language Model
Ressoning Tokens
Action Tokens
Unowen Background
Robot Reasoning
Open Sourced
Example of Reasoning
Shitting
Robett Pick up rubik's cube.
Generalization
Robot: Grabbing
toy blue car.
Interpretability
Zoro Shot Picking
Grabbing
por
Contectual Reasoning
Adeptation to New Embodiment
Rober:
Fast Inference
Reasoning Injection Module
Chatting
Describa
82Hz
DIVLA-28
DIVA: rubli'ss cube on right a yellow object.
42Hz
DIVLA-7B
Diffusion Model
User What is the object?
OpenVLA-78
DVVA is not clear, But looks like tulips.
Figure 1: Our proposed Diffusion VLA model unifies autoregressive diffussin modeling enable self-reasoning robot policy learning. This approach generalizes effectively to visual changes, supports zero-shot bin picking, adapts to new robot morphologies, performs visual question-ansseringg and generates actions with high speed.
Abstract
ing with diffusion policies to address the lim- itations of existing methods: while autoregres- sive Vision-Language-Attion (VLA) models lack precise and robust action generation, diffusion- based policies inherently lack reasoning capabil- ities. Central to our approach is autoregressive reasoning a task decomposition and explana- tion process enabled by a pre-trained VLM to guide diffusion- action policies. To tightly couple reasoning with action generation, we in- troduce reasoning injection module that directly embeds self- self-generated reasoning phrases into the
In this paper, we present DiffusionVLA, a novel framework that integrates autoregressive reason-
'Equal contribution, work done during Junjie Wen and Min- jie Zhu's internship at Midea Group. 'Midea Group, Shanghai, China Fast China Normal University, Shanghai, China SShanhhai University, Shanghai, China. Correspondence to: Yichen Zhu
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
policy learning process. The framework is simple, flexible, and efficient, enabling seamless deploy- ment across diverse robotic platforms.
mance in manipulation tasks by modeling action sequence generation noise-denoising proces.. This approach bet- ter captures the multimodal nature of robotic actions and enables faster sequence generation compared to NTP-based VIA models. However, despite the advantages of diffusion models for policy learning, they lack the reasoning capabili- ties crucial for VLA models to solve complex tasks effec- tively, a component that evidently improves LLMs. This motivates us raise the question: can we bring together the advantages of both parties, specifically the reasoning power of autoregressive models and the robusiness of high- frequency action generation offered by diffusion models?
We conduct extensive experiments using multi- ple real robots to validate the effectiveness of Di- VLA. Our tests include a challenging factory sort- ing task, where DiVLA successfully categorizes objects, including those not seen during train- ing. The reasoning injection module enhances interpretabilityy enabling explicit failure diagno- sis by visualizing the model's decision process. Additionally, we test DiVLA on zero-shot bin- picking task, achieving 63.7% accuracy on 102 previously unseen objects. Our method demon- strates robustness to visual changes, such as dis- tractors and new backgrounds, and easily adapts to new embodiments. Furthermore, DiVLA can follow novel instructions and retain conversa- tional ability. Notably, DiVLA is data-efficient and fast at inference; our smallest DiVLA-2B runs 82Hz on a single A6000 GPU. Finally, we scale the model from 2B to 72B parameters, show- casing improved generalization capabilities with increased model size.
In this work, we propose a unified model, named Diffusion- VLA (DiVLA in short), that integrates autoregression with diffusion model. The autoregressive component provides reasoning over the query, while the diffusion model controls the robot. Specifically, DiVLAA builds upon pre-trained Vision-Language Model (VLM), retaining its autoregressive capabilities for text-based reasoning. We extend this founda- tion by integrating a diffusion model that facilitates learning robotic actions through a noise-denoising process. This setup empowers DiVLA achieve both language-driven reasoning and robust action generation in robotic contexts. However, simply combining these elements does not fully exploit the reasoning potential, as there is often an implicit gap between logical reasoning and actionable robot policies. To bridge this gap, we propose a reasoning injection module, which reuses reasoning outputs and embeds them directly into the policy head, thus enriching the policy learning pro- cess with explicit reasoning signals. This innovation allows us to directly incorporate reasoning into action generation, enhancing the model's dexterity, robustness, and general- ization across various scenarios. Our experiments confirm that DIVLA delivers the following advantages: 1) Visual generalization via self-generated reasoning: DiVLA can recognize and categorize previously unseen objects via self- generated reasoning, showcasing its ability to generalize to novel visual inputs.
1. Introduction
Recently, Vision-Language-Action (VLA) models have emerged as a promising direction in robotics (Bu et al., 2024; Liu et al., 2024b; Li et al., 2024c;a; Zhang et al.,
2024; Zheng et al., 2025; Pertsch et al., 2025; Brohan et al., 2023; Kim et al.; Wen et al., 2024; Black et al., 2024; Octo Model Team al., 2024; Deng et al., 2025; Liu al., 2025; Cui et al., 2025; Chen al., 2025; Liu et al., 2024a). Among these VLAs, a prominent approach frames action prediction as a next-token prediction (NTP) task, mirroring the dom- inant autoregressive paradigm in Large Language Models (LLMs), which operates by sequentially predicting discrete tokens, with each token's 's generation conditioned on the preceding ones. While these models, such RT-2 (Brohan et al., 2023) and Open VLA (Kim al.), have demonstrated notable success, they suffer from inherent limitations. First, discretizing continuous action data into fixed-size tokens can disrupt the coherence and precision of actions. Second, NTP is inherently inefficient for action generation, particu- larly in real-time robotic applications where performance is critical.
2) Strong action interpretability: Our reasoning injec- tion module provides insights into the end-to-end policy's ss decision mecksion-making... explaining robot actions and facilitating failure analysis.
3) Adaptability to novel instructions and conversational capability: Our approach can execute novel instructions while maintaining conversational fluency, offering versa- tile response range in interactive scenarios.
4) Fast adaptation to other embodiment: DIVLA can quickly fine-tune for deployment on new embodiment, such as bimanual robots, achieving high performance with mini- mal adjustments.
Meanwhile, building on the success of diffusion models in content generation (Rombach et al., 2022; Peebles & Xie, 2023; Podell et al., 2023; Ho et al., 2020; 2022), diffusion- based models for learning visuomotor policies (Chi et al., 2023) have gained significant popularity over the past two years. Numerous methods have demonstrated strong perfor-
5) Fast inference speed: With inference rates of 82Hz for DiVLA-2B and 42Hz for DiVLA-7B on a single A6000 GPU, our method ensures real-time responsiveness even in
2

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
high-demand environments.
rather than diffusion for action generation. Our approach introduces reasoning- crucial element in language mod- els-into the diffusion-based vision-languagg-action (VLA) model.
6) Enhanced visual generalization: DiVLA is unaffected by visual distractions or novel backgrounds, demonstrating robustness in visually dynamic settings.
7) Scalability: The scalable DiVLA family (2B, 7B, and 72B) demonstrates that generalization and performance im- prove with model size, consistent with established scaling laws.
Robot foundation models. Existing works (Belkhale et al., 2024; Brohan et al., 2022; 2023; Gu et al., 2023; Jiang et al., 2022; Jang et al., 2022; Fu et al., 2024; Huang et al.) lever- age RL (Yuan et al., 2024) and LLM (Liang et al., 2023) to decouple the multimodal understanding and embodied con- trol. Another line of research leverages pre-trained Vision- Language Models (VLMs) and fine-tunes them directly on robotic data (Brohan et al., 2023; Zawalski et al., 2024; Kim et al.; Wen et al., 2024). Our work follows this ap- proach, unifying autoregressive and diffusion models for both reasoning and manipulation tasks.
This work introduces a novel end-to-end framework for robot policy learning. By unifying an autoregressive objec- tive with a diffusion model, our approach enables the model to both "talk" and "act." Critically, our proposed reasoning injection module promotes generalization to novel objects and provides interpretable actions. While the concept is simple and straightforward, the combination of ingredients and the application to robot learning is novel. Empirical evaluation across multiple complex real-world robot tasks demonstrates a level of generality surpassing that of previ- ous single-model approaches.
Unified rugo-segressvve model and image generation. Re- cent work has focused on unifying multimodal understand- ing with image generation. These efforts include using auto- regressive methods to generate images (Li et al., 2024b; Sun et al., 2024; Wang et al., 2024c;a), diffusion models to pro- duce text, or combining both approaches into unified mod- els (Ge et al., 2024; Lu et al., 2024; Zhao et al., 2024; Wu et al., 2024a), such as Show-O (Xie et al., 2024), Transfu- sion (Zhou et al., 2024), and Vila-U (Wu et al., 2024b). Un- like these methods, our framework explores unifying next- token prediction with diffusion models, which enhances the robot model's reasoning abilitics. This reasoning capability, in turn, improves the model's generalization to various tasks and environments.
2. Related Works
Autoregression models. Predicting the next token has been regarded as key approach toward general artificial intelli- gence due to its success in training language models (Tou- vron et al., 2023a;b; Achiam et al., 2023; Bi et al., 2024; Team, 2024). RT-2 (Brohan et al., 2023) pioneered the ap- plication of next-token prediction in robot learning which predicts actions by converting continuous actions into dis- crete tokens for learning robotic motion. Building this, VLA (Kim et al.) introduced an open-source, im- proved, and smaller version of RT-2 (Brohan et al., 2023) with similar approach, while ECoT (Zawalski et al., 2024) developed a chain-of-thouuht method. However, research has shown that next-token prediction may not be the opti- mal approach for robot models, especially when adapting to various embodiments. In this work, we leverage the strength of next-token prediction specifically for reasoning tasks.
3. Methodology
Our ultimate goal is to create a unified framework that combines autoregressive models, which excel at predict- ing language sequences for reasoning, with diffusion mod- els, which are highly effective at generating robotic actions. Developing such an integrated model presents substantial challenges, with key issues centered on: (i) designing an architecture that seamlessly and efficiently integrates both autoregressive and diffusion mechanisms; and (ii) leverag- ing self- generated reasoning to enhance action generation without adding inference computation overhead. In this sec- tion, we introduce the overall framework of our method in Section 3.1 and explore the design choices that inform our model architecture in Section 3.2.
Diffusion models. Diffusion models have become domi- nant in the field of visual generation. The Diffusion Pol- icy (Chi et al., 2023) extends the application of diffusion models to robot learning, demonstrating their effectiveness in handling multimodal action distributions. Subsequent work has advanced the Diffusion Policy (Zhao et al.: Wang et al., 2024d: Prasad et al., 2024: Reuss et al., 2024; Ue- hara et al., 2024a;b; Black et al., 2023a;b; Dasari et al., 2024; Lin et al., 2024) by applying it to 3D settings (Ke et al., 2024; Ze et al., 2024b;a; Yan et al., 2024), scaling it up (Zhu et al., 2024), increasing its efficiency (Jia et al.; Wang et al., 2024e), and introducing architectural innova- tions. For instance, VLA (Wen al., 2024) integrates diffusion models with lightweight vision-language mod- els, while no (Black et al., 2024) leverages flow matching
3.1. Architecture
Given any sequence of interleaved images, text, and video, we first encode the images into dense visual features using SigLIP (Zhai et al., 2023). These encodings are then trans- formed into a fixed number of N visual embeddings through a Transformer. It's worth noting that typical visual inputs
3

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Factory Sorting
Experiment Setup
Bin Picking
Stuffed Toys
Hex Keys
Knit Gloves
Toy Cars
(a) Box with four sectors
(b) Seen
Mixed Seen&Unseen
Cluttered
(e) Cluttered Seen&Unseen
Figure 2: Environmental Setup for the Franka Robot and Experimental Configuration for Factory Sorting, Left: For factor sorting following categories: stuffed toys,
tasks, (a) The target sorting box divided into four distinct sectors, each designated for one keys, knit gloves, and toy cars, (c) The seen objects in the train data, (d) mixing the seen and unseen object for evaluation, (e) cluttered scene for seen objects, (f) cluttered seene for mixing seen and unseen objects. Middle: We use a Franka robot arm equipped with two external Zed cameras and Realsense 435i wrist camera. Right: The setup for zero-shot bin picking.
Table Experimental Results for Multi-Task Learning Real Robot. We report the count of pre-trate trained trajectories. We also report average success rate evaluation on both in di-iistribution and ouito--distriuttinn. Task Select the appropriate object based on the user's intent. Task 2: Flip the vertically placed pot. Task Pick up cube and place into the [yellow/blue] hox. Task 4: Place the cup onto the plate. Task 5: Place the cube into the box.
Pre-trained Trajectory
Distribution
Visual Generalization
Model Tasks
Task
Task
Task 3
Task 4
Task
Avg.
Task
Task
Task 3
Task
Task 5
Avg.
Diffusion Policy (Chi al., 2023)
66.7
36.4
36.4
27.9
11.1
11.1
22.2
8.9
VLn (Wen et al., 2024)
72.7
45.5
36.4
45.5
27.3
45.5
44.4
44.4
11.1
22.2
22.2
28.9
Octo (Octo Model Team et al., 2024)
970K
57.6
27.3
9.1
27.3
24.3
44.4
11.1
11.1
0
22.2
17.8
OpenVT A 7B (Kim et al.)
970K
69.7
18.2
18.2
36.4
54.5
39.4
55.6
11.1
33.3
33.3
26.7
DiVLA-2B
39K
100
100
63.6
63.6
90.9
83.6
44.4
66.7
44.4
66.7
66.7
57.8
in robot learning often include multiple camera views. To manage this, we applied the shared SigLIP visual backbone to each view and subsequently concatenated the resulting visual tokens.
LayerNorm. This projection module functions similarly to those found in conventional vision-language models like LLaVA (Liu et al., 2023b;a), bridging the VLM's output embedding to the diffusion model and aligning their output dimensions. The diffusion model itself adheres to the standard Diffusion Policy design (Chi al., 2023). with randomly initialized weights This component also incorporates reasoning from the LLM, which we describe in detail below. An MLP layer is attached to the last layer at the bottom of the action decoder to predict the robot's s joint space. multiple embodiments are evolved, instead of making copy of a separate action decoder (Octo Model Team et al., 2024), we simply initialized new MI.P layer for training and evaluation. This step ensures that the knowledge from the pre-trained data is preserved and thus can quickly adapt to new embodiment.
For vision-language processing, we leveraged Qwen2- VL (Wang et al., 2024b), state-of-the-ar vision-language model available in three sizes: 2B, 8B, and 72B parameters. We initialized the VLM backbone with the publicly released checkpoint. It also possible to use any other pre-trained VLM as backbone, since we decouple the vision- vision-language understanding with action generation, making the overall architecture flexible to fit for advanced new models.
Projection layer for action tokens. Following the final embedding layer of the VLM, a fixed number of action tokens are generated. These tokens are then fed into a projection module, comprised of two MLP layers with
Reasoning injection module. The core of our approach

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Factory Sorting
100
DP
Octo
OpenVLA
TinyVLA
DIVLA-28
80
76.2
Success Rate
66.2
60
62.5
60.0
56.2
45.3
48.8
46.2
40
41.3
31.1
33
33.8
33.8
32.3
31.2
33.8
24.6
26.3
20
20.4
21.5
13.8
9.2
0
Average
Seen
Mixed
Cluttered Seen
Cluttered Mixed
Figure 3: Experimental Results Factory Sorting, We compared our DiVLA with Diffussin Policy, Octo, TLny and OpenVLA. DiVLA achieves the highest average success rate, outperforming the runner-up OLe by 20.9%.
lies in introducing explicit reasoning into Vision- LaniunnL- Action (VLA) models. Unlike most autoregressive VLAs, which require a recursive setup- - converting reasoning outputs into inputs for subsequent model runs our method proposes more efficient and streamlined integration of reasoning. By embedding reasoning directly within the policy model, we avoid the computational and operational complexities of iterative input-output cycles, enabling faster and more seamless reasoning incorporation.
weighted in the training process, allowing the model to learn effectively from both action and next-token prediction tasks.
Pretraining Data. We consider OXE (O' al., 2023) and Droid (Khazatsky et al., 2024) dataset for pretraining. We use Droid data to pre-train DiVLA-2B and DiVLA- 7B. Because larger models typically needs more data for training, we use OXE and Droid together for pre-training DiVLA-72B. The original Droid data contains only robotic actions, paired partially with observations and language instructions. These data contain only robotic actions, paired partially with observations and language instructions. To enhance our model's 's ability to generalize with language, we leverage GPT-4o to automatically transform these data into a form that includes reasoning. Consequently, the network architecture remains consistent across both the pre-training and fine-tuning stages.
Our reasoning injection module operates by taking the fi- nal embedding from the tokenized output of the reasoning component and directly injecting it into the policy model through Feature-wiss Linear Modulation (FiLM) (Perez et al., 2018). This injection technique, inspired by methods in RT-1 (Brohan et al., 2022) and YAY (Shi et al., 2024), enables us to modulate the policy network's 's layers based on the reasoning signal. We refer to this process as "injection" because, in our design, the policy network focuses primar- ily action-specific tokens, while the reasoning module functions as an auxiliary enhancement, providing contextual depth without dominating the primary docision-making flow. This approach ensures that reasoning is not only present but actively utilized during policy model training.
4. Experiments
In this section, we examine the effectiveness of DiVLA for embodied control. In Section 4.2, we compare Di- VLA against other sfate-of-the-ar models within a stan- dard multi-task setting, assessing its performance in both in-distribution and out-of-distribution scenarios. In Sec- tion 4.3, we evaluate DiVLA in the challenging factory sorting task, showcasing its remarkable performance and illustrating how reasoning enables the model to analyze robot actions and sort items accurately. In Section 4.5, we showcase DiVLA's impressive generalization abilities in a zero-shot bin-picking task involving over 102 unseen ob- jects. In Section 4.6, we illustrate DiVLA's adaptability to embodiments, successfully completing complex tasks that require bimanual coordination. We provide experimen- tal setup and implementation details in the Appendix.
3.2. Model Design Choices
We illustrate the training strategy and other techniques that we used to improve the efficiency and effectiveness of our method. We also discuss the selection pretraining data.
Training objectives. Given batch of input sequences, the overall training loss is formulated as combination of the diffusion loss and the next-token prediction (ntp) loss: T, Laifs + adnup,, where a is the syper-parameten weighting the loss term Intp- In our observations, the magnitude of Tntp consistently remains about ten times smaller than that of Laify- To balance the contribution of each component to the overall loss, typically set 10 in all experiments. This adjustment ensures that both terms comparably
5

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Zero-shot Picking on 102 Unseen Objects
4.2. Real-World Multi-Task Learning
DP
We begin with standard setting in which the model is trained on multiple tasks and completes each task based different user queries. We designed five tasks: object selection, flip the vertically placed pot, placing a cube into a designated box, placing a cup onto a plate, and placing a cube inside a box. Detailed descriptions of these tasks are provided in the Appendix. The exper- imental results can be found in Table We compare our method to the Diffusion Policy (Chi et al., 2023), Tiny VLA (Wen et al., 2024), Octo (Octo Model Team et al., 2024), and Open (Kim et al.). Notice that both Octo and Open VLA is pre- pre-trained on OXE (O O'Neill et al., 2023), which is 25 times larger than our pre-trained datasets.
Octo
OpenVLA
28.4
TinyVLA
23.5
DIVLA-28
63.7
Figure 4: Zero-shot Bin Picking on 102 Unseen Objects. Our method outperforms the state-of-the-are robot foundation models by a large margin.
Generalization to visual changes. We further eval- uate our method in multi-task setting with visual changes to assess its robustness and adaptability in diverse, dynamic environments. Specifically, we introduce three challenging scenarios designed to test the model's ability to handle visual variability: 1) adding additional distractors in the surroundings to increase visual clutter and complexity, 2) altering the background to test resilience against shifts in scene context, and 3) implementing colorful lighting effects to introduce varied illumination and color hues. These scenarios are shown in Figure 5 to illustrate the impact of each change on the visual environment. The experimental results are demonstrated in Table 1.
Figure 5: Examples of visual variations, including randomly placed distractors, different backgrounds, and distracting lighting. Di- VLA is robust to visual changes in different scenarios.
Our evaluation of these scenarios reveals that while all meth- ods experience decline in performance due these visual changes, our method consistently maintains the highest av- erage success rate across five different tasks. This outcome highlights the model's inherent robustness and adaptabil- ity, despite the absence of any specific data augmentation techniques during training.
4.1. Experimental Setup.
Implementation details and pretraiend data. The model pre-trained on the Droid (Khazatsky et al., 2024) dataset. We then finetune our model on evaluation tasks, similar to the setting To (Black et al., 2024). We use LoRA (Hu et al., 2021) to fine-tune the VLM models. We use 2e-5 as fixed learning rate to train the model, similar to Open VLA. The visual encoder and VLM are frozen. We apply LoRA on VLM for fine-tuning.
4.3. End-to-End Sorting on Real Robot
We evaluate the capability of DiVLA in an industrial setting, where a robot is tasked with sorting items into designated sectors within a large box based on object category. Specifi- cally, we categorize items into four classes: (1) toy cars, (2) knit gloves, (3) stuffed toys, and (4) hex keys. The language instruction provided is "Sort all items into corresponding areas". A total of 500 trajectories are collected as training data. The task is considered successful only if the robot successfully grasps the object and places it in the correct sector. The experimental setup is illustrated in Figure 2.
Data for finetuning. We explore four experimental settings: factory sorting, bin picking, multi-task learning, and table bussing. The first three settings are conducted with the Franka robot, while the table bussing task utilizes the bimanual AgileX robot. Our dataset includes 500 trajectories for the factory sorting task and 580 trajectories for multi-task learning. The bin picking task is designed as zero-shot task, so no training data was collected for it. For the table bussing task, we gathered 400 trajectories, where objects are randomly placed on the table, often overlapping with each other. During the fine-tuning stage, all data corresponding to the same embodiment are trained together. For instance, the factory sorting and multi-tast data are combined for training purposes.
This task poses several challenges, requiring both precise object grasping and accurate category identification. We as- sess our method under two difficulty settings: easy and hard. In easy mode, fewer than 5 items are placed on the table, whereas in hard mode, 6 to 11 items are randomly arranged.
6

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Furthermore, both seen and unseen objects are mixed in these scenarios. In the cluttered scene, items may overlap or be randomly distributed across the table, increasing the complexity of the sorting task.
Grab the toy blue car
Grab hex key, put into right- bottom sector
The experimental results are illustrated in Figure 3. Di- VLA demonstrates robust performance with an average suc- cess rate of 66.2% across all experimental settings. While other methods show significant performance degradation as scene complexity increases (i.e., higher object count and clutter level), particularly evident in DP's sharp decline to 9.2% success rate in highly cluttered mixed scenarios, DiVLA maintains a substantial 60% success rate. This sus- tained performance underscores our approach's capability to effectively handle complex and dynamic real-world sce- narios.
Figure 6: What internal processes guide model's actions? We illustrate this using an example DiVLA's reasoning, inferred from shifts in its behavior based on changes to the target object. The model initially intends to grasp "toy blue car", but when presented with a hex key, its reasoning instantly redirects to sorting the "hex key" instead. This demonstrates one approach inter- preting robot's 's actions.
4.4. Behavior Analysis of Robot Foundation Model
Deep learning has demonstrated superior performance and generalization capabilities compared to traditional methods. Despite this, its nature "black box" algorithm often raises concerns regarding trustworthiness. The lack of trans- parency into the model's recision-mmking process makes it difficult to understand its actions. This section explores applications of our reasoning injection module, designed to improve model interpretability by illuminating its internal reasoning and explaining why certain actions might fail.
Figure 7: Some of the unseen objects used for evaluation in the scoo-shot bin- din-picking tasks.
How does the model generalize to new objects? We con- duct experiments on sorting tasks. Specifically, we ask the model to identify and sort unseen objects, that are not in the train data. We place four previously unseen objects on the plate, a stuffed toy cat, a pair of green gloves, a dark toy car, and screwdriver.
illustrated in Figure 6, the model might initially identify toy car and generate the reasoning, "grabbing the toy car," indicating its intent to pick it up. However, if we
intervene by placing a hex key in the gripper instead, the reasoning dynamically shifts to "grabbing the hex key." This adaptive reasoning allows the model to adjust its subsequent actions, correctly sorting the hex key despite the unexpected change. This dynamic reasoning not only enhances the transparency and interpretability of the model's actions but also contributes to its robustness. The integrated reasoning module allows for form of self-correction, where the generated rationale guides and refines the action output, leading to more reliable performance even in the face of unexpected situations or perturbations. This capability suggests that the model is not simply executing pre propprogrammed actions, but rather engaging in more flexible, context-aware decision-making process, akin to an internal dialogue.
From this experiment, we observe several interesting find- ings. First, the model demonstrates the ability to analogize to new objects. For instance, while the model has not been trained on a screwdriver, it correctly categorizes it as a hex key due to their visual similarity. This indicates that the model can generalize to new objects, not by direct recogni- tion, but by comparing the object's semantic features with those of known objects. Second, the model can identify the color of some objects; for example, it categorizes the green glove as "green glove" and the stuffed toy cat as "brown toy cat." Although these experiments are not exhaustive, they represent an important step towards the explainable generalization of robot models.
Failure case analysis via self-generated reasoning. A key advantage of our model is its ability to generate natural language rationales alongside its output actions, providing valuable insight into its decision-making process. By observing these reasoning phrases, we can understand what the model is "thinking" at each step. For instance, as
4.5. Zero-Shot Bin Picking of Unseen Objects
This section evaluates instance generalization for DiVLA, focusing specifically on the bin-picking task a standard benchmark for assessing robot model performance. In our
7

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
size
toy dragon
Table 2:
Experimental results for Table Bussing on real biman- ual robot. Our method significantly outperforms both Diffusion Policy and Open VLe by a large margin.
gripper
Rublk's cube
screwdriver
tape
Scenarios
Diffusion Policy
VLe
DIVLA-2B
Seen
45.8
72.9
Figure 8: Examples of various unseen objects in zero-shot bin- pickine tasks. The unseen objects vary significantly in size.
Mixed
31.2
70.8
objects. This task was adapted for a bimanual robot setup: all tableware should be placed panel to the left, while trash items are dropped into bin on the right. Similar to our factory sorting task, we evaluated the model's perfor- mance using both seen objects and a combination of seen and unseen objects. The environment setup, along with all objects used for training and evaluation, shown in Fig- ure 9, Our evaluation consisted of twelve trials, with three to five objects randomly placed on the table. The success rate is computed by how many objects are correctly placed.
Bimanuol Robor Semup
tni Setup for Tadée Bussing
10 Been Teblenare
H|| Unseen Tebfemore
(6) Soon Trash
II Unsoon Tresh
Figure 9: (a) Environmental setup for the bimanual robot, (b) Table bussing sctup, (c-f) All tableware and trash items used in the bussing table task evaluation.
Our results show that our model successfully completes tasks in most cases when the object has appeared in the training data, achieving an average success rate of 72.9% on seen objects. In contrast, the Diffusion Policy and Open- VLA achieve 45.8% and 0% success rates. For tasks involv- ing both seen and unseen objects. DiVLA achieves up to 70.8% success rate, slight decrease in success rate com- pared to the seen one, showing remarkable generalization to objects with different colors and shapes. Finally, Di- VLA demonstrates the ability to recognize unseen objects, particularly by responding to object color. For example, it categorizes a Sprite can "green can" and correctly places it in the trash bin. This observation further supports the idea that reasoning contributes to generalization.
evaluation, we use 102 unique objects, none of which were included in the training data. Some of these objects are shown in Figure 7. We modify the task instruction to "move any object on the right panel to the left basket." Figure 2 (right) illustrates the experimental setup. We show some of the objects used in this test in Figure 7. The challenge in this evaluation lies in the significant variability across objects, which includes not only differences in dimensions but also distinct color patterns, textures, and degrees of deformabil- ity. Such variability aims to emulate real-world scenarios where robots must adapt diverse and unpredictable items. Figure provides examples of five differently sized objects from this experiment.
5. Conclusion
The experimental results are depicted in Figure 4. Di- VLA achieves success rate of 63.7%. In comparison, the success rates of the Diffusion Policy, Octo, Tiny VLA, and VLen are 8.9%, 19.6%, 23.5%, and 28.4%, respec- tively. These results indicate that DiVLA is resilient across broad range of object shapes and sizes, other models often fail due to reliance on object-specific features that may not generalize well to new instances. It highlights the poten- tial for applications in dynamic, unstructured environments where robots encounter unfamiliar objects and must perform tasks with minimal human intervention.
In this work, we present DiVLA, a state-of-the-art vision- language- acttin model that delivers strong performance in both simulations and real-world scenarios, including single- arm and dual-arm robots. The core of our method lies in combining the next-token prediction objectives and diffu- sion models: the former for task reasoning and the latter for action prediction. We introduce a reasoning reuse module to enhance action generation. Through extensive evaluations in both simulations and across multiple real-world embod- iments, we demonstrate that DiVLA outperforms several SOTA robot models. Additionally, we show that DiVLA has robust generalization capabilities, adapting effectively to new instructions, tasks, and environments, Our research offers a novel perspective on designing VLA models, en- couraging a rethinking of how reasoning can be reused to facilitate end-to-end policy learning.
4.6. Adapt to World Bimanual Robot
In this section, we examine DiVLA's adaptability to a dual- arm robot. Inspired by no (Black et al., 2024), we designed
a table bussing task that involves clearing table with various
8

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, This work is supported by the Sci-Tech Innovation Initiative by the Science and Technology Commission of Shanghai Municipality (49000,,,,,) the National Science Foun- dation of China (12471501), and the Science and Tech- nology Innovation Action Plan of Shanghai under Grant 22511105400.
Acknowledgements
X., Choromanski, K., Ding, TT, Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-actiin transfer web knowledge to robotic control, arXiv preprint arXiv:2307. 15818, 2023.
models
Bu, Q- Li, H., Chen, L.., Cai, J., Zeng, J., Cui, H., Yao, M., and Qiao, Y. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXiv:2410.08001 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, Y., Tian, S., Liu, S., Zhou, Y.. Li, H., and Zhao, D. Conrft: A reinforced fine-tuming method for vla models via consistency policy. arXiv preprint arXiv:2502.05450, 2025.
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burch- fiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0m 04137, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Cui, C., Ding, P., Song, W., Bai, S., Tong, X., Ge. Z., Suo, R., Zhou, W., Liu, Y., Jia, B., et al. Openhelix: A short survey, empirical analysis, and open-source dual- system vla model for robotic manipulation. arXiv preprint arXiv:2505.039) 2025.
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., and Sadigh, D. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024.
Dasari, S., Mees, O., Zhao, S., Srirama, M. K., and Levine, S. The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410. 10088, 2024.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C.,, Ding, H., Dong, K., Du Q., Fu, Z., al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv: 2XXi:2401.002554 2024.
Deng. S., Yan, M., Wei, S., Ma, H., Yang, Y., Chen, J., Zhang, Z., Yang, T., Zhang, X., Cui, H., et al. Graspvla: grasping foundation model pre-traine teaitraind billion-scale synthetic action data. arXiv preprint arXiv: 2505. 2505.0333,, 2025.
Black, K., Janner, M., Du Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 2023a.
Fu, 7.,, Zhao, T. Z.,, and Finn, C. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.0211 2024.
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint ar 20.:23300.1639 2023b.
Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi-granun granularituu comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
Black, K., Brown, N., Driess, D.,, Esmail, A., Equi, M., Finn, C. Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubezak, S., Jones, T., Ke, I.,, Levine, S., Li- Bell, A. Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X., Tanner, J., Vuong, Q., Walling, A., Wang, H., and Zhilinsky, U. no: vision-language-action flow model for general robot control, 2024. URL https: // //arxivv org/ ab/// 24164.
Gu, J., Kirmani, S., Wohlhart, P., Lu, Y., Arenas, M. G., Rao, K. Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z., et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.0197. 2023.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6857 2020.
Brohan, A., Brown, N., Carbajal, J.. Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu. J., et al. Rt-1: Robotics transformer for rorl-world control at scale. arXiv preprint arXiv:2212.00 12.06877, 2022.
Ho, J.. Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. Advances in

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Neural Information Processing Systems, 35:8633-8646, 2022.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 11838, 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 222.0000966. 2021.
Li, Liu, M., Wang, D., Liu, J., Kang, B., Ma, X., Kong. T., Zhang, H., and Liu, H. Towards generalist robot policies: What matters building assion-lagguage-attin models. arXiv preprint arXiv:2412.1405: 2024c.
Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y.,
Li, Q.. Zhu, S.-C., Jia, B., and Huang, S. An embodied generalist agent in 3d world. In ICLR 2024 Workshop: How Far Are We From AGI.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Lin, F., Hu, Y., Sheng, P., Wen, C., You, J., and Gao, Y. Data scaling laws in imitation learning for robotic manipula- tion, 2024. https: /taxivxxvvv.////e// org/ h/sp//////// 2410. 18647.
Jaa, X., Wang, Q.. Donat, A., Xing, B., Li, G., Zhou, H., Celik, O., Blessing, D., Lioutikov, R., and Neumann, G. Mail: Improving imitation learning with selective state space models. In 8th Annual Conference on Robot Learning.
Liu, H., Li, C., Li, Y., and Lee, J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv:3310.03744 2023a.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruc- tion tuiigg In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https: / //oeerreveew net/ forum? wtw.m///d//H2GH.....
Jiang, Y. Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., and Fan, L. Vima: General robot manipulation with multimodal prompts. arXiv preprint arwiv:2210.0000000 2(3):6, 2022.
Liu, J., Liu, M., Wang, Z., An, P., Li, X., Zhou, K., Yang, S., Zhang, R., Guo, Y., and Zhang, S. Robomamba: Efficient vision-language-cttinn model for robotic reasoning and manipulation. Advances in Neural Information Process- ing Systems, 37:40085 4015544110 10, 2024a.
Ke, T.-W.. Gkanatsios, N., and Fragkiadaki, K. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint ar arXiv:2402.10SS.
2024.
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., et al. Droid: A sarge-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. 2024.
X., Liu, J., Chen, H., An, P., Liu, Z., Zhang, R., Gu, C., Li,
Guo, Z., Chen, S., Liu, M., al. Hybridvla: Collaborative diffusion and autoregression in unified vision- lsiiun-lagguage action model. arXiv preprint arXiv:2503. 10631, 2025.
Liu, S.,. Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., and Zhu, J. Rdt-lb: a diffusion founda- tion model for bimanual manipulation. arXiv preprint 2024b.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model.
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and- play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., Kreiman, T., Tan, Y., Chen, L. Y., Sanketi, P., Vuong, Q., Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An sourne generalist robot policy. In Proceedings of Roboties: Science and Systems, Delft, Netherlands, 2024.
Li, Q. Liang, Y., Wang, Z., Luo, L., Chen, X., Liao, M., Wei, F., Deng, Y., Xu, S., Zhang, Y., et al. Cogact: A foundational vision iarion-languag--ace action model for synergiz- ing cognition and action in robotic manipulation. arXiv preprint Xrxiv:241 1. 19650, 2024a.
10

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint 1rXiv:2310.08864, 2023.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. $195-4205, 2023.
Uehara, M., Zhao, Y., Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L., Tseng, A. M., Biancalani, T., and Levine, S. Fine-tuning of continuous uatin diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194, 2024a.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general con- ditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Uchara, M., Zhao, Y., Black, K., Hajiramezanali, E., Scalia, G., Diamant, N. L.,, Tseng, A. M., Levine, S., and Bian- calani, T. Feedback efficient online fine-tuning of diffu- sion models. arXiv preprint arXiv:2402. 2024b.
Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for lissionlanguage-action models. arXiv preprint arXiv:2501.09744 2025.
Wang. D., Hart, S., Surovik, D., Kelestemur, T., Huang, H., Zhao, H., Yeatman, M., Wang, J., Walters, R., and Platt, R. Equivariant diffusion policy. arXiv preprint arXiv:2407.018/1 2024a.
Podell, D., English, Z., Lacey, K.,, Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint ar arXiv:2307.0/9522 2023.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, 7., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Prasad, A., Lin, K., Wu, J., Zhou, L., and Bohg, Consis- tency policy: Accelerated visuomotor policies via con- sistency distillation. arXiv preprint arXiv:2405.07503, 2024.
Wang. X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q. et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.1888 2024c.
Reuss, M., Yagmurlu, Ö. E., Wenzel, F., and Lioutikov, R. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. 2024.
Wang, Y., Zhang, Y., Huo, M., Tian, R., Zhang, X., Xie, Y., Xu, C., Ji, P., Zhan, W., Ding, M., et al. Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning. arXiv preprint arXiv:2407.00 0055013,,, 2024d.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High- rigghrreolltion image synthesis with latent diffuuion models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang, Z., Li, Z., Mandlekar, A., Xu, Z., Fan, J., Narang, Y., Fan, L., Zhu, Y., Balaji, Y., Zhou, M., et al. One-step diffusion policy: Fast visuomotor policies via diffusion distillation. arXiv preprint arXiv:2410.21257, 2024e.
Shi, L. X., Hu, Z., Zhao, Z., Sharma, A., Pertsch, K., Luo. J., Levine, S., and Finn, C. Yell at your robot: Improving on-the-fly from language corrections. arXiv 2024.
Wen, J., Zhu, Y., Li, J., Zhu, M., Wu, K., Xu, Z., Cheng, R., Shen, C., Peng. Y., Feng, F,, et al. Tinyvla: Towards fast, data-efficient assion-lngguage action models for robotic manipulation. arXiv preprint arXiv:2409.1255 2024.
preprint arXiv:2403.1299
Sun, P., Jiang. Y., Chen, S., Zhang, S., Peng, B., Luo, P. and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint MxXiv22406.06522 2024.
Wu, C.,, Chen, X., Wu, 7., Ma, Y., Liu, X., Pan, 7., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024a.
models. arXiv preprint arXiv:2405.099 08 2024. Team, C. Chameleon: Mixed-mode early-fusion foundation
Touvron, H., Lavril T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv: 139711 2023a.
Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y., Zhu, L., Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation, arXiv preprint arXiv:2409.0444 2024b.

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K. Q.. Gu, Y., Chen, Z.,, Yang, Z.,, and Shou, M. Z.. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Zhu, M., Zhu, Y., Li, J., Wen, J., Xu, Z., Liu, N., Cheng, R., Shen, C., Peng, Y., Feng, F,, al. Scaling diffusion policy in transformer to billion parameters for robotic manipulation. arXiv preprint arXiv:2499.14411 2024.
Yan, G., Wu, Y.-H., and Wang, X. Dnact: Diffusion guided multi-task tast 3d policy learning. arXiv preprint arXiv:2403.044 15, 2024.
Yuan, Z., Wei, T., Cheng, S., Zhang, G., Chen, Y., and Xu, H. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learning. arXiv preprint 1881v:2407.15815, 2024.
Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic control via embodied chain-of of of-thought reasoning. arXiv preprint arXiv:2407.08693 2024.
Ze. Y., Chen, Z., Wang, W., Chen, T., He, X., Yuan, Y., Peng, X. B., and Wu, J. Generalizable humanoid manipu- lation with improved 3d diffusion policies. arXiv preprint arXiv:2410. 10803, 2024a.
7e, Y., Zhang, G., Zhang, K., Hu, C.,, Wang, M., and Xu, H. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Ma- nipulation, 2024b.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pp. 11975-11966, 2023.
Zhang, Z., Zheng, K., Chen, Z., Jang, J., Li, Y., Wang, C., Ding, M., Fox, D., and Yao, H. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411. 19309, 2024.
Zhao, C., Song, Y., Wang, W., Feng, H., Ding, E., Sun, Y., Xiao, X., and Wang, J. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409. 16280, 2024.
Zhao, T. Z., Tompson, J., Driess, D.. Florence, P., Ghasemipour, S. K. S., Finn, C., and Wahid, A. Aloha unleashed: A simple recipe for robot dexterity. In 8th Annual Conference Robot Learning.
Zheng, J., Li, Liu, D., Zheng, Y., Wang, Z., Ou, Z., Liu, Y., Liu, J., Zhang, Y.-Q., and Zhan, X. Universal actions for enhanced embodied foundation models, arXiv 2025.
preprint arXiv:2501.
Zhou, C., Yu, I.., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
12

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Supplementary Material
Place cube into yellow/blue Box. In this task, we assess the model's ability follow instructions and demonstrate spatial reasoning. The robot is presented with an instruction specifying a designated box, either yellow or blue, and is required to accurately identify the correct box based on the given instruction. Once identified, the robot must then place cube inside the chosen box. This task tests not only the model's s com- prehension of language-based directives but also its ability to integrate spatial awareness and execute pre- cise actions.
6. Evaluation Tasks and Detailed Results
6.1. Evaluation Tasks
As described in Section 4, we evaluate our DiVLA and baselines on multi-task learning, factory sorting, zero-shot bin-picking, and table bussing. In this section, provide details on these tasks. All of these tasks are depicted in Figure 10 and 11.
For multi-task learning, we evaluate all methods on 5 tasks for multi-task learning and visual generalization tasks. We evaluate each method with a total of 77 trials for multi-task learning and 45 trials for visual generalization. We provide the number of demonstrations and the average length of these tasks in 3. A brief description of these tasks is as follows:
Setup for visual generalization. In this setting, we assess the model's robustness and ability to generalize its visual perception to different environmental conditions. The robot is required to perform manipulation tasks under challenging visual variations, including randomly placed distractors, distracting lighting conditions, and a colorful background. These variations test the model's capacity maintain focus the primary task while filtering out irrelevant visual noise and adapting to dynamic visual environments. The goal is to ensure the robot can accurately identify and interact with objects despite significant deviations from standard settings.
Object selection. In this task, we evaluate the model's ability to comprehend and act on user intent effectively. To do so, we design three distinct instructions, each requiring the robot to identify and pick up the appropri- ate object based on the user's expressed intent. These instructions are intentionally diverse test the model's versatility and robustness in understanding varying con- texts and nuances of user commands.
Moreover, detailed descriptions for factory sorting, zero- shot bin-picking, and table bussing are provided in Sec- tions 4.3, 4.5, and 4.6, respectively.
Flip the vertically placed pot. In this task, we evaluate the model's 's commonsense reasoning abilities. Specifi- cally, the model must determine whether a vertically placed pot is oriented to the right or left and accurately identify its direction to position it correctly. This test assesses the model's 's capacity to understand spatial ori- entation and apply logical reasoning achieve the desired outcome.
Summarization for the number of demonstrations and Table 3:
average trajectory length for real-world tasks.
Tink
of Demomsttations
Avg Trij. Longth
Object selection
160
91
Flip the vertically placed por
120
137
Place cube inside box with closed lid
50
146
Place onto the plate
100
107
Place into yellow/blue Box
100
Place cube inside box with closed lid. In this task, we evaluate the model's ability reason about and exe- cute multi-step operation that involves manipulating objects with physical constraints. The robot must first identify the box with the closed lid, open the lid, and then place the cube inside the box. This task assesses the model's planning, understanding of spatial relation- ships, and ability to perform actions in a sequence to achieve a desired goal.
6.2. Detailed Results
In this section, we present the complete evaluation results for both the Franka and Bimanual AgileX robots, as detailed in Table 4. DiVLA-2B consistently demonstrates superior performance across the majority of tasks. Additionally, our findings highlight DiVLA-2B's robust visual generalization capabilities, effectively adapting to changes in the surround- ing environment.
Place cup onto the plate. In this task, we test the model's ability to perceive and reason about the spatial arrangement of objects in a structured environment. A plate is placed on a two-tiered shelf, and the robot must first identify which tier the plate is located on. And the robot needs to carefully place the cup onto the plate without disturbing the surrounding setup. This task evaluates the model's ' spatial reasoning and precision in action.
7. Implementation Details
For the baselines, we follow a uniform training strategy to ensure consistency. For OpenVLA, the original implemen- tation uses only single camera view. Since our method use three camera views on both single arm Franka robot and bimanual AgileX robot, it would be unfair to compare the vanilla implementation of OpenVLA that uses only a single camera view. Therefore, for the purpose of fair com-

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Multi-Task Learning
Visual Generalization
Object
Selection
Flip the
vertically
distractors
placed pot
Place cube
inside box with
closed lid
w/ distracting lighting
Place cup onto the plate
w/ colorful background
Place cube into yellow/blue Box
Figure 10: Multi-task Learning and Visual Generalization. We evaluate each method on multi-task learning and visual generalizations, including adding additional distractors, changing the background, and implementing colorful lighting. Each of three images represents the initial state, intermediate state, and final state of one trial.
parison, we extend OLe to utilize three camera views, processing each view through the same visual encoder and concatenating their outputs for further analysis. Open VLA's pre-trained weights are used, and we fine-tune the model on our dataset with learning rate of 2e-5. Training is con- ducted over 20 epochs, as we find that Open VLA typically requires longer training times for convergence. For the Dif- fusion Policy, we employ DistilBERT' to encode language instructions, adopting methodology similar to YAY (Shi et al., 2024). Table 7 shows the performance of the single- view OpenVLA bascline. In the sorting task, the success rate drops significantly from 45.3% with three views just 12.7% with a single view, highlighting the importance of our proposed multi-view approach.
8.2. Efficient Inference
Fast inference speed is critical for deploying VLA models in real-world applications. However, as models grow in size and complexity, their inference speed tends to decrease significantly on server-grade hardware. To address this challenge and enhance model performance, we deploy our models on the vLLM framework (Kwon et al., 2023). This approach leverages optimized infrastructure maximize in- ference efficiency. In Table 5, we present comparative anal- ysis of control frequencies for DiVLA-21 2B, DiVLA-7B, and Open VLA (a 7B model), Our results demonstrate remark- able speed improvements: the DiVLA-2B model achieves an impressive 82 Hz on an A6000 GPU, showcasing ex- ceptional performance. Similarly, DiVLA-7B achieves a control frequency of 42 Hz, which is 8 times faster than
8. More Experiments
OpenVLA at the same model size. These findings under- score the effectiveness of our optimizations in maintaining scalability without sacrificing speed, paving the way for broader real-world applicability of VLA models.
8.1. View Shifting Generalization
Imitation learning often struggles to generalize its capabil- ities to novel viewpoints. In this study, we evaluate the view generalization performance of DP, VLe, and DiVLA-2B in factory sorting task. The experimental setup is illustrated in Figure 12. As shown in Table 6, Open- VLA exhibits poor generalization to view shifts. In contrast, DiVLA-2B demonstrates notable robustness in view gen- eralization, achieving a success rate of 60%. This result highlights the advantages of leveraging pretraining on large- scale robotic datasets to enhance generalization capabilities.
While vLLM accelerates our method, the improvement in inference time is less dramatic than reported for LLMs. Even without vLLM, DiVLA-2B and DiVLA-7B achieve respectable frame rates of 74Hz and 30Hz, respectively, Notably, this is still six times faster than Open which has a similar model size (7 billion parameters).
Another key observation that our model experiences sig- nificant performance degradation when evaluated at lower
2

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Factory Sorting
Zero-Shot Bin-Pich Bin-Picking
Seen
YY
Move any object on the right panel to the left basket
Cluttered
Seen
Table Bussing
Mixed
Seen
rY
Cluttered Mixed
Mixed
Figure 11: Factory Sorting, Zero-Shot Bin-Picking and Table Bussing. We further evaluate all robot policies on additional challenging tasks, including factory sorting, zero-shot bin-picking, and bimanual table bussing. These tasks involve previously objects with diverse textures, varying heights, and different degrees of deformability (as illustrated Figure 7 Figure 8). Each of three images represents the progression of a single trial, showcasing the initial state, intermediate state, and final state.
the model to pick up unseen items and follow sequential instructions. Specifically, the instruction templates are "Pick up [obj}." and "Pick up {obj) first, then pick up the {obj), finally pick up {obi]." We tested the model on four objects: 1) a watermelon, 2) lemonade, 3) a blue paper trash, and 4) red pepper. This an extremely challenging task, as these novel instructions are absent from both the Droid dataset and collected data. We evaluated four new instructions, with results summarized in Table 9.
View Shining
Nov
Origia
Figure 12: View Generalization. We evaluate DP, Open VLA, and DiVLA-2B in the view shifting setting, where we use completely different camera positions to capture images. The blue part indi- cates the original camera positions, while the red part indicates the new camera positions.
Our findings indicate that both Open and DiVLA-2B can recognize these unseen objects and perform basic pick- and-place tasks. However, when it comes to complex se- quential tasks, Ope fails interpret the instructions accurately, instead randomly selecting items. In contrast, our method correctly follows the instructions, picking up objects in the specified sequence. We hypothesize that by learning to decompose long tasks into subtasks, our method acquires a generalized ability to understand complex, multi- step instructions. While Open can execute simpler commands like "Pick up the watermelon," " struggles with more advanced instructions requiring item selection specific order. Additionally, ing precision when the model processes novel instructions, indicating that the novelty of instructions introduces further complexity to task exccution.
bit precisions, such as 8-bit and 4-bit. This aligns with findings VLe which highlight performance fluctua- tions introduced by quantization. These results suggest that current VLA models may require specifically designed quan- tization methods to maintain performance while achieving fast inference speeds at low precision.
observe decrease in grasp-
8.3. Following Novel Instruction
The previous section primarily evaluated the model's per- formance on in-distribution task commands. In this section, we assess the model's ability to follow novel instructions, specifically focusing on its generalization to previously un- seen commands. We introduce new instructions that prompt
3

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Table Detailed Experimental Results on both Franka and AgileX Aloha.
Type
Task
Trails
DP
Tiny VLA
Octo
OpenVLA
DiVLA-2B
Object Selection
33
22
23
19
33
Flip the Vertically Placed Pot.
11
4
5
3
11
Multi-Task Learning
Place Cube into Yellow/Blue Box
11
0
3
3
10
Place Cup onto Plate
11
4
5
1
7
Place Cube inside Box with Closed Lip
11
0
4
2
7
Object Selection
1
4
4
4
Flip the Vertically Placed Pot.
1
4
1
6
Visual Generalization
Place Cube into Yellow/Blue Box
9
0
2
0
6
Place Cup onto Plate
2
2
0
6
Place Cube inside Box with Closed Lip
9
0
1
2
0
4
Factory Sorting (Seen)
80
27
39
33
45
61
Factory Sorting (Mixed)
80
11
25
27
36
50
Factory Sorting (Cluttered Scen)
65
16
21
17
30
43
More Challenging Tasks
Factory Sorting (Cluttered Mixed)
65
14
15
40
Zero-Shot Bin-Picking
102
10
24
4
29
65
Table Bussing (Seen)
48
22
35
Table Bussing (Mixed)
48
15
0
34
Table 5: Inference speed for DiVLA. We report inference speed for DiVLA on A6000 GPU,
Table 7: Ablation study on OpenVLA using one camera view and three camera views. For a fair comparison. our main experi- ments evaluate OpenVLA using three camera views with existing methods. This section presents ablation study comparing these results with those obtained using single-camera view to demon- strate benefits of our multi-view approach.
Method
DiVLA-2B
DiVLA-7B
OpenVLA-7B
Control Frequency
82Hz
42Hz
5H2
Table Experimental results for view shifting generalization. We conduct view shifting generalization on factory sorting task. The experimental setup is shown in Figure 12. We report the success rate for each policy. Each method is evaluate with 5 trails.
Method/Sartng
Seen
Mixed
Clutered Seen
Clattered Mixed
Avg.
OpenVLA OpenVLA (1 view)
26.1
17.5
4.8
2.1
12.7
Table 8: Ablation study on reasoning injection module.
Task Method
DP
VLe
DiVLA-2B
In-Distribution Task
Factory Sorting
60
Model 1 Tasks
Task
Avg.
DiVLA-2B
100
100
63.6
63.6
90.9
83.6
wlo reasoning injection
66.7
66.7
45.5
45.5
27.3
50.3
8.4. A Ablation Study on Reasoning Injection Module
Our core contribution is the proposed reasoning module, which enables the network to complete long-horizon tasks via direct prompting. This section presents an ablation study of this module, following the real-world experimental setup described in Section 4.2. We train a model using the same methodology, but without the reasoning injection module. Table 8 presents the results. We observe significant per- formance drop compared to our baseline model when the reasoning injection module is removed. We hypothesize that the reasoning module facilitates task decomposition, result- ing in more fine-grained action generation, which simplifies the learning process.
model size and data volume grow, the performance of the model should ideally improve. To evaluate this scalability, challenging
conducted additional experiments tasks: factory sorting and zero-shot bin-picking. These tasks provide rigorous benchmarks measure how increased model capacity and pre-training data influence performance. The detailed experimental results are presented in Table 10.
Our analysis highlights significant performance improve- ments when larger models and more extensive pre-training datasets are utilized. For example, the DiVLA-7B model demonstrates substantial gains over the DiVLA-2B model, achieving improvements of 8.7% and 3.0% in success rates for the factory sorting and bin-picking tasks, respectively, This underscores the benefits of scaling model size, which enables better representation learning and decision makinng capabilities.
8.5. Model Scaling
An important consideration for pure end-to-end neural net- works is assessing their ability scale effectively. As the

---

Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Table Experimental Results for Following Novel Instruction. We report the success rate, calculated as the number of successful trials divided by the total trials, for the novel abilities of OpenVL. VLA and DiVLA-2B. Our method successfully picks up items based given instructions and follow sequen- tial instructions to retrieve items the correct order.
Table 1I: VQA for DIVLA. test DiVLA's ability to answer questions based on visual signals.
Queston
Ocject/Serre
Answer
Corvect
Toy Tiger
What object?
Telip
Model Tinks
Fellowing Novel Insinccionn on Four
Watermelon
Watermelon - Blue Poper Trah- - Lemonade
Oharnge
Op (Kim ata)
(X3
DVVIA-28
2/3
2/
Brown
Model Tooks
Red Pepper
Blue Cup Watermelon - Lemonade - line Paper Troh
séjjetts coler?
OpenVLA (Kin al)
03
IV3
DiVLA-28
18
1/
Green
Deseribe the scene.
The cube on the right side of yellow pepper
Moreover, scaling up to 72B parameters yields even greater performance boosts. These larger models not only achieve higher success rates in in-distribution scenarios but also exhibit superior generalization out-of-distribution setups. This improvement reflects the models' enhanced ability to handle variability in task environments, unseen object properties, and dynamic conditions.
The the top of boldes
reveals limitations in processing broader visual features for complex tasks. To further assess its capabilities, we tested the model with simple scene description task, requiring it to describe spatial relationships between objects. The model successfully interpreted spatial relationships, such as identifying that one object is on the right or on top of another. However, it often failed recognize objects correctly. For example, DiVLA misclassified football as regular ball.
Table 10: Experimental results for model scaling,
Tasks/Models
DiVLA-2B
DiVLA-7B
DiVLA-72B
Sorting
66.2
74.9
82.4
Bin Picking 63.7
66.7
75.9
Notably, these objects were not included in the model's pre- training or fine-tuning data. However, the pre-trained VLM may have encountered similar objects during its pre-training phase, enabling it to recognize them correctly in some cases. This observation underscores the importance of leveraging pre-trained VLMs foundation for end-to-end visuomo- tor learning, they provide a strong prior understanding of visual concepts that can significantly enhance downstream performance in complex tasks.
8.6. DiVLA can do qutual-ueestion-Anweeinng
Previous works, such as RT-2 (Brohan et al., 2023) and ECoT (Zawalski al., 2024), have suggested that co- training with vision-language data helps preserve basic con- versational functionality for Vision-Language Action (VLA)) data. In this section, we demonstrate that despite not being co- co-trained with vasion-language data, our method retains its chatting capability and is proficient in performing common souual-ucstion-answeeeee
(VQA) tasks. However, these works have neither provided concrete examples of chat in- teractions nor open-sourced their results, leaving it unclear to what extent conversational capabilities are retained in such models. We give few examples in Table 11.
When prompted to identify objects, we observe that the model demonstrates the ability to recognize some items ac- curately, such as tulips and oranges. However, struggles with more nuanced distinctions; for instance, DiVLA mis- takenly identifies toy dragon toy tiger, likely due to its reliance color features rather than other distinguishing characteristics like shape or texture.
Interestingly, we observe that the model demonstrates strong sensitivity to color, consistently providing correct answers to all three questions focusing on an object's color. This highlights a notable strength in color recognition but also
5


Reviews:
[{"review_id": "RO80Ctu24v", "reviewer": "Reviewer_fKzV", "summary": "DiffusionVLA unifies autoregressive reasoning with diffusion-based action policies to build robust vision–language–action models for robotic control. By injecting self-generated reasoning directly into the policy head, the framework improves interpretability and decision-making. Extensive experiments on tasks like factory sorting and zero-shot bin picking demonstrate strong generalization and fast inference, outperforming several state-of-the-art baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and valuable comments. We address each question below.\n\n## 1. Training efficiency for DiVLA compared to baselines\n\nThank you for your valuable feedback and insightful advice. Since DP and Octo perform significantly worse across most tasks, our comparison focuses on competitive VLAs.\n\n1). **Pretraining Computations:** OpenVLA requires 27 epochs (21,500 A100 GPU hours) for action token prediction on the OXE dataset, while DiVLA achieves language rationale generation in just 2.5 epochs (155 H800 GPU hours) on part of Droid dataset. DiVLA uses only 39K pretraining samples, 25 times fewer than OpenVLA's 970K, yet still delivers interpretable reasoning, demonstrating superior efficiency.\n\n| model | Pre-trained Data | GPU hours |\n| --- | --- | --- |\n| TinyVLA | / | / |\n| OpenVLA | 970K | 21500 |\n| DiVLA-2B | 39K | 155 |\n\n2). **Finetuning Computations:** All models are trained with the same batch size on 8×H800 GPUs. TinyVLA takes 12 hours to adapt, while DiVLA outperforms it by 29.8% in factory sorting and reduces finetuning time compared to OpenVLA, achieving a 20.9% improvement. DiVLA shows superior efficiency and adaptability in finetuning.\n\n| model | OpenVLA | TinyVLA | DiVLA-2B |\n| --- | --- | --- | --- |\n| Finetune Time (hours) | 25 | 12 | 15 |\n| size | 7B | 1.3B | 2B |\n| GPU memory | 66 | 29 | 38 |\n\n3). **Model Size and Memory Usage:** As shown in the table, DiVLA is smaller than OpenVLA (2B vs. 7B parameters) and requires less GPU memory, making it more reproducible and accessible.\n\nOverall, DiVLA demonstrates superior computational efficiency in both pretraining and finetuning, achieving significant performance gains and highlighting the advantage of our approach.\n\n## 2. Ablation on projection layer of action tokens\n\nThanks for pointing it out. We ablate two projection layer, as requested for action tokens on 3 tasks (described in page 4 line 189-199). For each variants we evaluated them for 11 trails for each task.\n\n| task/module | FiLM | two-MLP+LayerNorm(ours) |\n| --- | --- | --- |\n| task 1 | 63.7 | 100 |\n| task 2 | 45.4 | 100 |\n| task 3 | 36.4 | 63.6 |\n| Average | 48.5 | 87.9 |\n\nAs shown in table, using two-MLP+LayerNorm as projection layer performs relatively better than FiLM which highlights our choice is effective.\n\n## 3. Ablation on three reasoning injection modules (MLP, Q-Former, and FiLM)\n\nWe thank the reviewer so much for pointing this out. We ablate three reasoning injection modules, 1) Plain MLP like LLaVA [1], 2) Q-former as in BLIP [2], and 3) FiLM. For each variants, we evaluated on three tasks, each with 11 trials. \n\n| task/module | Plain MLP | Q-Former | FiLM(ours) |\n| --- | --- | --- | --- |\n| Task 1 | 36.4 | 18.2 | 100 |\n| Task 2 | 45.4 | 45.4 | 100 |\n| Task 3 | 18.2 | 27.3 | 63.6 |\n| Average | 33.3 | 30.1 | 87.9 |\n\nAs demonstrated in the table, DiVLA with the FiLM module significantly outperforms plain MLP and Q-former, confirming the efficacy of our design. Intuitively, reasoning tokens act as conditioning factors, enhancing robot action generalization without dominating predictions. The FiLM architecture effectively fulfills this conditioning role.\n\n## 4. Do we use 2e-5 for all module?\n\nYes, we use initial learning rate of 2e-5 to all layers, including newly initialized projection modules. We didn’t do hyper-parameter search in our experiments. \n\n## 5. It is interest that Diffusion-VLA can generate language rationales alongside its robot action, how does it compare to OpenVLA and pi0 in terms of  interpretability\n\nThank you for raising this point. DiVLA is the only model that generates language rationales alongside robot actions, making it interpretable. In contrast, OpenVLA and Pi0 cannot generate language. DiVLA uses reasoning for dynamic scene understanding and task planning, revealing the model's decision-making process and improving long-horizon task completion by making failure reasons visible to users.\n\nOur experiments shows the importance of making VLA interpretable. Specifically, as illustrated in Figure 3, on the sorting task involving 4 progressively challenging settings, DiVLA can generate real-time reasoning traces (e.g., identifying objects and categorizing them). It achieves 20.9% improvement over OpenVLA across all settings. Furthermore, as shown in Figure 4, DiVLA exhibits remarkable zero-shot generalization, outperforming OpenVLA by 35.3% on 102 unseen objects. These experimental results shows that introducing interpretability to VLAs significantly enhances their ability to execute long-horizon tasks and generalize to novel objects.\n\n## 6. For format, section renaming and figure annotation\n\nThanks for pointing this out. We will improve these formating problems, section renaming and figure annotation in updated version.\n\n[1]. Improved Baselines with Visual Instruction Tuning\n\n[2]. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", "claims_and_evidence": "-  The paper claims that combining autoregressive reasoning with diffusion policies and a dedicated reasoning injection module yields superior and interpretable robotic control compared with prior VLA models.\n\nWhile the empirical results show improved task performance and interpretability, similar capabilities (e.g., natural language rationale generation) have been explored in comparable works. A more detailed ablation or comparison that isolates the effect of the reasoning module would strengthen the claim (e.g., the choice of projector, the effect of data source).", "methods_and_evaluation_criteria": "- Training Efficiency and Model Size:\n\nHow does DiffusionVLA’s training efficiency and model size compare against baseline methods? Quantitative comparisons (e.g., training time, parameter counts, memory usage) would provide valuable context.", "theoretical_claims": "There are no novel theoretical contributions.", "experimental_designs_or_analyses": "- Projection Layer for Action Tokens:\n\nHave the authors experimented with different projection layers, such as FiLM, instead of the two-MLP+LayerNorm design? A comparison would help justify the module selection.\n\n- Reasoning Injection Module:\n\nSimilarly, for the reasoning injection module, did the authors try simpler alternatives (e.g., a plain MLP or an attention-based mechanism) and compare their effectiveness? How do the authors guarantee that the chosen design is optimal?\n\n- Learning Rate Details:\n\nIs the fixed learning rate (2e-5) uniformly applied to all layers, including the newly initialized projection modules?", "relation_to_broader_scientific_literature": "A deeper discussion is needed on how this work situates itself within the broader literature on diffusion policies and autoregressive reasoning in robotics. How do its capabilities (especially regarding interpretability and efficiency) compare with models like OpenVLA, or π0?", "essential_references_not_discussed": "The reference list is comprehensive and appropriately covers related work.", "other_strengths_and_weaknesses": "### Strengths:\n\n- The unified framework effectively combines reasoning and diffusion-based action generation.\n- The reasoning injection module adds interpretability by generating natural language rationales alongside actions.\n- Empirical results show robust performance across varied real-world tasks with fast inference speeds.\n\n### Weaknesses:\n\n- Limited quantitative discussion on training efficiency and model size compared with baseline methods.\n- The paper does not provide comparative experiments on alternative projection or reasoning modules (e.g., using FiLM or plain MLPs).", "other_comments_or_suggestions": "- Add a period after “randomly initialized weights” for clarity.\n- In Figure 1, annotate the x-axis to indicate model size.\n- Consider renaming Section 3.2 from “Model Design Choices” to a title that more accurately reflects its focus on architecture and training objectives.\n- Clarify whether the fixed learning rate (2e-5) is applied uniformly across all layers, including newly initialized projection modules.\n- The claim that the model “generates natural language rationales alongside its output actions” is interesting; please discuss if any baseline models offer similar interpretability features and how DiffusionVLA’s approach compares.", "questions_for_authors": "See the comments and questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxrgxlmeja", "reviewer": "Reviewer_8bw9", "summary": "The authors propose combining the reasoning capabilities of LLMs with the robot action generalization capabilities of diffusion models, creating DiVLA. DiVLA extracts and interleaves tokens from visual input and text using SigLIP, concatenates them, and processes them in a VLM. The VLM generates action tokens that are projected and processed by the diffusion model. FiLM layers are used to incorporate reasoning tokens output by the VLM into the diffusion model. DiVLA is compared to previous VLAs on two real-world robot settings for various pick and place tasks, including seen and unseen objects and including a bimanual robot.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback.\n## 1. Simulated evaluation\nReal-world evaluation is more challenging than simulation. While our work emphasizes complex real-world tasks like long-horizon bin-picking and bimanual table bussing, we also evaluate DiVLA on two standard simulation benchmarks, Calvin and Libero. Compared with baselines including Diffusion Policy, Octo, OpenVLA, RT-1, Robo-Flamingo, and GR-1, DiVLA achieves the best performance under a unified training setup (results: https://i.postimg.cc/xTybS5p3/simulation.png), demonstrating its robustness across both real and simulated environments.\n\n## 2. Ablation on key modules\n\nWe do agree that ablation study is necessary for better support our claim. We have addressed this problem \nin response to **Reviewer fKzV#3**. Please refer to that section for details.\n\n## 3. Clarify difference between points 1 and 6 in the introduction section\n\nWe are sorry for the confusion between point 1 and point 6. Generalizing manipulation skills on novel objects and robust to dynamic environment are two main challenges in robot learning. Point 1 focuses on DiVLA’s ability to recognize novel objects through self-generated reasoning, while Point 6 highlights its robustness to distractors and dynamic environments.\n\n## 4. Citation for “Research has shown … various embodiments.”\n\nThanks for pointing this out. The next-token-prediction method used in OpenVLA struggles to complete tasks when adapting to new embodiments [1] and performs poorly when learning dexterous skills with high-frequency control [2].\n\n## 5. Limitation of current diffusion-based VLAs and Superiority of DiVLA\n\nThanks for your insightful question. We have addressed this problem \nin response to **Reviewer fKzV#5**. Please refer to that section for details.\n\n## 6. Prompt for gpt-4o\n\nThanks for pointing this out. The prompt is at here: https://i.postimg.cc/kXcDH4Dj/prompt.jpg.\n\n## 7. Clarify whether the model will confuse similar objects or not\n\nThanks for your reminder. While DiVLA shows strong generalization, it occasionally misclassifies unseen objects, such as confusing small stuffed toys with toy cars. However, DiVLA handles shape-similar, color-distinct objects well, thanks to its robust color perception. Real-time reasoning traces help diagnose misclassifications and offer insights for further improvement.\n\n## 8. Comparison on model size, inference speed, training speed with baselines\n\nThanks for pointing this out. We will discuss the model size and inference speed separately. \n\n1). **Model size:** Our DiVLA, which uses only 2B parameters—1/3 of OpenVLA’s parameters—achieves the best performance across all tasks, demonstrating our method's model efficiency.\n\n| model | DP | Octo | OpenVLA | DiVLA-2B |\n| --- | --- | --- | --- | --- |\n| size | 153M | 93M | 7B | 2B |\n\n2). **Inference speed:** As shown in table, DiVLA-2B can achieve 82hz control frequency at test time with vLLM and action chunk. This is 20x faster than OpenVLA. The results highlight the inference efficiency of diffusion-based VLAs over autoregressive VLAs.\n\n| model | DP | Octo | OpenVLA | DiVLA-2B |\n| --- | --- | --- | --- | --- |\n| inference speed(Hz) | 122 | 105 | 4 | 82 |\n\n3). **Training speed:** We have addressed this problem in response to **Reviewer fKzV#1**. Please refer to that section for details.\n\n## 9. Clarify details on reasoning and action tokens\n\nThanks for your feedback. \n\n1). **Difference between reasoning and action tokens:** Action and reasoning tokens serve as different conditions on action denoising. Specifically, action tokens mainly include visual embeddings providing global observations and raw instruction tokens. On the other hand, reasoning tokens encapsulate hierarchical substep-level features that decompose complex tasks into temporally executable steps. \n\n2). **How is VLM trained to output reasoning and action tokens:** Action tokens refer to vanilla vision tokens and tokens from raw instruction processed by VLM backbone while reasoning tokens are task-specific language generated by VLM. Thus, we only use standard next-token prediction loss for supervising reasoning generation.\n\n3). **Number of both tokens:** As previously noted, action tokens comprise tokens from raw instruction and visual tokens, whereas reasoning tokens are autonomously generated by the VLM. Action tokens are much longer than reasoning tokens as a large number of vision tokens.\n\n## 10. Typo problems\nThanks for your feedback. We will improve them in updated version.\n\n[1]. TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation\n\n[2]. FAST: Efficient Action Tokenization for Vision-Language-Action Models\n\n[3]. CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation\n\n[4]. RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation\n\n[5]. π0: A Vision-Language-Action Flow Model for General Robot Control", "claims_and_evidence": "Since the paper makes the claim that reasoning contributes to the model’s robustness in l. 365-368 col. 2 then this should be validated.", "methods_and_evaluation_criteria": "Yes but there are no synthetic benchmarks, implying that the model may not perform well on an environment with a vastly different set-up than the real-world experiments currently used.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "DiVLA builds directly off of Diffusion Policy and previous work in VLMs. It tackles a worthwhile problem of combining the best of VLMs and diffusion models for VLAs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nS1: Strong performance on two real world settings, especially on generalization to unseen objects.\n\nWeaknesses:\n\nW1: The components of DiVLA are not ablated. Since the paper makes the claim that reasoning contributes to the model’s robustness in l. 365-368 col. 2 then this should be validated. Additionally it is important to validate the choice of model architectures, since the main contribution of DiVLA is in the combination of existing components.\n\nW2: No synthetic benchmarks, such as Libero [1], that are standard for pick and place tasks in robotics. This implies that the model may not perform well on an environment with a vastly different set-up than the real-world experiments currently used.\n\n[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Liu et al. arxiv preprint arxiv:2306.03310, 2023.", "other_comments_or_suggestions": "- l. 306 col. 1 “pretraiend”→”pretrained”\n- l. 309 col. 1 “the setting as pi_0”→”pi_0’s setting”", "questions_for_authors": "1. What is the difference between points 1 and 6 in the introduction section?\n2. l. 140-144 “Research has shown … various embodiments.” Can you give some citations with evidence for this?\n3. What are problems with current diffusion-based VLAs?\n4. l. 248-252 col. 2 “To … reasoning.” What prompts do you send to GPT-4o?\n5. l. 364 - 376 Does this pose a problem when there are two or more similar objects in the scene, i.e. does the model confuse similar objects when both exist in the scene?\n6. How does the DiVLA’s size compare to other VLAs such as Diffusion Policy, Octo, OpenVLA etc? How long does it take to infer a single image and to train compared to the other models?\n7. What is the difference between reasoning and action tokens? How is the VLM trained to output them separately? Is there a different number of each type of token generated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Hk2qSZ8qd", "reviewer": "Reviewer_13rJ", "summary": "DiVLA is a VLA model that connects a VLM with a diffusion model to enable both reasoning and action generation in robotics. It builds upon a pre-trained Vision-Language Model (VLM) for text-based reasoning while incorporating a diffusion model to learn robotic actions through a noise-denoising process. DiVLA introduces a reasoning injection module, which embeds reasoning outputs directly into the policy head, enhancing decision-making and interpretability. This framework demonstrates strong generalization and robustness in basic tasks (e.g., pick and place) compared to previous methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and valuable comments. We address each question below.\n\n## 1. Why DiVLA-7B has similar number of parameters but is 8 times faster than OpenVLA-7B?\n\nThank you for pointing out this question.\n\n1) Diffusion-based VLA operates significantly faster than autoregressive VLA in robot action generation. In robotics, inference speed refers to the number of actions produced per second. For instance, using the Franka Emika with 7 degrees of freedom, OpenVLA must predict 7 tokens for each action. In a typical setup, the model needs to predict 30–60 actions for each incoming observational state, requiring 210–420 tokens for prediction. In contrast, diffusion-based VLA, such as DiVLA, only needs 10 denoising steps to predict 30–60 actions. This conclusion is supported by FAST [1], which finds that diffusion-based VLA is much faster than autoregressive VLA. In summary, DiVLA benefits from significantly faster inference times due to its use of a diffusion process, confirming the accuracy of our inference tests.\n\n2) To further ablate the impact of vLLM, we report the following test on an A6000 GPU, where T_ntp and T_diff represent the time spent on the NTP and diffusion processes, respectively.\n\n| Model | T_ntp \\ T_diff (s) | Chunk Size |  Inference speed |\n| --- | --- | --- | --- |\n| OpenVLA | 0.27 \\ - | 1 | 4HZ |\n| DiVLA w/o vLLM | 0.5 \\ 0.06 | 16 | 29HZ |\n| DiVLA w/ vLLM | 0.3 \\ 0.07 | 16 | 43HZ |\n\n## 2. Clarify the role of reasoning and action tokens in DiVLA\n\nWe appreciate you highlighting this. \n\n1). **Are both tokens used as conditions for diffusion model?** Yes, both action and reasoning tokens serve as different conditions on action denoising. Specifically, action tokens mainly include visual embeddings providing global observations and raw instruction tokens. On the other hand, reasoning tokens encapsulate hierarchical substep-level features that decompose complex tasks into temporally executable steps. \n\n2). **How action tokens are generated?** Action tokens refer to vanilla visual tokens and tokens from raw instruction processed by VLM backbone. They are not generated autoregressively.\n\n## 3. Whether DiVLA's ability to perform zero-shot bin picking is attributed to its generalization capability or spatial memory?\n\nThank you for your insightful question. To answer your question, the DiVLA’s ability to perform zero-shot bin-picking definitely stems from its strong generalization capability.\n\nAs outlined in page 8 line 410-420, the bin picking experiment used 102 unique objects that were entirely absent from the training set, ensuring that the model had to generalize to novel objects.  In Figure 8, we shows that the objects have varying size and height. Object like white tape is only 1cm high, while the toy dragon is 10cm. Also, in Figure 7, we present some unseen objects that we used for zero-shot bin picking tasks. It can be observed that these objects exhibited substantial variations in shape, size, color, texture, and deformability, ensuring that the model could not rely on simple heuristics such as shape consistency or fixed spatial features. Our model, DiVLA, achieved a 63.7% success rate, substantially outperforming baselines (Diffusion Policy: 8.9%, Octo: 19.6%, TinyVLA: 23.5%, OpenVLA: 28.4%), underscoring its superior generalization for unseen objects.\n\n[1]. FAST: Efficient Action Tokenization for Vision-Language-Action Models", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Please refer to 'Other Strengths and Weaknesses.'", "theoretical_claims": "Application paper without making theoretical claims.", "experimental_designs_or_analyses": "Please refer to 'Other Strengths and Weaknesses.'", "relation_to_broader_scientific_literature": "Effectively constructing a VLA model for robotics.", "essential_references_not_discussed": "This paper provides a thorough discussion of related works.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Attaching the diffusion module after the VLM to construct a VLA model is the mainstream action prediction approach, similar to TinyVLA and Pi_0.\n\n2. Unlike previous diffusion-based methods, DiVLA further incorporates Reasoning Tokens to enhance action generation, representing an innovative exploration.\n\n3. DiVLA exhibits strong zero-shot bin-picking capabilities.\n\nWeaknesses:\n\nQ1 My primary concern is the inference speed.\n1.1 The authors claim that DiVLA-7B's inference speed is 8 times faster than OpenVLA at the same model size, which appears unrealistic. OpenVLA performs next-token prediction for 7 action tokens (discrete end-effector poses), whereas DiVLA additionally requires language next-token prediction (e.g., \"Pick up the Rubik’s cube\"). Given this, the inference speeds of both models should be comparable. The authors should retest and verify DiVLA's speed.\n1.2 If LLM acceleration tools such as vLLM were utilized, this should be explicitly stated in the main text.\n\nQ2 Are Reasoning Tokens and Action Tokens both used as conditions for the Diffusion Model? If so, while Reasoning Tokens are obtained through next-token prediction, how are Action Tokens generated? Are they directly derived from the question token output of the VLM?\n\nQ3 DiVLA-2B and DiVLA-7B were pretrained on the Droid dataset, which is relatively limited compared to OXE. Could the model’s zero-shot bin-picking ability be attributed to its memory of the gray experiment tray (e.g., picking all objects within the tray) rather than genuine generalization to object semantics? For example, how would the model perform if objects were placed at different positions on the table without the tray?", "other_comments_or_suggestions": "It is recommended that the authors properly retest the model's inference speed. If DiVLA generates language reasoning during inference using an autoregressive next-token prediction approach, then DiVLA-7B's inference speed should be comparable to OpenVLA.\nFor instance, the process of DiVLA-7B generating 7 words in its LLM follows a similar inference logic to OpenVLA generating 7 action bins. However, if DiVLA does not generate language reasoning during inference, then how are Reasoning Tokens constructed?", "questions_for_authors": "Please refer to 'Other Strengths and Weaknesses.'", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvAooKbGWG", "reviewer": "Reviewer_bZhD", "summary": "This paper proposes a novel VLA framework that integrates NTP with a diffusion process. The writing is logical, and the experimental evaluation is thorough. While the core idea is intriguing and somewhat similar to $\\pi_0 $[1], the latter was released in October, close to the ICML submission deadline in January, which may explain the absence of a direct comparison. However, it would still strengthen the paper if the authors highlighted the specific differences between their approach and $\\pi_0 $[1],\n\n[1] Black, Kevin, et al. \"$\\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control.\" arXiv preprint arXiv:2410.24164 (2024).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback. We have addressed each point below. Please see the following responses for details.\n\n## 1. The inference speed of DiVLA(Same as R3#1)\n\nThis issue was partially addressed in our response to R13rJ#1. Bellow, we fristly clarify why DiVLA-7B has similar number of parameters but is 8 times faster than OpenVLA-7B, followed by an explanation of how vLLM optimizes inference efficiency in VLMs.\n\n1) **Inference Speed.** Diffusion-based VLA operates significantly faster than autoregressive VLA in robot action generation. In robotics, inference speed refers to the number of actions produced per second. For instance, using the Franka Emika with 7 degrees of freedom, OpenVLA must predict 7 tokens for each action. In a typical setup, the model needs to predict 30–60 actions for each incoming observational state, requiring 210–420 tokens for prediction. In contrast, diffusion-based VLA, such as DiVLA, only needs 10 denoising steps to predict 30–60 actions. This conclusion is supported by FAST [1], which finds that diffusion-based VLA is much faster than autoregressive VLA. In summary, DiVLA benefits from significantly faster inference times due to its use of a diffusion process, confirming the accuracy of our inference tests.\n\n2) **DiVLA with vLLM.** Regarding our DiVLA with vLLM, the inference speed of VLM can achieve approximately 2× speedup. vLLM enhances VLM inference efficiency through PagedAttention, which prevents memory fragmentation by storing KV cache in fixed-size pages, reducing memory overhead and improving stability for long-sequence generation. Additionally, optimized CUDA kernels accelerate attention computation, resulting in significantly faster inference compared to traditional frameworks, often achieving 2–3× speedup.\n\n| Model | T_ntp \\ T_diff (s) | Chunk Size |  Inference speed |\n| --- | --- | --- | --- |\n| OpenVLA | 0.27 \\ - | 1 | 4HZ |\n| DiVLA w/o vLLM | 0.5 \\ 0.06 | 16 | 29HZ |\n| DiVLA w/ vLLM | 0.3 \\ 0.07 | 16 | 43HZ |\n\n## 2. Details architectural diagram for DiVLA\n\nWe apologize for the absence of the detailed architectural diagram. As you correctly pointed out, the “projection layer\" and \"action decoder\" are crucial components of our framework but are not represented in Figure 1. To address this, we will include a more comprehensive diagram that clearly illustrates the entire architecture, including these specific modules and their interactions. \n\nThe projection layer consists of two MLP layers with LayerNorm. It bridges the VLM's embedding to the diffusion model, aligning their dimensions. Additionally, our action decoder follows the standard diffusion policy design [2]. It takes the embeddings from the VLM as conditions and predicts the noise based on the noisy actions, repeating this process *k* times to obtain the final action chunk.\n\n## 3. Clarify on Language Generation and  the L_NTP loss function\n\nIn Section 3.2 Model Design Choices, we state that our training loss comprises tow part: the diffusion loss L_diff and the NTP loss L_ntp. The ground truth for L_ntp is based on language supervision. Consequently, we can generate reasoning tokens (i.e. language output) during inference.\n\n## 4. Failure case analysis\n\nWe have failure case analysis (**page 7 lines 378-384**) which systematically examines how dynamic reasoning enables action self-correction. To extend this discussion, we propose a **dual-aspect failure taxonomy**: **erroneous reasoning chains** and **action execution errors**. As demonstrated in Table through controlled experiments on the bin-picking task, we evaluate 4 trials per object across both seen and unseen categories. Seen objects are appeared in robot data while unseen objects are not. Our model could successfully pick up those objects when facing seen objects and we empirically observe that action always fails when the model **cannot recognize the unseen objects** though. We attribute the recognition degradation to robot-data overfitting as exclusive training on robotic demonstrations diminishes the base VLM's open-vocabulary recognition capacity. Thus a potential solution is co-training the VLA with robot data and vision-text data, preserving both generalizable visual concepts and action-specific skills.\n\n| success/object | Watermelon(unseen) | dragon(unseen) | Orange(unseen) | toy car(seen) | hex key(seen) |\n| --- | --- | --- | --- | --- | --- |\n| Reasoning | 50 | 0 | 75 | 100 | 100 |\n| Action | 50 | 0 | 50 | 100 | 100 |\n\n## 5. Typo problem\n\nThank you for your careful review. We appreciate your feedback regarding the misreferenced table and figure. These issues have been corrected in the next version of the manuscript, which will be updated accordingly.\n\n[1]. FAST: Efficient Action Tokenization for Vision-Language-Action Models\n\n[2]. Diffusion policy: Visuomotor policy learning via action diffusion.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "Related", "essential_references_not_discussed": "Black, Kevin, et al. \"$\\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control.\" arXiv preprint arXiv:2410.24164 (2024).", "other_strengths_and_weaknesses": "Refer to Other Comments Or Suggestions", "other_comments_or_suggestions": "- Inference speed: After reviewing the supplementary materials, it appears that the high inference speed primarily arises from the use of the vLLM framework rather than the design of the method itself. To enhance clarity, it would be beneficial to explain in the main paper how vLLM is used to speed up inference and provide more details about its underlying implementation. \n\n- Method details:\n  1. The paper introduces a new VLA architecture but does not include a detailed architectural diagram. Specifically, the \"Projection layer\" and \"action decoder\" mentioned starting at Line 212 are absent from Figure 1, making it difficult to understand their functions and how they interact with other components. Including a comprehensive architecture figure that clearly illustrates these modules and their connections would greatly improve clarity.\n  2. In Figure 1, it appears that the reasoning tokens are transformed into language output. However, the method section does not mention any language output. Additionally, it remains unclear whether the ground truth (GT) for $L_{NTP }$ is based on action supervision or language supervision.\n\n- Others:\n  1. The paper references Table 1 on page 6, but the table is actually located on page 4, which disrupts the flow of the experimental discussion. A similar issue occurs with Figure 3. Please ensure that the references match their corresponding figures/tables to improve readability.\n  2. Although the model can perform corrections, it remains unclear why certain failures still occur. The authors should include a failure case analysis section to clarify the limitations of the model and suggest potential improvements.", "questions_for_authors": "Refer to Other Comments Or Suggestions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose DiffusionVLA (DiVLA), a hybrid model integrating VLM-based reasoning with diffusion-based action generation for robotics via a reasoning injection module (FiLM layers). The core idea involves using a VLM to generate explicit reasoning tokens (natural language rationales) based on visual input and instructions, which are then injected into a diffusion-based policy head via a FiLM module to condition the action generation process. The claim is that autoregressive action generators often lack the level of precision required for robotics, while diffusion models do better; but on the flip side, diffusion models are not good at reasoning. This approach is then tested on several real-world tasks such as pick and place, sorting, bin picking etc., showing significant improvements over baselines such as Diffusion Policy, OpenVLA among others.

Initial reviews were mixed, and towards the end still leaned negative (one weak accept, three weak rejects). Strengths identified by the reviewers included the core idea, strong real-world/zero-shot results, and interpretability via rationales. There were also some questions regarding the novelty of the interpretability feature, and concerns about lack of ablations/comparisons within this context. But the major weaknesses that were pointed out were the lack of crucial ablations and standard synthetic benchmarks. 

However, the authors provided a detailed and effective rebuttal that substantially addressed these key concerns. To summarize, the rebuttal addressed the following:

1. Missing Ablation Studies: Authors provided quantitative ablation results comparing different choices for the action token projection layer (showing their MLP+LayerNorm outperformed FiLM) and the reasoning injection module (showing their FiLM choice significantly outperformed Plain MLP and Q-Former), justifying their design choices (addressing R-fKzV#2, R-fKzV#3, R-8bw9#2).
2. Lack of Simulation Benchmarks: Included new results demonstrating state-of-the-art performance on standard simulation benchmarks (CALVIN/LIBERO), broadening the evaluation beyond the initial real-world tasks (addressing R-8bw9#1).
3. Training Efficiency & Model Size: Provided tables comparing DiVLA's pre-training requirements (significantly fewer GPU hours and data than OpenVLA), fine-tuning time (competitive), model size (smaller than OpenVLA), and GPU memory usage against key baselines like OpenVLA and TinyVLA (addressing R-fKzV#1, R-8bw9#8).
4. Inference Speed Justification: Offered detailed explanations differentiating diffusion-based action generation (fewer steps for sequences) from autoregressive methods (token-per-DoF). Provided quantitative breakdowns of inference time with and without vLLM acceleration, clarifying the sources of speedup and quantifying vLLM's impact. Explained the action chunking strategy used (addressing R-13rJ#1, R-bZhD#1).
5. Clarification on Reasoning/Action Tokens: Detailed the distinction between action tokens (visual/instruction embeddings) and reasoning tokens (generated language steps), how each is generated (backbone processing vs. next-token prediction), their relative counts, and confirmed both serve as conditioning inputs for the diffusion model (addressing R-8bw9#7, R-13rJ#2). Provided a code link for reasoning generation.

Authors also addressed generalization vs. memory concern and other methodological details. While the initial submission had gaps, I believe that the rebuttal successfully addressed several of them, strengthening the paper's claims regarding performance, implementation details, particularly zero-shot generalization, and efficiency. Some of the subsequent concerns raised by reviewer 13rJ were also addressed by the authors in a followup comment, but the discussion was not entirely finalized, and I do believe the authors addressed the reviewer's concerns in that thread satisfactorily.

The core contribution of integrating explicit reasoning into diffusion policies is sound and now better supported empirically. Given the combination of the novel approach, strong demonstrated performance across varied tasks (real-world and simulation), and the valuable interpretability feature - all bolstered by a fairly strong rebuttal and the willingness of the authors to elaborate on details of their technique; pushes the paper towards acceptance. Therefore, despite initial reservations, I recommend this paper to be accepted, contingent on careful revision to incorporate the extensive new results and clarifications in the final version of the paper.
Decision: Accept (poster)