Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better
Authors: Danny Driess
Jost Tobias Springenberg
brian ichter
LILI YU
Adrian Li-Bell
Karl Pertsch
Allen Z. Ren
Homer Walke
Quan Vuong
Lucy Xiaoyang Shi
Sergey Levine
Venue: NeurIPS 2025 spotlight ()
Keywords: vision-language-action models
robotic manipulation
robot learning
Primary area: deep_learning

Abstract: Vision-language-action (VLA) models provide a powerful approach to training control policies for physical systems, such as robots, by combining end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training. However, the constraints of real-time control are often at odds with the design of VLMs: the most powerful VLMs have tens or hundreds of billions of parameters, presenting an obstacle to real-time inference, and operate on discrete tokens rather than the continuous-valued outputs that are required for controlling robots. To address this challenge, recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads, which typically require adding new untrained parameters to the pretrained VLM backbone. While these modules improve real-time and control capabilities, it remains an open question whether they preserve or degrade the semantic knowledge contained in the pretrained VLM, and what effect they have on the VLA training dynamics. In this paper, we study this question in the context of VLAs that include a continuous diffusion or flow matching action expert, showing that naively including such experts significantly harms both training speed and knowledge transfer. We provide an extensive analysis of various design choices, their impact on performance and knowledge transfer, and propose a technique for insulating the VLM backbone during VLA training that mitigates this issue. Videos are available at https://pi.website/research/knowledge_insulation and open-source model weights are available at https://github.com/Physical-Intelligence/openpi.

Knowledge Insulating Vision-Language-Actionn Models: Train Fast, Run Fast, Generalize Better
Danny Driess
Jost Tobias Springenberg
Brian Ichter
Lili Yu
Adrian Li-Bell
Karl Pertsch
Allen Z. Ren
Homer Walke
Quan Vuong
Lucy Xiaoyang Shi
Sergey Levine
Physical Intelligence
Abstract
iision-language-acttion (VLA) models provide powerful approach training control policies for physical systems, such as robots, by combining end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training. However, the constraints of real-time control are often at odds with the design VLMs: the most powerful VLMs have tens hundreds of billions of parameters, presenting an obstacle to real-time inference, and operate on discrete tokens rather than the continuous-valus for controlling robots. To address this challenge, recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads, which typically require adding new untrained parameters to the pretrained VLM backbone. While these modules improve real-time and control capabilities, it remains open question whether they preserve degrade the semantic knowledge contained the pretrained VLM, and what effect they have the VLA training dynamics. In this paper, we study this question in the context of VLAs that include a continuous diffusion or flow matching action expert, showing that naively including such experts significantly harms both training speed and knowledge transfer. We provide an extensive analysis of various design choices, their impact performance and knowledge transfer, and propose technique for insulating the VLM backbone during VLA training that mitigates this issue. Videos insulation and open-source
outputs that are required
are available at https://pii hwwp:/sseeeeeeeerc///oooo/o model weights are available at
Introduction
The success of large language models (LLMs) can be attributed to the availability of large-scale datasets combined with powerful model architectures such as transformers that trained with next-token prediction objective on trillions of tokens. LLMs can be prompted to solve all sorts of tasks, from writing poems and code solving competition-level math problems, and can further be adapted to solve visual reasoning problems when extended with multi modll encoders to produce vision-language models (VLMs). A natural next step bring the power of LLMs the physical world is to further extend them to take physical actions, resulting in vision- language action (VLA) models that can control robots to follow language commands, combining the power of end-to-end robotic learning with the semantic knowledge distilled from web-scale vasion-language pretraining 62] 24 However, adapting I.MSs and VL.Ms to real-world control requires addressing number of new challenges. Most physical systems (e.g., robots) require continuous and precise commands, such as joint angles or target poses, that must be produced in real time at a high frequency. Autoregressive decoding of discrete tokens is poorly suited to this kind of high-freyuency continuous control, both because of the limited resolution of discretized actions and because the computational cost of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

autoregressive loss Cevu
language prediction
discrete actions
continuoux actions
flom natching loss Crana
-1.7
2.23
3.16
2.42
pick
the Sleeve
+17
12
34
1A2
bidirectiona w/o loex
stop gradient
VLN BACKBONE (3B)
ACTION EXPERT (300H)
high-level
azorpt
NOTSE
ro3d
the
ghaxt
The key idea of our approach is train the VLM backhone with next-token prediction loss on discretized actions and general VLM data learn good representations, while the action expert is trained with flow-matching on continuous actions. Gradients do not flow from the action expert to the backbone, insulating the knowledge of the backbone. At inference time, generating continuous actions with the smaller action expert is desirable for fast and precise control, while representation learning with discrete actions and general VLM data makes the model train fast and leads to better generalization by transferring knowledge from VLM data into robot actions. As experiments show, having both action representations at training time is crucial.
Figure 1:
autoregressive decoding with large models, challenge only exacerbated by ever larger models. Furthermore, physical systems typically produce more complex observations than VLMs are trained for, such multi-view images and proprioceptive states. These differences necessitate modifications to the original VLM architecture to accommodate robotic control.
Consequently, the robotics community has developed architectures that are particularly well-suited to the demands of real-time continuous control [55] I 46 [6, 33
6 & 25 22]. While a number of different designs have been successful, common theme is that models adapted for effective dexterous control typically augment a transformer or VLM backbone with some sort of adapter for continuous inputs and outputs, with the latter most often utilizing, for example, diffusion flow matching with action chunks (short sequences of future actions) continuous action distributions, select very precise actions, and capture dexterous high- high-freeuency skills. However, when these additional modules are added to a pre- trairra VLM to create VLAs, they typically need to be initialized from scratch, and the VLA training process must "graft" them onto the VLM backbone. This raises an important question: how much do VLAs augmented with these continuous state and action adapters actually inherit and benefit from web-scale pre-training?
[55 This enables the model to represent complex
In this work, we observe that prior approaches for finetuning VLMs with continuous outputs can, perhaps unsurprisingly, lead to significantly worse training dynamics, they rely gradients from continuous adapters (e.g. diffusion heads) for the training signal. This can degrade both their ability to interpret language commands and the overall performance of the resulting VLA policy. To address this challenge, we propose a training recipe that addresses these issues, which we refer to as knowledge insulation. The key idea behind knowledge insulation is to fine-tune the VLM backbone with discretized actions while simultaneously adapting an action expert to produce continuous actions (e.g., via flow matching diffusion) without propagating its gradients back into the VLM backbone. We illustrate this in Figure|] [ In effect, the discrete action tokens provide a substitute learning signal that is unaffected by the uninitialized weights of the action expert, such that the VLM still Iearns appropriate representations for robotic control, but without the disruption that would stem from gradients from the action expert. This approach has additional advantages: first, using next-token prediction makes the model learn much faster and more stably. Second, using an action expert still enables fast inference. Third, our recipe enables us to co-train a model on general vision- language data, bringing the advantages of VLAs back into our model. Our experimental evaluation provides an extensive analysis of the various modeling choices in continuous action VLAs, building on the To model architecture [7] We evaluate complex, long-horizon robotic manipulation tasks, including mobile bimanual robots, as well as open-source benchmarks such as DROID and LIBERO.
2 Related work
Multi-modal large language models. In this work, we investigate how to integrate robot actions as a new modality into pre-trained VLMs. In the literature, common technique is to embed new input modalities into discrete or continuous ("soft") tokens [32 26 9 43). Following earlier works on multi-modal cross-attention
[2]. recent work showed that for multi-moda generative modeling, e.g..

---

interleaved image, text and speech prediction, separating modalities into poddality-speciffe "expert" networks that cross-attend to each other can prevent interference and lead to higher quality predictions [8
50 While these works trained vision-language-ppee speech models, we are interested in using similar architectures for fusing a new modality, robot actions, into pre-trained VI.Ms.
Vision-language-acee
models (VLAs). Vision-languuge-actinn
models have recently been pro-
posed as promising approach for generalizable robot control [4] 62 24
51
57
35
27
3
60]. The core idea of VLAs is to fine-tune pre-trained vision- language models (VLMs) for action prediction. Such VLAs scale favorably to large-scale robot
42
38
6
21
61
datasets 12
15 49
37
23
17
40 II, and have been shown to transfer knowledge from web-scale
VLM pre- pre-ttrining to improve policy generalization 62_ [4 61 60). To enable VLM ine-tuning on action data, VLA training pipelines typically map continuous actions to a sequence of discrete action tokens, either through simple binning discretization 62 24 or more advanced, compression- based tokenization approaches [88. The VLA is then trained via standard autoregressive next-token prediction. While this strategy effective on simpler, low-frequency control tasks, it has two draw- backs: (1) the mapping from continuous to discrete action tokens can be lossy, and (2) decoding actions autoregressively leads to slow policy inference [38 31. As a result, modern autoregressive VLAs typically run at control frequencies lower than 2Hz (38], making them impractical for many high-frequency tasks.
Fast and continuous action decoding mechanisms in VLAs. To address these concerns, multiple prior works have explored alternative action decoding mechanisms for VLAs that retain continuous action outputs and fast inference. These approaches typically introduce new weights and losses during VLA action fine-tuning, often via diffusion prediction heads 35 52
K or flow-matching based that attend to features in the VLM backbone. While these approaches enable fast inference, simply adding new weights and training losses during fine-tuning comes with its own issues: such VLAs are often significantly slower to train than their autoregressive counter-parts and suffer from reduced web data transfer [38]. Liu et al.
33
"action experts" 7 g
33 introduce a hybrid autoregressive-difiun
training approach but still require slow autoregressive action decoding at inference time. Several works, including OLAnVLA-OFT (25) and To.5 [22], employ a two-stage procedure, where the model is first trained with autoregressive discretization, and then fine-tuned to target domain with continuous outputs.
To and To.5 models. We build on the 7 and FASTA 38 VLAs. wo introduced continuous action expert, which can capture complex continuous distributions over action chunks, allows for efficient inference, and enables continuous control of dexterous tasks, such as folding laundry, However, the mo recipe by itself, as we show in our experiments, leads to degradation terms of both language following and training speed, the gradients from the action expert degrade the pre-trained VLM backbone. TT-FAST addresses this by using tokenized actions, using a DCT-based tokenizer that allows for efficient discretization of complex action chunks, but at the cost of requiring expensive autoregressive inference and degrading the ability to perform delicate and dynamic tasks, as we also illustrate in our experiments. To.5 22 first trains with only FAST tokenized actions, and then adds randomly initialized action expert in post-training for fine-tuning on mobile manipulation data (by joint-training) Our work formalizes the approach of To.5 and extends it to develop single-stage training recipe, where the VLM backbone is adapted for robotic control with discrete tokens while the action expert simultaneously trained produce continuous actions, providing the best of both worlds. We rigorously ablate different mechanisms for knowledge preservation and co-training in our experiments. We thus propose the first VLA recipe that trains quickly, retains VLM knowledge, and supports high-freqeecy control with continuous action outputs.
3 Standard vision-langunge-action (VLA) model training recipes
We describe standard recipes for building and training vision-language-actio models (VLAs). The idea of most VLAs is to adapt a vision-language model (VLM) to output robot actions a E I conditioned on image observations Iı,, the robot's proprioceptive state q E R and natural language instruction f as input, i.e. a ~ x(-[4vv q, E). The promise of VLAs is to inherit knowledge of the underlying VLM pre-trained on internet-scale data when finetuning it to robot actions.
Action representations. Robot actions a Rd are, in most cases, real-valued vectors that typically represent robot joint angles or end-effector coordinates. A common strategy is to employ so-called
3

---

action chunking [55]. i.e. to predict trajectory of robot actions 61:H relative to the current robot state. To adapt a VLM to a VLA, there are multiple choices of how to represent those action chunks.
Naïve discretization. In the simplest case, each dimension of each action in a chunk is discretized, and then each discretization bin is associated with special text token [62] This way, chunk 01:H is mapped into ! d tokens. Robot action prediction then is framed as next-token prediction problem and the model can be trained as if was a non-robot specific VLM with cross-entropy loss.
Temporal action abstractions. The disadvantage of naive discretization that for high-frequene and high-dim. systems the number of tokens to represent actions grows quickly, which greatly increases the computational cost and leads to slow training convergence. Recent work, e.g. PRISE [58], FAST [38, mitigate this effect by applying a transformation that compresses information in time. We use FAST for encoding actions, which applies a discrete cosine transform to each dimension in the action chunk, followed by quantization and byte-pair encoding [88] produce action tokens.
Diffusion and flow matching, number of recently proposed VLA models have used diffusion or flow matching 22 36 to generate continuous actions, and our own experiments follow the design of To in using a flow matching "action expert" 77, as shown in Figure [ For the flow matching time index T E [0, IJ. the input to the model is a noised version of the action chunk T.u = Ta1-l + (1 Tw w ~ (0, I), and the model is trained to predict the flow w 01:H- At #1 inference time, this flow field is integrated to denoise to the final action chunk.
State representations. We consider three different representations for the robot's proprioceptive state, namely to represent it as text ("text state") after discretization, to use special tokens ("special token state") also with discretization, and by directly mapping the continuous state into the backbone with learned projection ("continuous state"). We refer to Sec. C for more details and discussion.
VLA architectures, training, & mixture of experts. Most VLAs are built from multimodal transformer, usually initialized with pre-trained VLM weights. Here, we describe a general form of transformer- based VLA architectures. Our model
T Tw q. E) (V (1 = (v So
((0()))))) (0(0))
(1)
maps sequence of n multimodal imput tokens v, to probabilities over sequence of n. multimodal out- put tokens y. For VLAs, typically y = y corresponds to action targets. Previous work has considered training one model jointly for action prediction and VLM tasks (for which y y is a tokenized text As indicated by its modality type p: -> (image, word, action, state, ............ each token can be a text token (xf E N), an image patch (x E R2xpx3), or a continuous input (z, E R such as robot states or actions. The tokens are embedded with different encoders Φ) : T, - Rdd, where Tj is the space of all multimodal tokens of type j. and de the embedding dimension of the model. Image patches are encoded with a trasforransfoormer, text tokens with an embedding matrix, and continuous inputs via affine projection. The attention mask A((((i) E indicates which tokens can attend to each other.
output) [4] 62)
{ {-o0, 0
A transformer 48 is a function f Rndd, + gaxd, that maps n input embeddings to n output embeddings. It built by stacking multiple blocks that themselves are composed of an attention layer, a feedforward layer, and normalization layers. Let X = Zinn E Raxds_ The attention E(X)Wy, Wy E Rdd.dd, where layer in a standard transformer is computed as attn(X) and Q(), K(), V(-) are the so-called E(X)= = P(X)V(X). P(X) soflmax(((X)) query, key, and value projections, e.g. Q(X) = XQm. Qm E Rd. Xd, with dy being the dimension of the projection. Compared to a standard transformer, our model processes different tokens with separate weights, as proposed in [28). As To [7]. we initialize the VLM from PaliGemma 4 and use a smaller set of weights for action tokens which significantly reduces the inference time when generating actions. The backbone and action tokens have their own query, key and value projections, but the dimensions dy. dg, d, of those projections are the same such that experts can interact with each other.
Most VLAs are trained on large robot behavior cloning datasets. For autoregressive architectures, the standard training procedure is to minimize the negative log- likelihodood of target tokens
LAR-VLA (0) = [(a log pe (21:n (r..))) = E(ayy) Σ Mj log pe (V/+1 111.), (2)
where M is a loss mask indicating which tokens should be predicted and D is a dataset (and we typically assume = y). In cases where flow-mate how-matching is used for action prediction the loss is

---

(FLOW-VLA()) EDD
[ww- 01:H f [a]))))))
(3)
4 Problems with standard VLA recipes
In Fig. we visualize problems with current recipes for training VLAs.
Autoregressive VLAs are slow. Autoregressive VLAs cast the problem of predicting real- real-valued actions as a discrete next-token prediction problem, which both limits the resolution of values the model can represent and results in slow, sequential inference. The inference time of F--FAST for predicting a 1-second action chunk is =750 ms on an RTX4090 GPU [38] which, as we show in the experiments, can lead dynamics mismatches and slow overall trajectories.
Robotic specific architectures and modality adapters don't benefit as much from VLM pre- training. Architectures like To 7 or GROOT [6] contain robotics specific modules that enable faster inference. For example, the action expert in the Ty architecture has fewer parameters than the VLM backbone, and hence Ty achieve a control frequency of 10 Hz, which is much faster than autoregressive VLAs (1.3 Hz). While parts of these models are initialized from pre-traie vre-traneed VLMs (e.g. the vision encoder or language model backbone), the robotics robotics-specific modules are initialized from scratch. We show that naive training with such a randomly initialized action expert harms the models' ability to follow language commands (presumably due to gradient interference).
VLM pretraining does not have suf- ficient representations for robotics- freezing doesn't work. Intuitively, the easiest way of maintaining the knowledge from VLM pretraining, and thus avoiding the problem men- tioned above, would be to freeze the pre-trained weights and only train the newly added, robotics-specific weights. However, current VLMs not pre-trained with robotics data. As result, their representations, when frozen, are insufficient for training highly performant policies, show in our experiments, cf. Fig. 4a and Fig. g (0% performance).
Oure
Wronn abjeeR
Tirm:
T1tm: 77k
langunge
Fast Trasning Fast Inference
Figure 2: Problems with standard VLA recipes. The robot instructed to bus the speon into the hin. To 7 (left) ignores the command and grasps piece of trash instead. T--FAST [38] (mid- dle) will eventually succeed but its inference time is very slow. Our recipe (right) solves the task, has fast inference, and the model converges very quickly to good performance (cf. Fig. (b).
Improving VLAs with co-training, joint-raiinnn & knowledge insulation
We consider number of measures in order overcome the limitations of prior VLA approaches outlined in Sec. 4 In particular, we propose:
1. to train the model on both autoregressive and flow- wow-maaching action prediction jointly at the same time (joint-training). The model uses the (smaller) action expert to produce continuous actions for fast inference at test time. The autoregressive objective is only used at training time as a representation learning objective, which enables the model to train much faster.
2. to co-train the model on non-action datasets such as general vision- lsngualangguag data, and robot planning data (VLM data co-traine no-training). Training on these data sources ensure that the model loses less of its knowledge when adapting to a VLA.
3. to stop the gradient flow between the action expert and the backbone weights. This way, when adapting the pre-trained VLM VLA, the newly initialized weights of the action expert don't t interfere with the pre-trained weights.
5.1 Co-training & representation learning with joint discrete/continous action prediction
To enable effective co-training with VLM data, enhance knowledge transfer from language to policies, and allow for fast training, we consider combining autoregressive language and discrete action predictions as well as fow-mmtchhig modeling of continuous actions all in one model. particular, we propose to learn a model from which we can sample both, real-valued action chunks @1.H. a, E gdd and text é, i.c. the output space of our model is y (a1.H, yf where d1: continuous
5

---

Figure 3: Evaluation setups. The left three tasks are evaluated completely unseen environments.
actions and yf denotes both language tokens as well as discretized action tokens. We use the FAST 38 tokenizer to convert continuous actions to discrete tokens. We can then jointly sample actions and text from our model, (o. 2) ~ T(., 1 q. E). and train the model with a combination of token prediction (ef. LAR VIA in and flow matching losses (ef. LLL in (3), simultaneously, i.e.
LCO.VIA (0) = RD T.
Mý log log po po (ff+1/11g)))) + a || 61: (0 T.0 "
(4)
where a a loss multiplier, trading off action prediction via flow-matching with the standard language modeling loss. M is language loss mask (indicating locations in the token stream which the language loss should be applied) and M is an action mask indicator specifying whether or not actions should be predicted for the given example. This loss construction allows us to flexibly co- co-training with data from different modalities. In particular, we combine VLM data (which has only images and text annotations) with action-only data (where the task is action prediction conditioned on images and text) as well as combined language and action prediction tasks (where we take action only data and additionally annotate with a language description of what the rohot should do next) 15 As we will see, mixing data of different modalities in this way enhances knowledge transfer in the resulting VLA. É contains both text (language) tokens and FAST tokenized action tokens. Crucially, we set the attention mask A such that no discrete FAST action token attend to continuous action tokens and vere-versa. We observe in our experiments that this joint training objective lets us combine the best of both worlds: we obtain fast convergence during training from using FAST action tokens learn good representations, and still obtain fast inference of continuous actions via a few steps of flow-integration.
5.2 Knowledge insulation gradient flow
Gradients from the action expert that is trained with flow matching can unfavorably influence the training dynamics of the image encoder and language model backbone; especially when adding new, randomly initialized, action expert to a pre-trained backbone. Therefore, we propose to stop the gradient flow from the action expert to the pre-trained weights the model. This is a sensible restriction if and only if the backbone is additionally trained to predict actions directly as part of its language outputs. Since we propose to train the model on discrete actions jointly, we can ensure that the combined activations of the transformer layers contain enough information to infer the actions. The pre-trained model backbone and action expert only interact via the attention layers. To stop the gradient flow from the action expert into the backbone, we need to modify the attention layers as follows. For the single head attention case, we can write the attention operation
PPss
Pao where X are the inputs to the attention layer, Q, K are the attention query and key projections, respectively. A is the attention mask as described above, and softmax the row-wise softmax. The result are attention probabilities over token features which decompose into probabilities where features from the VLM backbone attend to features from the backbone Pos. probabilities for action expert features attending to backbone features Pab and probabilties for action expert features attending other action expert features P.n. Given this we can restrict information flow as desired by implementing the softmax computation as
as P softmax (oftmax(((())((A)) (QX)) +4)
Pab
Pos 0 softmax Pab Pao =oftmax
(0o0 (X.))))) (OO
(x.( + +A) )
(5)
where sg denotes the doppgraaiient operator that restricts gradient-flow through this part of the computation. X; corresponds to all I; processed with the backbone weights, X, to the tokens processed with the action expert weights. The value embeddings are then computed by
Ebb Ea
PX.V)(X)) (Psss((V((X.)) + Pa P..))((.)))
E
(6)

---

and the final attention is (X) = PE. One additional advantage of this design is that we can simply set since now the diffusion loss term applies an independent set of weights.
Experiments
We evaluate method on dexterous, Long-horizon, manipulation tasks in the real world encom- passing multiple different robot embodiments (Figure 3 The tasks include cleaning a table ("table bussing"); folding shirts (referee as "shirt-folding" with bimanual, static robot; putting house- hold items in drawers with a single, static robot arm ("items in drawer"); and multiple tasks involving bimanual mobile manipulator. For the latter two, we exclusively evaluate the model in held-out scenes where the model has not seen any data. We further show results on the LIBERO simulation benchmark BT. as well as on DROID 23 in the real world. We train models both single robot embodiments as well as generalist models that are trained on a large mixture of data from many different robots on a large number of tasks, including non-action prediction tasks such as image captioning, bounding box prediction, and robot planning. We refer to Sec.
S [) details on tasks, datasets, and model training. Our experimental evaluation focuses on the following questions:
1. Performance. How does method compare to strong baseline VLAs xn M. -O-TAST 38
HybridVIA 33 OLenVL.A-OFT 25 in terms of absolute task performance?
2. Knowledge insulation. What is the effect of stopping the gradient flow?
3. Language following. A common limitation of many robot policies is that they pay much more attention to images than the language input [25]. Which modeling choices influence how well the model pays attention to language inputs, and thus the task at hand?
4. Convergence speed. How fast does our model train in terms of training steps?
5. Generalization. Our architecture enables us to train the model not only on robot action data, but also other data sources such VQA, image captioning, bounding box prediction. Can we transfer knowledge from these sources into generating actions with the action expert?
6. State representations. How do different robot state representations influence the model?
We consider the following baselines and ablations which we re-train on our data mixture:
1. <T 7 uses action expert, continuous actions, and is trained robot data only.
2. -F-FAST 38 an autoregressive VLA with token compression, only trained on robot data.
3. OpenVLA-OFT 25 modifies a standard autoregressive VLA to use parallel decoding with bidirectional attention. We adopt this approach herein, but do not use FiLM and keep text state.
4. Transfusion [59] denoises continuous inputs in the same transformer backbone. The original transfusion work applied their method to image generation. Here we adapt it to robot actions.
5. HybridVLA 3 trains VLAA with transfusion and naïve autoregressive tokenization simulta- neously. The autoregressive tokens can attend the diffusion inputs. We slightly modify this architecture to also use action expert for continuous tokens.
6. joint-traaiinng is the same our model but without the stop-gradient.
7. joint-training w/o VLM data. This ablation removes both the stop-gradient and co- training on VLM data from our proposed method, which can also be considered variant of HybridVLA where we train on both action representations simultaenously, but, compared to HybridVLA, the autoregressive tokens cannot attend to the flow-matching inputs.
8. Naive tokenization representation learning objective compared to FAST (see Sec. 3
Task performance & comparison to baselines. Our method consistently achieves the highest performance in the real world evaluations. For the "items in drawer" task, which requires both accurate language following (to pick the right object) and precise manipulation (to open a kitchen drawer), all baselines perform significantly worse than our proposed approach (Fig. 4a with common failure mode of being unable to open the drawer. Note that this task is evaluated in held-out environment. In particular, the joint-training baseline (no stop gradient) has problems following language, similar issues occur with Ty- AU-FAST moves slowly and fails to open the drawer with precision many cases. HybridVLA 33 is the baseline that is methodologically most similar to our approach, since it also jointly trains on both discretized and continuous actions, but allows autoregressive tokens to attend to continuous actions. This seems to hurt performance on this task significantly. Setting the attention mask propose leads to much better performance. A
7

---

Average Task Completion
100%
106%
p-8.040
80%
p-o 00.000
p-8.813
Language Rate
88%
peg een
68%
60%
40%
pea. 881
48%
20%
20%
aurg
joint-
training no
ny FAST
Hybrid
Trans-
fusion
Frozen Backbone
ours
joint-
training no
no -A-FAST
Hybrid VLA
Trans-
fusion
Frazen
Backbone
VLA
(a) Performance
(b) Language following
Figure 4: Comparison to baselines for the "items in drawer" task. Our method outperforms all other baselines both in terms of performance and the ability of the model to follow language instructions. Allowing gradients from the action expert to the backbone (joint-trinnie maintains good language following, its performance worse than our method. Neither HybridVLA freezing the backbone viable for this task.
or To) harms language following. While TO-FAST
Trans fosion state OpenVVAA-FF
text
tokens
curs stride notve cokons
joint-trsining
Ry-FAST
Average Task Completion and Language Rate
100%
1000
88%
800
Time 1
68%
600
48%
400
28%
200
Language Following
Time to Completion
Performance
Figure 5: Comparison of multiple models/architcctures "table bussing" task with specialist models trained single robot embodiment. Our model has the highest performance, low inference time, and follows language instructions well. Fo-FAST also follows language well and has good performance, of time (wall clock) to solve the task due to slow inference. To struggles with following language instructions. Open VLA-OFT follows language well and has low inference time, but has the lowest overall performance.
requires twice amount
detailed ablation of modeling choices made for our method well other baselines on the "table bussing" task is depicted Fig. As before, our method performs best, here joint conntttrrining also performs well. FAST is slow, requiring twice the amount of time to solve the task. Transfusi performs well but is slower than our method. Using parallel decoding (OpenVLA-OFT) also generally performs worse. Freezing the backbone is not viable option for knowledge insulation, since the representations in the pre-trained model are not sufficient for robotics, leading to low performance ef. Fig. 44 and shirt-folding, Fig. 8 where To also struggles due to being trained on large single embodiment data-mix. Again in this setting freezing the backbone or parallel decoding are not good strategies. We also evaluate our generalist set of tabletop manipulation tasks as in 338 Our method received a score of 0.55 + 0.09, mp received 0.49 + 0.09, and FASA achieved 0.45 + 0.09. We provide open-source model weights for DROID trained with our method at
the open source benchmark DROID 23 for the same
Finally, our approach achieves shown in Tab. This model has been finetuned on LIBERO from the generalist stop- cop-graddent + VLM data co-trained model since the generalist model did not have LIBERO data in its original training mixture.
state-of the-art in LIBERO-90 and LIBERO-S -SaRtal [3] as
Generalist VLA evaluation. While the previous results considered VLAs trained with data from the target embodiment only (though with more tasks than we evaluated for) we next shift to assessing how well our recipe works when training jointly on all data we have available for training. Fig. 6aa shows that for the "table bussing" task our recipe achieves comparable performance to the embod- iment specific results from above. In comparison joint-training degrades in task completion.
8

---

VLM data
w/ VEN data
Joint-trainkng
aura
FAST
ng
ours
Joint-trosning
1.0
VLM dota
Average Teak Completion
100%
8.8
80%
(ve 100
0.6
60%
0.4
40%
20%
0.2
0%
Language
8.0
20
48
108
158
//
1286
Performance Following
Treining Steps (K)
(a) Different training strategics
(b) Performance over number of training steps
Figure 6: Results on "table bussing" task with generalist model trained on many embodiments. Our model follows language well, and trains as quickly as To- -..... In comparison, my trains significantly slower, requiring 7.5 times as many training steps to reach a similar performance.
DUES #f VER deta trant-treiinng
QUT's W/O VLM Oata J5h1t training
1 FAST
62%
765)
(p-0.783)
189%
52%
(p-4. 0800
68%
(peé (M2)
40%
(ped. 20w.20011000000 001)
205
frowe tarkhons NFT
training
toke1a
Larguage Followinng
OLstr Abu ELon Per for marcz
Langurge Folloning
200 Per forannence
Figure 8: Performance "shirt folding".
Figure 7: Generalization to novel objects (mobile manipulator).
We also show that removing VLM data (e.g. ours w/o VLM data) leads to slightly worse task completion percentage. Interestingly when looking the rate at which the policy follows the human provided language commands for cleaning the table, removing VLM data has the biggest effect on joint- ttrining.. We hypothesize that this data especially needed to avoid catastrophic interfer- ence with pre-trained representations in this case. Finally, we evaluate our method on four mobile manipulation tasks (e.g.. placing dishes in a sink, see Sec. A.2.2 for details). Results are shown in Fig. 9 where similar trends emerge and our method trained with VLM data performs best. Notably To performs worse when evaluated after the same number of training steps; we elucidate why in Fig. 6b where we can see that training with now-matching loss only (Tp) requires many more steps to converge whereas training with our method is as quick as training with FAST.
Language following. In any scene, a robot can typically execute many sensible actions, for example grasping different objects. Here we evaluate whether different models, when given a specific task (provided via natural language instructions), produces actions that attempt to achieve this task. This particularly important for tasks which have a long-horizon goals such as cleaning kitchen counter where models could easily overfit on provided data to solving the long-horizon task by focussing on the image inputs alone. We hypothesize that if a VLA maintains more of its VLM pre-training knowledge, it should be more likely to pay attention to the actual language input. As one can see in Fig. [bb] stopping the gradient flow from the action expert is an effective way of improving language following compared to zo and joint-training without stop-gradient and without VLM data co-training. As mentioned above already, if the model is co-trained with VLM data, as shown without -rrgdint can also achieve good language following. Further, transfusion (Fig. 5) follows language better than To with the action expert, which can be explained by the fact that it reuses the backbone weights for continuous action generation and the only newly initialized parameters are the action projections. These results strongly support the hypothesis that gradients from randomly initialized robotics specific adapters unfavorably interact with the pre-trained VLM weights. Our proposed knowledge insulation techniques of stoping gradient flow, and/or co-training with VLM data are able to achieve better language following.
in Fig. [6a] Fig. 5 and Fig. 7 then point-traint
Transfer from VLM data into robotics. One of the main motivations for VLAs is transferring knowledge from non-robot data sources to robot policies. We perform an experiment with a mobile manipulation robot, where we investigate semantic generalization to new objects. The robot tasked with moving objects from kitchen counter into an (already open) drawer. The objects are not seen

---

p-0 2-0.806
pro-201
p-8.156
Spatial Object Goal 10 (Long)
90
70%
Raku
86.0
90.0
sors
Open'V LL-OFF
MoDE
97.6
98.4
97.9
94.5
94.0
95.0
1946 (01)
96.8
98.8
95.8
81.2
895
96.8
Ours (Trom scritch))
96.6
97.2
94.6
84.8
92.7
Ours (Trom generalist model)
98,0
97.8
95.6
85.8
96.0
outs
0079
1ortrr- jutort-trats trath.
Jeunt-rrair cratn.
Jount
Table 1: Success rates (%) on the LIBERO [31] benehmark. Our method achieves a state-of-the-art in LIBERO-90 and LIBERO-Spatial but is worse LIBERO-10.
YLM
Figure 9: Average performance 4 mobile manipulation tasks in unseen environments.
during training. As one can see in Fig. 7 under "OOD Follow Rate", co-training on VLM data is particularly important for this generalization. Note for the OOD tasks we include color references which makes the tasks easier for the baselines.
Investigation of other modeling choices. Our main recipe uses FAST 38 to represent discrete actions as a representation learning objective for the model backbone. One motivation for FAST is that it provides better learning signal compared to naive tokenization, and can help with faster inference (due fewer tokens). Since here we use the discrete action tokens only during training time, one may wonder whether simpler, naive tokenization is sufficient for learning good representations. To investigate, we exchange FAST with naive tokenization during training, but keep all other choices the same. The resulting model is still better than training with continuous actions alone, but worse than using FAST for representation learning (Fig. 5 (cf. naive tokens in the figure). Sub-sampling tokens (via stride of 5), in this case is better than dense naive tokenization. In the Appendix (Fig. [0] we compare different choices for the robot's proprioceptive state, cf. Sec. C on the table bussing task and find that our method works both with text and continuous state.
7 Discussion & Limitations
We analyze the performance, generalization, and language following capabilities of continuous- action VLAs that fine-tune VLMs output continuous actions, show that such models suffer from a significant loss of pre-trained knowledge, and propose a method that can greatly mitigate this degradation by shielding the pre- pre-trained VLM backbone during VLA training. The core idea in approach is to use discretized actions to provide a learning signal to fine-tune VLM representations, while simultaneously training continuous (flow matching) action expert withou propagating its gradient into the VLM. The VLM backbone is thus not damaged by backpropagation from the action expert but still receives learning signal (from discrete actions) that adapts its representations to the robot control task. Experiments across numerous real-world and simulated tasks provide strong evidence for our hypothesis about the degradation of the VLM backbone with naïve training, and clear indication that our approach mitigates this challenge.
Our method provides an effective recipe for training continuous- VLAs, but does have limita- tions. Training with both continuous and discrete outputs increases computational cost by about 20% during training. However, due to the increased convergence speed, this cost is offset such that our model still trains much faster (in wall-clock time) relative to purely diffusion based VLAs such as To- Additionally, while method improves language following it is still far from perfect, likely because correlations in the training data still cause the model to sometimes ignore language instructions.
Acknowledgments We thank Chelsea Finn for initial experiments regarding language following; Ury Zhilinsky, Karan Dhabalia, Haohuan Wang. Dibya Gosh, Kyle Stachowicz, Kevin Black for train- ing infrastructure; Noah Brown, Szymon Jakubczak, Adnan Esmail, Tim Jones, Mohith Mothukuri, James Darpinian, James Tanner for help with robot infrastructure; and Anna Walling, Chelsea Finn, Karol Hausman for help with robot, data and eval operations. We are grateful to the whole team of robot operators at Physical Intelligence for their enormous contributions to running data collection and policy evaluations. Finally, we thank Claudio Guglieri, Alex Krasikov, Spike Brehm, Lachy Groom and Karol Hausman for their help with visualizations for the website.
10

---

References
[1]
HelBbo--oold-Contibbute Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang. Yuxin Jiang, Cheng Jing, Hongyang Li. Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Hujjie Wang. Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang. Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXivv2503, 20X6v22503.06669, 2025.
[2]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35: 2022.
[3] Suneel Belkhale and Dorsa Sadigh. Minivla: A better vla with smaller footprint, 2024. URL https:///ithuup cow.suufooooottttoooooooooooo
Stt///ataafrrrrrra ILw.dd-IIAAA/ppae ILL.AADDppenlaammnni
[4] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 2024.
[5] Homanga Bharadhwaj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788 IEEE, 2024.
[6] Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 2025.
[7] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. no: A assion-language-attinn flow model for general robot control. arXiv preprint 24X1422410.24164, 2024.
[8] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang. Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503. 06669, 2025.
[9] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Pata-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8800--1040, 2022.
[10] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1XXiv:5504.0032. 2015.
[11] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273662 2023.
[12] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019.
[13] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-thh-ar multimodal models. arXiv preprint arXiv:2409.17144 2024.
11

---

[14] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint N303:23333.03338,, 2023.
[15] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2 109.0099..... 13396, 2021.
[16] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-firsi International Conference on Machine Learning, 2024.
[17] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 653-660. IEEE, 2024.
[18] Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-335 1994.
[19]
Yash Goyal, Tejas Khot, Douglas Summers-Staa Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2017.
[20] Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto. Baku: An efficient transformer for multi-task policy learning. arXiv preprint wrXiv:2406.075500 2024.
[21] Huang Huang, Fangehen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, and Pieter Abbeel. Otter: A langon-llnnggggaaiiiiie model with text-aware visual feature extraction. arXiv preprint arXiv:2503.00 03734, 2025.
[22] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. To.5 vision- language-action model with open-world generalization. arXiv preprint arXiv:2504.160 16054, 2025.
[23] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi. Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson. Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony 7. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O' Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang. Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín- Martin,, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song. Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale --ld robot manipulation dataset. In Proceedings of Robotics: Science and Systems, 2024.
[24] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-languagc-actoon model. arXiv preprint arXiv:2406. S11::2400.0022466. 2024.
[25] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning laiin--anguaae-acction models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025.
12

---

[26] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2023.
[27] Qixiu Li, Yaobo Liang, Zeyu Wang. Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-actin model for synergizing cognition and action in robotic manipulation. arXiv preprint arX6v:2411.19650, 2024.
[28] Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer. Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis Wen-tau Yih, Luke Zettlemoyer, et al. scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.044 2024.
A sparse and
[29] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02777, 2022.
[30] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791 2023.
[31] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances Neural Information Processing Systems, 36, 2024.
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[33] Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, et al. Hybridvla: Collaborative diffusion and autoregression in unified vision-language-action model. arXiv preprint arXiv:2503. 10631, 2025.
[34] Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209. 14577, 2022.
[35] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- lb: a diffusion foundation model for bimanual manipulation. arXiv preprint 2rX1v:2410.07864, 2024.
[36] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2022.
[37] Open X- X-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog. Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Rafin,, Ayzaan Wahid, Ben Burgess-Limeriet Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su. Hao-Shu Fang, Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang. Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jachyung Kim, Jan Schneider, Jasmine Hsu, Jeannette Bohg. Jeffrey Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang. Joseph J. Lim, João Silvério, Junhyek Han. Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan. Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishnan Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Ott, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen,
13

---

Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martin-Martin, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song. Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yujin Tang. Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open X- X-Embodiment: Robotic learning datasets and RT-X models. abs aas/a/bs/////// 200.b///818888444444 2023.
https://arxiv,
[38] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models. Robotics: Science and Systems, 2025.
[39] Moritz Reuss. Jyothish Pari, Pulkit Agrawal, and Rudolf Lioutikov. Efficient diffusion trans- former policies with mixture of expert denoisers for multitask learning. arXiv preprint arXiv:2412.12953 2024.
[40] Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint arXiv:2311.16098. 2023.
[41] Weijia Shi, Xiaochuang Han, Chunting Zhou, and Weixin Liang. Xi victoria lin, luke zettle- moyer, and lili yu. Ilamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188 2024.
[42] Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal Ilms to generalist embodied agents: Methods and lessons. arXiv preprint arXiv:2412.08442, 2024.
[43] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/1, 2024.
[44] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Bean-Baptiste Alayrac Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Oscar Chang, Jose En- rique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, David D' Ambrosio, Sudeep Dasari, Todor Davchev, Coline Devin, Norman Di Palo, Tianli Ding, Adil Dostmo- hamed, Danny Driess, Yilun Du, Debidatta Dwibedi, Michael Elabd, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Leonard Hasenclever, Nicolas Heess, Brandon Hernaez, Alexander Herzog, R. Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan Julian, Dmitry Kalash- nikov, M. Emre Karagozler, Stefani Karp, Chase Kew, Jerad Kirkland, Sean Kirmani, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Isabel Leal, Alex X. Lee, Tsang-Wei Edward Lee, Jacky Liang, Yixin Lin, Sharath Maddineni, Anirudha Majumdar, Assaf Hurwitz Michaely, Robert Moreno, Michael Neunert, Francesco Nori, Carolina Parada, Emilio Parisotto, Peter Pastor, Acom Pooley, Kanishka Rao, Krista Reymann, Dorsa Sadigh. Stefano Saliceti, Pannag Sanketi, Pierre Sermanet, Dhruv Shah, Mohit Sharma, Kathryn Shea, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Jost Tobias Springenberg, Rachel Sterneck, Raz- van Surdulescu, Jie Tan, Jonathan Tompson, Vincent Vanhoucke, Jake Varley, Grace Vesom, Giulia Vezzani, Oriol Vinyals. Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Yuxiang Yang, Rui Yao, Sergey Yaroshenko, Wenhao Yu, Wentao Yuan, Jingwei Zhang, Tingnan Zhang, Allan Zhou, and Yuxiang Zhou. Gemini robotics: Bringing into the physical world, 2025. URL. https: / aaaxiiiv..v.... org/ abs/ hbss/253.2000
14

---

[45] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprini arXiv:2403.08295, 2024.
[46] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy, arXiv preprini urXivv2405. /2213, 2024.
[47] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems,
2024.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[49] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. BridgeData v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736. PMLR, 2023.
[50] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji. Zhuoyi Yang, Lei Zhao, Xixuan Song. Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2024. URL https 7/arxiv. wwg//rggg//..eee abt.ab/.../1111111
[51] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, and Jian Tang. Tinyvla: Towards fast, data-efficient vision-language-actionn models for robotic manipulation. arXiv preprint arXiv:2409. 12514, 2024.
[52] Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05885, 2025.
[53] Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu, Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14022-14032, 2024.
[54] Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of chain-of-thought reasoning, In Conference on Robot Learning, 2024.
[55] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[56] Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity, arXiv preprint 14X.v:2410.13126. 2024.
[57] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d lasion-language action generative world model. arXiv preprint arXiv:2403. 09631, 2024.
[58] Ruijie Zheng, Ching-An Cheng, Hal Daumé III, Furong Huang, and Andrey Kolobov. Prise: Learning temporal action abstractions as a sequence compression problem, 2024.
[59] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 11099, 2024.
[60] Zhongyi Zhou, Yichen Zhu, Junjie Wen, Chaomin Shen, and Yi Xu. Chatvla-2: Vision-language- action model with open-world embodied reasoning from pretrained knowledge. arXiv preprint 20Xiv:2505.21906, 2025.
15

---

[61] Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-action model. arXiv preprint arXiv:2502. 14420, 2025.
[62] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-actieo models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165-2183 PMLR, 2023.
16

---

100%
p=8.457
Average Task Progress
I
p=8.074 p=0.099
p=0.010
88%
60%
48%
20%
0%
ours text
ours
ours 7o cont. state
To text state
cont.
special
state
state
taken
state
Figure 10: Comparison of different state representations on "table bussing" task. Our method works well with both text and continuous state, while To works with both state representations. Therefore, the difference between To and method cannot be explained by the difference their original state representations, Using special tokens as state performs worse.
A Dataset & task details
A.1 Common public benchmarks
We evaluate method on the simulated LIBERO 31 benchmark, well as the real world DROID 23 benchmark. For these two benchmarks, we evaluate specialist models that are trained only on the respective datasets. For LIBERO, also evaluate model fine-tuned from generalist model that is trained with diverse robot embodiment data and non-robot data. The generalist model did not include LIBERO data during its original training.
We evaluate on the same set of real-world tabletop manipulation tasks as Pertsch et al. [38]. These tasks include picking and placing, wiping, and opening and closing drawers. The environment and objects are completely unseen. Each trial is scored based on task progress, and we report the average score.
DROID.
LIBERO. The LIBERO 31 simulation benchmark consists of four task suites: LIBERO-Spatial LIBERO-Object, LIBERO-Goal and LIBERO-100 (which is further split into LIBERO-90 and LIBERO-10, also known as LIBERO-Long). We evaluate LIBERO-Spatial LIBERO-Object, LIBERO-Gov and LIBERO-Long following the same setup as Pertsch et al. [38), jointly train- ing one policy on all four datasets. We additionally evaluate our method on LIBERO-90 with policy trained only on that dataset. All training datasets are prepared the same manner as Kim et al. [24] which includes re-rendering images at higher resolution of 224 x 224 px, and filtering out unsuccessful demonstrations and "no-op" actions. All methods in our comparisons use both third-person and wrist camera images as inputs.
A.2
Real-world tasks
For all our real world tasks, we evaluate 10 episodes per task per policy and report performance as outlined below. We report statistical significance according to two-sided t-test.
A.2.1 Tasks with static robots
Items in drawer. The task begins with three objects on a countertop. The static single-arm robot must open the drawer beneath the counter, place the items into the drawer, and close the drawer. One point is awarded for (1) opening the drawer, (2) putting one item into the drawer, and (3) closing the drawer. Thus the maximum number of points is 5. We evaluate a specialist model with data only from static single-arm robots. This dataset contains more than just this task. The evaluation environment for this task is unseen.
T-shirt folding. The task begins with shirt flat on the table. The static bimanual robot must fold into the usual folded form with collar facing up. The maximum number of points is 5, and it is scored
17

---

based on the squareness and amount of wrinkles of the final form. We evaluate specialist model with data only from static bimanual robots. The dataset contains many different tasks.
Table bussing. The task begins with 12 objects on the table and two receptacles, one for uten- sils/dishes and one for trash. The static single-arm robot must follow language commands to pick up the correct object and place it into the correct receptacles. One point is awarded for each correctly placed object. Thus, the maximum number of points is 12. We evaluate specialist model with data only from static single-arm robots, as well as a generalist model trained with diverse data.
A.2.2 Tasks with mobile manipulator robots
Make bed. The task begins with the bed partially unmade, with two gray pillows at the foot of the bed. The mobile bimanual robot must tidy the blanket and place the two pillows at the head of the bed. One point is awarded for (1) straightening the blanket so it covers the sheets, (2) placing one pillow at the head of the bed, (3) blanket being straightened very neatly, and (4) both pillows are placed very neatly. Thus the maximum number of points is 5. We evaluate the generalist model.
Dish in sink. The task begins with 4 dishes (e.g. plates, bowls, cutting boards, utensils) placed near a sink. The mobile bimanual robot must place all of them in the sink. One point awarded for (1) picking one item up, and (2) placing one item into the sink. Thus the maximum number of points is 8. We evaluate the generalist model.
Mobile items in drawer. The task begins with a household item on a countertop. The mobile bimanual robot must place the item into drawer beneath the counter. One point is awarded for (1) picking up the object, (2) opening the drawer, (3) putting the object into the drawer, (4) closing the drawer. Therefore, the maximum score 4 points.
Laundry in basket. The task begins with clothing lying the ground. The mobile bimanual robot has to pick up the clothing and place it in the laundry basket. One point is awarded for (1) navigating to and picking up the clothing, (2) placing the clothing into onto the laundry basket, (3)
the clothing is fully inside the basket. Therefore, the maximum score is 3 points.
A.3 Datasets for training the generalist model
The generalist model is trained on a large dataset encompassing 12 configurations of robot embodi- ments, including single-arm static manipulators (ARX, UR5, Franka), bimanual static manipulators (ARX, AgileX, Trossen, UR5), and bimanual mobile manipulators (mobile Trossen, ARX slate, Galaxea GI, Hexmove H1, Fibocom). This robot data includes a large diversity of tasks, going much beyond the evaluation tasks considered in this work, c.g.. grinding coffee beans hanging a towel on the oven handle) in diverse environments (both office-like ones and real home ones). We also include the open-source OXE dataset (37]
We also train the generalist model with a variety of general VLM tasks. The data involves image captioning (CapsFusion ( COCO [OO). visual-question -eaa-guuestionnnswerinng (Cambrian-7M) [47). PixMo 13, VQA22 [19), as well as object localization. For object localization, we further extend the standard datasets with additional web data of indoor scenes and household objects with bounding box annotations.
We refer to
22 for more details about the training tasks and mixture.
B Training details
We use the PaliGemma VLM architecture as the VLM backbone and initialize it with its pre- trained weights. The action expert is a smaller transformer that takes in a sequence of noisy actions a1s for an action horizon of 50, i.e. H = 50. The noisy action chunk is first projected to the transformer embedding dimension using a single linear layer. We use MLP to project and then applies adaptive RMSNorm to inject the timestep information to each layer of the action expert. The MLP takes in the form of swish(W2 swish(W, o(T))), where o R + R is sinusoidal
18

---

positional encoding function 48 and W1, W E R"x" The action expert outputs action tokens Fi. which are then decoded into the target vector field using final linear projection.
The dimensions of the VI.M backhone and action expert are as follows: (width=2048, depth=18, mlp 16,344 num 1om_heads=11, num_kv_ hey_head==1, head _dmm==66) for the 2B language model backbone, and the same except for [width=1024, 40p__iim=0996 for the action expert, leading to 300M parameters.
Embeddings from the VLM and action expert interact only through self-attention. full prefix mask is used images, language tokens, and text state; FAST action tokens attend to this prefix and ruto-regressivvly on previous action tokens. Embeddings from the action expert attend to the prefix and to another, but do not attend FAST action tokens avoid information leakage between the two representations of actions. In effect, information flows unidirectionally from the VIM to the action expert; VLM embedding attends to the action expert.
We follow To for sampling the fow-matching timestep т. In summary we deviate from standard uniform sampling instead use time-step sampling distribution that emphasizes low time-steps 77. given by p(T) =
U(0, 1) [29 34) methods emphasizing midrange timesteps [16]. and
Beta( -=;; 1.,, 3 1), s 0.999.
C State representations
We consider three different representations for the robot's proprioceptive state E IR..
Text state. This representation discretizes the into ni many bins, then converting the bins into numbers from 1 to Va. Those numbers are then input to the model as normal text. This has been the standard approach for previous VLAs. For tokenizers such as Gemma 45 this implies up to [logıo (nx) + 2) many tokens to represent the state as text.
Special token state. Similar to the "text state", this representation uses the discretized bins directly, associating each bin with special token from the VLM tokenizer. This requires tokens.
Continuous state. This representation inputs the real-valued vector y directly into the model by projecting it with a learned affine projection into de dimensional embedding space.
The text state approach has the advantage of it being the closest to what the backbone might have seen during pre-trane pre-training as is just a sequence numbers in natural text but it requires the most tokens. Both special token state and continuous state completely new inputs to the model with randomly initialized projections/emteedding..
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contribution.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The conclusion section contains a description of limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper does not contain formal proofs, but equations that show how stop-gradient for an action expert architecture is implemented. We state all necessary assumptions and provide enough notation detail to understand this procedure.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: While large parts of our experimental results rely on proprietary datasets / robot platforms, we also include results on the open-source benchmarks LIBERO (30 and DROID 23 which reuse previously published evaluation setups. We release model weights here https::/ gi thub. com haanm/hhsscallaa. Denl--Inelllll intelligence, pepence/openpi
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are unable to release the proprietary robot data for our real world experiments. However, one can re-implement our paper based on the informa- tion given in the paper well as openly available repositories such as https: // github.com wttp://ayssiacllnneelllleeeeeeee
com/ github. 10/ We re- lease model weights here httpps//giithbb com wwt.m//hhscaalaa. wwt.---nneeieeee//ppppnnn As such, we believe that sufficient reproducibility is given.
openpi and for data htttttttttttthhhit oitp://BRRRooottte/oooooooddeeed
Rreelong-RRoot-Learrie
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide all experimental detail in the appendix, including details about the used datasets, optimizers, hyperparameters etc.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
20

---

Justification: We provide error bars and statistical tests when presenting our results.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational resources in the appendix.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://.gggg..aaoo...ooooo/////a
Answer: [Yes]
Justification: We follow the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have a discussion of broader impacts at the end of our paper.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do release model datasets.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all benchmarks and baseline models used in this work.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not conduct any crowd-sourcing or research with human subjects.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
21

---

Justification: This work does not conduct any crowd-sourcing or research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We do make of LLMs conduct any research for paper writing. Our paper is however concerned with using LLMs VLMs as the backbones for training control policies, and we discuss this length in the paper.
22


Reviews:
[{"review_id": "onlRLJIqYC", "reviewer": "Reviewer_yt5X", "summary": "This paper investigates the integration of continuous action prediction and knowledge perservation of pretrained VLM. They demonstrates current approach significantly harms both training speed and knowledge transfer. This paper provides an extensive analysis of various design choices, their impact on performance and knowledge transfer, and propose a technique for insulating the VLM backbone during VLA training that mitigates this issue.", "questions": "1. L80-L81, \"The core idea of VLAs is to fine-tune pre-trained vision-language models (VLMs) for action prediction.\", which is very confusing.  Do you think VLA is limited to the finetuned VLM? How about the VLA models from scratch? I think it is also possible to jointly train the VLA model with both vlm dataset and the robot control data from scratch. Therefore, L80-L81 is quite confusing. Currently, the major attention of VLA lies on fine-tuning a VLM.\n2. This paper proposes to stop the gradient propagation from the continuous action expert, together with the discretized actions for gradient propagation. Did you have evaluate the training scheme without the discretized actions but with VLM supervisions? I found the text state refer to the discretized actions, and thus did not figure out which ablation is related to the experiment without discretized actions.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "pros:\n\n1) This paper investigates an interesting question: the integration of continuous action prediction and vision-language ability.\n2) This paper demonstrates a clean improvement compared to the recent state-of-the-art pi0.\n3) The robot experiments are extensive and solid.\n\ncons:\nThe paper claims the proposed approach achieves better knowledge insulting for VLA models from VLM. However, the evaluation is mainly based on manipulation performance according to experience. Meanwhile, this paper does not quantitatively demonstrate the knowledge preservation of VLM, e.g., evaluating it on the corresponding benchmarks.\n\nOverall, this paper is good and expands the knowledge boundary of vision-language-action models.", "was_revised": true, "final_justification": "The response has well addressed my concerns. I maintain my rating.", "rebuttal": "Thank you for your positive and detailed feedback!\n\n> L80-L81, \"The core idea of VLAs is to fine-tune pre-trained vision-language models (VLMs) for action prediction.\", which is very confusing. Do you think VLA is limited to the finetuned VLM? How about the VLA models from scratch? I think it is also possible to jointly train the VLA model with both vlm dataset and the robot control data from scratch. Therefore, L80-L81 is quite confusing. Currently, the major attention of VLA lies on fine-tuning a VLM.\n\nIn our understanding, the term VLA refers to a model that combines *large scale* language, vision-language, and robot action data. While it is possible to train such a model from scratch, we are not aware of this ever being done at scale comparable to VLM pretraining from scratch.\n\nWe will reword this sentence to say that “The idea of ***most*** VLAs is to fine-tune pretrained …”. \n\n> This paper proposes to stop the gradient propagation from the continuous action expert, together with the discretized actions for gradient propagation. Did you have evaluate the training scheme without the discretized actions but with VLM supervisions?\n\nIn Figure 4a and Figure 8  the “frozen backbone” experiment can be interpreted as a training scheme with (web) VLM supervisions only. Since the model has been pretrained on web VLM data, training it with VLM supervisions only (as you suggest) is very similar to just freezing the backbone. For the “items in drawer” task, this can have some success (around 30%), while for the “shirt folding” task, the success is 0, both indicating that VLM supervision alone is not sufficient.\n\n> I found the text state refer to the discretized actions, and thus did not figure out which ablation is related to the experiment without discretized actions.\n\nThe ablation that corresponds to training without discretized actions is the pi0 ablation in all figures. “Text state” refers to the representation of the robot’s proprioceptive state (joint angles) as text."}, {"review_id": "eV4os0SrdK", "reviewer": "Reviewer_oXwX", "summary": "This paper proposes a method for training generalist VLA models by co-training on large robot and VLM datasets. To address the challenge that naive co-training may hinder effective knowledge transfer from the VLM to robotics, this paper proposes a stop-gradient strategy. Specifically, it stops the gradient flow from the action expert to the backbone, allowing the backbone to be fine-tuned solely through a next-token prediction loss on both discretized actions and general language tokens, while the action expert is independently trained using flow-matching on continuous actions.", "questions": "1. Are reasoning language tokens generated at test time? \n2. The example of reasoning is missing in the demo. Can you provide some demo samples includes the reasoning generated during inference?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This paper conducts thorough experiments to demonstrate the importance of co-training in improving the language following ability of VLA. The experiments are challenging, the performance of the model is astonishing.\n2. The authors propose a novel training recipe, termed as knowledge insulation, which effectively mitigates gradient interference between the VLM backbone and the action expert in naive co-training strategies.\n\nWeaknesses:\n\n1. The training data used in the experiments is outlined in the paper, but further clarification or elaboration may be beneficial.\n\n2. The concept of knowledge insulation is intriguing, and similar phenomena have been explored in prior work. For example, ChatVLA[1] identifies that training exclusively on robot data causes the VLA to lose its multimodal capabilities entirely after fine-tuning, and it proposes a method for unifying multimodal understanding with robot control. ChatVLA-2[2] enables the VLA to preserve pretrained knowledge from the VLM while following reasoning in manipulation tasks. A discussion on how this approach relates to and differs from these works would be valuable.\n\n[1] ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model\n[2] ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and feedback!\n\n> The training data used in the experiments is outlined in the paper, but further clarification or elaboration may be beneficial.\n\nWe will expand the description of the training data for a revised version of the paper and include more detailed descriptions in the appendix.\n\n> The concept of knowledge insulation is intriguing, and similar phenomena have been explored in prior work. For example, ChatVLA[1] identifies that training exclusively on robot data causes the VLA to lose its multimodal capabilities entirely after fine-tuning, and it proposes a method for unifying multimodal understanding with robot control. ChatVLA-2[2] enables the VLA to preserve pretrained knowledge from the VLM while following reasoning in manipulation tasks. A discussion on how this approach relates to and differs from these works would be valuable.\n\nThank you for pointing us to these papers, we agree that they are relevant and will include a discussion of them in a revised version of the paper.\n\n> 1. Are reasoning language tokens generated at test time?\n> 2. The example of reasoning is missing in the demo. Can you provide some demo samples includes the reasoning generated during inference?\n\nFor the experiments with the mobile manipulators, the model gets as input a “high-level” task such as “clean the bedroom”, and then first outputs a “low-level” task in language such as “pick up the pillow”. The model then predicts an action corresponding to the “pick up the pillow” sub-task. Currently, those are two separate inference calls to the same model (one call for autoregressive generation of the subtask followed by a call to generate the actions through integrating the flow field). We will be more explicit in a revised version of the paper."}, {"review_id": "dGHoFtTzAo", "reviewer": "Reviewer_LSGX", "summary": "This paper primarily investigates how to better train VLA models that incorporate diffusion or flow matching action experts. The proposed approach involves training the VLM backbone with discrete actions, training the action expert with continuous actions, and stopping gradient flow from the action expert to the VLM backbone. The authors conduct extensive real-world experiments to validate both the performance of their model and the effectiveness of the proposed training scheme.", "questions": "Questions\n\n1. If the VLM backbone is first pretrained using discrete action data (with VLM data), and then frozen while only finetuning the action expert, would this yield similar results to jointly training the VLM backbone and action expert with gradient stopping?\n\n2. In Figure 7, why do Pi0 and joint training without VLM data perform better in the OOD setting than in the ID setting? Additionally, the performance gap between ID and OOD for the proposed method is relatively small. Does this suggest that the OOD tasks are too simple or not sufficiently distinct from the ID tasks?\n\n3. Section A.3 of the supplementary material lists several VLM tasks used during training. To what extent does the object localization task contribute to the model’s overall performance?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. The paper clearly states its motivation and is well-written and easy to follow.\n\n2. It proposes an effective training method for VLA models that use diffusion or flow matching action experts.\n\n3. The experiments are thorough and well-designed, providing strong empirical evidence to support the paper’s claims and demonstrating the advantages of the proposed model.\n\nWeaknesses\n\n1. It appears that part of the robot action data used to train the model is not open-sourced, which may hinder reproducibility.", "was_revised": true, "final_justification": "The paper is technically solid, and the authors have addressed my concerns. Therefore I am keeping my score", "rebuttal": "Thank you for your positive review and questions!\n\n> It appears that part of the robot action data used to train the model is not open-sourced, which may hinder reproducibility.\n\nWe present experiments on LIBERO and DROID, datasets and setups which are fully open source. We will open-source a model checkpoint that has been trained on the DROID dataset. Further, while our whole dataset is very large and hard to open-source, we will also open-source a model checkpoint that has been trained on our whole data. The release will be accompanied by fine-tuning scripts for further model training. We hope this should address your concerns.\n\n> If the VLM backbone is first pretrained using discrete action data (with VLM data), and then frozen while only finetuning the action expert, would this yield similar results to jointly training the VLM backbone and action expert with gradient stopping?\n\nThis is an excellent point, thanks for raising it. In principle, it is possible to do what you suggest and we found that it yields good results. However, the total training time in this case, both in terms of wall-clock and total number of gradient steps increases significantly. For example, we tried first training only the VLM backbone and then training the action expert on the full data mixture, but even after 160% of the training time (i.e. allowing for another 60% of total steps to train only the action expert) the models are not yet fully comparable (even though we would assume convergence to yield a policy that is as good eventually). In contrast, the computational overhead of adding the action expert with stop-gradient is minimal and does not lead to a significant reduction in the train steps/s, hence it is significantly more efficient to have the action expert in the model from the beginning.\n\n> In Figure 7, why do Pi0 and joint training without VLM data perform better in the OOD setting than in the ID setting? Additionally, the performance gap between ID and OOD for the proposed method is relatively small. Does this suggest that the OOD tasks are too simple or not sufficiently distinct from the ID tasks?\n\nThe OOD tasks contain objects not seen in the robot training data and were designed prior to knowing the performance of the policies on these objects. In the OOD setting, we prompt the model with both the object category and object color, while in the ID setting with the object category only. Adding the color reference indeed seems to make the task easier for “Pi0” and “joint training without VLM data”. It turns out that for our method, removing the color reference leads to the same performance. We will include details in the appendix of each set of objects and a picture of the two tasks for clarity.\n\n> Section A.3 of the supplementary material lists several VLM tasks used during training. To what extent does the object localization task contribute to the model’s overall performance?\n\nPerforming this ablation cleanly would involve a full pretraining run, which is very resource intense. Some of the web datasets such as Cambrian also include localization as part of their diverse mixture. Preliminary experiments have qualitatively shown that object localization helps with robustness of the mobile policies in unseen environments."}]

Author rebuttal:

Meta-review: The paper investigates the integration of VLM into robotics, noting that to adapt the VLM architecture to a continuous robotic control, often ad-hoc adapters are trained from scratch, and fine-tuning the VLM backbone results in performance degradation. The paper finds that the gradient coming from continuous actions does not adapt well to the VLM's discrete nature, and proposes to train a continuous flow matching expert while stopping the gradient flow between it and the VLM. Such a technique is presented with the name of "knowledge insulation", and is tested on both simulated and real-world tasks.

Reviewers have anonymous positive opinions: the research question has been found relevant, experiments satisfactory, with a clear improvement on the previous approaches, and overall the paper is well motivated and written. The main weaknesses emerged during the reviewing process appear to be minor clarifications (e.g., discussion of previous works) and the fact that the dataset will not be entirely open-sourced. During the discussion phase, all the reviewers showed satisfaction with the provided answers, and the general opinion remained unanimously in favour of acceptance.

After careful examination, the AC does not find reasons to overturn the reviewers' decision. The paper can be considered for a spotlight, since the addressed problem can be of interest for a large audience, and the simplicity of the idea could be easily picked up by future works.
Decision: Accept (spotlight)