4D-VLA:  Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
Authors: Jiahui Zhang
Yurui Chen
Yueming Xu
Ze Huang
Yanpeng Zhou
Yu-Jie Yuan
Xinyue Cai
Guowei Huang
Xingyue Quan
Hang Xu
Li Zhang
Venue: NeurIPS 2025 poster ()
Keywords: VLA
LLM
Primary area: deep_learning
TL;DR: This paper introduces 4D-VLA, a pretraining framework that enhances spatial and temporal reasoning in robotics by aligning coordinate systems with RGB-D sequences and efficient memory sampling.

Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset’s action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution—an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce Memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance.  In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA.To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

4D-VLA: Spatiotemporal Vision-Language-Actiinn Pretraining with Cross-Scene Calibration
Jiahui Zhang!
Yurui Chen
Yueming Xu
Ze Huang'
Yanpeng Zhou?
Yu-Jie Yuan²
Xinyue Cai
Guowei Huang
Xingyue Quan
Hang Xu²
Li Zhang'
School of Data Science, Fudan University
Huawei Noah's Ark Lab
OpenVLA pretrain
Issue with insufficient input
4D-VLA pretrain
Enpui
Cooedinate
Frame ambiguity
Unique frame via coonds
Coordinate embedddings
P(4.
"Move mug near
"Movee the stascberry theennng
Bobut base unseen
High Ver(A))
Lowe Ver(h))
Input domaln
Input
Historical images
VLA Po
"Open and does cabrner
:[X |4,00 Aptiny
Hf 4400)) Aghaer
FEXt 1,00 40
Action: F ((ippc)
Sunilee RCS diff actions
Nonsmoott A/O)
Locally smooth A.O)
LIBERO
Real-Worid
Meal-Wordd MV
76.
85
40
Figure 1: Top: Our pretraining design philosophy highlights that prior methods often lack key cues in their input for accurate action inference. This leads to target action distributions Ac(-) exhibiting high variance non-smoothness, which negatively impacts pretraining performance. A rough analysis shows that in the DROID dataset, 67% of the samples have the robot's base occluded, causing coordinate system chaos. Bottom: We verify our method in both simulated and real-world robotic settings and report the performance for the OLe baseline and our 4D-VLA approach.
Abstract
Leveraging diverse robotic data for pretraining remains critical challenge. Exist- ing methods typically model the dataset's 's action distribution using simple observa- tions inputs. However, these inputs are often incomplete, resulting in dispersed conditional action distribution issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency, To address this, we propose 4D-VLA, novel approach that effectively integrates 4D information into the input mitigate these sources of chaos. Our model introduces
"Equal contribution. 'Corresponding author (lizhangfd0fudan. edu. cn).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves significant increase in success rate over Open [1]. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi- view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.
1
Introduction
The emergence of pretrained vision-language models has established an effective framework for aligning human language with visual data, advancing embodied intelligence. Meanwhile, the open- sourcing of diverse robotic datasets [2, 3, 4, 5] has enabled data-driven robot control. However, efficiently extracting useful information from these datasets remains a challenge for improving generalization across diverse scenarios.
real-world action distribution can be interpreted as a response function conditioned on observations or imput, denoted as A (input). The objective of pretraining learn model Fş (input) that approx- imates this funetion using large-scale data Adata (input). However, existing pretraining paradigms often suffer from incomplete or under-informative input, lacking critical contextual cues required for reliable action reasoning. Consequently, the resulting target distribution Ax(-) may exhibit undesirable sharacteristics--uchh as lack of smoothness, high variance, or multimodality- wwich hinder the model's ability to learn robust and generalizable behaviors.
Previous approaches, such as Open VLA [1], use only a single RGB image and textual instruction as input. This limited input setting leads to two prominent types of chaos. The first is coordinate system chaos, which arises when actions are defined in the robot's coordinate frame, yet the visual input lacks sufficient spatial context. For instance, if the image does not fully capture the robot's body, it becomes challenging to infer the robot's ' exact position and orientation. The second is state chaos, which arises in scenarios where single frame lacks the necessary temporal contextual cues to resolve action ambiguity. This includes symmetric trajectories where is difficult to infer the direction of motion-as well cases where visually similar observations correspond entirely different actions. These ambiguities hinder the effectiveness of pretraining, as illustrated in Fig. 1. In contrast, HPT [6] extends the input by incorporating some dataset-specific parameters, which helps address the issue of inconsistent coordinate systems across different datasets. However, this approach lacks scalability and increases the complexity of training.
To address this, we propose 4D-YLA, a framework that integrates 4D spatiotemporal information to resolve such ambiguities. By combining spatial coordinate embeddings with 3D-aware module, it generates spatial vision tokens that align the robot's coordinate system with the scene, enhancing 3D perception. This also enables efficient encoding of multiple historical frames, improving temporal reasoning. Additionally, we introduce memory bank sampling, a frame sampling strategy that selects key frames based on historical similarity, boosting model efficiency.
To further investigate the spatial understanding and generalization of VLA models, we introduce MV-Bench, a multi-view dataset that evaluates performance across diverse viewpoints. Our approach enables robust pretraining, improving generalization to novel scenarios while outperforming baselines.
Our contributions are: (i) We propose 4D- VLA, an efficient VLA model that integrates spatial module with vision features generate 3D-aware spatial vision tokens, effectively mitigating coordinate system and state chaos, thereby significantly enhancing pretraining efficiency. Additionally, we introduce memory bank sampling, simple but effective method for historical information sampling. (ii) Our model has been validated both in the simulated and real-world environment, demonstrating its superiority. (iii) We develop a multi-view simulation dataset MV-Bench to evaluate spatial understanding and generalization, which our model achieves outstanding performance.

---

2
Related works
Vision-language models Recent advancements in vision-language models (VLMs) have significantly enhanced the integration of vision and language understanding across diverse domains. Models like Flamingo [7], LLAVA [8] and BLIP-2 [9] focus on aligning text and image feature spaces to facilitate effective image understanding. More recently there has been growing interest in expanding language models to support multi-image inputs, enabling them to handle more complex tasks and real-world scenarios. For instance, [10, 11, 12, 13] utilize multi-image sequences to capture temporal and action-related dynamics across frames. providing robust video comprehension. In addition, some recent models have begun incorporating 3D information to bolster spatial reasoning, as seen in 3D- LLM [14] and Scene-LLM [15]. These models leverage 3D inputs to enhance understanding of spatial relationships within scene. However, none of these models explicitly integrate 4D spatiotemporal information to fully capture both spatial and temporal dynamics within the architecture.
Vision-language-action models The vision-language-action (VLA) model represents significant advancement in vision- lasoon-anguuae research, enabling more complex and interactive tasks aimed at facilitating real-world environment interactions. [16, 17, 18] complete tasks by directly predicting trajectories, [19, 20, 21, 22, 23, 24] predict the robot's ' current actions to enable closed-loop control, while [25, 26] enhance action prediction by training a world model to forecast future states. Some works [27, 28] also drives VLA policies using simple concatenation of historical observations.
Recent works leverage diverse robotic datasets from various scenes and robot types to pretrain models for better generalization in novel environments. [29, 30, 3, 1] use the single image as input and pretrained large-scale datasets, while Octo [31] incorporates historical context and uses diffusion head to predict the next n actions. HPT [6] addresses the heterogeneity among different datasets by introducing dataset-specific parameters during pretraining improve training efficiency. However, these methods overlook that the inefficiency in prior pretraining arises from insufficient input context, resulting in high variance of the conditioned action distribution A.(-) and ultimately hindering pretraining effectiveness. Our approach tackles this issue by introducing 4D information to mitigate coordinate system chaos and state chaos. This enables the model to learn meaningful action distributions from diverse datasets, thereby enhancing performance.
3
Method
This section provides comprehensive overview of our proposed 4D-VLA As shown in Fig. 2, our model processes sequential RGB-D images as input, converting them into corresponding spatial vision tokens. These tokens, together with task-specific text tokens, serve as feature inputs for the subsequent Transformer. After decoding through the VLM Transformer and action head, it ultimately generates the action output.
3.1 Preliminary
Problem definition The vision-language action (VLA) model takes language instruction input and aims to control a robot to accomplish the specified task. Specifically, VLA with low-level control policy refers to class of models that use the current observations as input to predict action for the robot in its present state, enabling end-to-end, closed-loop control. The action is defined by three components: Ar 3R3 AO R3, and g E [0, 1]. representing the control translation, rotation offset, and the open-close state of the robot's 'ss end-effector, respectively.
Vision-language model backbone We leverage pretrained large vision-language model (VLM) as the backbone, specifically Intern [12], which consists of text tokenizer T, a vision encoder E, and a Transformer decoder D. The vision encoder processes visual observations, which are subsequently compressed by an MLP projector P to generate vision embeddings, while text inputs are tokenized and embedded to form structured textual tokens. These multimodal tokens are then fed into the decoder D for next-token prediction. This backbone provides robust foundation for aligning visual information with the shared semantic space, enabling effective robot action generation.
3

---

sequential inpuls
Sampled inputs
Vision encoder
Memory bank sampling
Spatial PE
Siiina
Fame indence
Ioars
Tokun from Borna
00000
Seathv
Temporal PE
000-000
Predicted actions
LIM Transformer
Ax Grip
Action hoad
Figure 2: Our VLA pipeline. Our memory bank sampling method selects informative frames from sequential RGB-D imputs. vision encoder with 3D coordinate embeddings generates spatial-award aware tokens, which are fused into 4D spatiotemporal representation. Combined with text tokens, these are processed by the L.M to decode actions via an action head.
3.2 Spatial-aware visual tokens
reasonable action prediction requires awareness of both semantic perception and spatial perception of the scene. A fundamental aspect of spatial perception is that the robot must accurately determine its relative position within the scene, effectively resolving coordinate system chaos. To address this issue while enhancing spatial perception without compromising semantic awareness, we fuse the coordinate information derived from depth with vision tokens, resulting in spatial vision tokens.
In our method, the input image I E R3xhx is first encoded by & into a feature map with a downsampling rate of c, yielding f, = E(T x where & denotes the feature dimension. Next, we obtain downsampled depth map D A which assigns depth values to each corresponding feature volume. Using the camera's 's extrinsic [R[T] and intrinsics K, we back-project the depth value into 3D coordinates P.. E R3x x within the world (or robot) coordinate system:
Pu Pu( = R ((u,,))
0
+ +T.
We apply a learnable positional embedding Es encode the 3D coordinates and integrate it with the original visual feature map via element-wise addition, forming spatial vision features that enhance spatial representation. These features are then processed by the MLP projector P within Intern VL [12], generating spatial vision tokens ST (I + {P(Pw))
3.3 4D representation with multi-frame encoding
As spatial vision tokens are designed represent information from single frame, and the LLM is fine-tuned with video data, we can naturally extend our encoding process to incorporate multi- frame information using the same approach. This allows us to integrate coherent 4D spatiotemporal representations seamlessly. naive approach is to use the spatial vision tokens from the current frame along with those from uniformly sampled historical frames as input {eeTT i 0, 1,2,..., 1}. where denotes the total number of temporal frames used, and represents the index of the current frame in the sequence. These tokens, together with the corresponding text tokens from task instructions, are then fed into the VLM's Transformer for decoding.
Memory bank sampling. However, our experiments reveal that the performance of the model highly sensitive to both the sampling interval and the total temporal window While denser frame sampling improves performance, it significantly increases memory consumption and reduces inference speed. Moreover, since robot's s movement speed varies over time, naive uniform sampling strategy often leads to redundant information, resulting in inefficiencies.
To effectively exploit temporal information, we propose an adaptive historical frame sampling method based on memory bank, aiming to capture rich historical information with minimal number of frames. Specifically, given current timestamp I, all image observations {Tr_ | 0, 1, 2,....... n 1} with a temporal window n, memory bank sampling M returns set of & sampled timestamps H. MMt, (I), k, Φ). where & is the number of sample frames based on certain feature extractor

---

The algorithm is detailed in Alg. 1, which sequentially traverses image groups while maintaining similarity queue, ensuring each newly added frame has lower similarity than the current maximum.
Temporal positional encoding. Since memory bank sampling follows a non-uniform strategy, is essential encode the temporal position of each sampled spatial vision token relative the current frame. To enhance the model's flexibility and generalization, we introduce a time encoding token e"". which captures the relative temporal offset between the historical and current frames, ej Et(t j). where. denotes the timestamp of historical frame, i represents the current observation time, and & the learnable temporal encoding function. Accordingly. the final input token set is structured as:
X'= = U'eU
| (e
(2)
iCH
where elezt represents the text instruction tokens. Each sampled frame i contributes token pair [e." e, with the temporal encoding token preceding the spatial vision token. This design ensures that the model effectively captures temporal relationships while maintaining spatial-awarence
3.4 Loss functions
To accelerate control policy genera- tion, we use action head with two MLP layers that predict the action [Aé, A g] using the hidden features of the VLM Transformer's last token.
Algorithm 1: memory bank sampling
Input: t. {Ir-) j = 0, I...,.... 1}. sample size l. feature extractor 6
Output: A set of sampled timestamps H
Initialize H.
[t]
Start with the current frame b. SSiilarity list
Initialize < inf] for j = 1 to
Our total training loss can be written as follows:
do 8 = Similarity (x( (0k4-11),.(.,.. o(I-)))
L= L. + C, + + Adaa, (3) where the translation loss L.
if len( < then
Append f - to H: Append s to S
| Aæ/|2, the rotation loss L,
else
| 2 and the grip loss Cg
===-a arg max(S)
BCE(g. g). Since the translation
if s < Si then
> Insert and reorganize
|Aal in action is often small, place greater emphasis on directional awareness within the action by intro- ducing a directional loss:
Append H: Append s to S
8 Similarity(@(I.) ( [i[m- 11), oI kmm++))
Remove H [m]
Replace S pm + 1] with s': Remove S(m] > Replace the last frame (IH(e-)1)
else
La ||d (á)) d( 12,
(4)
s Simillarit(@@
where d(x) s 2- is small value ensuring smoothness at zero.
Remove H[-1] Append ( - j to H: Append s' to
return H
3.5 MV-Bench
We propose the MV-Bench to provide comprehensive evaluation of model capabilities in learning control policies across diverse viewpoints and generalizing to novel views, while also assessing the model's s spatial understanding and adaptability.
Benchmark settings. We build multi-view dataset based on LIBERO-SPATIAL [4]. For each trajectory, we sample training and testing viewpoints uniformly within 270 front-facing range. Evaluation includes two tasks: In- View, where training and testing use the same views; and Cross- View, where testing is done on unseen viewpoints. Our camera settings are shown in Fig. 3.
4 Experiments
We first introduce the datasets and simulation environment, then describe pretraining and fine-tuning. Our model is pretrained real-world data and fine-tuned with both simulation and real-world trajectories. We run closed-loop evaluations in diverse environments and report task performance.
In addition, we evaluate on the ARM4R [32] benchmark for a direct comparison with ARM4R. Note that ARM4R pretrains with 3D inputs, while our approach differs supervision and architecture; results and detailed discussion provided in Appx. 7.1.
5

---

(a)
(b)
Arglect
Training Vice
Figure 3: Our MV-Bench camera setting. We select diverse viewpoints as training views and render images for all LIBERO-SPATIAL tasks. Novel inference views are placed near the training views. To avoid occlusion from the black box, test views blocked areas are excluded.
Method
Spatial
Object
Goal
Long
Avg.
UmiAct-0.5B [33]
64.5
77.5
68.0
46,5
64.1
SparseVLM [34]]
79.8
67.0
72.6
39.4
64.7
FastV [35]
51.6
VLA-Cache [36]]
83.8
85.8
76.4
52.8
74.7
DiflusionPolicy [16]
78.3 + 1.1
92.5 + 0.7
68.3 1.2
50.5 + 1,3
72.4 + 0.7
Trace VLA [37]
84.6 +
85.2
54.1
74.8 0.5
VL. [38]
88.2 0.5
89.9 1: 0.7
78.6 0.6
55.5 1: 1.0
78.1 +: 0.7
Octo [31]
78.9 1.0
85.7 + 0.9
84.6
0.9
51.1 +:11::0 1.3
75. 0.6
OpenVLA [1]
88.4
79.2
53.7
ODurs)(Ours)
88,9 1 0.5
95.2 0.3
90.9 0.4
79.1
88.6 0.3
Table 1: Evaluation of success rate LIBERO. Bold indicates the best-performing model. Our
model significantly outperforms other competitors, with an average success rate 12.1 higher than OpenVLA. DDnotees no available standard deviation data.
4.1 Datasets and simulation environments
DROID [2] diverse real-world robot manipulation dataset with 76,000 demonstration trajectories,
or 350 hours of interaction data, spanning total of 564 scenes and 86 tasks, each featuring RGB-D data from two third-person and one wrist-mounted camera.
LIBERO [4] The LIBERO benchmark is simulation suite with 4 task sets designed advance life- LIBERO-OBJES and LIBERO-GON explore knowledge transfer in spatial reasoning, object understanding, and task goals. LIBERO-100 includes 90 short-horizon (LIBERO-90) and long-horizon LLIBER--LONG tasks, covering 130 subtasks, each with 50 trajectories captured from both main and wrist-mounted camera.
long learning in robotic manipulation.
4.2 Pretraining setups
Data process. We pretrain our model on the DROID [2] dataset. RGB-D frames resized to 448 252, and each trajectory is uniformly downsampled to 100 actions. We remove frames with unchanged proprioception, specifically the stationary frames, and exclude trajectories with total action count exceeding 600. Actions are defined as the difference between the end-effector's 's current and target states, with translation scaled by 15 and rotation (Euler angles) by for normalization.
Pretraining details. Our pretrained model is based on Intern VL.4B. We use a temporal window of 20 and apply memory bunk sampling to select 5 past frames along with the current frame. The RGB-D inputs are processed by the original vision encoder in InternVL. with 3D positional encodings incorporated before being passed to projector from the pretrained Intern weights. To handle sparse or incomplete depth from DROID, we compute the mean depth within each vision patch. If over 90% of patch is masked, we skip adding 3D features, effectively acting as dropout and data augmentation.
MLP with downsampling rate of 4, initialized
In the training process, we freeze the vision encoder but training all other parameters. Ad is set to We utilize a cosine learning rate scheduler with an initial learning rate 2e-5. Our model was trained for epoch with batch size of 512, requiring around 20k steps complete. Training was conducted on 8 NVIDIA A6000 GPUs over 96 hours. Inference with FlashAttention bf16 requires approximately 8 GB of GPU memory.

---

Taski: Spatial generalization
Task2: Robustness distractors
Pladng hoba steecn spora
Pladi giees bedepossal
Task3: Precise placement
Task4: Instruction following
(akeggityyllovvaeeeeeee onto tbe
bellmttgafooerreepppoooog Instrction
Figure 4: Our real-world experiment settings. These settings aim to evaluate the model's spatial generalization, robustness to distractors, precision in placement, and ability to follow instructions. Each row presents 3-frame execution snapshot.
LIBERO evaluation
After pretraining, we fine-tune and conduct close-loop testing the LIBERO simulation environment, using task success rate as the metric to evaluate the performance.
Evaluation protocol. For each task in LIBERO, there 10 test subtasks, and each subtask contains 50 different object layouts. This requires 1500 simulation tracks to evaluate single task. The random seed is used alter the initial state of the objects. To evaluate each task, we randomly sample different seeds to calculate the mean and standard deviation of the success rate.
Fine-tuning details. Unlike the pretraining phase, used the simplest input settings enable our model to learn the interaction effects between 3D information and historical data on actions. In the subsequent fine-tuning phase, set the number of sequential frames A with window size n = 20. The current state RGB-D image is resized to 448 x448, while historical images are resized to 224 224 optimize memory usage and model efficiency. Following [1], normalize the ground truth action label using the 99th and 1tt percentiles. We employed a cosine learning rate scheduler with learning rate of 4e-5, using batch size of 128 and training for 20 epochs. A is set to O. During training, we activated all network parameters except the vision encoder.
Experimental results. As shown in Tab. 1, our 4D-VLA shows clear performance gains over prior methods, with 5.2% over Open VLA[1] on and 6.3% over Octo [31] LONG task, it surpasses Ope by 25.4%. On average, 4D- VLA improves success rate by 12.1% than Open VLA, demonstrating stronger stability and spatiotemporal reasoning in complex settings.
2.7% over DiffusionPolicy [16] on On the most challenging LIBERO-
4.4 MV-Bench evaluation
As shown in Tab. 2, our model achieves 81.0% success rate in the In- View setting, demonstrat- ing its capability to handle diverse training views effectively. This result shows that the integra- tion of spatiotemporal information enables our model to manage challenging and conflicting perspectives, outperforming VLenVLA [1] by significant margin of 28.8%. In the Cross- View evaluation, model also achieves the best per- formance, indicating a notable generalization capability across novel viewpoints. highlights the robustness of our model in handling diverse viewpoints.
Method
In-View, A0"
Avg.
Angle
60
120
271
3007
330
OpenVLA
4D-YLA (Ours)
83.2
87.0
79.5
70.2
75.8
90.2
Cross: View, 211"
333
Angle
15.
45"
75"
105
307
50'
OpenVLA
4D-VLA (Ours)
$3.4
83.2
74.0
65.6
75.8
60.8
73.8
Table 2: Evaluation of success rate on MV-Bench.
A symbol representing the angular deviation from the nearest training viewpoint along the z-axis.
4.5 Real-world evaluation
To evaluate models in real-world scenarios, we conducted physical experiments using Franka robotic arm. Specifically, we designed representative tasks to assess the model's capabilities, using success rate the primary evaluation metric. For each task, manually collected 50 diverse trajectories for

---

Method
Task 1
Task
Task 3
Task 4
Avg.
Open [1]
45.00
22.50
30.00
13.33
27.70
Base VLA
35.00
20.00
5.00
2.67
15.67
Pretraining
60.00
60.00
40.00
28.00
47.00
Pretraining + Coord.
75.00
60.00
85.00
34.67
63.67
Pretraining Hist.
80.00
77.50
70.00
36.00
65.88
Pretraining Coord. Hist. (fall model)
90.00
82.50
90.00
80.00
85.63
Table 3: Real-world evaluation results. We incrementally improve the Base VLA by adding
pretraining, coordinate encoding, and historical frames selected via memory bank sampling.
training and trained the model for 20 epochs. For Task 4, involving color-ordered pick-up instructions, we used 5 color sequences with 10 demonstrations each, totaling 50 trajectories.
Task descriptions. As shown in Fig. 4, we design real-world manipulation tasks to evaluate different aspects of the model's s spatial reasoning and generalization capabilities. Task 1: Spatial generalization. The robot tasked with placing a yellow cube into plate from positions not seen during training, evaluating its ability to generalize across novel spatial positions. Task 2: Robustness to distractors. The robot is asked to place two green cubes into the plate in scenes with cluttered backgrounds. This task evaluates the model's robustness to environments distractors. Task 3: Precise placement. The robot is asked to precisely stack the yellow cube onto the red one. This task emphasizes the need for fine-graine, action prediction. Task 4: Instruction following, The robot is asked to execute color-ordered assesses the model's ability to follow structured instructions correctly.
sak--add-llee commands (c.g., red + green + blue). The task
Evaluation metries. Tasks and are evaluated by success rate (i.e., successful trials total trials). For Task 2, each correctly placed green block earns I point (maximum 2 per trial), with performance measured as a total score out of 40. Task 4 involves following color-ordered instructions across combinations (5 trials each). Each correctly placed block earns I point (maximum 3 per trial), for total score out of 75. The final performance is reported as score divided by 75.
Experimental results. We set VL-4B with single RGB image inputs followed by an action head as our Base VLA model. Based on this, we incrementally add different modules to investigate their individual contributions to performance. OpenVLA [1] is set our main competitor.
As shown in Tab, 3, the base VLA model without pretraining underperforms OpenVLA on all tasks. Our spatially grounded pretraining significantly boosts performance- even with single RGB-D frame confirming its effectiveness. In short-horizon tasks (Task and 3), adding coordinate information notably improves results, especially in Task 3, which demands precise spatial alignment. This shows that coordinate encoding enhances spatial grounding and action accuracy. In long-horizon tasks (Task 2 and 4), the model often succeeds in the first step but fails the second without access to history, due to latent state ambiguity. memory bank sampling alleviates this by providing temporally relevant frames, improving multi-step reasoning. Overall, we find that matching the downstream input to the pretraining setting (c.g., coordinate- aware, temporally structured) leads to better transfer of spatial representations. Even with mismatched inputs, model still outperforms baselines, showing strong generalizability. Coordinate encoding offers explicit spatial cues, while multi- frati-frmme inputs provide temporal context to disambiguate action intent.
4.6 Multi-view real-world evaluation
In this section, we conduct additional real-world experiments under a multi-view camera setup. We design two more challenging tasks to evaluate the model's generalization ability with respect to: (i) variations in object locations together with the changed background environments; (ii) inputs from novel camera viewpoints. We use 4 fixed cameras to capture each demonstration from different angles, collecting 50 trajectories per task per camera- reesuling in total of 200 trajectories per task for training. All models are trained for 20 epochs, and performance is measured by success rate.
Task descriptions. These two more challenging tasks are shown Fig. 5. Task Out-of- distribution generalization. The robot is tasked with placing a yellow cube into plate under conditions where both the spatial configuration and the surrounding environment differ from those seen during training. These variations include changes in the plate's position, the presence and location
8

---

Training
Rind
Reed
Viend
Virid
a Vand
A
TrIgH
Vinal
Flacing the yellow cube Into the
Inference Task
View?
Placing yellow cube plate.
Placing the yellow cabe into the plate. (Novel-vto, d'feeet oibes gats peatioal
Figure 5: Our multi-view real-world experiment settings. These settings aim evaluate the
model's out-of-distribution and novel-view generalization ability.
Method
In- View
Cross- View
Avg.
Angle
0"
90
180
225
A 15")
A 25")
A455(1359)
VLenVLA []]
25
15
30
10
30
10
5
18
4D-VLA (Ours)
60
65
40
Table 4: Real-world multi-view evaluation. We test our model's spatial generalization across
varying viewpoints and object layouts. 4D-VLLA shows strong in-view and cross-view performance, highlighting its robustness under real-world distribution shifts.
of distractor objects (e.g., other cubes). This task evaluates the model's s ability to generalize to unseen object arrangements and background contexts, testing robustness real-world deployments beyond the training distribution. Task 2: Novel-view generalization. Similar to Task 1, the robot asked place yellow cube into a plate, with spatial setup and surroundings differing from training. However, during inference, data input is captured exclusively from an additional novel, unseen camera viewpoint that was not used during training. This task evaluates the model's viewpoint robustness-in ability to generalize across camera perspectives and accurately interpret the scene from unfamiliar angles. Success this task reflects strong spatial understanding and invariance to viewpoint changes, both critical for weal-world target block only moved within a small spatial range, while the background is fully randomized.
deployment. To simplify the setup, the
Evaluation metrics. Each multi-view task is evaluated over 20 trials. In every trial, both the back- ground and object positions are randomly shuffled to assess the model's robustness and generalization. The evaluation metric is the task success rate, computed the ratio of successful trials to the total number trials.
Experiments results. As shown in Tab. 4, 4D-VLA significantly outperforms VLA VLA in both in-view and cross-view settings, demonstrating strong generalization to viewpoint shifts and layout variations. In the in-view setting, where the camera is fixed but object layouts change, our model maintains consistently high success rates, indicating robustness to spatial perturbations. In the more challenging cross-view setting involving unseen viewpoints, 4D-VLA continues to perform stably across different angles. Although performance slightly drops at larger viewpoint shifts (e.g., A45"), remains stable compared to VLe whose success rate fluctuates more severely under such conditions. These results suggest that our model effectively captures spatial consistency across views, leading to more reliable visuomotor control real-world environments.
Discussion
Building upon previous experiments, we further analyze three key questions: (i) The Role of historical information -How does historical context influence model's effectiveness? (ii) Ablation study model components -How do different architectural components contribute to the overall

---

performance of the model? (iii) The impact of coordinate system chaos -How does coordinate system inconsistency affect the model's performance. and can the introduction of 3D representation mitigate this issue? The first question is addressed in Sec. 5.1; the second is covered in Appx. 7.2, and the third in Appx. 7.3. Our discussion begins with the simple model, i.e., Intern VI.-4VLL.-BB with an MLP action head, using single RGB image as the vision input.
5.1 Exploring historical information utilization
As highlighted in Octo [31], incorporating historical context can significantly boost model perfor- mance. However, this has been underexplored. We conduct a comprehensive analysis to assess its true impact model effectiveness. We define window size the number of historical frames available to the model and & as the number of frames sampled from them. To systematically investigate their impact, design experimental settings: (i) fixing while progressively increasing k, and (ii) fixing i while expanding n2
Success
72
Encoding
Position
Fusion
Success rate
71.
learnable
relative
concat
75.6
learnable
relative
additive
74.8
i-5, MBS
sinusoidal
relative
additive
71.6
k:S, Uniform
learnable absolute additive
0.0
Am10, Uniform
-No temporal encoding-
63.0
60.6
No nistorical Image
29.5
Table 5: Ablation on temporal encoding method.
20 Window
Figure 6: Historical image analysis. Larger points indicate lower efficiency.
As illustrated in Fig. 6, our findings reveal that the historical window size n plays a pivotal role in determining performance, whereas the effect of I is comparatively minor. This suggests that uniform sampling may introduce excessive redundancy, adversely diminishing the model's s efficiency. memory bank sampling strategy, which effectively reduces redundancy and improves performance, even with a smaller k; showcasing its effectiveness maximizing the utility of historical context.
Ablation on temporal encoding method. Since MBS employs non-uniform sampling, the absence of explicit temporal position encodings can lead to ambiguity the historical sequence, making difficult for the model fully leverage past information. To address this, we investigate the effect of different temporal encoding strategies on model performance, as summarized Tab. 5. Specifically, additive means the temporal encoding is directly added to the image tokens, while concal appends it before the image token, and absolute refers to encoding based on the sampled timestamps rather than relative to the current frame. Clearly, the concat method achieves the best performance, with the key advantage seamlessly integrating with spatial vision tokens to enhance the model's representation.
Conclusion
In this paper, we present 4D-VLA, which incorporates 4D information to address challenges in leveraging diverse robotic datasets for pretraining, such as coordinate system chaos and state chaos. Our model encodes sequential RGB-D images into visual features with corresponding 3D coordinates, aligning the robot's coordinate system with the scene. This alignment enables strong spatiotemporal reasoning with minimal training. We also introduce memory bank sampling, a frame sampling strategy extract informative and diverse key frames from sequences, improving efficiency. In the simulated LIBERO environment, 4D-VLA outperforms existing methods. Additionally, our multi- view simulation dataset, MV-Bench, demonstrates superior spatial perception and generalization to novel viewpoints, surpassing current approaches. A limitation of our approach is its reliance on RGB-D input, which introduces hardware restriction.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
10

---

References
[1] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint, 2024. 2, 3, 6, 8,9
[2] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park; Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe. Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma. Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O' O'Neill, Rosario Scalise, Derick Seale, Victor Son. Stephen Tian, Emi Tran. Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang. Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martnn, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024. 2,6 6
[3]
Open X- X-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerice Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su, Hao-Shu Fang, Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang. Igor Mordatch, Iijaa Radosavovic, Isabel Leal, Jacky Liang, Jachyung Kim, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang, Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishnan Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Ou,, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martin-Martin, Russell Mendonea, Rutav Shah. Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song. Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yuch hua Wu, Yujin Tang, Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open X-Emmodiment: Robotic learning datasets and RT-X models. https:/ // //tpx//xvv//////////ee//eaaaa..a wwwriii.ogg/.eeeee 23100. 088644, 2023. 2,3
Beomjoon Kim, Bernhard
[4] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone, Libero: Benchmarking knowledge transfer for lifelong robot learning. In NeurIPS, 2024. 2, 5,
11

---

[5] Oier Mees, Lukas Hermann, Erick Rosete-Best Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditiong policy learning for long- oong-horizon robot manipulation tasks, In RA-L, 2022. 2
[6] Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-vssue learning with heterogeneous pre-trained transformers. arXiv preprint, 2024. 3,
[7] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. 3
[8] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023. 3
[9] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023.
3
[10] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. In NeurlPS, 2022. 3
[11] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint, 2023.
[12] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie Ssuaal-inguustic tasks. In CVPR, 2024. 3,
[13] Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint, 2023. 3
[14] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. In NeurlPS, 2023.
[15] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint, 2024. 3
[16] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023.
3, 6,
[17] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint, 2023. 3
[18) Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint, 2024.3
[19] Chuan Wen. Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint, 2023.
[20] Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. arXiv preprint, 2024.
[21] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint, 2022. 3
[22] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint, 2023. 3
12

---

[23] Mengda Xu, Zhenjia Xu, Yinghao Xu. Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow the cross-domain manipulation interface. arXiv preprint. 2024. 3
[24] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML, 2024.
[25] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong. and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. In ICMI., 2024.
[26] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In ICLR, 2024. 3
[27] Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token prediction. Advances in neural information processing systems, 37:79307-773324 2024. 3
[28] Iliaa Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In CoRL, 2023.
[29]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong. Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Roboties transformer for real-world control at scale. arXiv preprint, 2022. 3
[30] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision- action models transfer web knowledge to robotic control. arXiv preprint, 2023. 3
[31] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https:
//octo-models github. 1o, 2023. 3, 7, 7,17 10
[32] Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, and Roei Herzig. Pre-training auto-regressive robotic models with 4d representations. In Forty-second International Conference on Machine Learning. 5, 22
[33] Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Universal actions for enhanced embodied foundation models. arXiv preprint, 2025.
[34] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint, 2024.
[35] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- lasgune models. In ECCV, 2024. 6
13

---

[36] Siyu Xu, Yunke Wang, Chenghao Xia, Dihao Zhu, Tao Huang, and Chang Xu. Vla-cache: To- wards efficient vision- ciroon-llguag-actttn model via adaptive token caching in robotic manipulation. arXiv preprint, 2025. 6
[37] Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances tpatoraltempora awareness for generalist robotic policies. arXiv preprint, 2024.
[38] Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-langaage-action model. arXiv preprint, 2025.
[39] Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J Davison. Coarse-to-fine q- attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739 13748, 2022. 22
[40] Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. In Conference Robot Learning, pages 3333-3355, PMLR, 2025. 22
[41] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023. 22
[42] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu. Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, al. Longvu: Spatiotempo- ral adaptive compression for long video-language understanding. In Forty-second International Conference on Machine Learning. 22, 23
[43] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth tokens large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024. 22, 23
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly stated the contribution and scope of the paper in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work the conclusion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

We have provided the theoretical assumption and proof of this paper in the method part.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have disclosed all the information needed to reproduce the main experi- mental results of the paper in the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We have not released code this paper in the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpp.cccppbblll/ http:ccccpppccccccecceeccccccccc httpublicguuuieeesseeeeppp guides/ /CoeeSubmdeibiii ssi ssisssoolooooooooo oonooiiyyyylolooooiomoooomiiiilp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details of this paper in the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported appropriate information about the statistical significance of the experiments in the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided sufficient information on the computer resources of this method in the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / //t::///nnee /tt.riss....p/eee//eee/ee cc htcppbbllc/Echccssuueeeeeeee
Answer: [Yes]
Justification:
We have conducted research under the NeurIPS Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the societal impacts of this paper in the introduction and method parts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has not released data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used in the paper are properly licensed. Our paper cites the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets, including data and models. We also did not publish or submit the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research topic of this paper is vision language action model, not involving crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research topic of this paper is vision language action model, not involving crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: L.M is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientifie rigorousness, originality of the research
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: : //nerrurpp.... wwtccc////ooonecceeeeeee Cet.ooeeenneeee225/////// for what should or should not be described.
21

---

Appendix section
7.1 Comparison with ARM4R
Our approach differs from ARM4R in both design and practice. Instead of representation learning with full 3D point clouds and proprioception, we adopt an end-to-end, action-centric pretraining pipeline that maps vision-language inputs directly to low-level actions. Inputs are RGB with depth- aligned spatial coordinates (2D patches (xc, y, 2) embeddings), keeping the model lightweight and VL.M-comatibbe while avoiding point-cloud encoders.
Method
Meet-off-grill
Push-buttons
Place-wine
Open-drawer
Avg.
C2FARM-BC [39]
20.0
72.0
18.0
20.0
32.5
LLARVA [40]
80.0
56.0
12.0
60.0
52.0
PerAct [41]
84.0
48.0
12.0
80.0
56.0
ARM4R [32]
94.4
67.2
36.0
88.8
71.9
4D-VL
95.2
79.2
45.6
89.6
77.4
Table RLBench results under the ARM4R protocol.
Analysis. To balance efficiency and spatial coverage, strategy that keeps 5 history frames within 20-frame context. The current step uses dual views (front wrist), while history stores only the front view. We also adopt OpenVLA's random-crop augmentation. This configuration provides strong spatial grounding and good runtime, offering a strategy different from ARM4R yet comparably effective its benchmark.
streaming memory bank sampling
Note on supervision. Our method leverages calibrated intrinsics/extrinsics during training and inference. While such calibration is standard in many robotic datasets, it is an additional source of supervision not explicitly used by ARM4R.
7.2
More ablation study
Pretrain
Coord.
Prop.
Success rate
Action head
FPS
Success rate
X
X
29.5
MLP
12.6
29.5
X
X
V
16.4
auto regression
0.6
28.1
X
X
36.6
diffusion
8.7
27.0
X
45.0
Table 7: Ablations on heads and inputs (Libero-Long). Left: action head vs. FPS and success (MLP, autoregressive, diffusion). Right: effect pretraining, 3D coordinate embedding, and proprioceptive tokens on success.
Sampling Method
Success
Lateney (ms)
Peak Mem (MB) +
Single frame
0.738
76.5
8342.5
Adaptive Pooling [42]
0.604
150.7
8949.1
Grid Pooling [43]
0.620
208.8
8852.7
Q-Former (per-frame)
0.556
223.3
8812.8
MBS (Ours)
0.866
160.0
8682.9
Table 8: Frame sampling ablations on Libero-Spatial. MBS attains the highest success (0.866)
with competitive efficiency, while single-frame is fastest and most memory-light but less accurate.
Action heads. We conduct additional ablations on LIBERO-LONG. For the action heads (Tab. left), the MLP head yields the highest inference speed with a relatively high success rate. In contrast, the autoregressive head preddicinniiinggggg text tokens for actions -unns runs slower due multi-token reasoning.
Pretrain and spatial components. Results for other components/operraiions are shown Tab. (right). Proprioceptive tokens hurt performance, likely because proprioceptive states (e.g., joint positions, velocities) are highly correlated with action labels, which encourages overfitting to dataset- specific motion patterns rather than learning generalizable visuomotor features.
22

---

Memory bank sampling. We compare our memory bank sampling with three video-style samplers: Adaptive Pooling [42], Grid Pooling [43], and a per frame Q-Former, using IntemVL2.5-4B on Libero-Spatial (20 epochs). We report success, latency, and peak memory in Tab. 8. Adaptive Pooling caches the full trajectory and uses two thresholds (0.99 for frame selection and 0.96 for spatial compression). We omit its second-stage cross-modal filter due to low language diversity (10 tasks) and observe hyperparameter sensitivity and variable-length tokens that increase memory and reduce accuracy, Grid Pooling replaces language tokens with 8 learnable queries and compresses each frame to 2x2 grid (4 tokens), yielding 20 (4+8) historical tokens; it is simple but nonadaptive and can drop key spatial cues. Q-Former (per frame) uses 10 learnable queries per frame extract features but converges slowly and discards too much spatial context, which hurts success. MBS (ours) operates online with window of 20 and memory of 5. It selects informative key frames from the stream, stores only 5 history frames at 224 x 224, and keeps the current frame at 448 x 448. As a result, MBS achieves the highest success (0.866) with competitive cost (160.0 ms and 8,682.9 MB). Single frame is faster (76.5 ms) but weaker (0.738). The other baselines are slower and less accurate. MBS fits VLA's streaming and causal nature because it avoids full-clip caching, uses relative temporal encoding, and remains compatible with closed-loop control. It yields stronger long horizon performance without extra retraining or heavy compute.
7.3 Coordinate system chaos impact analysis
To assess the impact of coordinate system chaos on VLA's performance, we conduct controlled experiment. The experiment consists of two main steps: First, we deliberately introduce controlled chaos into the LIBERO environment. Then, we compare the performance of models with and without 3D information. Our findings demonstrate that chaotic coordinate transformations significantly degrade model performance while incorporating 3D information effectively alleviates it.
Chaos generation. To simulate the diverse view- points in the pretraining waheeet-where the robot's body is absent from the image, and its coordinate system varies unpredictably from a specific task in LIBERO-SPATIE each trajectory from 30 distinct viewpoints. For each trajectory and its corresponding viewpoint, we intro- duce coordinate system chaos by applying a random translation t E IR's and rotation q E S3(3) to the robot's coordinate frame.
100
select trajectories and render
40
30
Success Rase (36)
10.
w/b 30
After applying the coordinate transformation, the gripper's grasping state in the ground-truth action remains unchanged. However, the rotation offset Ao and translation Az, along with the proprioceptive and camera pose information, undergo the following transformation:
Level 0
Level 1
Level 2
Level
Chaos Level
Figure 7: Success rates under varying coor- dinate chaos levels.
AO + (qb(A)))))
Aw' q
0' = + (q))
x' = q + t, T' = qT +t.
R' qR,
(5)
Here, the function to R33 - (3( maps an Fuler angle to its corresponding rotation matrix. The terms AO' and Aæ' denote the transformed action values, while 0', O and x', x represent the transformed and original rotation and position, respectively. Additionally, [R'[T"] and [ correspond to the transformed and original camera poses.
In the subsequent training process, we utilize the transformed action values A and Ax', along with the transformed camera parameters, for model training.
Implementation details. We employ a simple baseline, controlling for the involvement of 3D information. The baseline model extracts tokens from single RGB view, while the alternative model converts an RGB-D frame into spatial vision tokens as input for the L.M Transformer. During testing, we do not apply random rotations or translations to the world coordinate system.
23

---

Experimental results. We control chaos levels by adjusting the magnitude of random rotations. Level 0 applies no rotation, while levels 1-3 introduce random z-axis rotations of 15 309, and 90", respectively. Translation is randomly set within range of 0.5. As shown in Fig. 7, without chaos, both models perform well, with 3D information further boosting success rates. Notably, the 3D model shows lower variance across viewpoints. As chaos increases, the non-3D model's performance drops sharply, while the 3D model maintains relatively high success highhighng the value of 3D cues in handling coordinate system chaos.
24


Reviews:
[{"review_id": "jRs9NlEa26", "reviewer": "Reviewer_jMXX", "summary": "VLA pre‑training pipelines suffer from two kinds of input incompleteness: coordinate‑system chaos and state chaos.  To tackle both, the authors augment RGB with depth and an adaptive history sampler. The proposed 4D-VLA tries to bridge the gap of image‑only systems.", "questions": "1. Cite the window‑based sampler used in VLMs and briefly explain why MBS is lighter or more effective.\n\n2. Are you planning to release your data, and what's your release plan?", "limitations": "This method assumes the hardware to be RGB-D.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Authors show their results on LIBERO, MV‑Bench and real hardware with performance gains over baseline methods.\n2. Viewpoint generalisation provided by the method is validated in multi‑view evaluation.\n\n3. 3‑D positional embedding plus a lightweight memory bank sampling queue introduces few learnable parameters.\n\nWeakness:\n1. There is hardware dependence on RGB‑D. Many large‑scale demonstration datasets are RGB‑only.\n2. Window-based keyframe sampling and spatio-temporal encoding are already widely adopted in vision-language model (VLM) research and have demonstrated effectiveness across multiple benchmarks. The paper would benefit from an elaboration of architectural novelty beyond these established techniques.", "was_revised": false, "final_justification": "", "rebuttal": "## Q1: There is hardware dependence on RGB‑D.\n\nThank you for raising this point. We would like to clarify that **our method does not depend on hardware RGB-D sensors**. The key requirement of our framework is the availability of **calibrated camera poses**, which are used to construct a consistent robot-centric coordinate system across time and views.\n\nWhile we leverage depth information to back-project image features into 3D space, this depth **need not be obtained from RGB-D hardware**. In RGB-only scenarios, monocular depth estimation (e.g. recent scaled VGGT) can serve as an effective substitute. Such techniques are well-established and can be seamlessly integrated into our pipeline with minimal additional cost.\n\nFurthermore, we empirically demonstrate the flexibility of our approach in **real-world experiments** (see Table 3 of the paper), where our model achieves superior performance over strong baselines **even when using only RGB inputs during inference**. This result highlights the robustness and transferability of our pretraining strategy beyond RGB-D settings.\n\nWe will revise the manuscript to emphasize this point and avoid any impression of hardware dependency.\n\n## Q2: Window-based keyframe sampling and spatio-temporal encoding are already widely adopted in vision-language model (VLM) research\n\nThank you for the suggestion. While our frame sampling strategy may appear simple on the surface, it is specifically designed for the **streaming nature of Vision-Language-Action (VLA) tasks**, where observations arrive sequentially and memory constraints are critical. Unlike traditional keyframe sampling methods used in video QA or captioning, our approach focuses on **stream-aware frame selection** that aligns with **temporal context and physical continuity**.\n\nIn particular:\n\n- **Our memory-bank sampling strategy** enables the model to retain only the most informative historical frames, maintaining a fixed memory size (e.g., 5 frames) regardless of total history length. This design respects the **online inference constraint** of VLA tasks and is **tailored for frame-by-frame robot policy rollout**, unlike prior works that assume full video access.\n- Our method supports **longer temporal context at lower frame rates**, which enhances efficiency and improves performance, as validated in our experiments (see figure 5 in our paper). To our knowledge, **prior works have not systematically explored streaming-aware history sampling** under memory constraints for VLA tasks.\n\nRegarding the spatial encoding, our design is not merely an architectural enhancement, but a **crucial solution to a fundamental pretraining challenge**. In VLA tasks, the target action is defined in the **robot-centric coordinate system**, yet in many existing RGB-based datasets, the robot’s base frame is not visible in the image. **This means that even for identical instructions (e.g., “move the cup forward”), the corresponding action vectors may vary drastically depending on the unknown and unobserved orientation of the robot base.**\n\nIn such cases, learning action outputs without access to the coordinate origin is equivalent to **learning from missing or noisy input**, which introduces severe ambiguity. This **renders the supervision signal inconsistent**, and undermines the effectiveness of pretraining—especially when using large-scale datasets with diverse scene layouts and robot configurations.\n\nOur spatial encoding addresses this issue by injecting **geometric priors aligned with the robot’s canonical frame**, allowing the model to disambiguate actions even when the base is not directly observed. Despite its simplicity, this solution is both effective and essential. We visualize and analyze the consequences of coordinate misalignment and the benefits of our approach in **Appendix Figure 6**.\n\nWe will further clarify these points in the revised version to better highlight the novelty and motivation behind our design choices.\n\n## Q3: Cite the window‑based sampler used in VLMs and briefly explain why MBS is lighter or more effective.\n\nThank you for the suggestion. We will include relevant citations in the revised version. Notably, to the best of our knowledge, we are the first to adopt a **window-based and memory-efficient sampling strategy specifically tailored for VLA tasks**.\n\nOur proposed **Memory Bank Sampling (MBS)** is both **lighter** and **more effective** than common baselines:\n\n- **Lightweight**: MBS maintains a fixed-size memory bank (e.g., 5 frames), requiring **only pairwise similarity comparisons** without any learnable modules (e.g., no Q-former or transformer layers). Its sampling process is computationally cheap and easily parallelizable, making it suitable for real-time or streaming settings. Additionally, it avoids re-encoding every historical frame at each timestep—unlike attention-based memory methods.\n- **Effective**: MBS selectively retains **diverse and informative historical frames** by minimizing redundancy. As shown in **Algorithm 1**, it dynamically filters frames that are most dissimilar from the current one, maintaining temporal diversity and long-term context. In our experiments (see Figure 5), **MBS outperforms uniform sampling baselines even when using half the number of frames**, demonstrating its efficiency in both memory usage and downstream accuracy.\n\nBy contrast, prior VLM works often rely on either fixed interval sampling or full sequence attention, both of which are less adaptive and computationally heavier. Our MBS strikes a practical balance between temporal coverage and efficiency, making it a compelling design for VLA reasoning.\n\n## Q4: Release plan\n\nYes, we will make all the data and tools associated with this work publicly available to facilitate any further research.\n\nSpecifically:\n\n- We will release our **multi-view robot-centric dataset**, including RGB, depth, and pose information, shortly after the review process.\n- We also plan to open-source our **processed version of the DROID pretraining dataset**. To our knowledge, currently available versions of DROID data either lack depth and camera parameters, or are only available in large raw formats (e.g., DROID-raw), which contain redundant trajectories and are difficult to use directly.\n- Finally, we will release **MVBench**, our multi-view evaluation suite, which includes rendered training data under the LIBERO simulator and a complete benchmarking pipeline for consistent evaluation.\n\nThese releases will enable reproducibility and provide the community with high-quality resources for further exploration of vision-language-action learning in spatiotemporal settings."}, {"review_id": "f1bHYykDoi", "reviewer": "Reviewer_GHr5", "summary": "The paper proposes 4D-VLA. The proposed approach is an improved VLA architecture with 4D representations. It involves RGB-D inputs, a memory bank, coordinate alignment, and pre-training. The method shows consistent gains in tested settings.", "questions": "Please see the weaknesses section.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The proposed approach leads to consistent improvements in tested settings\n- Different design decisions are ablated individually\n\nWeaknesses:\n- The proposed approach involves a number of engineered design decisions and is quite specific to robotic manipulation. It may be a better fit for a robotics venue than a general machine learning one.\n- Discussion of and comparison to prior methods that address \"state chaos\" is missing. For example, [1] and [2] include past context for low-level observations and image observations, respectively.\n- The demonstrated tasks are fairly simple and likely achievable with simpler methods that act on 2D observations. It would be good to demonstrate a result where 4D representations enable behaviors that have not been shown with 2D observations, which would help justify the added method complexity.\n- The \"system chaos\" and \"state chaos\" terminology may be a bit too dramatic and something more descriptive may make the paper easier to parse.\n\n[1] https://proceedings.neurips.cc/paper_files/paper/2024/hash/90afd20dc776bc8849c31d61a0763a0b-Abstract-Conference.html\n\n[2] https://proceedings.mlr.press/v229/radosavovic23a/radosavovic23a.pdf", "was_revised": true, "final_justification": "I acknowledge I read the rebuttal I maintain my original recommendation.", "rebuttal": "## Q1: The proposed approach involves a number of engineered design decisions and is quite specific to robotic manipulation. It may be a better fit for a robotics venue than a general machine learning one.\n\n**We respectfully disagree. Our work is not narrowly tailored to a specific manipulation task, but rather addresses a fundamental and generalizable challenge in vision-language-action (VLA) pretraining.**\n\nAt the core of our method is a key observation: in VLA settings, the **target action is defined in the robot-centric coordinate system**, but many existing RGB-based datasets fail to observe or encode the base orientation of the robot. This introduces a critical ambiguity—**even for the same instruction** (e.g., “move the cup forward”), **the ground-truth action vector may differ dramatically** depending on the robot’s unobserved base pose. This inconsistency results in **chaotic supervision**, which significantly undermines pretraining effectiveness.\n\nOur proposed solution—**injecting consistent 4D spatial information (multi-view, time, and pose-aligned input)**—is not task-specific engineering, but rather a **crucial solution to a core problem in general VLA representation learning**. This insight applies broadly to a wide range of embodied tasks, including navigation, interaction, and instruction following—not just manipulation.\n\nIn addition, we show (Fig. 5) that **not all historical frames are equally informative**, motivating our memory-efficient frame sampling strategy. Unlike prior work that encodes fixed-length context windows, we propose a **streaming-aware, high-utility memory bank** that selectively retains useful frames under compute constraints.\n\nMoreover, our experiments span **both simulation and real-world settings**, and demonstrate strong generalization, suggesting that the proposed pretraining strategy captures **robust spatial priors transferable across domains**.\n\nWe believe these contributions, particularly the identification and resolution of coordinate-system inconsistency in VLA data, are of **broad interest to the machine learning community**, especially as interest in generalist embodied agents continues to grow.\n\n## Q2: Discussion of and comparison to prior methods that address \"state chaos\" is missing. For example, [1] and [2] include past context for low-level observations and image observations, respectively.\n\nThank you for the references. While both [1] *Humanoid Locomotion as Next Token Prediction* and [2] *Robot Learning with Sensorimotor Pre-training* incorporate past context, their focus differs from ours. **In terms of temporal modeling**, our work focuses on how to **efficiently and effectively leverage sequential information in streaming VLA tasks**. Specifically, we investigate how to **retain and select the most informative historical observations under memory and compute constraints**, which is crucial for scalable and real-time VLA model deployment.\n\nSpecifically:\n\n- **[1] Humanoid Locomotion** uses a **short history window** of RGB frames and proprioceptive states, which are all **re-encoded at each step** and passed through the full Transformer for autoregressive prediction. The method assumes all past observations are available (non-streaming) and incurs significant computational cost. In contrast, our design is optimized for **streaming inference**: we use **window-based keyframe selection** to maintain a **fixed-size memory bank** of only the most informative frames, achieving both high performance and low latency, as shown in our experiments and in response to Reviewer 7q7T Q2.\n- **[2] Sensorimotor Pretraining** encodes the **most recent 16 steps** of low-level state and RGB input for behavior prediction. However, it uses a fixed history length and lacks any explicit sampling or memory mechanism. In contrast, our **memory bank sampling (MBS)** explicitly selects high-value frames tailored to the streaming VLA setting, maintaining spatial and temporal diversity within a bounded memory size (e.g., 5 frames). Our ablation (Figure 5 in our paper) shows that this leads to better efficiency and generalization than uniform or FIFO sampling.\n\nIn summary, while prior work utilizes past context primarily to aid behavior prediction within fixed observation windows, **our method focuses on improving how historical information is selected and reused**. By introducing a memory-efficient sampling strategy tailored for streaming VLA inputs, we enable more effective use of past observations without incurring significant computational overhead. **This distinction highlights a key contribution of our work in temporal modeling for VLA.** \n\nWe will cite and discuss these work in the revised version of manuscript.\n\n[1] Humanoid Locomotion as Next Token Prediction, NeurIPS 2024\n\n[2] Robot Learning with Sensorimotor Pre-training, CoRL 2023\n\n## Q3: The demonstrated tasks are fairly simple and likely achievable with simpler methods that act on 2D observations.\n\n**Thank you for the comment. While the individual tasks may appear simple in isolation, our goal is to examine the effects of spatiotemporal misalignment that arise in multi-scene, multi-view pretraining—challenges that simpler 2D approaches often overlook.**\n\nIn our supplementary material, we provide **a controlled toy experiment** showing that **spatial inconsistency across scenes (e.g., from unaligned coordinate frames)** leads to significant performance degradation. This demonstrates that even “simple” behaviors become brittle when the underlying representation fails to enforce spatial consistency.\n\nWe also include **multi-view pretraining experiments**, where the baseline model trained on conflicting viewpoints shows a dramatic drop in success rate compared to the single-view case. In contrast, our method, which enforces consistent 4D spatial alignment, **maintains high success rates even under large viewpoint shifts (e.g., 45° yaw) in zero-shot settings**. These results highlight the critical role of spatial grounding in enabling generalization beyond narrow training distributions.\n\nOn the **temporal side**, we observe that single-frame models often get stuck or exhibit unstable behavior. In contrast, our method—via memory-efficient access to historical observations—enables **retry and recovery behaviors**, which are signs of **stronger sequential reasoning and long-term credit assignment**.\n\nWe will expand these illustrative cases in future versions to more clearly highlight **how 4D representations (3D + time) contribute to robustness, generalization, and behavior persistence**—capabilities that are difficult to achieve with purely 2D models.\n\n## Q4: The \"system chaos\" and \"state chaos\" terminology may be a bit too dramatic\n\nGreat suggestion! We appreciate the concern that terms such as “system chaos” and “state chaos” may sound overly dramatic.\n\nThese terms were deliberately introduced to provide **succinct and intuitive shorthand** for the **underlying pretraining challenges** we aim to address—namely, the **inconsistency of coordinate frames across scenes** and the resulting **ambiguity in action supervision**. In large-scale VLA datasets, where camera poses and robot orientations vary significantly and are often not directly observable, such misalignments can severely degrade learning. We found that naming these phenomena helped us clearly **frame the core motivation of our method** and distinguish it from prior works that focus solely on local temporal context.\n\nBy naming these issues explicitly, we aimed to clearly frame the **motivation behind our method** and highlight the **distinction from prior work** that often assumes spatial alignment or focuses only on local temporal cues.\n\nThat said, we appreciate the reviewer’s concern and will revise the terminology in the final version to make it more **descriptive and accessible**, without losing the clarity of the underlying concepts."}, {"review_id": "zhnh4eWobo", "reviewer": "Reviewer_7q7T", "summary": "This paper introduces 4D-VLA, a VLA that leverages both depth information and historical context to improve the performance. They propose to convert depth information into point cloud in the robot frame and use learnable positional embeddings to encode the point cloud. The spatial embeddings are added to the 2D image embeddings to serve as spatial vision features.For temporal features, they propose a memory bank sampling strategy to balance the efficiency and effectiveness. They further propose a MV-Bench to evaluate the performance of the model across diverse viewpoints. Experiments on MV-Bench, LIBERO and real world show that 4D-VLA achieves state-of-the-art performance.", "questions": "Answer all the questions in Weaknesses.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and structured, making it easy to follow the proposed approach.\n2. The experiments are comprehensive, covering both synthetic and real-world datasets, which demonstrates the effectiveness of the proposed method.\n3. The motivation for using depth information and historical context is clearly explained, and the proposed method effectively integrates these features into the VLA framework.\n\nWeaknesses\n1. The way to use depth information resembles 3D-VLA[1]. It seems that both 3D-VLA and this paper convert depth information into point cloud using camera parameters and then use positional embeddings to encode the point cloud, which are later fused with the 2D image embeddings. It seems the only difference is that this paper uses learnable positional embeddings while 3D-VLA uses sin/cos positional embeddings. The authors should clarify the difference between the two methods.\n2. The baselines used to verify the effectiveness of the proposed memory bank sampling strategy are not sufficient. The authors only compare with the uniform sampling while there are many other sampling strategies in video understanding, such as spatiotemporal adaptive compression [2], grid pooling [3], using Q-Former, etc. The authors should include more baselines to demonstrate the effectiveness of their sampling strategy.\n3. Though the teaser shows scenarios where actions can be ambiguous without considering the history, the experiments do not include such scenarios. It seems that tasks in LIBERO benchmark and in real world experiments are all deterministic, i.e., the actions can be determined by the current observation without considering the history. The authors should explain why history helps these tasks, especially the Task 4 in real world experiments.\n4. The authors should provide more details about the proposed MV-Bench, such as how many demonstrations are used for each task, why LIBERO-10, LIBERO-goal and LIBERO-objects are excluded.\n5. Many relevant VLA works are not cited and discussed in the paper.\n\n\n[1] 3D-VLA: A 3d vision-language-action generative world model.\n[2] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding\n[3] LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models", "was_revised": true, "final_justification": "While the rebuttal addresses some concerns, key issues remain regarding the novelty of the work. The authors seem to have a misunderstanding of 3D-VLA—I still believe 4D-VLA and 3D-VLA use depth information in a similar way. Contrary to the rebuttal, 3D-VLA does not rely on a point cloud encoder; instead, it uses 3D-LLM’s backbone, which explicitly extracts visual features from 2D image encoders and merges them with 3D positional embeddings. Given these points, I respectfully maintain my original rating of 3.", "rebuttal": "## Q1: The way to use depth information resembles 3D-VLA.\n\nThank you for raising this point. While our pipeline may appear superficially similar to 3D-VLA in that both methods project depth into 3D and apply positional encoding, our approach is fundamentally different in **motivation, architectural design**, and **pretraining objective**.\n\n**First**, 3D-VLA is centered around *goal imagination*: it uses a **diffusion model to hallucinate future point clouds** and relies on a separate point cloud encoder to extract geometry-aware features, which are then fused with image features to enrich scene understanding. Depth in 3D-VLA serves primarily as an **auxiliary input for prediction**, and the supervision is goal-centric.\n\n- In contrast, our method is explicitly designed for direct, end-to-end **action prediction** in real-world and simulation environments. Our use of depth is not for scene generation or imagination, but rather to address a **core problem in VLA pretraining**: the **coordinate inconsistency across scenes**, or what we call \"coordinate system chaos\". In many VLA datasets, robot base poses are unobserved, leading to inconsistent supervision—e.g., the same language command (“move the cup forward”) may correspond to drastically different action vectors depending on camera and robot orientation. This inconsistency severely disrupts pretraining.\n\nRather than treating depth as a geometric side channel, we use it to **reconstruct the spatial frame** and **consistently ground actions** across scenes. This allows us to transform VLA from a partially-observable policy learning problem into a well-posed, spatially grounded one.\n\n**Architecturally**, we also differ: 3D-VLA encodes the entire 3D scene (or predicted goal state) using a diffusion pipeline and point cloud encoder. In contrast, we extract **3D patch positions in real-time via lightweight fusion of depth and vision embeddings**, with **learnable spatial embeddings aligned to the robot-centric frame**, enabling streaming deployment and policy inference.\n\nTo summarize:\n\n- **3D-VLA**: Uses depth to **generate goals** and enrich image features with auxiliary 3D features (for understanding) . It introduces considerable latency, which limits inference efficiency in VLA tasks..\n- **Ours**: Uses depth to **resolve spatial inconsistency** and stabilize the action space across diverse VLA data (for control).\n\nThis distinction in both **purpose and implementation** reflects a fundamentally different design philosophy, tailored to scalable, real-world action learning.\n\n## Q2: The baselines used to verify the effectiveness of the proposed memory bank sampling strategy are not sufficient.\n\nThank you for the valuable suggestion. To further evaluate our proposed Memory Bank Sampling (MBS), we implemented and compared it against several additional sampling baselines adapted from the video understanding literature, including adaptive compression [3], grid pooling [4], and Q-former-based token selection. All methods were evaluated on the Libero-Spatial benchmark using the same base model (InternVL25-4B), fine-tuned for 20 epochs.\n\nThe results are summarized below:\n\nTable1: Performance and Efficiency Comparison of Frame Sampling Strategies\n\n| Sampling Method | Success Rate | Inference Latency (ms) | Peak Memory (MB) |\n| --- | --- | --- | --- |\n| Single frame | 0.738 | 76.5 | 8342.5 |\n| Adaptive Pooling [3] | 0.604 | 150.7 | 8949.1 |\n| Grid Pooling [4] | 0.62 | 208.8 | 8852.7 |\n| Q-Former (per-frame) | 0.556 | 223.3 | 8812.8 |\n| **MBS (Ours)** | **0.866** | 160.0 | 8682.9 |\n\n**Implementation Details and Comparisons:**\n\n- **Adaptive Pooling [3]:** We cache the full trajectory and apply the method at each timestep using a 0.99 threshold for frame feature extractor and 0.96 for spatial token compression. The second-stage cross-modal filtering was omitted due to limited language diversity (10 tasks only), which caused instability. The method is hyperparameter-sensitive and memory-heavy due to variable-length input tokens, and underperformed in our setting.\n- **Grid Pooling [4]:** We replaced language tokens with 8 learnable queries (due to low text diversity). Each frame was compressed into a 2×2 token grid (4 tokens), leading to 20×(4+8) historical tokens. While efficient in theory, it lacks adaptive selection, and the compression may lose key spatial cues.\n- **Q-Former (per-frame):** We used 10 learnable queries per frame to extract token representations. While popular in VLMs, Q-former converged slowly and discarded too much spatial context, resulting in negative optimization and poor task success.\n- **MBS (ours):** With a window size of 20 and memory bank size of 5, we sample keyframes dynamically based on streaming input. Only 5 frames are stored and downsampled (224×224), while the current frame remains at 448×448. This design achieves the best success rate and competitive efficiency without retraining or excessive compute.\n\n**Why MBS excels for VLA:**\n\nUnlike prior methods that assume access to full video clips, our MBS is tailored for VLA’s streaming and causal nature. It operates online, selecting informative history frames incrementally without storing the entire trajectory. Furthermore, our approach incorporates **relative temporal encoding** and is fully compatible with closed-loop control settings.\n\nIn summary, MBS delivers superior performance with fewer frames, lower memory, and higher inference efficiency—making it particularly suitable for long-horizon, real-time VLA tasks.\n\n[3] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding, ICML 2025 \n\n[4] LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models, ECCV 2024\n\n## Q3: Though the teaser shows scenarios where actions can be ambiguous without considering the history, the experiments do not include such scenarios.\n\nThank you for the thoughtful question. While the teaser presents an extreme case of ambiguity, our motivation for incorporating history goes well beyond isolated corner cases. In practice, **temporally extended observations help simplify the action function** $A_t(input)$, making it smoother, more robust, and easier to learn.\n\nAlthough many tasks in LIBERO or our real-world experiments may appear deterministic at first glance, **single-frame observations often miss important context**—such as object trajectories, prior interactions, or occluded elements—that affect decision-making. In such cases, **historical frames enrich spatial and temporal context**, helping the model disambiguate subtle variations and better estimate scene dynamics.\n\nFor example, in **Task 4**, the robot needs to interact with multiple objects in sequence. While each sub-action may appear locally deterministic, the overall task involves **longer-horizon dependencies and state transitions** (e.g., remembering what has already been picked up). Including historical frames acts as a **lightweight memory mechanism**, allowing the model to maintain context over time and avoid redundant or erratic behaviors.\n\nWe observe empirically that **models with access to temporal context are more stable**, less likely to get stuck, and demonstrate **higher success rates**, even in tasks that may not explicitly require memory in a rule-based sense.\n\nIn summary, even in tasks that seem deterministic in design, **history helps bridge perception gaps**, supports smoother policy learning, and improves robustness in real-world execution.\n\n## Q4: The authors should provide more details about the proposed MV-Bench.\n\nWe appreciate the reviewer’s interest in MV-Bench. Below we clarify our benchmark design and dataset choices:\n\n**Benchmark Scope and Dataset Selection**\n\nMV-Bench is built upon the full **LIBERO-SPATIAL** benchmark, which is specifically tailored for evaluating **multi-view spatial reasoning** in VLA tasks. We chose to exclude **LIBERO-10**, **LIBERO-GOAL**, and **LIBERO-OBJECT** because rendering multi-view variants of these tasks would significantly increase training and evaluation costs, without introducing fundamentally new spatial challenges. LIBERO-SPATIAL already emphasizes **viewpoint variation**, making it a representative testbed for multi-view VLA. We plan to extend MV-Bench to include these additional subsets in future work to improve benchmark coverage.\n\n**Training Setup**\n\nFor each of the **10 tasks** in LIBERO-SPATIAL, we use the **provided 50 demonstrations per task**, and render each trajectory from **6 different viewpoints**: **0°, 60°, 120°, 270°, 300°, and 330°**. This results in a total of **300 training trajectories per task**.\n\nTo ensure view quality, we **exclude severely occluded views**, such as the **120° angle for Task 10**, where key interactions are blocked.\n\n**Evaluation Protocol**\n\nThe evaluation is split into two parts:\n\n- **In-View:** We evaluate on the **same 6 viewpoints** used during training, using the standard LIBERO-SPATIAL test layout. We similarly remove 120° for Task 10 due to occlusion.\n- **Cross-View:** We introduce **6 novel viewpoints** that were not seen during training: **15°, 30°, 45°, 75°, 90°, and 105°**. This tests generalization to unseen perspectives. We exclude **105° for Task 7** due to significant occlusion.\n\nThis protocol ensures both **robust multi-view training** and **cross-view generalization evaluation**, which are essential for assessing spatial consistency and viewpoint-invariance in VLA models.\n\n## Q5: Many relevant VLA works are not cited and discussed in the paper.\n\nThank you for pointing this out. We acknowledge that the current version does not cover all relevant VLA literature. We will revise the related work section to include and discuss additional recent works, particularly those focusing on VLA pretraining and multi-modal policy learning, in the revision."}, {"review_id": "Ts5vAWTkYs", "reviewer": "Reviewer_rfTc", "summary": "This work introduces 4D-VLA, a novel approach that leverages 4D information to overcome the challenges of diverse robotic datasets for pretraining, specifically addressing coordinate system chaos and state chaos. The method encodes sequential RGB-D images into visual features with corresponding 3D coordinates, aligning the robot's coordinate system with the scene. Furthermore, 4D-VLA incorporates Memory bank sampling, an efficient frame sampling strategy designed to extract informative and diverse key frames from sequences. The authors also propose MV-Bench, a multi-view simulation dataset. Extensive experiments validate the effectiveness of their method.", "questions": "mentioned above.", "limitations": "Overall, I find the core idea compelling; however, similar challenges appear to have been addressed more comprehensively in prior work, such as ARM4R. I am open to the authors' feedback and other reviewers' opinions.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. In general, the paper is clear and the technical details are easy to understand.\n2. The main idea is interesting.\n3. The reported performance is good.\n\n\nWeaknesses:\n\n**Novelty.** ARM4R has proposed a framework that integrates 4D spatiotemporal information for pretraining as the authors are suggesting. I don’t see any new benefit this work has been proposing on top of what ARM4R [1] already showed. I am happy to get clarifications about the comparison and the differences between the two works.\n\nAnother point that remains unclear is the distinction the authors make between 'coordinate system chaos' and 'state chaos.' The rationale behind this specific split is not well explained, and it is unclear why the authors chose to focus on these two aspects in particular\n\n[1] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n\n**Evaluation.** The authors should include comparisons beyond the OpenVLA baseline, as it is unclear whether the observed issues are specific to OpenVLA or representative of broader challenges across VLAs. For a more comprehensive evaluation, please consider including results from recent VLAs such as LLARVA [1] (another VLA) and ARM4R [2] (the first 4D-VLA) to assess how alternative approaches perform under similar conditions.\n\n[1] LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning, CoRL 2024.\n\n[2] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n\n**Ground-truth annotations.** It appears that ground-truth camera intrinsics and extrinsics are available. Were these annotations not expensive to obtain? This could pose challenges for scaling to large datasets.\n\n**Memory bank sampling.** Why not identify the keyframes in advance, selecting a minimal set of frames necessary to represent the video effectively? \n\n**Real world experiments.** The authors state that they manually collected 50 diverse trajectories. Given the limited number of demonstrations, it is understandable that baselines such as OpenVLA in Table 3 may struggle to perform well under such low-data conditions.\n\n\n**Related Work.** As previously noted, the paper would benefit from discussing more recent and relevant studies. Several new VLAs (e.g., LLARVA [1], LLARA) have been proposed, and notably, the recent ARM4R [2] work demonstrates how 4D representations can effectively address the pretraining challenges highlighted in this paper for robotic foundation models. Without proper citation and comparison to these works, it remains unclear how the proposed approach differs from or advances beyond existing methods.\n\n[1] LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning, CoRL 2024.\n\n[2] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n[3] LLaRA: Supercharging Robot Learning Data for Vision-Language Policy, ICLR 2025.", "was_revised": true, "final_justification": "After reviewing the authors’ feedback and other reviewers’ comments, I would like to thank the authors for their rebuttal. Most of my raised concerns have been addressed, thus I am raising my score to 4.", "rebuttal": "## Q1: **Novelty.**\n\nThank you for the opportunity to clarify. While both our work and ARM4R [1] utilize spatiotemporal information, the **pretraining paradigms, data settings, and core motivations differ substantially**.\n\n### Key Differences from ARM4R:\n\n- **Pretraining objective**:\n    \n    ARM4R adopts a **representation learning paradigm**, pretraining on diverse videos and later fine-tuning for action prediction. In contrast, our method follows an **end-to-end action-centric pretraining framework**, where the model directly learns to map from vision-language inputs to low-level actions during pretraining. This results in fundamentally different supervision signals, architectures, and deployment goals.\n    \n    **ARM4R uses full 3D point clouds** as structured input for pretraining a 3D point track prediction objective. This involves **dense point-level processing**, specialized encoders, and architectural complexity tailored to 3D trajectory regression. In contrast, **our model directly encodes RGB images with depth-aligned spatial coordinates**—represented as **2D patches augmented with (x, y, z) positional embeddings**. This lightweight formulation avoids full point cloud processing and **enables efficient spatial grounding in a vision-language transformer framework**.\n    \n- **Data assumptions and embodiment**:\n    \n    ARM4R pretrains on the **MonoHuman** dataset—primarily **human video data** in controlled monocular views. Our setup uses the **DROID robot dataset**, which contains diverse camera placements, egocentric views, and no guarantee of visible robot base or consistent reference frame. This introduces a unique challenge of **coordinate frame ambiguity** that ARM4R does not address.\n    \n- **Core motivation — coordinate system inconsistency**:\n    \n    Our work is centered around solving a **critical but underexplored challenge in robotic VLA pretraining**: in many RGB-based datasets, the robot’s base orientation is not visible, making the same instruction (e.g., “move the cup forward”) correspond to **inconsistent action vectors** across environments. We refer to this issue as **\"coordinate system chaos\"**, and we show in Appendix 7.1 (Fig. 6) that it severely degrades pretraining performance. ARM4R does not encounter or resolve this issue due to its dataset setting.\n    \n- **Depth vs. proprioception**:\n    \n    ARM4R does not use depth; instead, it uses **proprioceptive features** (joint angles, velocities) to model embodiment. In contrast, we explicitly use depth (or estimated geometry) to **anchor visual observations to a shared 3D coordinate system**, improving cross-scene generalization. This geometric alignment is especially crucial for **robot-centric control prediction**, which ARM4R does not attempt.\n    \n- **Temporal modeling design**:\n    \n    ARM4R processes historical context by **concatenating past frames** and feeding them into a Transformer, which can be computationally expensive and unstructured in memory use.\n    \n    We instead propose a **MBS** strategy that **selectively retains informative frames** in a lightweight, streaming-friendly queue (see 3.3 and Algorithm 1). This enables efficient long-horizon reasoning with bounded memory, and achieves better performance than uniform sampling under the same frame budget.\n    \n\nWhile both terms describe different forms of ambiguity, they **complement** each other and **jointly motivate** our design: 4D inputs (spatial + temporal + aligned geometry) help **resolve coordinate inconsistency** and **simplify policy learning**.\n\nWe believe these distinctions—in data, motivation, architectural design, and deployment focus—underscore the novelty and practical value of our work beyond prior efforts like ARM4R.\n\n## Q2: Another point that remains unclear is the distinction the authors make between 'coordinate system chaos' and 'state chaos.'\n\nThank you for raising this question. We introduce the terms **coordinate system chaos** and **state chaos** to clarify two distinct and pervasive sources of input inconsistency in VLA pretraining—both of which we have encountered in large-scale embodied datasets such as OpenVLA.\n\n- **Coordinate system chaos** refers to the spatial misalignment introduced by **invisible or unobserved robot base poses**, which leads to **inconsistent action supervision**. For example, the same instruction (e.g., “move forward”) can correspond to drastically different action vectors depending on the robot’s orientation—information often missing from RGB-only datasets. This is a dataset-level issue that arises due to **cross-scene variability and lack of global spatial anchoring**.\n- **State chaos**, on the other hand, arises from **temporal aliasing or partial observability**. That is, given only a single frame, it is often ambiguous to infer the agent’s true context or intent. For example, if the robot is returning from a failed attempt, the next best action may differ from that in a fresh state. This motivates our incorporation of **temporally extended inputs**, which we show empirically helps produce **smoother, more learnable policy functions** $A_t$ .\n\n## Q3: **Evaluation.**\n\nThank you for the valuable suggestion. We agree that broader comparisons would provide a more comprehensive understanding of the proposed method's effectiveness.\n\nWe have already completed the training of relevant baselines such as LLARVA and ARM4R under our evaluation setup. However, due to the tight rebuttal timeline, we were not able to finalize the inference and analysis in time for inclusion in this response.\n\nIf the review process allows for an extended discussion or revision period, we would be glad to update our submission with these results.\n\n## Q4: **Ground-truth annotations.**\n\nThank you for the question. Our pretraining relies on **robot action datasets**, where **camera intrinsics and extrinsics are routinely recorded** as part of the standard data collection pipeline. These calibrations are essential for tasks like visual reasoning or 3D reconstruction, and thus are **already available** in most real-world robot platforms.\n\nWe note that obtaining such annotations is **not prohibitively expensive**, as many modern camera systems (e.g., RealSense, Azure Kinect) provide tools for **automatic calibration** or come with factory-calibrated parameters. In fact, most datasets used in recent robotic learning works already include this metadata by default.\n\nTherefore, our approach **does not introduce additional annotation cost**, and we believe it remains scalable to large real-world datasets collected from embodied systems.\n\n## Q5: **Memory bank sampling.**\n\nThank you for the question. While selecting keyframes in advance is effective for **offline video summarization or fully observed settings**, our method targets a **closed-loop, streaming vision-language-action (VLA) setup**, where the model receives observations **causally** and must make decisions **in real-time**.\n\nIn this **online setting**, the full video is not available ahead of time, so **pre-identifying the \"minimal\" keyframes is not feasible**. Instead, we propose a **Memory Bank Sampling (MBS)** strategy that incrementally selects and retains the most informative frames, based on their utility for action prediction.\n\nThis design better reflects practical deployment scenarios (e.g., onboard embodied agents), where:\n\n- Observations arrive sequentially;\n- Memory and compute are limited;\n- The agent must reason with **partial past** and make forward predictions without future access.\n\nWe believe this streaming-compatible design is both **more realistic** and **more broadly applicable** to scalable, real-world VLA systems.\n\n## Q6: **Real world experiments.**\n\nThank you for the comment. While it is true that our real-world evaluation involves only 50 human-collected trajectories, **all baselines—including OpenVLA—are fine-tuned using exactly the same limited set**, ensuring a fair and controlled comparison.\n\nDespite this low-data regime, our model demonstrates **significantly stronger generalization and robustness**, which we attribute to our **4D-consistent pretraining strategy**. The ability to adapt quickly with minimal supervision is crucial for real-world deployment, and we view this as a key strength of our approach.\n\nWe also note that low-data evaluation is **common in prior embodied learning benchmarks**, as it better reflects practical constraints in robot deployment where large-scale annotation is expensive or infeasible.\n\n## Q7: **Related Work.**\n\nThank you for highlighting recent progress in VLA. We appreciate the contributions of ARM4R and will include a thorough discussion of such works in our revised related work section.\n\nWhile both our method and ARM4R incorporate 4D spatiotemporal signals, the **pretraining paradigms differ fundamentally**:\n\n- **ARM4R** adopts a **representation learning + fine-tuning** pipeline, using RGB, point clouds, and proprioception to learn general-purpose embeddings.\n- **Our method is end-to-end**, directly predicting actions from vision-language input, enabling **joint alignment of vision, language, and action** during pretraining.\n\nMoreover, **ARM4R relies on point clouds and robot-centric proprioception**, which, while powerful, limits compatibility with existing VLMs and broader datasets. Our method is **VLM-compatible by design**, and does not require 3D sensors—making it easier to scale and transfer across 2D vision datasets and real-world scenarios.\nIn addition, we introduce:\n\n- A **lightweight, streamable temporal sampling strategy** (Memory Bank Sampling), supporting online decision-making.\n- A concrete solution to **coordinate inconsistency** across large-scale robot datasets—an underexplored but critical issue that undermines pretraining effectiveness.\n\nIn summary, while sharing similar goals, our work offers a complementary, action-centric, and scalable alternative to ARM4R."}]

Author rebuttal:

Meta-review: This paper proposes 4D-VLA, a vision–language–action pretraining framework that addresses two key sources of input inconsistency in embodied learning: coordinate system misalignment and temporal ambiguity. The method augments RGB with depth (or estimated geometry) and introduces a memory-bank sampling (MBS) strategy tailored to streaming VLA tasks. The authors also propose MV-Bench, a multi-view evaluation suite. Experiments on LIBERO, MV-Bench, and real robots show consistent improvements over baselines.
Strengths

The paper tackles a fundamental and under-explored challenge in VLA pretraining: the lack of consistent robot-centric coordinate frames in existing datasets. The integration of depth/geometry for coordinate alignment and the lightweight MBS strategy are well-motivated and effective for streaming policy inference. Experimental validation is broad, spanning simulation, multi-view benchmarks, and real-world robot rollouts. The rebuttal provides detailed clarifications, extensive additional comparisons (e.g., against LLARVA and ARM4R on RLBench), and transparent discussion of limitations.

Several reviewers questioned novelty, noting similarities to prior works such as 3D-VLA and ARM4R, and raised concerns about limited evaluation scope and task simplicity. Some reviewers felt the terminology (“coordinate system chaos,” “state chaos”) was overstated and could be made clearer. The method leverages calibrated camera intrinsics/extrinsics, which is an additional source of supervision compared to some baselines. Real-world experiments used only 50 demonstrations, which some reviewers felt was too small to fairly assess baselines.

The authors’ rebuttal was detailed and constructive. They clarified that: 1) Their use of depth is fundamentally different from 3D-VLA, emphasizing spatial consistency for action prediction rather than scene imagination. 2) Differences from ARM4R in pretraining paradigm, input modalities, and architectural focus. 3) Why MBS is lightweight and streaming-oriented compared to common video sampling methods. They also added further RLBench comparisons, demonstrating superior performance and efficiency, and acknowledged limitations (e.g., reliance on camera calibration, small-scale real-world evaluation).

Three reviewers (jMXX, GHr5, 7q7T) raised concerns but did not further engage after rebuttal. Reviewer rfTc engaged extensively, acknowledged the clarifications, and ultimately raised their score to borderline accept. While the paper received borderline or negative initial ratings, the ACs believe the rebuttal and subsequent clarifications address the main concerns to a satisfactory degree, including those raised by reviewers who did not respond during rebuttal. The core contribution of resolving coordinate frame inconsistency in VLA pretraining is promising, and the experimental evidence supports its effectiveness. 

To conclude, this work makes a meaningful and timely contribution to the growing area of VLA pretraining and embodied foundation models. The ACs encourage the authors to incorporate their rebuttal clarifications, additional comparisons, and a more transparent discussion of limitations into the final version.
Decision: Accept (poster)