Title: 4D-VLA:  Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
Venue: NeurIPS 2025
Accepted: yes
Keywords: VLA

LLM
Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset’s action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution—an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce Memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance.  In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA.To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces 4D-VLA, a novel approach that leverages 4D information to overcome the challenges of diverse robotic datasets for pretraining, specifically addressing coordinate system chaos and state chaos. The method encodes sequential RGB-D images into visual features with corresponding 3D coordinates, aligning the robot's coordinate system with the scene. Furthermore, 4D-VLA incorporates Memory bank sampling, an efficient frame sampling strategy designed to extract informative and diverse key frames from sequences. The authors also propose MV-Bench, a multi-view simulation dataset. Extensive experiments validate the effectiveness of their method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. In general, the paper is clear and the technical details are easy to understand.\n2. The main idea is interesting.\n3. The reported performance is good.\n\n\nWeaknesses:\n\n**Novelty.** ARM4R has proposed a framework that integrates 4D spatiotemporal information for pretraining as the authors are suggesting. I don’t see any new benefit this work has been proposing on top of what ARM4R [1] already showed. I am happy to get clarifications about the comparison and the differences between the two works.\n\nAnother point that remains unclear is the distinction the authors make between 'coordinate system chaos' and 'state chaos.' The rationale behind this specific split is not well explained, and it is unclear why the authors chose to focus on these two aspects in particular\n\n[1] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n\n**Evaluation.** The authors should include comparisons beyond the OpenVLA baseline, as it is unclear whether the observed issues are specific to OpenVLA or representative of broader challenges across VLAs. For a more comprehensive evaluation, please consider including results from recent VLAs such as LLARVA [1] (another VLA) and ARM4R [2] (the first 4D-VLA) to assess how alternative approaches perform under similar conditions.\n\n[1] LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning, CoRL 2024.\n\n[2] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n\n**Ground-truth annotations.** It appears that ground-truth camera intrinsics and extrinsics are available. Were these annotations not expensive to obtain? This could pose challenges for scaling to large datasets.\n\n**Memory bank sampling.** Why not identify the keyframes in advance, selecting a minimal set of frames necessary to represent the video effectively? \n\n**Real world experiments.** The authors state that they manually collected 50 diverse trajectories. Given the limited number of demonstrations, it is understandable that baselines such as OpenVLA in Table 3 may struggle to perform well under such low-data conditions.\n\n\n**Related Work.** As previously noted, the paper would benefit from discussing more recent and relevant studies. Several new VLAs (e.g., LLARVA [1], LLARA) have been proposed, and notably, the recent ARM4R [2] work demonstrates how 4D representations can effectively address the pretraining challenges highlighted in this paper for robotic foundation models. Without proper citation and comparison to these works, it remains unclear how the proposed approach differs from or advances beyond existing methods.\n\n[1] LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning, CoRL 2024.\n\n[2] Pre-training Auto-regressive Robotic Models with 4D Representations, ICML 2025.\n\n[3] LLaRA: Supercharging Robot Learning Data for Vision-Language Policy, ICLR 2025.\n\nQUESTIONS:\nmentioned above.\n\nLIMITATIONS:\nOverall, I find the core idea compelling; however, similar challenges appear to have been addressed more comprehensively in prior work, such as ARM4R. I am open to the authors' feedback and other reviewers' opinions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces 4D-VLA, a VLA that leverages both depth information and historical context to improve the performance. They propose to convert depth information into point cloud in the robot frame and use learnable positional embeddings to encode the point cloud. The spatial embeddings are added to the 2D image embeddings to serve as spatial vision features.For temporal features, they propose a memory bank sampling strategy to balance the efficiency and effectiveness. They further propose a MV-Bench to evaluate the performance of the model across diverse viewpoints. Experiments on MV-Bench, LIBERO and real world show that 4D-VLA achieves state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-written and structured, making it easy to follow the proposed approach.\n2. The experiments are comprehensive, covering both synthetic and real-world datasets, which demonstrates the effectiveness of the proposed method.\n3. The motivation for using depth information and historical context is clearly explained, and the proposed method effectively integrates these features into the VLA framework.\n\nWeaknesses\n1. The way to use depth information resembles 3D-VLA[1]. It seems that both 3D-VLA and this paper convert depth information into point cloud using camera parameters and then use positional embeddings to encode the point cloud, which are later fused with the 2D image embeddings. It seems the only difference is that this paper uses learnable positional embeddings while 3D-VLA uses sin/cos positional embeddings. The authors should clarify the difference between the two methods.\n2. The baselines used to verify the effectiveness of the proposed memory bank sampling strategy are not sufficient. The authors only compare with the uniform sampling while there are many other sampling strategies in video understanding, such as spatiotemporal adaptive compression [2], grid pooling [3], using Q-Former, etc. The authors should include more baselines to demonstrate the effectiveness of their sampling strategy.\n3. Though the teaser shows scenarios where actions can be ambiguous without considering the history, the experiments do not include such scenarios. It seems that tasks in LIBERO benchmark and in real world experiments are all deterministic, i.e., the actions can be determined by the current observation without considering the history. The authors should explain why history helps these tasks, especially the Task 4 in real world experiments.\n4. The authors should provide more details about the proposed MV-Bench, such as how many demonstrations are used for each task, why LIBERO-10, LIBERO-goal and LIBERO-objects are excluded.\n5. Many relevant VLA works are not cited and discussed in the paper.\n\n\n[1] 3D-VLA: A 3d vision-language-action generative world model.\n[2] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding\n[3] LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models\n\nQUESTIONS:\nAnswer all the questions in Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes 4D-VLA. The proposed approach is an improved VLA architecture with 4D representations. It involves RGB-D inputs, a memory bank, coordinate alignment, and pre-training. The method shows consistent gains in tested settings.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed approach leads to consistent improvements in tested settings
- Different design decisions are ablated individually

Weaknesses:
- The proposed approach involves a number of engineered design decisions and is quite specific to robotic manipulation. It may be a better fit for a robotics venue than a general machine learning one.
- Discussion of and comparison to prior methods that address "state chaos" is missing. For example, [1] and [2] include past context for low-level observations and image observations, respectively.
- The demonstrated tasks are fairly simple and likely achievable with simpler methods that act on 2D observations. It would be good to demonstrate a result where 4D representations enable behaviors that have not been shown with 2D observations, which would help justify the added method complexity.
- The "system chaos" and "state chaos" terminology may be a bit too dramatic and something more descriptive may make the paper easier to parse.

[1] https://proceedings.neurips.cc/paper_files/paper/2024/hash/90afd20dc776bc8849c31d61a0763a0b-Abstract-Conference.html

[2] https://proceedings.mlr.press/v229/radosavovic23a/radosavovic23a.pdf

QUESTIONS:
Please see the weaknesses section.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
VLA pre‑training pipelines suffer from two kinds of input incompleteness: coordinate‑system chaos and state chaos.  To tackle both, the authors augment RGB with depth and an adaptive history sampler. The proposed 4D-VLA tries to bridge the gap of image‑only systems.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Authors show their results on LIBERO, MV‑Bench and real hardware with performance gains over baseline methods.
2. Viewpoint generalisation provided by the method is validated in multi‑view evaluation.

3. 3‑D positional embedding plus a lightweight memory bank sampling queue introduces few learnable parameters.

Weakness:
1. There is hardware dependence on RGB‑D. Many large‑scale demonstration datasets are RGB‑only.
2. Window-based keyframe sampling and spatio-temporal encoding are already widely adopted in vision-language model (VLM) research and have demonstrated effectiveness across multiple benchmarks. The paper would benefit from an elaboration of architectural novelty beyond these established techniques.

QUESTIONS:
1. Cite the window‑based sampler used in VLMs and briefly explain why MBS is lighter or more effective.

2. Are you planning to release your data, and what's your release plan?

LIMITATIONS:
This method assumes the hardware to be RGB-D.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-04