Pre-training Auto-regressive Robotic Models with 4D Representations
Authors: Dantong Niu
Yuvan Sharma
Haoru Xue
Giscard Biamby
Junyi Zhang
Ziteng Ji
Trevor Darrell
Roei Herzig
Venue: ICML 2025 poster (2025)
Keywords: Auto-regressive Robotic Models
Pre-training
4D Representations
Primary area: applications->robotics
TL;DR: We introduce a novel robotics pre-training approach that leverages low-level 4D representations by tracking 3D points over time in videos to achieve effective pre-training for robotics.

Abstract: Foundation models pre-trained on massive unlabeled datasets have revolutionized natural language and computer vision, exhibiting remarkable generalization capabilities, thus highlighting the importance of pre-training. Yet, efforts in robotics have struggled to achieve similar success, limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world. In this paper, we introduce ARM4R, an **A**uto-regressive **R**obotic **M**odel that leverages low-level **4**D **R**epresentations learned from human video data to yield a better pre-trained robotic model. Specifically, we focus on utilizing 3D point tracking representations from videos derived by lifting 2D representations into 3D space via monocular depth estimation across time. These 4D representations maintain a shared geometric structure between the points and robot state representations up to a linear transformation, enabling efficient transfer learning from human video data to low-level robotic control. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.

Pre-training Auto-regressive Robotic Models with 4D Representations
Dantong Niu
Yuvan Sharma" Haoru Xue
Giscard Biamby'
Junyi Zhang' Ziteng Ji
Trevor Darrell f
Roei Herzig t
htps://arm4r.githu...i
Abstract
models exhibit impressive zero-shot and few-shot learning capabilities (Radford et al., 2021; Ouyang et al., 2022; Wei et al., 2021; Chung et al., 2024), highlighting the power of pre-training on generic data. However, numerous attempts in robotics (Xiao et al., 2022: Kim et al., 2024; Zhen et al., 2024b; Niu al., 2024; Ye al., 2024) have yet to achieve the same pre-training success seen in other domains. This could potentially be attributed to the scarcity of large-scale, diverse robotic data, unlike the abundance of text and image data available for vision and language FMs.
Foundation models pre-trained on massive un- labeled datasets have revolutionized natural lan- guage and computer vision, exhibiting remarkable generalization capabilities, thus highlighting the importance of pre-training. Yet, efforts in robotics have struggled to achieve similar success, limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world. In this paper, introduce ARM4R, an Auto-regres Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre- trained robotic model. Specifically, we focus on utilizing 3D point tracking representations from videos derived by lifting 2D representations into 3D space via monocular depth estimation across time. These 4D representations maintain shared geometric structure between the points and robot state representations up to a linear transformation. enabling efficient transfer learning from human video data to low-level robotic control. Our exper- iments show that ARM4R can transfer efficiently from human video data to robotics and consis- tently improves performance on tasks across vari- ous robot environments and configurations.
The lack of robotic data poses a significant bottleneck in training foundation models that can effectively generalize across diverse robotic platforms and tasks. To overcome this limitation, several recent approaches (Xiao et al., 2022; Ye et al., 2024) employ representation learning by pre-training on an abundance of human data, enabling transfer to robotic systems. These approaches aim to recognize the inherent similarities between human and robot manipulation tasks and exploit the vast repositories of human video data avail- able on the internet. Yet, these approaches have not been able demonstrate effective generalization to downstream tasks. In part, this is due to their representations lacking an understanding of the physical world (Zhen et al., 2024a), and therefore being less effective for robotics.
In contrast with these methods, LirioonLLnguage-Aeee (VLAss models take slightly different approach, implic- itly leveraging human data in robotics by incorporating pre-trained components from Vision-and-Language Models (VLMs). In particular, they use language decoders pre- trained on tasks like visual question answering (e.g., RT- 2 (Brohan et al., 2023a)) and image captioning (e.g., Open- VLA (Kim et al., 2024)). Despite such efforts, there is a discrepancy between these models' high-level pre-training objective and the goal of enabling robotic models to handle low-level action prediction. While these initial objectives are valuable for comprehending visual and linguistic content, they don't "on't directly address the nuances of low-level robot control, which involves aspects like precise manipulation and spatial reasoning. To address this, this paper's 's method employs a lower-level pre-training objective by starting with model that utilizes next-token prediction learn 4D rep- resentations from human video data. These representations
1. Introduction
Recently, foundation models (FMs) have shown remarkable success, particularly in the domains of language (Brown al., 2020; Touvron al., 2023), vision (Kirillov et al., 2023), and multi-modal models (Chen et al., 2022; Alayrac al. 2022; Liu et al., 2023; Li et al., 2023; OpenAI, 2023) pre-trained on vast amounts of vision and text data. These
Equal contribution. Equal advising. Berrklee AI Re- search, UC Berkeley. <bias_88@ beo@bbrreley.cdu...
Correspondence to: Dantong Niu
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Pre-Train on 4D Representations from Human Video Data
"install the brake shoe"
Monocular Human Video
3D Point Tracks Prediction
Robotic Control
Execute Action Prediction
Input Tokens
Ouput Tokens
ARM4R
Zt
ZerT
"play basketball"
robot proprios
2t11
Final state
Figure I: Overview of ARM4R. We introduce an Auto-regressive Robotic Model that leverages low-level 4D Representations (3D point tracks across time) learned from human videos to yield better pre-trained robotic model.
can then be transferred to more specialized scenarios by fine- tuning on robotic scenes and subsequently on proprioceptive data, while maintaining same training objective.
method on average surpasses baselines like PerAct (Shrid- har et al., 2023) on RLBench, and LLARVA (Niu et al., 2024), OpenVLA (Kim et al., 2024), and To - (Pertsch et al., 2025) on real tasks with 7-DoF Kinova Gen3 robot; (iv) Our model also exhibits several advantageous proper- ties, including cross-robot generalization and 3D point track prediction for out-of-domain human and robotic videos.
In this paper, introduce ARM4R (Auto-regressive Robotic Model with 4D Representations). The key in- sight behind ARM4R is to learn low-level representation from the abundance of human video data that can capture properties of the physical world. This involves lifting 2D representations to 3D using monocular depth estimation and subsequently tracking the 3D points. The resulting rep- resentations maintain shared geometric structure - up to linear transformation between the 3D points and robot state representations used downstream, enabling efficient transfer learning from human video data to robotic manip- ulation tasks. Surprisingly, pre-training our method solely on human data yields superior results compared to other models like VI (Kim et al., 2024) that are pre-trained on robotic data such OpenX (Collaboration al., 2023).
2. Related Work
Vision-I Lanoonn-anna Models. VLAS are a type of robotic model that combines visual perception, language understanding, and action generation capabilities. VLAs take input visual observations along with language in- struction, and output a sequence of robot control actions. Several VLAs, such as LLARVA (Niu et al., 2024), Open- VLA (Kim et al., 2024). LLaRA (Li et al., 2024a), and RoboPoint (Yuan et al., 2024) directly fine-tune VLM to predict robot actions, often using special tokens to represent the action space. These models differ in the choice of VLM and the specific method used to encode robot actions, but they share the underlying principle of adapting a pretrained VLM for robotic control. A similar model is 3D-VLA (Zhen et al., 2024a), which consists of components for generating future states of an environment based on data that includes 3D information, such as point clouds. These existing VLAs utilize language decoders that have been pre-trained for high-level tasks like image captioning (Kim et al., 2024)
We summarize main contributions as follows: (i) We in- troduce novel robotics pre-training approach that incorpo- rates low-level 4D representations that enhance understand- ing of the physical world while also learning from unlabeled videos. (ii) Our approach shows that pre-training solely on human video data can lead better performance than other methods that are pre-trained only on robotic data; (iii) Our
ARM4R pronounced "armor".
2

---

Pre-training Auto-regressive Robotic Models with 4D Representations
and VQA (Brohan et al., 2023b), which may be inadequate for low-level robotic environments. In contrast, show that leveraging low-level vision representations from human video data can result in a better pre- rre-trained robotic model.
To address the data issue, another prominent pre-training approach is to leverage large-scale datasets of human videos. This harnesses the abundance of freely avail- able human activity data on the internet, offering a scal- able alternative to collecting expensive robot demonstra- tions. For example, Track2Act (Bharadhwaj et al., 2024) trains a 2D point-tracking model on human videos from Piticenss (Damen et al., 2018) and Something- Something-v2 (Goyal et al., 2017), then re-purposes it to guide robotic manipulation. Any-Point Trajectory Modeling (ATM) (Wen et al., 2024) similarly utilizes small set of human demonstrations to aid cross-embodiment transfer, though in a more task-speciiic setting and still relying on 2D motion. By contrast, our approach lifts 2D observations into 4D representations (3D plus time), which not only enhances spatial awareness and occlusion handling, but also allows pre-training on human videos at scale, providing broader applicability and more robust policy learning in robotics.
3D Motion Fields. Motion estimation spans from 2D opti- cal flow (Horn & Schunck, 1981) and object tracking (Wu et al., 2013) to recent dense point tracking (Harley et al., 2022). Moving from 2D 3D further enriches the geomet- ric understanding. Early work scene flow (Vedula al., 1999) estimates short-term 3D motion based on explicit 3D structure (Menze & Geiger, 2015) or depth images (Teed & Deng, 2021). More recently, SpatialTrackkr (Xiao et al., 2024) tackles long-range 3D point tracking by lifting 2D pix- els into 3D with monocular depth estimates and iteratively refining 3D trajectories with as-rigid-as-pe aspossible motion pri- ors. This 3D-driven strategy greatly improves occlusion robustness and yields impressive 3D point tracking results.
In robot learning, 2D motion fields have been used to enable fine-grained control, guiding manipulation and imitation learning (Goyal et al., 2022; Vecerik et al., 2023; Gu et al., 2023; Yuan et al., 2024; Zheng et al., 2024; Bharadhwaj et al., 2024; Xu et al., 2024). Despite their success, these approaches remain limited by the lack of geometric cues and less spatial awareness. In contrast, 3D motion fields of- fer more spatially grounded representations, enabling more efficient policy learning. ToolFlowNet (Seita et al., 2022) leverages scene flow to estimate tool trajectories in behavior cloning, though it uses only a relatively coarse 3D signal. We instead adopt dense 3D point tracking on diverse human videos, and use these rich 4D representations (3D points tracked across time) to pre-train a general auto-regressive robotic model with robust and versatile action generation.
3. Auto-regressive Robotic Models
To address the challenge of leveraging pre-trained vision representations from human video in robotic models, we present an auto-regressive model that relies on low-level 4D) representations. The model is trained in three stages. The first stage the pre-training stage focuses on learning gen- eralized low-level representations through 3D point tracking from human vidcos. In the second stage, the model is fine- tuned for the same task, but using a small amount of data for the robot that we intend to use in downstream tasks. Finally, the third stage fine-tunes the model for robotic control. We begin by discussing the preliminaries (Section 3.1), then introduce the architecture (Section 3.2), and the training procedures (Section 3.3). Our method is shown in Figure 2.
RVT (Goyal et al., 2023) and RVT-2 (Goyal et al., 2024) are recent transformer-basee methods that use a more direct approach for learning 3D information, leveraging ground- truth RGB-D images to reconstruct a scene's point cloud and then predict keyframes. This point cloud reconstruction relies on similar principle to our method, with ARM4R instead using pre-training on 3D point tracking to learn scene structure.
3.1. Preliminaries
4D Scene Representations. Our 4D representations result from solving the 3D point tracking problem, which involves finding the 3D coordinates of discrete points across time, given a monocular video consisting of T discrete frames. Formally, the objective is find pt as defined below:
Pre-training for Robotic Models. Pre-training has emerged as a crucial technique for improving the perfor- mance and generalization capabilities in robotics. Large- scale datasets such OpenX (Collaboration et al., 2023) contain diverse sensor modalities, tasks and action spaces across various robots. Models trained with these datasets, such as RT-1-X (Brohan et al., 2023c), RT-2-X (Brohan et al., 2023a), Octo (Team et al., 2024). VLe (Kim et al., 2024) and LLARVA (Niu et al., 2024), can be applied in various robot embodiments and tasks. Yet, these robot pre-training datasets are still orders of magnitude smaller than the data that current LLMs and VLMs are trained on.
Pt = (Zazt Mat 2)t) 0 j n
(I)
where n is the total number of points being tracked, and 0 < t < T. In solving this tracking problem, the identities of the points are fixed and consistent across all frames: the j-th point in Pt refers to the same physical point in 3D space across all time steps t E [0, T). To initialize these points, we define square grid of size g x y on the first frame (frame = O), resulting in = 9² points. The task is to track the 3D coordinates of these initial queried n points throughout the video while maintaining their unique identities.
4D Representations for Robotic Manipulators. One of
3

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Decoder
3D Point
Fine-Tuning:
Track Training
Causal Transformer
Robot Control
Pi+1
8
Attention Pooling
Language Encoder
Image Encoder
State Encader
Future State Encoder
3D Point Track Pre-training and Fine-tuning Stages
check temperature
x
x2296
x 1296
language nput
Image Input 2
Current Point Coordinates 1t
Future Point Coorcinates 1t+1
Robot Control Fine-tuning Stage
Future Robot
play basketball
Robot Proprios
Proprios
Language nput
Image input
Current Robot State St
Future Robot State 8
Figure 2: ARM4R trained in three stages. Top Grey Box: The first two stages focus on learning a scene-wide 4D representation by predicting 3D points across time, where Stage pre-trains on large egocentric human dataset (Epic- Kitchens 100), and Stage 2 fine-tunes tracking to robotic scene and camera. Bottom Grey Box: Finally, the model is fine-tuned to predict robot proprioceptive states rather than 3D points to enable robotic control.
smaller dataset (1-2K demonstrations) of robotic scenes, adapting the point
the main benefits of using 4D representations is that the 3D point tracks robotic setting are described by linear transformations constructed using the robot states. We now present simple proof of this claim.
ized by temporal sequences that capture the robot complet- ing particular task. The task is described by the language instruction The temporal sequences typically consist of visual observations 1o:7-1 and proprioceptive states s0:7-1- which can lie in Cartesian space joint position space. Then, the objective is to learn policy that predicts one or more future actions, conditioned on finite number of previous timesteps, to successfully complete given task.
Consider n-DoF open-chain manipulator, initially in its reference configuration 0, 0, 0,02 = 0...., On Suppose the manipulator is commanded to new position 0, = 04, 02 02 On = 0z.- Next, we take any arbi- trary point on the body of the manipulator lying between joints i, 1. Let p be described by the transformation (0) in joint i's frame. Then, p's new position in the base frame can be described by:
0.
Inputs. Given any video, we structure our models' input at timestep f, into three parts: the language instruction the image input i, and the current 3D coordinates of the tracked points, Pt. These elements together provide the contextual, visual, and spatial information necessary for 3D point track- ing. The output is the future 3D coordinates of the tracked points,
(0) cf ... c::0 gt.p
When fine-tuning the model for robotic control (see Section 3.3), we replace the tracked input points with the robot's current state &t, and the output points with the next state, $1+1- We hypothesize that the shared geomet- ric structure up linear transformation between the points and robot state representations enables efficient transfer learning between the second and third stages.
where ε, are the twists associated with each joint. Since this product of 4x4 SE(3) matrices represents a linear transfor- mation, points on the robot body as well as attached rigid bodies evolve through linear transformations in terms of the robot states, Additionally, for overactuated manipulators (7-DoF), the robot learning problem simplifies to tracking arbitrary trajectories, configuration space limitations are less restrictive. This allows the model to benefit more effec- tively from the 3D point tracking pre-training task.
3.2. Architecture
In the first and second training stages, objective is to develop auto-regressive model capable of predicting
Robotic Episodes. Robotic control can be formulated as a finite-horizon Markov Decision Process (MDP), character-

---

Pre-training Auto-regressive Robotic Models with 4D Representations
3D point tracks. The predictions are conditioned on the input (1, it, Pt) from a context window of C timesteps:
cessing points Pt with similar MLPs for processing robot states 8t- Additionally, when processing multiple images in the fine-tuning stage, we combine the observation tokens by concatenating linear transforms of the different views to get a single zybs token. The model is also trained to pre- dict multiple future proprioceptive states. The rest of the architecture is kept the same (e.g. loss function). For more details, please see Appendix C.I.
Tl, i (-----111,,---- Pt-C+1a) + Pt+1
(2)
During control fine-tuning, the objective changes slightly, as the model conditions on, and predicts proprioceptive states:
(1, 8t- St-G+1a) 1:t) + 1
(3)
Before being fed into the causal transformer for next-token prediction, each part of the input and output must be pro- cessed and projected into the same latent space. To achieve this, separate encoders are used for the language, image, points, and robot states, as we discuss below.
3.3. Training
As previously mentioned, ARM4R is trained in three stages: the first two stages focus on the 3D point tracking task for human and robot videos respectively, and the last stage focuses on robotic control. Next, we describe these stages.
Language Encoder. We use a frozen CLIP (Radford et al., 2021) text encoder trained on LAION-2B (Schuhmann et al., 2021) to process text, with learnable linear projection layer added at the end get the language token 21.
Stage 1: Human Videos Pre-training. In the pre- pre-training stage, we focus on learning 3D point tracking, since this task allows our model to leverage large-scale human video data with a representation that also transfers over to the robotic domain. Specifically, we train our model on 76K videos from the Epic- Sale-Kitchens1000 100 dataset (Damen et al., 201 8), which contains rich human-object interactions with 97 verbs and 300 noun classes. By training to predict 3D point tracks for such large-scale human data, ARM4R gains a deeper understanding of the spatial dynamics and physical interactions of different bodies and objects, knowledge that is critical for enhancing robotic models.
Image Encoder. To process the image, we use standard Vision Transformer (Dosovitskiy et al., 2021) to get the im- age token 2!Th. This VIT is frozen while training our model, and is pre-tr using CrossMAF on a combination of ImageNet (Deng et al., 2009) and the OpenX dataset (Col- laboration et al., 2023). This enables the vision transformer to learn to encode both non-robotic and robotic data, which is important since our pre-training stage emphasizes the former, while fine-tuning targets the latter.
To extract pseudo- annottaionnsssnns for 3D point tracks, an off- off-the-shelf tracker that generates 3D tracks for points arranged on a 9 X 9 grid. Points on the grid are initialized in the first frame of the video and tracked throughout the sequence. We note that the pscudo-labeled tracks are gener- ated in the camera coordinate frame, inherently capturing both object and camera motion due to the egocentric nature of the human videos. In contrast, our robotic applications typically involve stationary cameras and different object- hand interaction patterns, introducing discrepancies in both camera dynamics and embodiment. To reconcile these dif- ferences and ensure smooth transfer to robotic domains, we introduce a fine-tuning stage focusing on 3D point tracking in the downstream robotic setup.
4D Representations. To encode the point coordinates, we use a standard 2-layer MLP. The resulting feature 2/! is combined with zi via an attention pooling layer get the current observation token, 2ghhs A separate MLP is used to encode the next timestep's 's point coordinates and get é.
Causal Transformer. For each timestep, we get three tokens, one each for language, current visual observa- tion and prediction, which are fed into the transformer. In our implementation, we use a randomly initialized causal Transformer (ViT-Base). The Transformer is trained for standard next-token prediction on the sequence ..), ), with loss only being calcu- (zI, zobs St, R1, *t+1, , ...,,,..)
_obe
lated for 2, During inference, we input (zz, 2005) at timestep 0, and the model predicts & for every timestep.
Stage 2: Fine-tuning on 3D Point Tracking for Robotic
Decoder and Loss Function. We calculate the loss using only é, as predicting 2 and 2ل is not the objective in either 3D point tracking or robotic control tasks. The predicted token is decoded using two-layer MLP into the predicted point tracks pe+1- The L.1 distance between P and the ground truth point tracks pi+1 is used as the final loss:
Settings. After the pre-training stage on human video data, we fine-tune ARM4R for the same 3D point tracking task with videos from the robotic setup we use in the downstream application. We note that this fine-tuning only needs to be performed once for every robot setup for all tasks combined, with a modest amount of data (se 5 5-10% compared to Stage 1). This step helps transition from the camera dynamics and embodiment gaps between the human video pre-training and the control fine-tuning in the next stage.
1 n
L(p411, P(41)
liet1 P1+1l1a
(4)
Adaptation for Robotic Control. We note that when fine- tuning for robotic control, we replace the ML.ss for pro-
Stage 3: Fine-tuning for Robotic Control. Having trained
5

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Table 1: Success rate ( (%) on RLBench Multi-Task setting. We compare ARM4R's performance against several related baselines on 12 tasks from the RLBench benchmark. We use 25 episodes per task and 5 random seeds, averaging the results to get the success rate. ARM4R achieves the best performance on of 12 tasks and the best average success rate.
Task
Method
open
meat off' turn put push sweep slide close screw place reach and stack
Average Success Rate (%)
drawer
grill
tap
money buttons dustpan block jar
bulb
wine
drag
blocks
Image-Be BC (ViT)
16
0
16
0
2.67
C2FARM-BC
20
20
68
12
72
16
24
8
18
24
4
23.83
ManiGaussian
76
60
56
20
64
24
28
92
12
48.00
LLARVA
60
80
56
44
56
84
100
28
8
12
52
0
48,33
PerAct
80
84
80
44
48
56
72
60
24
12
68
36
55.33
ARM4R
88.8
94.4
61.6
92.0
67.2
72.0
85.6
24.0
10.4
36.0
77.6
4.0
59.47
the model 3D point tracking, we then fine-tune it for robotic control. In this stage, collect a number of robotic demonstrations depending on the downstream tasks. We note that we use significantly fewer demonstrations for real robotic tasks than other baselines (See Section 4.3). After collecting successful data of the robot performing the target task, we replace the current and predicted point tracks in the training process with current and predicted robot states.
2023). A task is defined as a collection of demonstrations of the robot interacting in given scene, with object variations (such as color or size). We train ARM4R for each task using 190 successful demos for every variation of the task (for more details, see Appendix D), and evaluate using 25 episodes per task in the validation set. Every episode is scored either 0 for failure or 100 for success. We use seeds, which are averaged get the final success rate.
Baselines. We compare to several baselines for our sim- ulation evaluation. Image-BC (ViT) is 2D language- conditioned baseline model that uses a ViT vision encoder, reported in PerAct (Shridhar et al., 2023). To compare against two different methods that use 3D representations, we select C2FARM-BC (James et al., 2022) and PerAct, which use voxels as 3D input calculate robot actions. To compare to a method with 3D temporal tracking similar to ours, we evaluate against ManiGaussian (Lu et al., 2025), which uses a dynamic Gaussian splatting representation to predict robot actions. Lastly, LLARVA (Niu et al., 2024) is a recent state-of-thh- thrt VIA that directly predicts low-level robot actions given an image and proprioceptive information as part of language prompt.
4. Experiments and Results
We evaluate ARM4R on 12 tasks in RLBench (James et al., 2020) and compare to relevant 2D and 3D baselines. We also test and ablate our model on two real robots: a 7-DoF Kinova Gen3 robot, and a 7-DoF Franka Emika Panda robot.
4.1. Implementation Details
ARM4R is implemented using PyTorch (Paszke et al., 2019). We use ViT-Base as our vision encoder, which is pretrained as described in Section 3.2. We use Spatial et al., 2024) as our off-the- 3D point tracker. We note that the model uses a maximum context window C, which is the number of previous timesteps it considers when pre- dicting the next action. In practice, we use C = 16 for most tasks, increasing it to C' 32 for some long-horizon tasks (details in Appendix D). The model is also trained to predict the next 16 actions, but we only execute the first prediction during evaluation. In both our simulation and real settings, we use end-effector control, with the model predicting the Cartesian position and rotation of the end-effector, and bi- nary value for the gripper. Finally, we use 4 NVIDIA A6000 GPUs for training and a single NVIDIA A6000 GPU for evaluation. More information, like training and fine-tuning recipes, is in Appendix C.2.
(Xiao
We report simulation results in Table 1. ARM4R achieves the highest average success rate across all the tasks, and the best success rate for 4 out of 12 tasks. In particular, ARM4R surpasses PerAct, which directly uses voxel information from the simulation environment as input. This approach is not scalable since voxel data is expensive to collect in the real world. Instead, ARM4R learns to model the 3D world by pre-training on 3D point tracking, and the impressive performance highlights the model's 'ss strong grasp of physical understanding. We also note that ARM4R's 's su- perior performance compared to LLARVA - a VLA which uses a pre-trained language decoder - emphasizes the ef- fectiveness of our representation and pre-tratee pre-training approach.
Results.
4.2. Simulation Evaluation
Failure Cases Analysis. We also found some abnormal behaviors when evaluating ARM4R on RLBench, which can be further summarized into two categories. (1) Un-
Experimental Setup. evaluate ARM4R on 12 RLBench tasks, and follow the settings in PerAct (Shridhar et al.,
6

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Table 2: Success rate (%) on the real Kinova Multi Task setting. We compare ARM4R's performance to four related baselines on 13 real tasks grouped into five categories. We use 25 episodes per task for evaluation, averaging the results over 3 seeds to get the final success rate, ARM4R outperforms both baselines on all the tasks.
pick cube up
destack
stack
Method
yellow
cyan
green
yellow
cyan
yellow on cyan
cyan on yellow
ATM
5.3 3,5
6.7 + 2.7
9.3 + 1.3
4,0 2 2.3
9.3 + 3.5
1,3 + 1.3
2.6 +: 1.3
LLARVA
44.4 = 6.4
41.6 4.2
54.2 + 11.0
8.3 = 4.8
10.3 2.6
5.6 2.8
12.8 + 2.6
To-FAST
63.0 33.0=3. 3.7
33.3 + 8.3
15.0:000
5.5 +2.8
2
22.2 = 2.8
25.6 16.8
OpenVLA
77.8 + 6.4
45.8 + 4.2
91.7 +: 8.3
55.6 +: 2.8
51.3 + 2.6
27.8 + 2.8
38.5 + 4.4
Ours
92.6 + 3.7
100 + 0.0
95.8 + 4.2
94.4 + 2.7
94.9 + 5.1
63.6 + 5.2
59.5 + 2.4
pick toys then place to target
push
Average
Method
spiderman
penguin
pig
play basketball
push red button
push red then blue
ATM
5.3 1.3
6.7 + 1.3
5.3 3.5
24.0 + 4.6
4.0 = 2.3
0.0 = 0.0
6.4 = 2.2
LLARVA
9.3 + 1.3
9.3 + 1.3
8.0 +
10.7 + 3.5
20.5 + 5.1
2.8 2.8
18.3 + 3.9
To- To-SATT
2.3
17.3 + 1.3
9.3 2.7
13.3 3.5
20.5 + 2.6
0.0 - 0.0
21.2 1: 3,0
OLe
2.7 + 1.3
17.3 + 1.3
2.7 2.7
49.3 +: 3.5
23.1 + 4.4
0.0 0.0
37,2 + 3,4
Ours
90.7 = 1.3
94.7 + 1.3
93.3 + 1.3
92.0 + 2.3
84.6 + 4.4
25.0 +: 4.8
83.1 + 3.0
natural rotation: we examined a new task,"pu "put knife" in RLBench and found our model struggles to succeed this task. Interestingly, this simulated task features an unnatu- ral whole-arm rotation to grasp the knife handle, based the expert demonstrations created by the simulator's 's mo- tion planning. As the movement between two key points is very long, we hypothesize that our model has difficulty learning this rotation. (2) Lack of Precision: we observed that ARM4R struggles with the 'screw bulb' task, pro- cedure requiring precise insertion into the bulb holder. In contrast, model effectively performs other standard pre- cision tasks, such as 'open drawer' which necessitate only standard precision.
FAST and OpenVLA are state-of-the-art VLA models pre- trained on the OpenX dataset, while ARM4R trained on significantly smaller dataset, with pre-training consisting exclusively of human video data. More details for these implementations are in Appendix D.1.
Results. Table shows that ARM4R outperforms all base- lines across all tasks, achieving an average success rate of 83.1 % compared to Open 37.2% and ATM's 6.4% %% ATM in particular does not perform well in our real setting despite training with significantly larger number of demon- strations than we use in our fine-tuning. We believe that this significant gap in performance is due to how we track points: ARM4R utilizes 3D coordinates, while ATM relies on 2D. The use of 3D coordinates provides a more natural and accurate representation for robotic tasks, which may contribute to our model's improved performance.
4.3. Real Robot Evaluation
Experimental Setup. For our real experiments, we use
7-DoF Kinova Gen3 robot mounted with Robotiq 2F-85 adaptive gripper. We test our model and the baselines on 13 total tasks, grouped into five broad categories based on the dominant action: pick, destack, stack, pick and place, and push. For each task, training performed using 190 episodes of every variation. Evaluation is conducted over 25 episodes per task, with results averaged across three different seeds to calculate the final success rate.
In contrast to ATM, Open and To -AST use a similar number of fine-tuning episodes to our evaluation setting. However, we believe that our superior performance over these baselines can again be attributed to our use of low-level 4D representations, which enable 3D scene understanding.
4.4. Ablation Studies
We conduct ablations to assess the importance of human video pre-training (Stage 1), and the robotic fine-tuning (Stage 2). All model versions in this section include robotic control fine-tuning (Stage 3). For this, we train the following versions: (i) Stages 1, 2, and 3; (ii) Stages and 3; (iii) Stages 2 and 3, and (iv) Stage 3 only.
Baselines. We evaluate our model against four baselines in real-world settings: ATM (Wen et al., 2024), LLARVA (Niu et al., 2024), FAST (Pertsch et al., 2025) and Open- VLA (Kim al., 2024). ATM utilizes hierarchical frame- work to predict 2D point trajectories, which are then used to condition a policy, while LLARVA uses 2D point pre- diction as an auxiliary task for learning action prediction. In contrast, ARM4R predicts 3D point trajectories, a more intuitive and natural representation for robotic tasks. To-
Human Video Pre-Training. Figure 3 shows that the model with all stages performs better on all tasks than the Stages 2+3 model, indicating that pre-training on the human dataset provides a large benefit compared to only training

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Pick
Destack
Stack
100
100
100
90
Success Rate (%)
Success Rate (%)
70
70
Success
60
60
60
50
50
50
40
40
Stages 1+2+3
Stages 1+3
Stages 2+3
3 Only
Figure 3: Ablation Study for Stages and We train ARM4R on three real tasks in the Kinova setting, ablating Stages 1 and 2. The results indicate that while both stages improve performance, Stage has more significant impact.
Table 3: Pre-training approaches comparison. We com- pare ARM4R to several other robotic models that leverage pre-training on three tasks with a Kinova robot. We find that approach yields the best average success rate.
robustness noise. More experiments are in Appendix A.
The Effectiveness of Pre-training. In order to study the effectiveness of pre-training on the 3D point track predic- tion task, we take three tasks from our real setting: pick cube, destack cubes, and stack cubes, and compare to other works that use pre-training. MVP focuses on pre-training the vision encoder using human data, while RPT focuses pre-training with visual and proprioceptive states. Octo, which is transformer-
Method
pick cube
stack cubes
destack cubes
MVP
75.00
18.75
81.25
RPT
87.50
31.25
93.75
Octo
56.25
12.50
37.50
ATM
7.11
2.00
6.67
policy, is pre-trained on the OpenX dataset, similar to the VLA models LLARVA and Open VLA. Lastly, ATM pre-trains a 2D point track trans- former whose output used condition a policy.
VLe
68.75
31.25
53.33
LLARVA
93.75
56,25
100.00
ARM4R
96.0 + 2.3
61.3 + 1.3
94.7 + 1.3
The results are shown in Table 3. can be seen that our pre- training improves performance over the baselines. ARM4R outperforms other representation learning based pre-training methods, such as MVP, RPT, Octo and ATM, validating the benefits of using a 4D point-tracking based representation. In addition, while the two VLA baselines (OpenVLA and LLARVA) perform well, our approach still surpasses their results, possibly demonstrating the importance of using low- level representations opposed to language decoders that were pre- traitreined high level vision- vision-language language tasks.
for 3D point tracking on robotics videos. The performance boost observed when adding Stage 1 to Stage 3 is greater than the boost from adding Stage 2 to Stage 3, indicating that 4D pre-training on human videos provides a larger in- crease in performance than robotic videos. The key resulting insight is that when sufficient robotic pre-training data is unavailable, human video data can be viable alternative, provided the proper 4D representations are used.
Robotic Video Fine- TaneTTning.. The ablation results shown in Figure 3 reveal that adding robotic video fine-tuning (Stages 2+3; green) leads to improved performance over models trained solely for robotic control (Stage pink). Adding Stage to the training regime still improves perfor- mance, as the model performing all stages (blue) yields the highest success rate. As mentioned in Section 3.3, Stage is useful addressing the distribution shift and embodiment gap when switching from human to robotic data.
Table 4: Success rate (%) of ARM4R on cross-robot set- ting. We fine-tune the pre-trained model for motor control on different robots and report success rates of cube tasks.
Pre-train Epic
FT
Robot
pick
stack
destack
Kinova Kinova 96.0 + 2.3 61.3 + 1.3 94.7 + 1.3
Franka 73.3 4: 2.7 49.3 + 5.8 65.3 + 3.5
Epic
Kinova Franka 93.3 + 1.3
56.0 + 2.3
97.3 + 1.3
Kinova Franka 81.3 1.3 52.0
2.3 73.3
4.5. Additional Experiments
We conduct additional experiments to assess the benefits of pre-training, generalization of 3D point representations, and
Generalization from Kinoya to Franka. In order to study how our low-level 4D representations can help model
8

---

Pre-training Auto-regressive Robotic Models with 4D Representations
generalize across different robots, we perform an ablation experiment involving fine-tuning ARM4R on Kinova robot videos, and fine-tuning for control on 7 DoF Franka Emika Panda robot. We note that besides having different robots, the two setups also have very different configurations, as the Kinova robot is mounted on a stand as part of a bimanual setup, while the Franka robot is mounted on a table,
Table 6: Performance under different real-world robust- ness conditions.
Noise Source
Pick
Stack
Destack
Standard
96.0.1..3
61.3.11.3
94.7.11.3
Dim Light
1..7
52.0+2.3
81.3+2.7
Background Distractor
9.3
1..31:.3
90.711..
Table-top Distractors
81.3111.
48.0:2.3
74.72:1.3
Despite these significant differences, the results in Table 4 show that adding the human video pre-training and Kinova video fine-tuning improves the average performance on the Franka robot by 19.6%. This supports our claim that the 4D representations are sufficiently generalizable to transfer across different robotic setups.
Robotic Model that leverages low-level 4D representations by lifting 2D representations into 3D using monocular depth estimators, and tracking 3D points in videos. Our results in simulation and real-world setups show that our method consistently outperforms existing methods across diverse robotic tasks, showcasing its superior transferability and generalization capabilities. More broadly, our approach shows that training solely on human video data can lead to better performance than methods like Open VLA that are pre-trained on robotic data alone. This suggests that ef- fective pre-training can be achieved without the need for large-scale robotic datasets by bridging the gap between human-centric visual data and robotic applications, unlock- ing new possibilities for scalable and data-efficient robotics. As we continue to explore the boundaries of representation learning for robotics, ARM4R lays a foundation for future research into autonomous systems that can learn from the vast repository of human experience available in video data.
Robustness Analysis. We conduct an experiment to test how ARM4R handles dynamic changes in its environment. We test the model three cube tasks (Pick, Stack, and Destack) in our real Kinova setting, manually moving the cube during rollout when the arm reaches 1/3 and 2/3 of its descent. The results are presented in Table 5, and show con- sistent performance despite movement of the target object. This highlights the model's robustness to environmental and object-level changes, which we believe is partly due to the fact that it predicts each step in trajectory and thus interacts with the environment continuously.
Table 5: Performance in dynamic environments. The cube disturbed at varying phases of the robot trajectory.
Disturbance Time
Pick
Stack
Destack
6. Limitations and Future Work
No Disturbance
61.3 96.0=2.3 94.7=3.5
94.7+11.3
At 1/3 Descent
60.0+0.0
1.3
While ARM4R offers substantial benefits for pre-training with human video data for robotic learning, it is impor- tant to recognize certain limitations that accompany our approach. Our approach tracks 3D points in camera co- ordinates, leading to learned representations that combine object and camera motion. This coupling makes it difficult for the model to disentangle the two, leading to potentially inaccurate predictions due to a lack of invariance to cam- era intrinsics and motion. An improvement addressing this concern could involve pre-training 3D tracks in world co- ordinates, leveraging recent dynamic SLAM methods such as MonST3R (Zhang et al., 2024) or MegaSAM (Li et al., 2024b), which we leave for future work. Other improve- ments could include scaling the pre-training data to help the model generalize better to different camera viewpoints, or using multi-view fusion reduce the dependency on a sin- gle viewpoint to improve robustness to occlusions. Another line of future work could focus on selectively tracking only relevant or moving points instead of fixed uniform grid across frames. This would allow greater resolution in areas with small objects, and also help the model focus on objects critical to the task, improving its ability to disentangle object motion from background noise and camera movement.
At 2/3 Descent
92.0=2.3
1..3
9..7
To further assess ARM4R' robustness in dealing with factors like camera noise, occlusions, and sensor drift, we perform additional evaluations on the three cube tasks under the following conditions: (1) Dim lighting: ambient light is reduced 50%, (2) Background distractors: dynamic back- ground changes, such as people walking by or movement of background curtains, and (3) Tabletop distractors: two random objects are placed near the target. The results are presented in Table 6. Overall, ARM4R demonstrates robust- ness to lighting and background changes. This is likely due to the use of attention pooling, which guides the model to focus on the target region rather than background features. This hypothesis is supported by the observed performance drop when tabletop distractors are introduced.
5. Conclusion
In this work, we demonstrate that our pre-training ap- proach from human video data to robot learning is effective in addressing longstanding challenges of robotic learning pre-training. We introduced ARM4R, an Auto-regressive

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Acknowledgments
July 2023a. URL /tattpp//axiiii org/ abs 15818. arXiv:2307.15818 [es].
We would like to thank Abrar Anwar, Chancharik Mitra, and Yida Yin for their helpful feedback and discussions, as well as Rodolfo Corona for his assistance with data collection. We also thank Nicole Walters for creating our lovely logo. This project was supported in part by BAIR's industrial alliance programs.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding. T., Driess, D., Dubey, A., Finn, C., Florence, P., Fu, C., Arenas, M. G., Gopalakr- ishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, L., Lee, T.-W. E., Levine, S., Lu, Y. Michalewski, H., Mordatch, I., Pertsch, K., Rao, K., Reymann, K., Ryoo, M., Salazar, G., Sanketi, P., Sermanet, P., Singh, J., Singh, A., Soricut, R., Tran, H., Vanhoucke, V., Vuong, Q. Wahid, A., Welker, S., Wohlhart, P., Wu, J., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. RT-2: Vision-language-actinn models transfer web knowledge to robotic control, 2023b. URL http: http//aavvvv.//// abs 2307 15818.
Impact Statement
In this paper, we demonstrate how pre-training on low- level 4D representations from human video data can benefit robotic action prediction. This suggests that effective pre- training can be achieved without the need for large-scala scale robotic datasets by bridging the gap between human-centric visual data and robotic applications, unlocking new possi- bilities for scalable and data-efficient robotics. Finally, we note that this paper presents work whose goal advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Her- zog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jack- son, T., Jesmonth, S., Joshi, N. J., Julian, R., Kalash- nikov, D., Kuang, Y. Leal, I., Lee, K.-H., Levine, S. Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K., Ryoo, M.. Salazar, G., San- keti, P., Sayed, K., Singh, J., Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke, V., Vega, S., Vuong, Q.. Xia, F., Xiao, T., Xu. P., Xu, S., Yu, T., and Zitkovich, B. RT-1: Robotics Transformer for Real-World Control at Scale, August 2023c. URL http: / aarpaaaxvvvv.o.....aa..a.// abs/ 06817. 7. arXiv:2212.06817.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M.. Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198, 2022. URL https: / wwwpppissmttttttttaaeeeee.
Brown, T., Mann, B. Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry,
G., Askell, A., Agarwal, S., Herbert- A., Krueger,
Bharadhwaj. H., Mottaghi, R., Gupta, A., and Tulsiani, S. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation, 2024. URL
G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https::/procedingss neurips. htw.ppssccc/papeee/00///1ee 14 1457000bbfoo443771111bballlll112 pdf.
http:://arx...orr/aass///2. 2aw..b//22550055.00
httt:://axivve wwtp:/wwwwrg/////////
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., Florence, P., Fu, C., Arenas, M. G., Gopalakr- ishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N., Julian, R., Kalashnikov, D., Kuang. Y.. Leal, I., Lee, L.. Lee, T.-W. E., Levine, S., Lu, Y., Michalewski, H., Mordatch, I., Pertsch, K., Rao, K., Reymann, K., Ryoo, M. Salazar, G., Sanketi, P., Sermanet, P., Singh, J., Singh, A., Soricut, R., Tran, H., Vanhoucke, V., Vuong, Q. Wahid, A., Welker, S., Wohlhart, P., WU, J., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. RT-2: Vision- Laiion-anguage-ACce Models Transfer Web Knowledge to Robotic Control,
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint wwXiv2229.0674/4 2022.
10

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetune language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
T., Chung, T., Jain, V., Vanhoucke, V., Zhan, W., Zhou,
W., Burgard, W. Chen, X., Chen, X., Wang. X., Zhu, X., Geng, X., Liu, X., Liangwei, X., Li, X., Pang, Y., Lu, Y., Ma. Y. J., Kim, Y., Chebotar, Y., Zhou, Y., Zhu, Y., Wu, Y., Xu, Y., Wang, Y., Bisk, Y., Cho, Y., Lee, Y., Cui, Y., Cao, Y., Wu, Y.-H., Tang, Y., Zhu, Y., Zhang, Y., Jiang, Y. Li, Y., Li, Y., Iwasawa, Y., Matsuo, Y., Ma, Z., Xu, Z.. Cui, Z. J., Zhang, Z.. Fu, Z., and Lin, Z. Open X- Embodiment: Robotic learning datasets and RT-X models. https: /tt:://axxii www.xiivorrg/oo www.oog///....ooooo..//........e atl///as/2223 /21..././...0.......... 2023.
Collaboration, O. X.-E., O'Neill, A., Rehman, A., Mad- dukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang. A., Kolobov, A., Singh, A., Garg, A., Kem- bhavi, A., Xie, A., Brohan, A., Raffin, A., Sharma, A., Yavary, A., Jain, A., Balakrishna, A., Wahid, A., Burgess- Limerick, B., Kim, B., Schölkopf, B., Wulfe, B., Ichter, B., Lu. C., Xu, C., Le, C., Finn, C., Wang, C., Xu, C., Chi, C., Huang, C., Chan, C., Agia, C., Pan, C., Fu, C., Devin, C., Xu, D., Morton, D., Driess, D., Chen, D., Pathak, D., Shah, D., Büchler, D., Jayaraman, D., Kalash- nikov, D., Sadigh, D., Johns, E.,, Foster, E., Liu, F., Ceola, F., Xia, F., Zhao, F., Frujeri, F. V. Stulp, F., Zhou, G., Sukhatme, G. S., Salhotra, G., Yan, G., Feng, G., Schi- avi, G., Berseth, G., Kahn, G., Wang. G., Su, H., Fang, H.-S., Shi, H., Bao, H., Amor, H. B., Christensen, H. I., Furuta, H., Walke, H., Fang, H., Ha, H., Mordatch, I., Radosavovic, I., Leal, I., Liang. J., Chakaa, J., Kim, J., Drake, J., Peters, J., Schneider, J., Hsu, J., Bohg, J., Bingham, J., Wu, J.. Gao. J., Hu, J., Wu, J., Wu, J., Sun, J.. Luo, J., Gu, J., Tan, J., Oh, J., Wu, J., Lu, J., Yang, J., Malik, Silvério, J., Hejna, J., Booher, J., Tompson, J., Yang. J. Salvador, J., Lim, J. J., Han, J., Wang, K., Rao, K. Pertsch, K. Hausman, K., Go, K., Gopalakrishnan, K. Goldberg. K., B Byrne, K., Oslund, K., Kawaharazuka, K. Black, K., Lin, K., Zhang. K., Ehsani, K., Lekkala, K., Ellis, K., Rana, K., Srinivasan, K., Fang. K., Singh, K. P., Zeng, K.-H., Hatch, K., Hsu, K., Itti, L.., Chen, L. Y., Pinto, L., Fei-Fei, L., Tan, L., Fan, L. J., Ott, L., Lee, L., Weihs, L., Chen, M., Lepert, M., Memmel, M., Tomizuka, M., Itkina, M., Castro, M. G., Spero, M., Du, M., Ahn, M., Yip, M. C., Zhang. M., Ding, M., Hco, M., Srirama, M. K., Sharma, M., Kim, M. J., Kanazawa, N., Hansen, N., Heess, N., Joshi. N. J., Suenderhauf, N., Liu, N., Palo, N. D., Shafiullah, N. M. M., Mees, O., Kroemer, O., Bas- tani, O., Sanketi, P. R., Miller, P. T., Yin, P., Wohlhart, P., Xu, P., Fagan, P. D., Mitrano, P., Sermanet, P., Abbeel, P., Sundaresan. P., Chen, Q. Vuong, Q. Rafailov, R., Tian, R., Doshi, R., Mart' in- in, R., Baijal, R., Scalise, R., Hendrix, R., Lin, R., Qian, R., Zhang, R., Mendonca, R., Shah, R., Hoque, R., Julian, R., Bustamante, S., Kirmani, S., Levinee, S., Lin, S., Moore. S., Bahl, S., Dass, S., Son- awani, S. Song. S., Xu, S., Haldar, S., Karamcheti, S., Adebola, S., Guist, S., Nasiriany, S., Schaal, S., Welker, S., Tian, S., Ramamoorthy, S., Dasari, S., Belkhale, S., Park, S., Nair, S., Mirchandani, S., Osa, T., Gupta, T., Harada, T., Matsushima, T., Xiao, T., Kollar, T., Yu, T., Ding, T., Davchev, T., Zhao, T. Z., Armstrong, T., Darrell,
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J.. Per- rett, T., Price, W., and Wray, M. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 2188225,,00 Icee, 2009.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Goyal, A., Mousavian, A., Paxton, C., Chao, Y.-.., Okom, B., Deng. J., and Fox, D. IFOR: Iterative flow mini- mization for robotic object rearrangement, 2022. URL http:
org/ abs 2ttpb///222.00322. 00732.
Goyal, A., Xu, J., Guo, Y., Blukis, V., Chao, Y.-W., and Fox, D. Rvt: Robotic view transformer for 3d object manipulation. In Conference Robot Learning, pp. 694- 710. PMLR, 2023.
Goyal, A., Blukis, V., Xu, J., Guo, Y., Chao, Y.W.., and Fox, D. Rvt-2: Learning precise manipulation from few demonstrations, arXiv preprint EXXi:24466008455 2024.
Goyal, R., Kahou, S. E.,, Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Muuller-Freitage M., et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pp. 5, 2017.
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang. H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrish- nan, S. K., Ryan, F., Sharma, J., Wray, M., Xu, M., Xu, E. Z., Zhao, C., Bansal, S., Batra, D., Cartillier, V., Crane, S., Do. T., Doulaty, M., Erapalli, A., Feichtenhofer, C., Fragomeni, A., Fu, Q., Fuegen, C., Gebreselasie, A., Gonzalez, C., Hillis, J., Huang. X., Huang, Y., Jia, W., Khoo, W., Kolar, J., Kottur, S., Kumar, A., Landini, E,, Li,

---

Pre-training Augo-regressive Robotic Models with 4D Representations
C.,, Li, Y., Li, Z., Mangalam, K., Modhugu, R., Munro, J. Murrell, T., Nishiyasu, T., Price, W., Puentes, P. R., Ramazanova, M., Sari, L.,, Somasundaram, K., Souther- land, A., Sugano, Y. Tao, R., Vo, M., Wang. Y., Wu, X., Yagi, T., Zhu, Y., Arbelaez, P., Crandall, D., Damen, D., Farinella, G. M., Ghanem, B., Ithapu, V. K., Jawa- har, C. V., Joo, H., Kitani, K., Li, H., Newcombe, R., Oliva, A., Park, H. S., Rehg. J. M., Sato, Y. Shi, J., Shou, M. Z., Torralba, A., Torresani, L., Yan, M., and Malik, J. Ego4d: Around the World in 3,000 Hours of Egocentric Video. CoRR, 10.///100.0058.. 2021. URL ht tittps://arxit http://axxxvvv....ree. abs/
W.-Y., Dollar, P., and Girshick, R. Segment anything. In Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV), pp. 4015-4026, October 2023.
Li J., Li, D., Savarese, S., and Hoi, S. BLIP-2: boot- strapping language-imann image pre- pre-training with frozen image encoders and large language models. In ICMI, 2023.
Li, X., Mata, C., Park, J., Kahatapitiya, K., Jang, Y. S., Shang, J., Ranasinghe, K., Burgert, R., Cai, M., Lee, Y. J., and Ryoo, M. S. LLaRA: Supercharging robot learning data for vision-language policy, 2024a. URL httt://arxiv.co wttp//////////.//ee//e//// 20095.
Gu, J., Kirmani, S., Wohlhart, P., Lu, Y., Arenas, M. G., Rao, K., Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z.. Sun- daresan, P., Xu, P., Su, H., Hausman, K., Finn, C., Vuong, Q. and Xiao, T. RT-Trajectory: Robotic Task Gen- eralization via Hindsight Trajectory Sketches, Novem- ber 2023. URL http: /tts://axxiiee arwaa.iivv.oo. www.iii.org//ass.. 22311 01977. arXiv:2311.01977 [cs].
Li, Z., Tucker, R., Cole, F., Wang, Q.. Jin, L.. Ye, V. Kanazawa, A., Holynski, A., and Snavely, N. Megasam: Accurate, fast, and robust structure and motion from ca- sual dynamic videos. arXiv preprint arXiv:2412.04463. 2024b.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023.
Harley, A. W., Fang, Z., and Fragkiadaki, K. Particle video revisited: Tracking through occlusions using point trajec- tories. In ECCV, pp. 59-75. Springer, 2022.
Lu, G., Zhang, S., Wang, Z., Liu, C.,, Lu, J., and Tang, Y. Manigaussian: Dynamic gaussian splatting for multi- task robotic manipulation. In European Conference on Computer Vision, pp. 449-366. Springer, 2025.
Horn, B. K. and Schunck, B. G. Determining optical flow. Artificial intelligence, 17(1-3):185-200 1981.
Menze, M. and Geiger, A. Object scene flow for autonomous vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3061-3070, 2015.
Hu, E. J. Shen, Y., Wallis, P., Allen-llnniiee Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.096 09685, 2021.
Niu, D., Sharma, Y., Biamby, G., Quenum, J., Bai, Y., Shi, B., Darrell, T., and Herzig, R. LLARVA: Vision-action instruction tuning enhances robot learning, 2024. URL http: /ttp://arxian wwtpp/wwwwo/////////....../ 11815.
James, S., Ma, Z., Arrojo, R., and Davison, A. J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3066, 2020.
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https::/appp.... http://pp..seenannccccccoooooo// org http:/oopp1I:::5555222
James, S., Wada, K., Laidlow, T., and Davison, A. Coarse- to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13739-13748, 2022.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems,
Karaev, N., Rocco, I., Graham, B., Neverova, N.. Vedaldi, A. and Rupprecht, C. Cotracker: It is better to track together. In ECCV, pp. 18-35. Springer, 2025.
2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Kim, M. J., Pertsch, K., Karamcheti, S. Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., Vuong. Q.. Kollar, T.. Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. OpenVVLA An open-source lision-lagguag-eee action model, 2024. URL http: /ttp://axxiii org 220/ 09246.
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, I.,, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for vision- vision-language-eee language-action models. arXiv preprint arXiv:2501.09) 097447 2025.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo,
12

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 5748-8763. PMLR, 2021.
Wen, C., Lin, X., So, J., Chen, K., Dou, Q. Gao, Y., and Abbeel, P. Any-point trajectory modeling for policy learn- ing. 2024. URL http: /aaxiv.iivvv.////a org/ abs/ 2401. 00025.
Wu, Y., Lim, J., and Yang. M.-H. Online object tracking: benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2411-2418, 2013.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111. .2..02214,,,, 2021.
Xiao, T., Radosavovic, I., Darrell, T., and Malik, J. Masked visual pre-training for motor control. arXiv preprint wrxiv:203.06173. 2022.
Seita, D., Wang. Y., Shetty, J., L.,E, Y., Erickson, Z., and Held, D. ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds, Novem- ber 2022. URL http: arxiv. org/ abs abs/2 09006. arXiv:2211 1.09006 [cs].
Xiao, Y., Wang, Q. Zhang, S., Xue, N., Peng, S., Shen, Y., and Zhou, X. SpatialTracker: Tracking any 2d pixels in 3d space, 2024. URL http:/t/tariiiiee 2404. 04319.
/abs/
Shridhar, M., Manuelli, L., and Fox, D. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Leurning, pp. 785-799. PMLR, 2023.
Xu, M., Xu, Z., Xu, Y., Chi, C., Wetzstein, G., Veloso, M., and Song, S. Flow as the cross-domain manipulation interface, 2024. URL https: / /aaa/aaxvvvv.o/..a/aaa abs/ 2407. 15208.
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
Ye, S., Jang, J., Jeon, B., Joo, S., Yang. J., Peng, B., Man- dlekar, A., Tan, R., Chao, Y.-W., Lin, Y., Liden, L., Lee, K. Gao, J., Zettlemoyer, L.. Fox, D., and Seo, M. Latent action pretraining from videos, 2024. URL https: wwwaaaaivvvvv.. org abs / 221.2111111111
Teed, Z. and Deng, J. Raft-3d: Scene flow using rigid- motion embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9374-8384, 2021.
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R. Murali, A.. Mousavian, A., and Fox, D. RoboPoint: vision-languan model for spatial affordance prediction for robotics, 2024. URL http:/t///// /ttp//xxvvvvoo// org /bs/ 2406610722
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Ham- bro. E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13371, 2023. URL https://api.sematticccc...
Zhang, J., Herrmann, C., Hur, J., Jampani, V., Darrell, T., Cole, F., Sun, D., and Yang, M.-H. Monst3r: A simple ap- proach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825 2024.
Zhen, H., Qiu, X., Chen, P., Yang, J., Yan, X., Du, Y., Hong, Y., and Gan, C. 3d-VLA: A 3d vision-language-action generative world model, 2024a. URL /tat:://arxx: org/ abs 22402.43..0 09631.
Vecerik, M., Doersch, C., Yang, Y., Davchev, T., Aytar, Y., Zhou, G., Hadsell, R., Agapito, L., and Scholz, J. Robo- TAP: Tracking arbitrary points for few-shot visual imita- tion, 2023. URL http: http::://aaxvv...///e/eepe wwwpwiwww.........aaoaereebebeee wwt.:/o////a/////eeeee 15975.
Zhen, H. Qiu, X., Chen, P., Yang, J., Yan, X., Du, Y., Hong, Y., and Gan, C. 3D-VLA: 3D Vision-Language-Action Generative World Model, March 2024b. URL http:// arxiv. wwwp 2403 0 09631. arXiv:2403.06631 [cs].
Vedula, S., Baker, S., Rander, P., Collins, R., and Kanade, T. Three-dimensional scene flow. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pp. 722-729. IEEE, 1999.
Zheng, R., Liang, Y., Huang, S., Gao, J., III, H. D., Kolobov, A., Huang. F., and Yang. J. Tracevla: Vi- sual trace prompting enhances spatial-temporal aware- ness for generalist robotic policies, 2024. URL https: / //arxivv waa.ii...rg/bb///41111
Wei, J., Bosma, M., Zhao, V. Y.. Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2/09. 009.0165,, 2021.
13

---

Pre-training Auto-regressive Robotic Models with 4D Representations
Appendix
tasks. It features diverse range of object interactions, fine- grained action labels, and naturalistic, unseripted activities, making it particularly valuable for studying human-object interactions and long-term temporal dependencies.
Here, we provide qualitative results of our 3D point tracking for both in-domain and out-of-domain videos (Appendix A), statistics of the used datasets (Appendix B), implementation details (Appendix C), evaluation details (Appendix D), and robotic setup details (Appendices E and F).
The dataset includes diverse hand-object interactions, de- scribed by combinations of 97 verbs (for the hand mo- tions) with 300 nouns (for the object categories). In our human video pre-training stage, we use almost all the la- beled episodes available in the original dataset. We first subsample videos at f0s an experimentally chosen rate, as the original 50 50,fps provides unnecessary redundancy for slow movements. We then model the duration distribution of all 75,886 episodes, and filter out Rs 1% of episodes that are of length > 256 frames. As a result, we get a final set of 75,041 episodes for pre-training. For each episode, we use simple 'verb + noun' instruction derived from the official annotation files.
A. Additional Experiment Results
A.1. Qualitative Results of 3D Points Tracking.
We conduct additional experiments to evaluate our pre- trained model's ability track 3D points. Specifically, run inference few randomly chosen episodes from Spic-Kitchens100 (Damen et al., 2018) (in-domain human videos), Ego4D (Grauman et al., 2021) (out-of-domal hu- man vidcos), Kinova robot vidcos (in-domain robot videos) and Open X Embodiment (Collaboration et al., 2023) (out- of-domain robot videos).
B.2. RLBench Robot Episodes
In Figure 4, we present the tracking results on human videos from a version of our model that has undergone human video pre-training (Stage 1). The top two rows show the results for an episode from PitcheKs (in-domain human videos) with the action "stir potatoes." The bottom two rows display monocular human videos and their corresponding 3D point tracking predictions for an episode from Ego-4D (out-of- domain human videos) with the action "pick up plate".
RLBench (James et al., 2020) is large-scale benchmark dataset for robotic learning, designed to facilitate research in vision- based reinforcement learning and imitation learning, It consists of a diverse set of robot manipulation tasks per- formed in a simulated environment using a Franka Emika Panda arm. The dataset provides high-quality demonstra- tions with multi-modal observations, including RGB images, depth maps, and proprioceptive data.
In Figure 5, we present the tracking results robot videos, from a version of our model that has undergone human video pre-training on BitieKKitcennsl as well as robot video fine-tuning on Kinova demonstration videos (Stage 1+2). The top four rows display monocular robot videos and their corresponding 3D point tracking predictions for two episodes from in-domain Kinova robot videos, with the actions "push red button" and "place spiderman into bowl" respectively. The bottom two rows show the results for episode from the Autolab subset of the OpenX Embodiment dataset out--of-domall robot videos) with the action "pick the tiger and place it into bowl."
In our experiments, we use 128 x 128 resolution images for training. For most tasks, we use the 'front rgb' and "wrist rgb' views for point track and control fine-tuning. However, in some cases, we find that using other views yields better performance (details on task-specific implementations are provided in Appendix C). For robot control, we use end- effector control:
(x. y; z, Oz, 0y. 0z), where (1r, y, is the position and (0z, Oy 0z) the Euler angles for orientation. We also have one- meediimnnioonl binary element to control the gripper. For language instructions, we use variation O from the official list of instructions for all tasks. We do not subsample episodes for our RLBench experiments.
These visualizations verify that our model not overfit to a certain dataset or robotic setup, but can in fact generalize well to new videos.
C. Additional Implementation Details
C.1. Architecture of Auto-regressive Model
B. Additional Dataset Details
Here, we provide details on processing the visual input of the auto-regressive model to support two views when adapting to robot control fine-tuning. The images from both views are fed separately into the image encoder to obtain the image embeddings 2;Th for each view. Each view is then pooled using attention pooling with the state embeddings to form the image tokens 2065, Next, we project each token to half of its original hidden dimension (768- - 384 in our implementation) and concatenate them to obtain the final
B.1. Spic-KKichhnsS00
Epic-Kitchens/00 (Damen et al., 2018) is a large-scale, ego- centric video dataset designed for action recognition and understanding in daily kitchen activities. Captured from a first-person perspective using head-mounted cameras, the dataset provides rich, untrimmed video recordings of in- dividuals performing various cooking and kitchen-related
14

---

Pre-training Auto-regressive Robotic Models with 4D Representations
"stir potatoes"
Spic-Kitchens
"pick up plate"
Ego4D
Figure 4: Visualization of ARM4R's 3D Point Track results on randomly chosen Epic-Kitcher (in-domain) and Ego-4D (out-of-domain) human videos.
image tokens, incorporating information from both views.
script for ARM4R training. In most cases, we follow the task setup of PerAct (Shridhar et al., 2023) and use the 'front rgb' and 'wrist rgb' views. We use C = 16 (C' is the context window of the auto-regressive model). The detailed task-level configuration is provided below.
C.2. Training Recipes
We used the following hyperparameters for the three stages of training:
Open Drawer. The task is to open one of three drawers. The success metric is full extension of the prismatic joint of the target drawer. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is -
Table 7: Training Hyperparameters for the three stages.
Hyperparameter
Stage
Stage
Stage
Learning Rate
10
10 4
10
Meat off Grill. The task is to take either a piece chicken or steak off the grill and put it on the side. The success metrie the placement of the specified meat on the side, away from the grill. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is C = 16.
Weight Decay
10
10
10
Batch Size
256
256
256
Number Epochs
20
10-50
note that for Stage 3, we trained our model for variable number epochs depending on the downstream task, until the loss converged.
Turn Tap. The task is to turn either the left or right handle of the tap. Left and right are defined according to the orien- tation of the faucet. The success metric is the joint of the specified handle being at least 90º away from the starting position. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is 16.
D. Simulation on RLBench
We evaluate our model on 12 tasks RLBench for our simulation setup. Each task includes multiple variations, and we generate 190 episodes using their data generation
Put Money. The task is to pick up the stack of money and
15

---

Pre-training Auto-regressive Robotic Models with 4D Representations
"push red button"
Kinova
"place spiderman into bowl"
Kinova
"pick the tiger and place it into bowl"
Autolab (OpenX)
Figure 5:
Visualization of ARM4R's 3D Point Track results randomly chosen Kinova (in-domain) and Open X- Embodiment (out-of-domain) robot videos.
place it on the specified shelf of safe. The safe has three shelves: top, middle, and bottom. The success metric is the placement of the stack money on the specified shelf in the safe. We use the 'front rgb' and "overhead rgb' views.
The context window of the model is C' 16.
Push Buttons. The task is to push the colored buttons the specified sequence. There are always three buttons present in the scene, whose colors sampled from 20 options, and
16

---

Pre-training Auto-regressive Robotic Models with 4D Representations
the number of buttons to press is between one and three. The success metric is all specified buttons being pressed in the right order. We use the 'front rgb' and "wrist rgb' views. The context window of the model is C' = 32.
specified color. The other three squares are considered distractors. The success metric used is some part of the block being inside the target's area. We modified this task to sample the target color from a list of three colors (maroon, magenta, teal). The colors for distractor squares are still sampled from 20 options. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is C = 16.
Sweep Dustpan. The task is to sweep the dirt particles into the specified dustpan. There are two dustpans, one short and one tall, and both are always present in the scene. The success metric is all five dirt particles being inside the specified dustpan. We modified this task by adding variation with a different-sized dustpan. We use the 'front rgb' view only, repeated twice, for this task. The context window of the model is C = 16.
Stack Blocks. The scene starts with 8 blocks and green platform. Four of the blocks are of target color, and the other four have a distractor color. The task is to stack N blocks of the target color on the green platform. The suc- cess metric is N blocks being inside the area of the green platform. We use the 'front rgb' and 'wrist rgb' views. The context window of the model = 16.
Slide Block. this task there is a block and four colored squares in the scene (green, blue, pink, and yellow). The task is to slide the block onto either the green or pink squares. The success metric used is some part of the block being the specified target square. The original task only had one target square, and we modified it by adding three additional colored squares- - one target and two distractors. We use the 'front rgb' view only, repeated twice, for this task. The context window of the model is C 16.
D.1. Baselines in Real Experiments
ATM. We reproduce ATM (Wen et al., 2024) as a baseline for our Kinova real-world experiment setup, following the provided code and instructions 2. In the first stage, we use all Kinova robot episodes (5 tasks, each with 200 episodes per variation) to train a track transformer using ground truth point tracks generated by Co-Tracker (Karaev et al., 2025). In the second stage, we take the best checkpoint of the track transformer to train a policy for each task separately, consistent with ARM4R's real-world setup. ATM uses a 7-dimensional joint pose and one-dimensional gripper state in its policy. To adapt it to data format, we modify the implementation to end-effector control, predicting 3- dimensional (z, U) position, a 4-dimensional quaternion rotation, and a 1 1-dimensional gripper state.
Close Jar. The task is to screw in the lid on the jar with the specified color. There are always two colored jars in the scene, one target jar and one distractor jar. The success metric used is the lid being on top of the specified jar and the robot gripper not grasping any object. We modified this task so that the target jar color is drawn from a list of three possible colors (red, maroon, and lime) ). The color for the distractor jar was still chosen out of 20 options. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is C' = 32.
We also test Open (Kim et al., 2024) on our Kinova robot setup, following their fine-tuning code and instructions 3. We fine-tune OpenVLA using LoRA (Hu et al., 2021), with rank 32 and batch size of 16, train- ing until convergence. To adapt Open to our control setting, we convert our absolute proprioceptive states to 3- dimensional delta position and 3-dimensional delta rotation (Euler angles), with an additional binary gripper dimension. We subsample our original collected data at ratio of 10, since the difference between consecutive steps in the orig- inal data is too small for delta control, given the accuracy limit of OpenVLA (10-3).
OpenVLA
Screw Bulb. There are two bulb holders of different colors, and the task is to pick up a light bulb from the stand specified by color and screw it into the bulb stand. The color of the target holder is sampled from two colors, while the color of the distractor holder is sampled from the original 20 color options. The success metric used is the bulb from the specified holder being inside the bulb stand. We modified this task to use three colors for the target holder (yellow, purple and silver) rather than 20 as in the original task specification. We use the 'front rgb' and 'wrist rgb' views. The context window of the model is C 16.
LLARVA. We reproduce LLARVA (Niu et al., 2024) as a baseline for our real-world Kinova experiment setup. fol- lowing the provided code and instructions". We fine-tune their released pre-trained checkpoint on our Kinova demon- strations for single-task policies (5 tasks, each with 200 episodes per variation). We use their default learning rate
Place Wine. The task is to pick up the wine bottle and place it at the specified location in a wooden rack. The rack has three locations: left, middle, and right. The success metric is the placement of the bottle on the specified location in the rack. We use the front rgb' and "wrist rgb' views. The context window of the model is C C=11 16.
Reach and Drag. The environment has cube, stick, and four possible colored target squares. The task is to pick up the stick and use it to drag the cube to the target square of
hntps://giitubb.com//aggg Toajoo/ooooaaeeerrrrrrr----oo TrajectryyModeelAATA Moode/ATTM
17

---

Pre-training Auto-regressive Robotic Models with 4D Representations
and batch size, and train for 4 epochs for each task.
task objectives, demonstrations, and accompanying task instruction labels. Domain randomization is applied to di- versify robot home position, grasping approach trajectory, and target pose.
FA-FAST. We fine-tune To FOSTAST baseline for our Ki- nova real-world experiment setup, following the provided code and instructions. We use their released base autore- gressive T--TA model for fine-tuning single-task policy (200 demonstrations per variation). We use joint position control instead joint velocity control, fine-tuning the pre- trained model to predict the absolute 7-DoF joint pose plus 1-DoF gripper status. We follow the default learning rate used in their implementation.
E.3. Task Building
We build 5 tasks under the Kinova real robot setup. The configuration of each task and its variations are shown in Fig- ure 7. The details of each task are described as follows.
Pick Cube. The episode starts with the arm in the home position. The robot moves pick up the cube, placed at random location on the table within the manipulator's workspace. The episode recording stops after the robot picks up the object and moves up by certain distance.
E. World Kinova Experiments
E.1. Hardware
head camera
The episode starts with the arm in the home position. After picking up the cube from point as the pick cube task, the robot stacks on another object already present in the scene at point B according the instruction. The episode recording stops after the robot releases the object and moves up by certain distance.
Stack Cubes.
side camera
Destack Cubes. The episode starts with the arm in the home position. After picking up the top cube from stacked pair at point A as in the grasping task, the robot moves the grasped cube to another location, point B. The episode recording stops after the robot releases the object and moves up by certain distance.
Pick & Place Toys/Baskethal. The episode starts with the arm in the home position. After picking up the toy described in the instruction from point A, the robot moves and places the toy into a bowl or basket at point B. The episode recording stops after the robot releases the object and moves up by certain distance.
Figure 6: The real-world experiment setup of Kinova robot.
For our primary real-world experiments,
a Kinova Gen3 7 DoF manipulator with a Robotiq 2F-85 gripper, shown in Figure 6. It is mounted on a base that mimics the human shoulder orientation and height.
Push Buttons. position. Following the instruction, the arm moves to specific height above the assigned button at point A, closes the gripper, pushes the button, then moves to push another button at point B. The episode recording stops after the robot releases the object and moves up by certain distance.
The episode starts with the arm in the home
We set up two cameras (Logitech BRIO 4K camera) to observe the table-top manipulation scene. One is mounted an ego-centric pose, and the other mounted on the side of the table.
We use the Movelt motion planning framework for inverse kinematics and end-effector position control. It takes the end effector position objective from the model, and executes linear trajectories in the Cartesian space.
F. Real- Franka Experiments
F.1. Hardware
We use Franka Emika Panda robot with a Franka gripper for real robot data collection and evaluations. The Log- itech BRIO 4K cameras positioned to the left and right of the Franka robot provides double-view RGB (without depth data) vision input to our model, as shown Figure 8. Camera autofocus is disabled, and the data is captured 640x480 resolution.
E.2. Data Collection
To collect task demonstrations, we develop an automated data collection procedure to record episodes of these demon- strations. In this procedure, we give the ground truth loca- tions of all objects on the table, and procedurally generate
"hanps//gihuu...om/Pyyyye http://hhaaccmmm/yyyccc/ee/ennna Intelligencelopnpii
18

---

Pre-training Auto-regressive Robotic Models with 4D Representations
pick cube
destack cube
stack cube
push button
pick & place spiderman
pick place penguin
pick place pig
play basketball
Figure 7: Task building of real-world Kinova setup.
in Figure 8. The details of each task are described as follows.
camera
Pick Cube. The episode starts with the arm in the home position. The robot moves to pick up the cube from point A. The episode recording stops after the robot picks up the object and moves up by certain distance.
right camera
The episode starts with the arm in the home position. After picking up the cube from point as in the grasping task, the robot stacks it on another object already present in the scene at point B according to the instruction. The episode recording stops after the robot releases the object and moves up by certain distance.
Stack Cubes.
Destack Cubes. The episode starts with the arm in the home position. After picking up the top cube from stacked pair at point A as in the grasping task, the robot moves the grasped cube to another location, point B. The episode recording stops after the robot releases the object and moves up by certain distance.
Figure 8: The real-world experiment setup of Franka robot.
F.2. Data Collection
We use the data collection code and process from SeaigihhbbccommMaaa//aaaaaaaaa
collect data for automated tasks. The script generates data for an arbi- trary number of episodes. For each episode, the process generates x-y positions on the table plane using a uniform random distribution for each axis. The script directs the robot to place the object at each location and then collects the camera and joint information as the robot moving.
F.3. Task Building
We build cube tasks under the Franka real robot setup. The configurations of each task and its variations are shown
19


Reviews:
[{"review_id": "v6j5lGcpa4", "reviewer": "Reviewer_aq5E", "summary": "The paper proposes to pre-train the vision-language-action models using the sparse 4D trajectories (obtained as pseudo-labels using pre-trained networks) of humans performing actions in ego-centric videos. Such pre-training is claimed to be helpful when the model is fine-tuned on the downstream task of action trajectory prediction. The paper is well written. The idea presented in the paper is interesting and well-motivated, and the aimed contributions are timely. The main problem of the paper is that the experimental results are not convincing.  Thus, the offered conclusion could be misleading.\n\n**Update** I have upgraded my rating to weak accept. Thank you for the rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We address the concerns raised below:\n1. **Table 1 Results:** As noted by the reviewer, the simulation results for ARM4R are slightly skewed by the high success rate for the “put money” task. We recalculate the success rate without this outlier for ARM4R and the closest competitor, PerAct: Our model achieves a success rate of 56.51%, while Peract achieves a marginally lower success rate of 56.36%. We note that PerAct also has access to ground truth voxel information from the simulator, while our model learns 3D understanding from its own training, only using RGB images at inference time.\n\nTo further evaluate our model against other 3D-based approaches, we also compare it with 3D-VLA—a 3D-based VLA model that generates goal point clouds. Since the official training and inference codebase is not publicly available, we compare ARM4R to 3D-VLA using three RLBench tasks for which results are reported in the 3D-VLA paper. The results are shown as follows:\n\n|Method|put knife|pick up cup|take umbrella|Average|\n|--------------------|-----------|-------------|----|-------------|\n|Image-BC ViT|0|4|0|1.33 |\n|C2FARM-BC|4|28|48|26.67|\n|LLARVA|4|52|32| 29.33|\n|PerAct|20|56|76|50.67|\n|3D-VLA|68|40|52|53.33|\n|**ARM4R**|28.8±1.50|99.2±0.80|82.4±2.03|70.13±1.53|\n\nWe see that ARM4R performs much better than all other tested models on 2/3 tasks, and the average performance is better as well. The poor performance on the put knife task is explained by the nature of data collected. This simulated task features an unnatural whole-arm rotation to grasp the knife handle, based on the expert demonstrations created by the simulator’s motion planning. Due to the large movement between the first two keyframes, we hypothesize that our model struggles to learn the rotation because the context window becomes saturated with local motion details, leading to a loss of global trajectory information. This is why LLARVA struggles with this task as well, since it has to predict the whole trajectory without skipping any step’s action. In contrast, 3D-VLA predicts keyframes and thus does not suffer from this issue. Finally, since our model is pre-trained on human data and this rotation is very unnatural, it might be more difficult to learn.\n\n2. **Table 2 Results:** We add two more baselines to our real Kinova setting: (1) LLARVA, a 2D-based VLA, and (2) Pi0-FAST, a recent SoTA VLA model. The results are below:\n\n|Method|pick yellow|pick cyan|pick green|destack yellow|destack cyan|stack yellow|stack cyan|\n|:-----------:|:--------------:|:--------------:|:---------------:|:---------------:|:---------------:|:---------------------------------:|:--------------:|\n|LLARVA|44.4 $\\pm$ 6.4|41.6 $\\pm$ 4.2|54.2 $\\pm$ 11.0|8.3 $\\pm$ 4.8|10.3 $\\pm$ 2.6|5.6 $\\pm$ 2.8|12.8 $\\pm$ 2.6|\n|Pi0-FAST|63.0 $\\pm$ 3.7|33.3 $\\pm$ 8.3|25.0 $\\pm$ 0.0|5.5 $\\pm$2.8|25.6 $\\pm$ 2.6|22.2 $\\pm$ 2.8|25.6 $\\pm$6.8|\n|**Method**|**pick spiderman**|**pick penguin**|**pick pig**|**play basketball**|**push red button**|**push red button, then blue button**|\n|LLARVA|9.3 $\\pm$ 1.3|9.3 $\\pm$ 1.3|8.0 $\\pm$ 2.3|10.7 $\\pm$ 3.5|20.5 $\\pm$ 5.1|2.8 $\\pm$ 2.8|   \n|Pi0-FAST|16.0 $\\pm$ 2.3|17.3 $\\pm$ 1.3|9.3 $\\pm$ 2.7|13.3 $\\pm$ 3.5|20.5 $\\pm$ 2.6|0.0 $\\pm$ 0.0|\n\nBoth models exhibit low performance: LLARVA / Pi0 achieves an average success rate of 18.3% / 21.2%. This is likely due to the amount of fine-tuning data used: LLARVA typically requires 800 episodes per task (we have here 13 tasks), Pi0-FAST shows good results when tuned for the whole DROID dataset (~76K demos), while we only have a total of 2,600 training episodes for our real setting. For robotics, collecting a large amount of demonstrations is expensive and difficult, so learning efficiency is a key performance indicator.\n\n3. **Table 4:** We added an extra experiment for Franka robot which includes: (1) Kinova 3D points tracking stage (2) Franka robot joint fine-tuning. The results are shown as follows:\n\n|pretrain|finetune|robot|pick|stack|destack|\n|----------|----------|--------|--------------|--------------|--------------|\n|-|Kinova|Franka|81.3$\\pm$1.3|52.0$\\pm$2.3|73.3$\\pm$2.7|\n\nThe results show that only training on Kinova 3D points tracking can also boost results when a Franka robot is used in the downstream setting, although the increase in performance is not as big as the one compared to when human video pre-training is also used. This aligns with the hypothesis that both Stage 1 and Stage 2 are important to achieve the best possible performance.\n\n4. **Dataset preparation:** We provide details on human video dataset preparation in Appendix B1. Regarding camera motion, static cameras are a subset of dynamic cameras; thus, our robotic settings are not out of distribution with respect to the pre-training. \n\n5. **Additional Experiments:** Due to space constraints, please refer to points 2 and 3 in response to reviewer vJYs for experiments on handling distractors and dynamic environments.", "claims_and_evidence": "No. The experimental results arguably support the claimed contributions, For example:\n\n1. In Table 1, the major gain is due to one single action, \"put money\". It is unclear if such high success, for one particular case, is merely due to statistical insignificance or due to the nature of that particular action. Otherwise, the proposed method does not seem to be helpful, contrary to the claim of the paper.\n\n2. Table 2 has limited methods, which obscures the complete picture. \n\n3. Table 4 does not show what happens if Epic is not used but Kinova is used.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are meaningful. The issue is not on this part.\nInstead, the problem stems from the fact that the statistical significance of the experimental results is questionable.", "theoretical_claims": "There is no theoretical claim that needs to be checked.", "experimental_designs_or_analyses": "The experimental designs are valid. The paper follows the standard evaluation protocol.", "relation_to_broader_scientific_literature": "The paper aims to make a timely contributions.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The presentation of the paper and the visualization within are of high quality and helpful.", "other_comments_or_suggestions": "The details of the human dataset preparation are missing. The fact that the human videos consist of distractors, moving camera/head, and bimanual manipulations, they may need to be taken into the account.", "questions_for_authors": "Please provide more details of the data preparation and validate your claims through more rigorous experimental results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZZYQVxeN3", "reviewer": "Reviewer_Zx5c", "summary": "The paper introduces ARM4R (Auto-regressive Robotic Model with 4D Representations), a novel approach to pre-training robotic models using 4D representations derived from human video data. The key idea is to leverage 3D point (generated by depth estimation) tracking from human videos to create a low-level representation that captures the physical world's properties. This representation is then transferred to robotic control tasks, enabling the model to generalize across different robotic environments and configurations. ARM4R is pre-trained on large-scale human video datasets (e.g., Epic-Kitchens100) to learn spatial dynamics and physical interactions, and the training process can be split into three stages: stage1: pre-training on human video data to learn 3D point tracking, stage2: fine-tuning on robotic video data to adapt the model to the specific robotic setup, stage3: fine-tuning for robotic control tasks using proprioceptive data. ARM4R outperforms existing methods on both simulated (RLBench) and real-world robotic tasks, demonstrating superior generalization and transfer learning capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We address the concerns raised below:\n\n1. **Comparison to RVT, RVT-2:** As the reviewer noted, RVT and RVT-2 are recent transformer-based methods that predict keyframes in robot trajectories. Importantly, both methods leverage RGB-D images to reconstruct a scene's point cloud and then predict keypoints. This point cloud reconstruction relies on a similar principle to our method, with ARM4R using pre-training on the 3D point tracking task to learn scene structure. We will add a proper discussion and comparison to this line of work in the final version of the paper.\n\nHowever, we would like to highlight that our approach contains some significant differences compared to RVT/RVT-2. First, our approach leverages 3D point tracking from human video using depth estimation, relying on pseudo labels. This enables the utilization of readily available web-based video data. In contrast, RVT and RVT2 depend on ground-truth RGB-D data and reconstructed point clouds captured by cameras. This dependence on ground-truth depth data inherently restricts the amount of usable video data. In addition, our approach uses 3D point tracking that can be easily extracted from human video data compared to keyframe-based approaches which leverage heuristics such as velocity changes to find keyframes.\n\n\n2. **Depth Estimation Model:** To obtain the 3D point tracks, we use SpatialTracker, which utilizes ZoeDepth [1] as the depth estimator. We note that since ZoeDepth predicts metric depth, it also predicts the absolute depth map. Additionally, as demonstrated in TapVid-3D [3]—a follow-up work of SpatialTracker—and DELTA [4], utilizing UniDepth [2], a more accurate monocular depth model, can enhance tracking performance. Lastly, as mono-depth methods further improve (such as UniDepth v2 [5]), 3D point tracking on monocular videos will also further improve.\n\n[1] Bhat, Shariq Farooq, et al. \"Zoedepth: Zero-shot transfer by combining relative and metric depth.\" arXiv preprint arXiv:2302.12288 (2023).\n\n[2] Piccinelli, Luigi, et al. \"UniDepth: Universal monocular metric depth estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Koppula, Skanda, et al. \"Tapvid-3d: A benchmark for tracking any point in 3d.\" arXiv preprint arXiv:2407.05921 (2024).\n\n[4] Ngo, Tuan Duc, et al. \"DELTA: Dense Efficient Long-range 3D Tracking for any video.\" arXiv preprint arXiv:2410.24211 (2024).\n\n[5] Piccinelli, Luigi, et al. \"Unidepthv2: Universal monocular metric depth estimation made simpler.\" arXiv preprint arXiv:2502.20110 (2025).", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Methods and evaluation criteria make sense.", "theoretical_claims": "The equations seem correct.", "experimental_designs_or_analyses": "The RLBench setting and real-world experiments make sense.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "### Strengths\n1. ARM4R demonstrates that pre-training on human video data can lead to better performance than models pre-trained solely on robotic data. \n2. ARM4R shows strong generalization capabilities across different robotic platforms (e.g., Kinova to Franka).\n\n### Weakness\n1. Lack of comparison with more competitive methods in RLBench, like RVT [1], RVT-2 [2].\n\n[1] Goyal, Ankit, et al. \"Rvt: Robotic view transformer for 3d object manipulation.\" Conference on Robot Learning. PMLR, 2023.\n[2] Goyal, Ankit, et al. \"Rvt-2: Learning precise manipulation from few demonstrations.\" arXiv preprint arXiv:2406.08545 (2024).", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How to acquire the point cloud for tracking training, which depth estimation model do you use? Are there any influences of using depth estimation that lacks absolute depth information?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h44rEnneq1", "reviewer": "Reviewer_vJYs", "summary": "The paper presents the ARM4R (Auto-regressive Robotic Model 4R), a novel approach to improve robotic pre-training by utilizing low-level 4D representations derived from human video data. The paper identify a significant challenge in the existing robotic models, which stems from a lack of large-scale, diverse datasets compared to those available in natural language and computer vision.\n\nARM4R addresses this gap by implementing a method that involves 3D point tracking, transforming 2D video representations into a format suitable for capturing essential physical world properties necessary for robotic manipulation. The main findings of the paper indicate that pre-training solely on human video data can yield superior performance compared to models trained exclusively on robotic data. Specifically, ARM4R surpasses established baselines, such as PerAct and OpenVLA, across multiple tasks in simulated and real environments with a 7-DoF Kinova Gen3 robot.\n\nKey contributions include:\n\n1. A novel pre-training approach that enhances understanding of the physical world through the utilization of low-level 4D representations from unlabeled human videos.\n2. Evidence that training on human video data can lead to improved performance in robotic tasks, supporting the idea of efficient transfer learning from human actions to robotic control scenarios.\n3. The model's capability to retain a shared geometric structure between 3D points and robot states, which facilitates better generalization across various robotic tasks and environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We address the concerns raised below:\n\n1. **Additional Limitations:** Here we provide a further analysis investigating more limitations of our approach.\n\n ***(i) Unnatural rotation:*** We examined a new task, “put knife” in RLBench. Interestingly, this simulated task features an unnatural whole-arm rotation to grasp the knife handle, based on the expert demonstrations created by the simulator’s motion planning. As the movement between two key points is very long, we hypothesize that our model has difficulty learning this rotation. \n\n ***(ii) Precision:*** We observed that ARM4R struggles with the 'screw bulb' task, a procedure requiring precise insertion into the bulb holder. In contrast, our model effectively performs other standard precision tasks, such as 'open drawer,' which necessitate only standard precision.\n\n2. **High Variability (Dynamic Environments) task performance:** ARM4R can handle dynamic changes as it predicts each step in a trajectory, thus interacting with the environment continuously. To further evaluate dynamic variability, we test three cube tasks (stack, destack, and pick), manually shifting the cube during execution when the arm reaches ⅓ and ⅔ of its starting height. See the results:\n\n|Generalization to Dynamic Environments|pick|stack|destack|\n|----------------------------------------|--------------|--------------|--------------|\n|no movement| 96.0$\\pm$2.3|61.3$\\pm$1.3| 94.7$\\pm$1.3|\n|cube moved when robot at ⅓ height|94.7$\\pm$3.5|60.0$\\pm$0| 93.3$\\pm$1.3|\n|cube moved when robot at ⅔ height|92.0$\\pm$2.3|57.3$\\pm$1.3|90.7$\\pm$1.3|\n\nResults show consistent performance despite target object movement, as long as sufficient z-axis space is available for adjustment. This highlights the model's robustness to environmental and object-level changes.\n\n3. **Robustness analysis:** To assess the robustness of ARM4R regarding factors like camera noise, occlusions, or sensor drift, we perform additional evaluations on the three cube tasks under the following conditions: (1) Dim lighting: ambient light reduced to 50%, (2) Background distractors: dynamic background changes, such as people walking by or adjusting curtains, and (3) Tabletop distractors: two random objects placed near the target. The results are presented below.\n\n| Real-world Robustness|pick|stack|destack|\n|-----------------------|--------------|--------------|--------------|\n|standard|96.0$\\pm$2.3|61.3$\\pm$1.3|94.7$\\pm$1.3|\n|dim light|86.7$\\pm$1.3|52.0$\\pm$2.3|81.3$\\pm$2.7|\n|background distractor|94.7$\\pm$1.3|57.3$\\pm$1.3|90.7$\\pm$1.3|\n|table-top distractors|81.3$\\pm$1.3|48.0$\\pm$2.3|74.7$\\pm$1.3|\n\nOverall, ARM4R demonstrates robustness to lighting changes and background distractors, likely due to the attention pooling (See Fig 2 in the main paper), which guides the model to focus on the target region rather than background features. This is further supported by the observed performance drop when tabletop distractors are introduced.\n\n4. **Dataset Bias:** We note that EpicKitchens is one of the main vision datasets containing various kinds of interactions between human hands and objects in kitchens. Thus, we choose to leverage it as a pre-training dataset for tabletop manipulation tasks. In future work, we plan to explore the benefits and effects of adding more pre-training datasets.\n\n5. **Domain Gap:** As the reviewer correctly pointed out, there exists a domain gap between human video data and robotic control environments. To address this issue, we introduced in the paper Stage 2 of our training pipeline: Fine-tuning 3D Point Tracking for Robotic Settings. This stage uses the same task as in Stage 1 but enables the model to adapt to differences in camera dynamics and embodiment from the target robotic environment. The impact of this domain gap is evident in our results (Section 4.4, Figure 3), where omitting Stage 2 leads to decreased success rates. These findings underscore the importance of Stage 2 in bridging the gap and improving downstream performance.\n\n6. **Number of demonstrations used:** In our real Kinova setting, we used 200 demonstrations (split into 19:1 training/validation sets) per task to finetune our model and all the other baselines except RPT and LLARVA in Tables 2 and 3. The RPT model uses 1920 demonstrations for its sensorimotor pre-training, and LLARVA uses 800 demonstrations per task. We note that for our newly added results on LLARVA, we used 200 demonstrations per task, following the same setting as the other baselines in Table 2. This may explain LLARVA’s poor performance on most of the tasks tested, as it typically requires more data. Overall, we found that training our model on 200 demonstrations is enough to achieve good performance in both real and simulation settings.", "claims_and_evidence": "The claims made in the paper regarding ARM4R's effectiveness in enhancing robotic pre-training are largely supported by specific performance metrics and comparisons with other models. Here are the key points of evidence that reinforce the claims, along with an evaluation of their clarity and convincing nature:\n\n1. **Performance Metrics**: The paper provides performance comparisons through tables that outline success rates for various robotic tasks. For instance, ARM4R shows a superior average success rate over other robotic models in specific tasks, such as picking and stacking cubes, achieving notable success rates of 96% for picking cubes and 61.3% for stacking cubes, which are higher than competitors like MVP, RPT, and others.\n2. **Methodological Clarity**: The paper details the methodology behind the ARM4R model, including its unique approach of utilizing low-level 4D representations from human video data for pre-training. The transformation of 2D video representations into 3D point tracking through monocular depth estimation is explained, presenting a clear rationale for why this method is beneficial for robotic manipulation.\n3. **Comparison with Existing Models**: The efficacy of ARM4R is further supported by comparative analyses, where it consistently outperforms baseline methods such as ATM and OpenVLA across various real-world tasks. These comparisons enable readers to gauge the relative improvements ARM4R offers, providing a compelling argument for its adoption.\n4. **Generalization Across Tasks and Environments**: The paper claims that ARM4R’s shared geometric structure between 3D points and robot states facilitates better generalization across different robotic environments and tasks. This theoretical claim is underpinned by empirical results, which lend credence to the assertion.", "methods_and_evaluation_criteria": "The proposed method, ARM4R, presents a well-structured approach to enhance robotic model pre-training by leveraging low-level 4D representations extracted from human video data. The choice to use 3D point tracking converted from 2D video via monocular depth estimation effectively addresses the common challenge of limited large-scale datasets in robotics, which contrasts significantly with more data-rich domains like natural language processing and computer vision. This is a relevant application, as it aims to bridge the gap between human-centric visual data and robotic control tasks.\n\nThe evaluation criteria employed in the paper are also appropriate and rigorous. ARM4R was tested across multiple real-world robotic tasks, categorized into specific actions such as \"pick,\" \"stack,\" and \"destack,\" which is essential for establishing a thorough understanding of the model's performance in practical scenarios. The success rates were reported for a variety of tasks, showing how ARM4R outperforms existing methods (like OpenVLA and ATM) that rely solely on robotic data, indicating a meaningful benchmark for comparison.", "theoretical_claims": "The support of proposed method appears to be on empirical evaluations and quantitative comparisons rather than theoretical proofs. Thus, there are no specific theoretical claims or proofs outlined that were checked for correctness in the paper. There are extensive experiments showing the effectiveness of proposed method.", "experimental_designs_or_analyses": "I have assessed the soundness and validity of the experimental designs outlined in the paper, especially focusing on the three training stages of the ARM4R model:\n\n1. **Stage 1: Human Videos Pre-training**: The model is trained on a large dataset (76K videos from Epic-Kitchens100) that captures rich human-object interactions. This pre-training stage is crucial as it builds a foundation for understanding spatial dynamics and physical interactions. However, it would be beneficial to see a clear justification of the choice of dataset and its relevance to the targeted robotic tasks.\n2. **Stage 2: Fine-tuning on 3D Point Tracking for Robotic Settings**: The model is fine-tuned on robotic videos, which is essential to adapt the learned representations to the robotic domain. The authors claim this stage only requires a modest amount of data (5-10% of Stage 1), but the criteria for selecting this data and whether it adequately captures the variations in robotic tasks should be justified. Additionally, potential discrepancies that may arise from differences in camera dynamics between human videos and robotic applications warrant further discussion.\n3. **Stage 3: Fine-tuning for Robotic Control**: The final stage involves refining the model with a focus on actual robotic control tasks. The described methodology is reasonable, but revealing more about the range and variety of tasks included in this stage would strengthen the claims. Furthermore, a discussion on how fewer demonstrations were used compared to previous baselines could provide insights into the model's efficiency.", "relation_to_broader_scientific_literature": "The paper proposed to use human videos for pre-training which is not used widely in previous literature. And this paper shows pre-training with human videos could boost the model performance for robotic tasks.", "essential_references_not_discussed": "I don’t find essential references that are not discussed.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. **Originality**: The paper introduces ARM4R, an Auto-regressive Robotic Model that utilizes low-level 4D representations derived from human video data. This approach represents a significant advancement in the application of representation learning to robotics, particularly by focusing on the transferability of learned representations from human actions to robotic control tasks. The novelty of leveraging dense 3D point tracking over time enhances the model's capability to generalize across various robotic platforms.\n2. **Significance**: The work addresses a critical bottleneck in robotics—the lack of large-scale, diverse datasets suitable for training foundation models. By demonstrating that it is possible to pre-train on human video data, the paper opens new avenues for developing robotic models that can learn from the rich visual and action data available on the internet. This approach is particularly valuable for scenarios where collecting robotic data is either challenging or costly, thereby expanding the potential for scalable and data-efficient learning in robotics.\n3. **Clarity**: The paper is well-structured, with a clear progression from the introduction of the problem to the methodology and results. The authors effectively communicate their contributions, and the use of relevant figures and diagrams enhances comprehension. Each section follows logically, making it easier for readers to grasp complex concepts associated with 4D representations and auto-regressive modeling.\n\n**Weaknesses:**\n\n1. **Limited Discussion on Limitations**: While the paper addresses some limitations, such as the coupling of object and camera motion affecting learned representations, it could benefit from a more comprehensive discussion regarding other possible weaknesses of the approach. For instance, greater consideration of potential biases in human video data, challenges in occlusion handling, and the model's performance on tasks with high variability could provide a more balanced perspective.\n2. **Potential Domain Gaps:** Since there are domain gaps between human videos and robotic control data, discussion on how to address this gap effectively would help make the paper more convincing.", "other_comments_or_suggestions": "I don’t have other comments or suggestions.", "questions_for_authors": "1. How fewer demonstrations were used compared to previous baselines in Stage 3: Fine-tuning for Robotic Control? What is the relation between demonstrations number and fine-tuned model performance? How many demonstrations are needed for a good performance at least?\n2. Does the domain gap between human videos and robotic control data have significantly influence on the model’s final performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwOkRHlh3Z", "reviewer": "Reviewer_kSQ4", "summary": "The paper presents ARM4R, an Auto-regressive Robotic Model that learns 4D representations from human videos to enhance robotic learning.\n- 4D Representation Learning: Transforms 2D videos into 3D point tracks, enabling temporal and spatial understanding.\n- Efficient Transfer to Robotics: Pre-training on human videos outperforms training directly on robotic datasets.\n- Improved Performance: ARM4R surpasses baselines like PerAct, OpenVLA, and ATM in both simulated and real-world tasks.\n- Cross-Robot Generalization: Demonstrates strong generalization across different robotic platforms (e.g., Kinova and Franka).\n\n## update after rebuttal\nThanks for the rebuttal. All my concerns are resolved. I maintain my initial rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We address the concerns raised below:\n\n1. **Causal Analysis & Optimality of 4D Representations:** As stated in the paper, one of the main benefits of using 4D representations is that the 3D point tracks in a robotic setting are described by linear transformations constructed using the robot states. We present a simple proof of this below:\n\nConsider a $n$-DoF open-chain manipulator, initially in its reference configuration $\\theta_1=0, \\cdots, \\theta_n=0$. Suppose the manipulator is commanded to a new position $\\theta_1=\\theta_1^d, \\cdots, \\theta_n=\\theta_n^d$. Next, we take any arbitrary point $p$ on the body of the manipulator lying between joints $i, i+1$. Let $p$ be described by the transformation $g_{i,p} (0)$ in joint $i$'s frame. Then, $p$'s new position in the base frame can be described by:\n$$\ng_{1,p} (\\theta) = e^{{\\hat{\\xi_1}}\\theta_1^d}\\cdots e^{{\\hat{\\xi_i}}\\theta_i^d} g_{i,p} (0)\n$$\n\nwhere ${\\hat{\\xi_j}}$ are the twists associated with each joint. Since this product of 4x4 SE(3) matrices represents a linear transformation, points on the robot body as well as attached rigid bodies evolve through linear transformations in terms of the robot states.\n\nOverall, this proof shows that learning 3D point track prediction is related to and can benefit robotic control, as validated by our ablation experiments. Additionally, for overactuated manipulators (7 DoF), the robot learning problem simplifies to tracking arbitrary trajectories, as configuration space limitations are less restrictive. This allows the model to benefit more effectively from the 3D point tracking pre-training task.\n\n2. **Comparison to 2D and 3D methods:** We note that in our paper (see Tables 1/2), we compare ARM4R with both 2D and 3D approaches. ARM4R outperforms 2D motion-based methods (e.g., LLARVA, ATM) and 3D-based methods (e.g., ManiGaussian). We also include an additional comparison below with a recent 3D method, 3D-VLA, further demonstrating the strength of 4D representations for robotic learning.\n\n|Method|put knife|pick up cup|take umbrella|Average|\n|----|--|------|---|-------------|\n|Image-BC ViT|0|4|0|1.33 |\n|C2FARM-BC|4|28|48|26.67|\n|LLARVA|4|52|32| 29.33|\n|PerAct|20|56|76|50.67|\n|3D-VLA|68|40|52|53.33|\n|**ARM4R**|28.8±1.50|99.2±0.80|82.4±2.03|70.13±1.53|\n\n3. **Geometric Alignment:** We clarify that our claim regarding geometric alignment refers not to alignment between human and robotic states, but between 3D point tracks in a robotic setting and the robot states. As shown in the proof above, these point tracks are described by linear transformations of the robotic states, which suggests that 4D representations are a natural choice for robotic control. We will clarify this in the final version.\n\n4. **2D-to-3D Lifting:**  An alternative to monocular depth estimation is to use multiview cues and triangulation, which is more appropriate for dynamic camera setups. However, since our deployment in both real-world and simulation environments involves static cameras, monocular depth estimation is more suitable. This is the common and simple practice used in many 3D tracking papers, such as SpatialTracker and SoTA method DELTA.\n\n5. **Scalability:** As the reviewer suggested, we add a scaling experiment, where we change the number of demonstrations used in our Epic-Kitchens pre-training. Specifically, we decrease the number of human videos in the pretraining to ⅓ and ⅔ of the original dataset used. We train the model for all three stages, and evaluate the performance on the three cube tasks. The results are summarized below:\n\n|Portion of the Epic Videos in Pre-Training|pick|stack|destack|\n|:------------------------------------------:|:------------:|:------------:|:------------:|\n|1/3|86.7$\\pm$1.3|56$\\pm$2.3|82.7$\\pm$3.5|\n|2/3|90.7$\\pm$1.3|58.7$\\pm$2.7|89.3$\\pm$2.7|\n|1|96.0$\\pm$2.3|61.3$\\pm$1.3|94.7$\\pm$1.3|\n\nThe results show that as the size of the pre-training set increases, the performance also tends to increase.\n\n6. **Real Robot Evaluation:** We note that the 3 tasks of pick, stack and destack in Table 3 are commonly used for evaluation in several baselines, including LLARVA, Octo, RPT, and MVP. The main reason we follow the same procedure in this experiment is that comparing ARM4R to 6 different pre-training approaches on all 13 tasks would require a significant amount of demonstrations, as every baseline would need to be fine tuned for a fair comparison. Specifically, fair fine-tuning for each baseline would demand approximately four times the data currently used. We plan to expand the evaluation to include additional tasks in the final version.\n\n7. **Monocular Depth Estimation:** See point 2 under Reviewer Zx5c.\n\n8. **Robustness Analysis:** See point 3 under Reviewer vJYs.\n\n9. **Failure Case Analysis:** See point 1 under Reviewer vJYs.\n\n10. **Generalization to Dynamic environments:** See point 2 under Reviewer vJYs.\n\n11. **Dataset Bias:** See point 4 under Reviewer vJYs.", "claims_and_evidence": "The paper makes strong claims about transfer learning in robotics using 4D representations, supported by empirical evidence, but some potential concerns remain:\n- Causal analysis: The causal relationship between 4D representations and robotic success is not thoroughly analyzed, leaving room for further investigation.\n- Optimality of 4D Representations: The paper lacks formal proof that 4D representations are the optimal choice for robotic learning. It does not compare 4D representations to alternative 3D motion-based methods, leaving this claim unverified.\n- Geometric Alignment: The assertion that linear transformation preserves geometric alignment between human and robotic states is not rigorously justified, raising questions about its theoretical validity.\n- 2D-to-3D Lifting: The approach of lifting 2D video data into 3D using monocular depth estimation is not thoroughly justified, leaving uncertainty about its robustness and generalizability.", "methods_and_evaluation_criteria": "- Failure Case Analysis: The study does not evaluate failure cases, such as camera noise, occlusions, or sensor drift, which are critical for assessing real-world robustness.\n- Generalization to Dynamic Environments: While experiments are conducted in controlled settings, their applicability to highly dynamic or unpredictable real-world environments remains unverified.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "- Performance Discrepancies Across Tasks: The model achieves high overall success rates, but specific tasks reveal performance inconsistencies. Rotational motion tasks (e.g., \"screw bulb\") show worse performance compared to the voxel-based model PerAct, while planar force-based tasks underperform relative to the LLM-based model LLARVA, which also uses intermediate 2D states as guidance. A deeper analysis of why these specific failure cases occur could lead to future improvements in representation learning.\n- Real Robot Evaluation: For pre-training models, 3-task subset may not unintentionally favor ARM4R performing best.\n- Scalability and Data Efficiency: There is no discussion on whether smaller human datasets (e.g., 5K or 10K videos) would still yield strong performance.\n- Potential Dataset Bias: The model is pre-trained on Epic-Kitchens100, which focuses on household activities, which raises concerns about bias.", "relation_to_broader_scientific_literature": "Good.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Null", "other_comments_or_suggestions": "Null", "questions_for_authors": "See the sections above for detailed feedback.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for robotic manipulation learning from human video via 3D point tracking in time, which relies on monocular depth estimation from 2D images. The evaluation is done in RLBench and on physical robot setups.

The reviewers found merits in the method's motivation and technical approach. At the same time, initially they also had a variety of concerns, ranging from the evaluation on RLBench being inadequate to lack of clarity around handling the embodiment gap between people and robots, to potential bias in the pretraining tasks, which are geared towards kitchen environments. 

Overall, the authors have successfully addressed these concerns in the rebuttals, partly by adding significantly more experiments and analysis during the rebuttal stage and partly by carefully positioning the proposed method, ARM4R, w.r.t. the related work. The metareviewer encourages the authors to incorporate the results and explanations from their rebuttals into the paper and has no major concerns about accepting this work.
Decision: Accept (poster)