Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
Authors: Lirui Wang
Xinlei Chen
Jialiang Zhao
Kaiming He
Venue: NeurIPS 2024 spotlight (2024)
Keywords: heterogeneous robot learning
heterogeneous pre-trained transformer
scaling law for robotics
robotic foundation model
Primary area: robotics
TL;DR: We propose Heterogeneous Pre-trained Transformers (HPT) that pre-train policy representation across different robot embodiments and tasks, scale it to 1B parameters and 50 datasets, and demonstrate transfer in simulation and real world evaluation.

Abstract: One of the roadblocks for training generalist robotic models today is heterogeneity. Previous robot learning methods often collect data to train with one specific embodiment for one task, which is expensive and prone to overfitting. This work studies the problem of learning policy representations through heterogeneous pre-training on robot data across different embodiments and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT), which pre-train a large, shareable trunk of a policy neural network to learn a task and embodiment agnostic shared representation. This general architecture aligns the specific proprioception and vision inputs from distinct embodiments to a short sequence of tokens and then processes such tokens to map to control robots for different tasks. Leveraging the recent large-scale multi-embodiment real-world robotic datasets as well as simulation, deployed robots, and human video datasets, we investigate pre-training policies across heterogeneity. We conduct experiments to investigate the scaling behaviors of training objectives, to the extent of 52 datasets. HPTs outperform several baselines and enhance the fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings. See the project website (liruiw.github.io/hpt) for code and videos.

Scaling Proprioceptive- Visual Learning with Heterogeneous Pre-trained Transformers
Lirui Wang'
Xinlei Chen
Jialiang Zhao' Meta, FAIR
Kaiming He'
MIT CSAIL
https://liruiw http:///iirux.ggituu....tc github.io/hp
Abstract
One of the roadblocks for training generalist robotic models today is heterogene- ity. Previous robot learning methods often collect data train with specific embodiment for one task, which is expensive and prone to overfitting. This work studies the problem of learning policy representations through heterogeneous pre- training robot data across different embodiments and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT), which pre-train large, shareable trunk of a policy neural network to learn task and embodiment agnostic shared representation. This general architecture aligns the specific proprioception and vision inputs from distinct embodiments to short sequence of tokens and then processes such tokens to map to control robots for different tasks. Leveraging the recent large-scale multi-embodiment real-world robotic datasets as well as simulation, deployed robots, and human video datasets, we investigate pre-training policies across heterogeneity. We conduct experiments to investigate the scaling behaviors of training objectives, to the extent of 52 datasets. HPTs outperform several basclines and enhance the fine-tuned policy performance by over 20% on unseen tasks in multiple simulator benchmarks and real-world settings.
Introduction
Building robotic policies today is hard: it often requires collecting specific data for each robot, task, and environment, and the learned policies do not generalize beyond these specific settings. historical lesson that has revolutionized machine learning is that pre-training [34, 27, 29] large-scale, high-quality and diverse data can bring general models that usually outperform specific models. Recent progress in open-source large-scale data collection [14, 76] has made this path possible, but the heterogeneity (such as varying robot hardware and different environments) present large-scale robotic data has posed a significant challenge. central question for the field now how to leverage the heterogeneous robot data to pre-train robotic foundation models [3].
Foundation models from natural language processing [60, 58] and computer vision [37] have shown paradigm to achieve general- general-purpose purpose task-agnostic models through pre-training on massive amounts and diversity of data. In addition the benefits from more data, training with diverse tasks also enforces the representation to be more generalized. These foundation models can achieve high task success rates for various tasks, more robust outliers, and are flexible for adapting to tasks. These approaches map input signals from distinct domains and tasks into high-dimensional representation space, and exhibit consistent scaling behaviors [34, 29]. After that, minimal fine-tuning is required to transfer the representation for downstream tasks to achieve good performance.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

embodiment- Figure I: The Heterogeneous Pre-training concept. It maps different embodiments, each with its own proprioception and vision sensors, onto a shared la- tent space by embodiment-specifie tokenizers (" ("stems"). This aligns the heterogeneous data from different em- bodiments into a joint representation space. This allows us to train shared Transformer trunk on the union of all heterogencous datascts. The pre-trained Transformer can be transferred to a new embodiment, with a small, new tokenizer learned at transferring time.
stem I
stem
anbockment
enbodiment
stem
The heterogeneity in robotics presents a distinct challenge: different robots are physically different embodiments' of hardware acting in different environments. Each embodiment can have a dis- tinct proprioception, including different degrees of freedom, end-effectors, motion controllers, and workspace configurations built for specific application. Another common heterogeneity in robotics is vision heterogeneity. Robots are often equipped with different camera sensors mounted at different places (e.g. wrist and/or third-person) and the visual appearance of each robot varies dramatically due to environments and tasks. Both proprioception and vision information are crucial for complex, contact-rich, Long-horizon behaviors in robotics. Poor learning of such information can lead to overfitting behaviors such as repeating motions for particular scene and task or even trajectory.
In this work, propose to address this issue by aligning the proprioception and vision information from different embodiments to a shared "language" of policies through helerogenous pre-training (Figure 1). With such shared representation, new embodiment only requires minimal data and training to "translate" its specific setup to the shared "languages". In other words, we want to pre-train task-agnostic and embodiment-age aggnostcc foundational models that can map raw sensor signals from individual embodiments into shared latent space. Previous works have made significant progress in pre-training only the vision part of the policy on human videos [49, 54, 35, 62] and pre-training the full policy [6, 14, 56] with a unified model and dataset format (e.g. using languages [5]). Additionally, they assume no proprioception in pre-training and add it post hoc transfer learning.
We introduce Heterogeneous Pre-trained Transformers (HPT), a family of architecture designed to scalably learn from data across heterogeneous embodiments. HPT modularizes a general policy network architecture (Figure 2) and pre-trains the policy representation of a latent transformer with supervised learning. Inspired by learning from multimodal data [1, 74, 20, 31], specific tokenizers, dubbed "stem", to align various sensor inputs such as camera views and pro- prioception inputs. The "trunk" is shared and pre-trained across datasets and is transferred when adapting to new embodiments and tasks that are unknown during the pre-training times. Moreover, we use task-specific action decoders, dubbed "head", to produce the action outputs. Crucially, after "tokenizing each embodiment", HPT operates on shared space of a short sequence of latent tokens. This hierarchy motivated by how humans handle feedback loops between specific motor responses and perceived stimuli at the level of the spinal cord's 's neural circuitry [69].
We extensively investigated the scaling behaviors and various designs of policy pre-training the extent of more than 50 individual data sources (2 times more than [56]) and model size of over 1 billion parameters. Analogous to the scaling laws [27, 29], we found that to some extent, HPT scales with the dataset quantity and diversity as well as the model and training compute.
In addition, heterogeneity can occur in different embodiment domains, such real robot hardware, simulation domains, and human videos. We incorporate many available embodied datasets in different embodiments such real robots [14, 76, 39], simulation [82, 90, 50, 21, 86, 81] and internet human videos [15] in the pre-training process and demonstrate the generality of our framework including embodiments beyond expensive real-world on-robot teleoperations.
Through transfer learning experiments across multiple simulation benchmarks [90, 50, 82] and real-world dexterous tasks, we compare with several baselines and the from-scratch counterparts. Overall, based on the pre-training objectives, HPT can scale with the model, data, compute, and the heterogeneity of the robotic datasets across real robots, simulations, and human videos. These pre-
"Embodiment can be defined differently according to the context of robotics and AL In this work, we consider robots equipped with distinct set of sensors and actuators with the associated observation and action space to be unique embodiment.

---

HPT architecture. HPT is modularized into stems, trunk, and heads. The stem, consist- ing of a proprioception tokenizer and a vision tokenizer, maps the vision and proprioception ob- servations of different embodiments to a fixed number (e.g. 16) of tokens. The shared trunk, which is Transformer, maps the concatenated tokens into shared representations. The head then maps the processed tokens to actions in different downstream tasks. For specific embodiment, stem/head pair activated (denoted by the switch). The trunk is shared and pre-trained on action-labeled data with supervised learning and then transferred to new embodiments. This proce- dure scales up 52 datasets and 1B parameters.
Figure 2:
head
head
Transformer
ambodiment
embodmment
ambadmennt III
stem
stem
stem
proprio
vision
training procedures and models can simplify building reliable robotic policies for new embodiments and new tasks in terms of data requirements and generalized performance. As an attempt to scale heterogeneous pre-training, our code and weights are open-sourced, and we hope that HPT can shed some light on learning robot representations from heterogeneous embodiments and tasks.
2 Related Works
Pre-training and Transfer Learning. Pre-training [2], through direct supervision [38] and/or self-supervision [57, 25, 12, 22, 10]. has been shown to learn representation useful for unseen downstream tasks in computer vision [7. 37] and natural language [58], and their intersections [60]. The representation learned from ImagetNet [16] or web-scale data [38, 60, 18] shows robustness to distribution shifts and can be transferred new tasks.
The recent surge of foundation models [3] scales these representation learning methods [27, 29] by applying task-agnostic objectives to multitask data. Moreover, recent works [46, 42, 45) show that small projection layers can be used to align the pre-trained feature spaces of the foundation models. Different from other fields, robotics has less data quantity and diversity but much more heterogeneity.
Alignment. Recent works such as Flamingo [1], Perceiver [31], and ImageBind [20] proposed ways to combine representations from multimodal data such image, language, and audio by aligning these different modalities to the same latent space in the pursuit of representation learning. Our architecture design is also motivated by methods such LLaVA[45 in the multimodal learning community. Very recently, GPT-4o [58], Gemini [77]. MM1 [52], X-VILA [89], and Chameleon [75] demonstrated the capabilities of heterogeneous pre-training universal transformer from and for multiple modalities. The idea of alignment, across modalities and/or embodiments, is important as we scale use heterogeneous embodiments and reuse data from distinct embodiments.
Representation Learning in Robotics. Representation learning has been explored in the robotic community. Previous works such as R3M [54], VC-1[49], Voltron[35], and Spatial [11] investigate visual representations by training the policy with human videos and robotic data [70]. Recent works [61, 17, 4, 88, 71, 40] also align representations from multiple modalities and data distributions for robotic tasks. After pre-training, transfer learning with the frozen representation and/or finetuning conducted in the target domains.
Generalist Policies. Large-scale policy learning in robotics has leveraged diverse data from real robots 16, 73J. human videos [54, 49], and simulation domain [33, 63, 83, 80] separately. There are also works in multi-task learning [65, 66, 85, 23], meta-learning [79, 55, 19], few-shot learning [84], and fleet learning [82]. Recently, RT-X, Octo, Open VLA [6, 14, 56, 36] train generalist lww---aa-gggaatiiii robotic policies on datasets from diverse robotic embodiments.
Compared with these works, HPT handles broader heterogeneity including proprioception and vision, explores scaling behaviors on more heterogeneous domains including real robots, human videos, and simulation data, and is evaluated at a larger scale in simulation benchmarks.
3

---

cross-attention
cross-attention
k,
k,
vision
encoder
propriaception
queries
vision
queries
Figure 3: Stem Architecture in HPT. In the HPT stem, the proprioceptive tokenizer uses an MLP to map proprioceptive information to feature which is then attended by 16 learnable tokens. The vision tokenizer uses pre-trained encoders similarly uses an attention mechanism to map vision features into fixed tokens. The architecture flexibly handles sequences inputs without increasing size tokens.
Mixture of Experts. Our architecture design is related works in conditional computation and MoE [51, 44. 72], where we create one expert for each embodiment, and the router (for the whole network) is determined by the embodiment. This technique has been used scale language models to substantial size [32].
3 Heterogenoues Pre-trained Transformers (HPT)
In heterogeneous robot learning with cross embodiments, the data generated from different domains such as simulation and real robots, across sensory modalities such as RGB images, language instructions, depth maps, 3D point clouds, and tactile images. Each robot is unique hardware embodiment with varying degrees of freedom, end-effectors, sensor configurations, controller and action spaces, and application-specffi physical setups.
In the following sections, we discuss the HPT network architecture and the training procedure to address the heterogeneity above. We modularize the network architecture (Figure 2) into the embodiment-specife stem, the shared trunk, and the task-specific heads. Intuitively, the stems, shown in Figure 3, are earlier layers of the neural network that align sensory inputs from heterogencous embodiment and modalities into the shared representation space. The shared middle part of the network is called the trunk, which processes the sensory representation into latent representation that can be used for multiple tasks. Finally, the last part of the network is the head, which maps that latent representation to the action space in individual tasks of interest. The training procedure, dubbed heterogeneous pre-training, assigns and aligns specific stem/head pairs based on the sampled embodiment and task data, and still enjoys the benefits joint training in the shared trunk. This can be thought of as tokenizing each embodiment using neural networks and alleviating the need to unify embodiments into homogeneous data form in standard training procedures.
3.1 Network Architecture
Stem. The stem 4stem (Figure 3) in HPT is composed of a proprioceptive tokenizer and vision tokenizer. These tokenizers map heterogeneous inputs from different embodiments a fixed number of tokens with fixed dimensions, which enables the trunk to treat them in the same manner despite large heterogeneity, as well as enjoy the scaling and inference benefits on fixed context length. The key idea is to leverage attention [78, 31, 9] attend fixed number of learnable tokens to features of the observations. Although we mainly focus on proprioception and vision, handling other kinds of sensor heterogeneity in tactile, 3D, and action inputs can be flexibly extended in stems.
Proprioception Tokenizers. In Figure 3 (left), for embodiment the proprioceptive tokenizer maps any sequence of robot proprioceptive information with dimension dž (e.g. 7 for end-effector pose) into Np (e.g. N₂ = 16) tokens with dimension d with values ranging from 128 to 1024. To achieve this, we first use an MLP to map the proprioceptive input into a feature space with dimension use attention across the state feature and the learnable tokens, to map into 16 tokens with dimension d. Proprioceptive information is critical in robot policy learning, its usage often simple feature concatenation with vision encoder [41].
We then apply sinusoidal position encoding and

---

Teleop
Simulation
Depth
Width
# #Attn. Heads
Param.
HPT-Small
128
8
3.1M
HPT-Base
256
HPT-Large
16
512
8
50.5M
HPT-XXaarge
226.8M
HPT-Huge
80
1024
16
1.1B
Table 1: Network Details of HPT. The width denotes the latent dimension size of the trunk transformer and the depth denotes the number of blocks. The default sctup is the HPT-Small model.
Wild
Human Video
# Dataset
# Traj.
# Samples
# Batch Size
Figure 4: Dataset Heterogeneity in Robotics. We show illustrations of dataset mixtures (each color is distinct embodiment) from different domains including real robot teleop [14], deployed robots [39], simulations, and human videos [15]. See Appendix Section A for dataset mixture details.
Default Scaled
27
16k
SM
256 2048
52
270k
155M
Table 2:
Dataset Details of Pre-train Settings. The default setup is trained with 27 datasets from RT-X with 16k trajectories (maximum 1000 trajectories each) and scaled setup involves more data and compute.
Vision Tokenizers. In Figure (right), the vision tokenizer can map any sequence of camera images (videos of multiple views) with dimension H x W x into N, (we use N, = 16 by default) tokens with dimension d. To achieve this, first use pre-trained frozen feature networks (e.g. 7 by features from ResNet) and then flatten the features. After that, we again use attention across these features and learnable tokens, to map the vision input into 16 tokens with dimension d.
After processing each modality individually in the time sequence order, concatenate all modality tokens and add additional modality embeddings and sinusoidal positional embeddings. This used as the input sequence to the trunk that introduce below. To avoid overfitting, the stem only has small number of parameters (one MLP and one attention layer).
Related works such as Octo [56] and others [54, 49, 6] mostly focus on pre-training the vision backbone of the policy through masking or self-supervision. They often stack sequences of single- view images along channels [6] for particular robot or use large number of tokens (256 in [56]). In contrast, HPT uses stems with pre-trained vision encoders to map arbitrary image sequences to short sequence of tokens (16). Moreover, rather than add in proprioception during transfer in related works, HPT jointly pre-trains the vision and proprioception parts, from heterogencous datasets.
Trunk. As the central component for pre-training, the trunk architecture follows a transformer, parametrized by (hhnnk in the latent space with dimension d. The output token sequence length L the same as the input token sequence length. The output token sequence simply pooled as the final combined feature for the observation. 'he trunk is shared across different embodiments and tasks to capture the complex input-output relationships (i.e. the number of trunk parameters is fixed independent of the number of embodiments and tasks).
Head. The policy head Gpead takes the output of the trunk transformer and maps it to the action space A in each dataset. For each embodiment and task, the policy head can be an arbitrary architecture (e.g. MLP) that takes input the pooled feature of the trunk and outputs a normalized action trajectory. The policy head is reinitialized for transferring to new embodiment.
3.2 Training Objective
Given a total of K datasets with heterogeneous embodiments sampled from different distributions DI, ..., De, ..., Dx, we let D. = (7() IssisM, denote a set of M₁ trajectories in dataset Dz. with -(1) {o()) aa JISIST denoting the i-th trajectory of maximum length T' of observation and action tuples. The objective is to minimize the following loss across datasets
(+)
min
L( grunk ghead Dx))-
(1)
5

---

C is behavior cloning loss computed as the Huber loss between the normalized action labels based on dataset statistics and the network's action predictions. 0 = U (0 O U Otrnk denotes the network parameters comprised of embodiment-speccfff stem and head 0% Obhead for dataset k, and single set of shared trunk parameters frunk across all embodiments. This training procedure has two axes of data scaling: the quantity Mig for one dataset D2 and the total number of datasets K. In the pre-training stage, only the trunk parameters are updated at every iteration, and the stems and heads for each heterogeneous embodiment and task are updated based the training batch sampling. See implementation details in Appendix Section A.3.
3.3 Transfer Learning
The policy transfer process is similar to aligning the features of the new domain (through pre-trained stem encoders) to the pre-trained embedding space of the trunk [42, 45]. Given new dataset Dx+1 from a new embodiment, the objective can be the same as pre-training or alternatives [13]. We reinitialize the head and stem parameters with embodiment-speccfic input and output dimensions (such as different proprioception and action dimensions), and freeze the weights of the trunk.
4 Experiments on Pre-training
In this section, we aim to answer the following question: Does HPT pre-training have a scaling behavior under heterogeneous data across domains?
Default Setting. We use 27 robot teleoperation datasets, including subset of the recently public Open-X Embodiment dataset [14] as the training corpus. By default, we use one camera view of the scene with the pre-trained frozen ResNet18 image encode to compute the vision features. We use proprioception information, such as end-effector poses and joint positions, whenever they are available and provided. We use maximum of 1000 trajectories from each dataset and a total number of 16k trajectories, and a held-out validation dataset with a maximum 200 trajectories per data source. Furthermore, we use a model with trunk size of 3.17 million parameters, which is denoted as HPT-Small (Table 1). The training uses a batch size of 256 for 80k iterations, which is around 0.65B tokens in the latent space that feeds into HPTs and around 5B tokens in the vision and proprioception token spaces thorizon-dependent) While we do not align or preprocess action space or observation space [56, 87] other than normalization, data cleanup and filtering would be very helpful.
Scaled Setting, We use 200k trajectories with 52 datasets, including simulation (e.g. [50]), deployed robots (e.g. [39]). human videos (e.g. [15]), from distinct embodiments in the training process. This includes many public and accessible robotic datasets. In addition to different tasks in different institutes, these heterogeneous mixtures of datasets (Fig. 4 and Fig. 13) come with multiple views, language inputs, and different observation inputs in different environments.
4.1 Protocol
We evaluate the HPT pre-training performance with the averaged validation loss (prediction errors on unseen trajectories) the last iteration of pre-training. These validation datasets are fixed independent of the trajectory counts and models during training. Unless particularly noted, the validation datasets come from the same 27 datasets the Default Setting. Note that it is unrealistic to evaluate the pre-trained models on many real-world robotic environments at scale and there are very few evaluation alternatives to measure large-scale pre-training if we ignore this objective. In fields such as NLP[30, 34], training loss objective (e.g. perplexity) is often used to measure the progress of pre-training. Admittedly, there are several caveats to this metric including the closed-loop performance gap and the task success rate gap. We will address these issues in Section 5 on HPT transfer learning. See Appendix Section A and Section D for more details and discussions.
4.2 Scaling Behaviors
Data Scaling, In Figure (a), we observe stable and scaling validation losses even on increasingly heterogeneous embodiments. Moreover, we found the compute (e.g. samples seen per training run) and the data amounts needed to scale in tandem [34] to get closer to convergence in the training process. In the red line in Figure 5 (a), we observe better validation losses we scale up the total number of trajectories, by using larger model and doubling the batch size every order of magnitude increase in trajectory counts. Strictly increasing data while keeping others bottlenecked (HPT-S

---

HPT-L
HPT-S
0.0780
Validation Loss
0.10
0.0765
Validation
0.08
Delaut
0.0750
0.06
0.0735
0.04
10
0.0720
10 20
30
Trajectory (a)
Dataset
(b)
Figure 5: Data Scaling. We scaling HPT experiments along dataset sizes and the number of datasets. Each
point is the validation loss of full training run. (a) We evaluate the losses on 27 datasets with the number of total trajectories ranging from maximum trajectories per dataset (170k in total). We compare two model sizes, HPTT-//L, where HPTTL is a bigger model trained with 4 times more tokens than HPT-S. (b) We compute the validation losses for fixed subset of 10 datasets with fixed number of epochs (2). We compute mean and standard deviations for 4 runs across model sizes from HPT-S HPT-XI. and across dataset counts from 10 52.
trajectories per dataset (270 total) maximum of 100000
Batci Sux
Batchs Sua
0.12
Datch Sae 128
0.12
Batch 128
256
256
0.10
Validation Loss
0.10
Batch
Validation
0.08
Batch
0.08
0.06
0.06
0.04
0.04
10a
10
10
10a
Batches Seen
Samples Seen (Million)
Figure 6: Epoch Scaling. We run scaling HPT experiments along the number of total samples. Each point is the validation loss full pre-training run. Setting: HPT-S, 27 datasets with maximum of 1000 trajectories for each dataset. Left) We scale up the number batch sizes and measure the changes validation losses. Right) Derived from the left figure, we multiply the batches seen by the number of samples in cach batch.
and fixed iterations) might cause an early plateau performance around 1000 trajectories max per dataset, as shown the blue line Figure number of datasets with fixed number of epochs and evaluate on the fixed subset (first 10 datasets). We hypothesize that training with more embodiments contributes to the generalization the trunk. scale to the extent of 200k trajectories and 52 datasets.
Figure 5 (b), we also pre-train on an increasing
These experiments
Model Scaling. In Figure 7, we fix the number of datasets (27) RT-X and use maximum of 1000 trajectories for each dataset. We scale along model size (from 1M to 1B) and gradually increase the batch sizes from 256 to 2048 (doubles every order of model size increase) and use the larger dataset with 170k trajectories. We observe that when scale to bigger models with larger amounts of compute (red line), the pre-training can achieve low validation losses until is plateaued. We do not find a significant difference between scaling depth or scaling width.
Epoch Scaling, In this experiment, we fix the number datasets (27) and use a maximum of 1000 trajectories for each dataset. In Figure 6, we observe that increasing batch sizes (Left), which effec- tively scales training tokens (Right), can generally improve the model performance until convergence. Another observation we have is to use distributed workers to load from many datasets as possible aggregate each batch. We hypothesize that the large variance of training on heterogeneous datasets can be reduced by using a large batch size. See Appendix B for more experiment details.
4.3 Pre-training on Synthetic Data and Internet Human Videos
We experiment beyond real-world robot teleop data, which is expensive to collect and scale. For the additional datasets, we consider simulation datasets across many popular simulators Drake [82], Mujoco [90, 50], Isaac Sim [21]. and PyBullet [86, 81], as well as Sapien [53] and Flex [67], with
7

---

0.068
Defret
16k tra) 170k traj
Figure 7: Model Scaling. run scaling HPT experi- ments along model sizes. Each point is a full training run. Setting: 27 datasets with maximum of 1000 tra- jectories for each dataset. We scale along model size (from 1M IB) for both blue and lines. The red line is trained with increasing data and epochs to reach convergence. Specifically, we gradually increase the batch sizes from 256 to 2048 (doubles every order of model size increase) and use 170k trajectories.
Validation Loss
0.065
0.062
0.059
0.056
10a
10a Model Paramnter (Million)
10
10a
Validation Loss
7.00
Figure 8: Joint Pre-training with Simulation and Hu- man Videos. The baseline denotes the default setting without simulation and human datascts. Sctting: We run the experiments with training corpus of datascts with 1000 trajectories maximum.
[xO (x0,00)
6.75
6.73
6.70
6.72
6.67
6.50
Baseline
Sim
Human Sim-+Human
image inputs and expert demonstrations. For the human datasets that lack proprioception and action information, we use poses and 2D positions as surrogates for the supervised policy learning objectives. We use in total 300 trajectories from EPIC kitchen [15] and PoCo [83] with maximum trajectory length 1000. See Appendix Figure 13 and Table 4 for more details on the dataset compositions.
In Figure 8, we use a maximum of 1000 trajectories for each dataset and compare against the baseline of 27 datasets with evaluation on all the pre-trained datasets. We show that pre-training on additional embodiment datasets such as simulation and human video datasets can be possible, despite the large embodiment gaps with real robots. These datasets provide complimentary embodiment data to pure teleop data, and they illustrate how much heterogeneity can be handled in the HPT framework.
5 Experiments on Transfer Learning
In the previous section, we evaluate pre-training using the validation losses. In this section, we answer the following question with task success rates in transfer learning: Can the pre-traines HPT model be transferred to new embodiments, tasks, and environments simulation and the real world?
5.1 Transfer to Embodiments in Simulations
Protocol. We evaluate the pre-trained representations on robot manipulation simulation benchmarks Meta-world [90]. RoboMimic [50], and Fleet-Tools [82]. Each training dataset uses from 20-100 trajectories per task and each testing covers 50 episodes with different initial conditions. The policies HPT-Small as the pre-trained trunk and reinitialize the stem and head for transferring.
During the evaluation phase, we compare the following models: No Trunk uses only the stem and head without the trunk in the middle and trains from scratch as common practice [41]. From Scratch trains the entire policy from scratch with the trunk, Pretrained Frozen uses and freezes the pre- trained trunk during transfer learning and Pretrained Finetuned loads the pre-trained HPT-Base trunk and finetunes the whole network end-to-end, and Pretrained Finetuned (HPT-XL) uses the same fine-tuning procedure with a pre- praitrdi HPT-XL trunk with lower pre- pre-training validation loss. To reduce the variance, we conduct independent training runs and evaluations times and average for each model. The inference time during transfer HPT-base and 19Hz for HPT--XI, while a more recent GPU like A 100 can be 3-4 times faster.
RTX 3070 GPU is 47Hz for
Experiment. In Figure 10 (a), we test the model on the downstream tasks in closed-loop simulation and observe improved task success rate using the pre-trained models ranging from HPT-B to HPT-XL, although pre- traiiiii for the simulation experiments only happens in the real-world embodiments.
In Figure 10 (b), we run HPT on the recently released Simpler [43] Benchmark, which allows for comparing with Octo [56]. RT1-X, and RT2-X [14] on high-fidelity simulation. We focus on three different tasks Close Drawer, Move Near, and Pick Coke Can the Google EDR embodiment. For each task, we test several different initializations with a total of over 300 episodes for all tasks.
8

---

Figure 9: Simulation Evaluation Tasks. We evaluate HPT several simulation benchmarks and show
policy rollout visualizations experiments. Experiment details can be found in Section 5.1 A.4,
89
79
RT-1X
RT-2X
56
32
Cose Drewm?
Pick Cola Can
Nove Wea"
Figure 10: Success Rates in Simulation Experiments. (a) We evaluate transfer learning performance of models from HPT-B to HPT-XL on tasks across different simulator benchmarks. (b) We compare with several generalist models in the recent Simpler [43] benchmark with Google GDR embodiment. The pre-trained trunks trained from the Scaled Settings. The success rates computed over 150 rollouts per approach.
Note that the pre- pre-training corpus of HPT-S does not include [6]. and simulation tasks have a focus on language conditioning and do not expose proprioception inputs, which is not suitable for HPT. To address these issues, we finetune HPT on the supervised datasets with around trajectories under the simulation protocol. We use HPT-base as the backbone for this experiment. We use the baseline results from [43]. See Section A.4 for more implementation and experiment details.
5.2 Transfer to Embodiments in the Real World
Protocol. For the real-world experiments, we evaluate the HPTs on two different embodiments for tasks in pet care and assembly, which are not covered in the pre-training datasets [14]. In particular, for these two robots, experiment with different observation spaces I camera v.s. 2 cameras as well as different action spaces relative pose v.s. absolute pose. For data collection, we experiment with both an Oculus Quest collect relative pose control as action labels as well as kinesthetic teaching. The episode lengths of real-world teleoperation vary from 50 steps to 150 steps with 10 Hz control frequencies. We experiment with the tasks Sweep Leftover, Fill Water, Scoop Food and Switch Insertion, which require 5-20 seconds of interactions with granular small objects with fine contacts, shown Figure for 15 trials measure the average success rate.
collect around 100 demos each task and evaluate them
Experiment. We adopt similar transfer learning method in the previous section and evaluate the pre-trained HPT representations under real-world evaluation protocols. We train the policy with 20000 iterations with batch size of 256 and learning rate 5e We defer implementation details Appendix Section A.5. Quantitatively in Figure 12, we observe pre-trained policies attain better success rate over the No-Trunk and the From-Scras -com-Scratch baselines in Fill-Water use the state-of-ther thrt---rt- diffusion policy architecture to illustrate the ficxibility of the pre-trained representations. In Figure 11, qualitatively, we observe better generalization and robustness to varying poses and numbers of granular objects, and varying camera configurations and lighting conditions with pre-trained HPT.
baselines. In particular, the
On Table 3, we perform ablation study for the Sweep Leftover task. We also compare with R3M [54], Voltron [35], and VC-1 [49]. We use finetuned model with the released backbone and weights. We note that these previous works focus on only pre-training the vision encoders of the policies with human videos. Finally, we compared with policies that train from scratch (From Scratch) and policies that do not use proprioception during pre-training (No Prop. Finetuned) and add in proprioception afterwards. All of our experiments use pre-trained encoders and the trainable parameters (stem and head) can be as few 2% of the parameters.

---

Lefoover
Switch Insertion
Figure 11: Real World Quali- tative Results. Pre-trained HPT policies can perform dynamic and long-horizon contact-rich preci- sion tasks in pet care and assem- bly. The policies show robust and generalized behaviors under scene changes and disturbances.
Water
Time Scoop Food
Time
Method
Success (%)
From Scratch No Prop.
26.7+3.3
From Scratch
43.3+3.8
R3M [54]
50.0+3.0
Voltron [35]
46.7113.88
0.2
For Scrtch
VC-1 [49]
53.3:1:2.6
No Prop. Finetuned
63.3:42266
0.0 Sweepp Leftower
Water
Scoop Food
Switch Insettion
HPT-B Finetuned
70.0++.0
Figure 12: Transfer Learning the Real World. We evaluate the pre-trained HPTs on four tasks two embodiments. The average success rate with standard deviations is computed for 45 trials per approach. We use the default pre-training setup with HPT- Base for this experiment. See Scction 5.2 for detailed descriptions.
HPT-XL Finetuned
76.7+3.33
Table 3: Comparison on the Sweep Lef Leveover. We compare the fine-tuned HPT models with several baselines in- cluding vision-only pre-trained models.
Conclusion
There is room for improvement for many aspects including the dataset curation and pre-training objectives. Specifically, the embodiment splits in our balanced dataset mixture are rather simple. Moreover, careful data filtering to ensure the data quality is under-explored in this work. Also, this work has focused on supervised learning as the pre-training objective and the data size tokens and training compute sizes in FLOPs only reach a moderate scale of LLM training to ensure full convergence. Although the model architecture and training procedure are modular and independent of embodiment setups, heterogeneous pre-training can converge slowly. For evaluation, both the simulation and real-world evaluation tasks are restricted to short-horizon manipulation tasks with a fixed embodiment, which might limit the benefits of using higher-capacity model. Furthermore, the learned policies still do not offer very high reliability on the tested tasks (typically below 90%). See Appendix $C for some failure modes.
Given the recent surge of scaled data, robot learning is still limited by its generality because of the heterogeneity, including different embodiments, tasks, and environments where the robots are operated. To handle the heterogeneity common in robotics, we propose HPT, a modular architecture and framework to embrace this heterogeneity through pre-training. We explore and scale HPT with heterogeneous datasets to over 50 available datasets. The learned representation can be transferred and improve performance in both simulation and the real world, and it shows correlations with pre-training performance. The code" is open-source for future research. We hope this perspective will inspire future work in handling the heterogeneous nature of robotic data for robotic foundation models.
Acknowledgement. We would like to thank Russ Tedrake for discussions and suggestions, Liane Xu for helping with real-world experiments, Tianhong Li for helping with cluster experiments, and Remi Cadene for helping with the LeRobot implementation. computing resources to process training data. This work is supported in part by the Amazon Greater Boston Tech Initiative and Amazon PO No. DD-06310236 Toyota Research Institute provided funds to partially support this work.
thank MIT Supercloud for providing
hhtps://eituub..o//iiiiww///// and https//giitub.com//iiwww//ce
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[2] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint (tt:::2008.025588 2021.
[4] Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, and Ashish Kapoor. Pact: Perception-action causal transformer for autoregressive robotics pre-training. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3621-3627. IEEE, 2023.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint wt..ix:22220088/111 2022.
[7] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
[8] Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, and Thomas Wolf. Lerobot: State-of-the- machine learning for real-world robotics in pytorch. https: // //github. http://uuggnnnfaceeeeee//o
2024.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213 222.2.2260 Springer, 2020.
[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922 992444, 2020.
[11] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401. 12168, 2024.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[13] Cheng Chi, Siyuan Feng Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint a3X1v:2303.04137, 2023.
[14] Open X-Embodiment Collaboration. Open X-Fmhodiment: Robotic learning datasets and RT-X models. https:://obont o
transformer- x git.ub...u...... io, 2023.
[15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. European Conference on Computer Vision (ECCV),

---

[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[17] Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross- embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408. 18/2, 2024.
[18] Alaaeldin El-Nouby. Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training large autoregressive image models. arXiv preprint 1rxXiv2241.08544 2024.
[19] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135 PMLR, 2017.
[20] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[21] Ran Gong, Jiangyong Huang. Yizhou Zhao, Haoran Geng. Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Long-Chun Zhu, al. Arnold: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- self-supervised learning, Advances in Neural Information Processing Systems, 33:21271 2020.
[23] Siddhant Haldar, Zhuoran Peng. and Lerrel Pinto. Baku: An efficient transformer for multi-task policy learning. arXiv preprint arXiv:2406.07533 2024.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint 1rXiv:2111.0633 2021.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE conference on computer vision and putlern recognition, pages 9729-9738, 2020.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[27] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010.14701, 2020.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6885 68511, 2020.
[29] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training computc-optimal large language models. arXiv preprint arXiv:2203. 2022.
[30] Chip Huyen. Understanding evaluation metrics for language models. The Gradient, 2023.
[31] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs outputs. arXiv preprint arXiv:2 14795,
2021.
12

---

[32] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.044 04088, 2024.
[33] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: Robot manipulation with multimodal prompts. 2023.
[34] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[35] Siddharth Karamcheti, Suraj Nair, Annic Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302. 12766, 2023.
[36] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09244 2024.
[37] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.026643 2023.
[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[39] FrodoBots Lab. Frodobots-2k dataset. https: FrodoBots- 2K, 2024. Accessed: 2024-05-27.
[40] Seungjae Lee, Yibin Wang. Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181, 2024.
[41] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):13341133,, 2016
[42] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2351. 2023.
[43] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 2XX::2405505941 2024.
[44] Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Gosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient carly-fusion pre- pre-training with mixture of modality-aware experts. arXiv preprint arXiv:2407.21770, 2024.
[45] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, arXiv preprint arXiv:2304..0 08485, 2023.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 36, 2024.
[47] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171 2017.
[48] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots real time, IEEE Robotics and Automation Letters, 2023.
13

---

[49] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? 2023.
[50] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martnn. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[51] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey, Artificial Intelligence Review, 42:275-200 2993, 2014.
[52] Brandon McKinzie, Zhe Gan, Phillippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training. arXiv preprint arXiv:2403.00 22240.099111, 1, 2024.
[53] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107. 14483, 2021.
[54] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203./2601 2022.
[55] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint arXiv: h0t::/803.000 180300999, 2018
[56] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine Octo: An open-source generalist robot policy. https: //octo-models. github. io, 2023.
[57] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03748, 2018.
[58] OpenAl, Gpt-4 technical report, 2023.
[59] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[60] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188)99 2019.
[61] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning, pages 683-693. PMLRR 2023.
[62] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426. PMLR, 2023.
[63] Ilija Radosavovic, Bike Zhang, Baifeng Shi. Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402. 19469, 2024.
[64] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[65] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenherg, et al. A generalist agent, arXiv preprint 2rX5v:2205.06115. 2022.
14

---

[66] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[67] Gautam Salhotra, I-Chun Arthur Liu, Marcus Dominguez-Kuhne, and Gaurav S Sukhatme. Learning deformable object manipulation from expert demonstrations. IEEE Robolics and Automation Letters, 7(4):8775-8782, 2022.
[68] Saumya Saxena, Mohit Sharma, and Oliver Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 2nd Workshop on Language and Robot Learning: Language Grounding, 2023.
[69] Lucia Seminara, Strahinja Dosen, Fulvio Mastrogiovanni, Matteo Bianchi, Simon Watt, Philipp Beckerle, Thrishantha Nanayakkara, Knut Drewing, Alessandro Moscatelli, Roberta L Klatzky, al. A hierarchical sensorimotor control framework for human-in-the-loog robotic hands. Science Robotics, x(88)cad)5434, 2023.
[70] Younggyo Seo, Danijar Hafner, Hao Liu. Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332-1344, PMIR, 2023.
[71] Rutav Shah, Roberto Martin-Martin, and Yuke Zhu. Mutex: Learning unified policies from multimodal task specifications. arXiv preprint arXiv:2309. 0202200994320,, 2023.
[72] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. arXiv preprint 1xXiv:1701.06500 2017.
[73] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. Conference on Robot Leurning, pages 785-799, PMLR, 2023.
[74] Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pages 10193-10202. PMLR, 2021.
[75] Chameleon Team. Chameleon: lixed-modal early-fusion foundation models, 2024.
[76] DROID Team. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[77] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[79] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18:77-95, 2002.
[80] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arxiv:2310.01366, 2023.
[81] Lirui Wang. Yu Xiang. Wei Yang, Arsalan Mousavian, and Dieter Fox. Goal-auxiliary actor- critic for 6d robotic grasping with point clouds. In Conference on Robot Learning, pages 70-80. PMLR, 2022.
[82] Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, and Russ Tedrake. Fleet policy learning via weight merging and application robotic tool-use, 2024.
[83] Linui Wang, Jialiang Zhao Yilun Du, Edward Adelson, and Russ Tedrake. Poco: Policy composition from and for heterogeneous robot learning, 2024.
15

---

[84] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[85] Philipp Wu, Arjun Majumdar, Kevin Stone, Yixin Lin, Igor Mordatch, Pieter Abbeel, and Aravind Rajeswaran. Masked trajectory models for prediction, representation, and control. arXiv preprint w.w.:233500006666 2023.
[86] Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, et al. Trifinger: An open-source robot for learning dexterity. arXiv preprint arXiv:2008. 03596, 2020.
[87] Jonathan Yang, Catherine Glossop, Arjun Bhorkar, Dhruy Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross-embodiment learning for manipu- lation and navigation. arXiv preprint arXiv:2402. 19422, 2024.
[88] Jonathan Yang, Dorsa Sadigh, and Chelsea Finn. Polybot: Training one policy across robots while embracing variability. arXiv preprint arXiv:2307.00 00.30077779,,. 2023.
[89] Hanrong Ye, De-An Huang. Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, al. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335 2024.
[90] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robol learning, pages 1094-1100 100. PMLR, 2020.
[91] Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward H Adelson. Transferable tactile transform- ers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406. 16400, 2024.
[92] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware, arXiv preprint arXiv:2304. 13705, 2023.
16

---

A Implementation Details
Experiment Details. We conduct pre-training experiments across several orders of magnitudes in computes and data. The number of trajectories and transitions in the dataset is limited by the maximum number of trajectories in each of the constituent datasets. We use maximum episode counts per dataset ranging from 10 trajectories to 100000 trajectories, and the total trajectories range from around 300 trajectories and 6000 transitions to around 300k trajectories and 5 million data points. When training with 80k iterations, the approximate training epochs with fixed batch size 512 range from 200 epochs to 2 epochs. In terms of tokens, our experiment model ranges from 0.5 million to 1 billion parameters, the dataset tokens from all modalities range from approximately 32 million tokens to 5 billion tokens, and the tokens in a batch range from 0.03 million tokens 2 million tokens (including sequence length). The compute FLOPs range from 0.03GFlops to 31GFlops. See Table for more details of the scale and see Figure 13 for example lists of dataset mixtures. To facilitate future research, we will open-source the data processing scripts.
Different from previous work [56, 87], we use minimal amounts of processing and cleaning of the observation and actions in the raw trajectories. Specifically, the default training setup is to train 80000 iterations with a batch size 256, which is around 0.65B tokens in the latent space that feeds into HPTs and around 5B tokens in the perception token spaces of the raw perception inputs (such as image patches). Due to resource limits, for some bigger datasets such Droid [76], we did not process the full size.
A.1
Dataset Details
Real Robot Teleoperation Dataset. In total, we use a subset of 42 datasets in the Open-X X Em- bodiment dataset [14], including the recent Droid [76] dataset. These public datasets have high heterogeneity including distinct embodiment, environments to operate on, tasks to solve, etc.
Simulation Dataset. For the additional 7 simulation dataset,
the simulator benchmarks across all popular simulators Drake [82], Mujoco [90, 50], Isaac Sim [21], and PyBullet [8 J.J. as well as Sapien [53] and Flex [67], with image inputs and expert demonstrations. These are used as additional training data from the simulation domains.
Human Video Dataset. Since the human datasets do not contain proprioception and action informa- tion, we use hand poses and 2D positions in the image space as surrogates for the supervised learning objectives. In the PoCo [83] dataset, we use 3D positions of the hand and use 6-Dof poses extracted by ICP as actions, and for KitChKit [15], we use normalized 2D centers of the detection box as proprioceptions and the difference to the next frame as actions. We use in total of 2000 trajectories of video clips from EPIC kitchen with a maximum trajectory length of 500.
Deployed Robot Dataset. To further increase the heterogeneity in the pre-training dataset, consider FrodoBot-2k [39] dataset that involves plays of driving robots in the wild for gaming. This dataset is composed of deployed mobile robots in the wild. We the front camera for this dataset. The action space of this robot is parametrized by linear and angular velocity actions and the proprioception space includes measurements from the IMU, We use total of 150 trajectories and each trajectory contains more than 500 steps.
See Figure 13 for visualization of some examples of heterogeneous dataset compositions. In practice when training on the mixture of these datasets, users can define customized sampling weights or apply stratified sampling methods. For generality, in this work, use a balanced weight sampling method, commonly used in multitask learning. For these datasets, we process the visual features separately and save them on the disks.
A.2 Network Details
Stem. For vision inputs, we resize each image to the standard square size (224x224) before feeding into a ResNet18 [26] into a 7x7 vision modality token. These tokens are specifically the features before the final global pooling. If multiple views are available, we create individual projector MLP for each image view and then concatenate the vision tokens. For the vision encoders. In Figure 18, We have experimented with multiple vision encoders such as MAE ViT base [24], Dino V2[59], and CLIP ViT Base [60]. We choose ResNet and the default image size for their simplicity and common
17

---

Datasets Episode Max
Episode
Episado
Episoda
100000 Episode
Trajectory
Episode Step
Sample Weight
Grouped Sample Weight
Huran Videe
Figure 13: Large-scale Dataset Heterogeneity in Roboties. We show different dataset mixtures at increasing scales (top row) trajectory counts, dataset sample counts, and sampling weights (bottom row). We also show illustrations the different embodiments including real robots, simulations, and human videos. By default. during training, we use uniform distribution to sample from each of the embodiment datasets.
usage in policy learning. The investigation of more complex fusion and processing for vision features is left to future works.
When language is used, use T5 [64] to encode trajectory-level language modality tokens. Rather than using raw data and computing the tokens each time, these tokens are processed offline and saved datasets before training.
For low-dimensional inputs such as proprioceptions and actions, we first flatten these vectors and then apply normalization to each dimension, as single token. We have also experimented with increasing the dimensions of these modalities by adding sinusoidal position embeddings. For multiple steps in the observation horizon, we concatenate the sequence for each modality separately. At inference time, these tokens are forwarded once and cached avoid multiple calculations.
We apply cross attention (with sinusoidal position encodings) and a shallow MLP projector [42, 46] within the stem map different various sequences of tokens into fixed number of tokens for that modality. The cross-aae cross-attentios attention layer has 8 heads and 64 hidden dimensions per head. We map the modality tokens into tokens, 16 tokens, and 8 tokens respectively for image, proprioception, and language.
Trunk. The trunk is parametrized by a decoder-only transformer architecture with embedding dimension h that we ablate from 64 to 2048, and with block numbers ranging from 16 to 80. Note that the number of parameters for the trunk scales quadratically with the dimension size and linearly with the number of layers. The trunk also supports loading from existing large language models. Refer to Table. for more details on model sizes.
The code is modularized that the trunk training and transfer are independent of the encoder architecture or pre-trained weights of the stem, which can be ImageNet pre-trained ResNet [26], R3M [54], Voltron [35], Dino v2 [59], etc, and can be fintuned during transfer learning. It is also independent of the head, which can be diffusion policies, MLP, or transformer.
Head. For the head architecture, we normalize the action spaces of each embodiment dataset to be between -1 and element-wise, based on the dataset statistics, such that the output scale of the heads and the loss and gradients remain similar scales. Since action trajectories and observation history can often help with the robotic problem, we pick fixed action horizon (length 8) and observation horizon (length 4) for each embodiment. We then apply random masking along the time dimensions for each batch during training to be suitable for downstream tasks with different horizons.
18

---

Dataset
Irajectory
Irajectory %
Sample
Sample
Austin Sallor Dosases
205
0.09%
290167
1.5%%
Hydra
Austin Rods Datnett
27894
0.18%
Austin Sinus Dataset
478
0.21%
235175
1.50%
Berkeley MVP
410
0.18%
34039
0.22%
2.0%%
JAMLAb CMU Pickup Insert
539
0.24%
L18055
0.75%
UT Austin Mutes
1283
0.57%
295042
188
Imperial College Sawyer Wrist Cum
145
0.06%
4519
0.03%
Language Toblee
29554
13.245
179855
1.12%
Kuka
15903
7.13%
73158
0.47%
BC-Z
4365
1,96%
559019
3.57%
DLR Sam Pour
171
0.0 0.08%
19462
0.12%
Stanfond Robuzook
2103
0,94%
73493
0.475
CMU Play Fusion
192988
1.23%
Bridge
1.1%
Fumiture Bench Dutsst
2763
1.244
24247003
48.48%
UCSD Pick And Place Dutzset
11:58
00.211
45162
0.29%
USC Cluch Sim
68.1
0.31%
60876
0.39%
Multimudal
2365
Robotark
1535
0.091
127523
0.81%
KAIST Nonprehensile
171
0.08%
25:788
0L1eS
ASU Table Top
0.14%
UTokyy
Barkeley Cable Routing
1266
0.57%
19328
0.12%
Droid
294377
13.1996
2800000
17.8%
UIUC D3Field
164
0.07%
9803
0.06%
1012887
QUT Desterous Manipulation
171
0.08%E
150698
0.96%
NYU Door Opening Surptising Elfectiveness
372
0.17%
11415
0.07%
Franka Dataset
25536
0.16%
Mimic
203738
1.94%
ManiSkill Dataset
21346
9.57%
2969893
18.96%
Calanbia Cairl.ab Puslit Keal
103
20375
0.13%
Hose Minipulation
0.08%
Shared Control
Barkeley GNM SAC Son
2526
1.134
183078
1.17%
Berkeley Autolab URS
706
1.3416
06425
0.42%
Mobile
2.57%
Agent Affocdunces
127403
0815
Eppcc Kitchen
28
0.034
L73012
1.10%
PoCo Hammer
220
1.10
12517
0,08%
Sparuia
0.05%
0.11%
PyBullet Grasping Image
1788
0.808
51852
0.33%
MiJoC'o MetaWorld
741
0,3346
34725
0.22%
MaJoCo RobaMimic
180
6735
0.04%
Isaac Image
PyBullet Trakinger
147
0.07%
85385
0.37%
MuJoCo Adroit
90
0.04%
8010
0.05%
FrodoRot
0.05%
0.08%
Table 4: Detailed Dataset Mixture. We include the detailed number of trajectories and the number of dataset samples in the training mixture. These include 41 dataset from Open-X [14], 7 datasets from simulation, 3 datasets from human video, and 1 from in-the-wild deployed dataset.
We support various types of policy heads in the network architectures such as standard MLP, trans- former decoder, and diffusion policy. For the MLP head, we pooled the trunk feature (e.g. averaging) and then applied a 3-layer MLP. For the transformer decoder, we concatenate learnable tokens to the tokens before feeding into the trunk and use ID convolution layer on these output tokens to regress actions. For diffusion policy in real world experiments, we use a diffusion head and train with DDPM [28]. Finally, the actions are unnormalized based on dataset statistics and Huber losses are applied for regression. The reason behind Huber loss is to balance between the "difficult frame" in robot trajectories and the easy but lengthy part of the trajectories.
As discussed, HPT is a meta-level architecture where the stem, trunk, and head code are modular and each supports various architectural changes with pre-trained weights. The inference time, which includes all the pre-processing and encoder time, on the local computer with an old NVIDIA RTX 3070 GPU is 39Hz for HPT-base and 33Hz for HPT-xlarge. A modern GPU such A100 will likely improve the speed by 3-4 times.
A.3 Pre-training Experiment Details.
We train HPT with Adam [47] optimizer with weight decay ratio 0.05, and a base learning rate of 0.0002 with a cosine learning rate schedule with warmups and dropouts. We apply proportional scaling of the base learning rate depending the batch size. To support various horizons during
19

---

Diusion
Tranefomet
Posree
Pooking
Figure 14: Flexible Head Architectures in HPT. We highlight that our HPT architecture is meta-leve level architecture for policy learning. and it can work with various head architectures. The important takeaway is the scalable transformer architecture in the middle of the policy absorh the diverse data and provide tokens for these policy heads to regress on the action outputs.
Method
Dataset
Trajectories Model Size
Heterogeneous Proprioception
RT-1 [6]
0.1M
16M
RT-2X [14]
12
55B
Octo [56]
0.8M
93M
VL.n [36]
25
1M
7B
HPT
52
0.2M
1.1B
Table 5: Experiment Statistics. By leveraging heterogeneous datasets, the embodiment diversity in data and training scales reaches across several orders. Note that the training flops and the number of tokens are approximated from a single iteration and the model size only counts the trunk parameters (stem and head only have small active parameter count. HPT is also provided with multiple open-source implementations and extensive simulation evaluation tasks across 6 different benchmarks.
transfer learning, we apply random masking along the time dimensions for each batch during training. Since action trajectories can have imbalanced losses along prediction horizons, we use Huber loss with 8 0.1 (empirically found). We found the pre-training stage to be stable across various hyperaarameters in learning rate schedules and optimizers, and the choice of the validation dataset. The code is open-sourced and the pre-released model can downloaded easily from Huggingface.
In practice, since training losses can vary across different datasets and our goal is to perform well on all embodiments and tasks, we apply weighted sampling procedure for data loading. For every training iteration, we sample a dataset with inverse probabilities based on an exponential of its dataset size temperature. Specifically, we compute the squared root of each dataset size and sum these sizes to compute normalization constant. For each batch item, we then sample from these dataset with the corresponding probability. This prevents large datasets from dominating full training epoch, which is common practice in multitask learning.
Note that the stem and head for each embodiment are updated in different frequencies than the trunk, similar to a mixture-of-expert [72] training procedure. Especially under distributed training settings, each stem and head is trained with data from particular embodiment and tasks, and the trunk will accumulate gradients from all batches from training workers. The compute resources for these pre-training experiments range from 8 V- V-100s to 128 V-100s and the training time spans from half a day to I month. The total dataset disk size is around 10Tb and the RAM memory requirement below 50Gb. See Table 5 for summary details of the experiment.
A.4 Simulation Experiment Details
For simulation benchmarks, we use the released datasets as the expert demonstrations [90, 82, 13].
In summary, Metaworld [90] uses wrist camera view, and Robomimic [50] as well as Simpler [43] uses third-person view, with their own proprioception definitions by the dataset. Floet-Tools [82]
uses both views as inputs and uses the end effector poses as the proprioception inputs. We encode the image using pre-trained frozen ResNet features and normalize the proprioception inputs before passing them into the stem. We train single-task policies for all these simulation benchmarks except for Metaworld.
For the Simpler [43] benchmark, we focus on the Close-Drawer, Move Near, and Pick Coke Can task and Google EDR embodiment with visual matching setting. We test different initializations with total of 218 episodes. Note that the simulation tasks have focus on language conditioning and do not expose proprioception inputs, which is not the most suitable testbed for HPT. To address
20

---

(a)
(b)
(c)
Veltationn Loss
3010
6.22
0066
Baseline
+Prex Action
-Mullbiii
*Larguage
Figure 15: Additional Architectural Ablation. (a) We found that architecture changes on HPT-Base such as adding previous actions as inputs, multiview as inputs, and language input can help with HPT pre-trat trainingg performance. (b,c) We ablate other policy head architectures such as discrete classification heads as well as action token heads by scaling along the number of trajectories. The experiment is conducted under the default setting with HPT-Base, fixed 27 datasets with 1000 max trajectories in each dataset.
these issues, we finetune HPT on the RTX supervised datasets with 79 trajectories as other simulation benchmarks. We use HPT-base the backbone for this experiment.
By default, we train with 20000 iterations with batch size 512 and small learning rate 1e 1e-5. The image and state stem are one-layer MLP with hidden dimension 128 and the head is two-layer MLP. We only use an observation window of length and MI.P as the policy head. Each training dataset uses from 10-100 trajectories per task and each test covers 50 episodes with different initial conditions. Each trajectory in the simulation has a slight difference in scene initialization. To reduce the variance, we conduct independent training runs times, and the average for each baseline. Figure 9, show illustrations of some simulation tasks we evaluated.
A.5
Experiment Details
Task Definition. We experiment with robotic tool-use tasks Sweep Leftover, Fill Water, Scoop Food and Switch Insertion across two different robot setups. While for both setups, we Franka Panda the robot, note that the sensor locations well the action spaces drastically different. We collect approximately 100 demos for each task and evaluate each task for 15 trials to measure the average success rate.
During evaluation, human supervises the robot at all times. evaluation episode can termi- nated due to safety concerns, robot faults, timeout, etc. An episode is considered successful if it accomplishes the task. In the Fill Water task, the success score of means some water poured into the bowl. In the Sweep Leftover tasks, success score of I means all the piles are pushed into the plate, and success score of 0.5 means some piles are pushed into the plate. In the Scoop Food task, success score of means some dog food is scooped and all is poured into the bowl and score 0.5 means some food is scooped. In the Switch Insertion ion task [91], a success score of 1 means the switch is precisely inserted into the three pins on the PCB board. The robot moves to a pre-defined pose before it attempts the insertion. We pick these challenging tasks as they require contact-rich interactions with tools and granular objects, and they require high-precision dense contacts. We make sure the initial conditions of the robots are the same. Due to the complexity of these tasks and human errors, the initial conditions of the object setups not exactly the same.
well as
Transfer Learning. For the policy head in the real-world experiments, we have experimented with both MLPs and diffusion policies [13]. Fine-tuning only has active parameters of no more than 3Mb, compared to much bigger models (e.g. 100M) often used for single task in other works. We use an observation history window of with small learning rate 5 We train with batch size 256 single NVIDIA RIX 2080Ti GPU for 20000 iterations (around 4 hours of wall time).
B Additional Experiments
In this section, we present some additional experiments and ablation studies.
B.1 Additional Simulation Experiments
Based on the training curves of the four baselines in Figure 16 (a), we observe that leveraging a pre-trained HPT representation can often achieve lower validation loss curve (lower) during fine-tuning.
21

---

Fleet-Tools
Robomimic
MetaWorld
Pnstraine Finetunee
2+30
3+ 30
Photramed Treeen
2011
From Scratch
Yunk
20
surt
2000
4000
C000
33000
3000
10385
Pulting Homattors
Figure 16: Transfer Learning Objective. We transfer learning across several simulator benchmarks [82, 50, 90]. We compare the validation loss curves of several baselines with and without pre-trained HPT trunks.
The pre-trained trunks are trained from the Default Settings.
Real Robot Dataset
Successs lales
-eepoontt
Aloha Dataset Sample Efficiency
Transfer Learning Loss Curve
100
78
76
84
80
0.35
pre-taliedd
Success Rate
Maka Wottan
0.30
large ow teetrinne
a-arge pre trained
DP
(Scratch)
HPTXL
HPT-XL
40
0.25
100
62
62
68
20
6:55
Succas
100%
70%
50%
20%
Data Ratio
ACT
(Scratch)
HPTXXL HPTXL
Taining Bararama
Figure 17: Simulation Task Performance compared with Single- Taak Policy in LeRobot Implementation. We do evaluation in a different implementation in unseen simulation benchmarks. Left) we show that an improvement in performance can be achieved with pre-trained HPT trunks and outperforms single-task state-of- the-art architectures. We note that HPT trunks have not been pre-trained with diffusion heads and transformer decoder heads. Middle) we show the sample efficiency ablation study for Hase. Right) We show model size ablation study in loss curves.
In Figure 10 (a), we run HPT on the Simpler [43] Benchmark, which allows for comparing with Octo [56], RTI-X, and RT2-X [14] on high-fidelity simulation. We focus on three different tasks Close Drawer, Move Near, and Pick Coke Can in the Google EDR embodiment. For each task, we test several different initializations with a total of over 300 episodes for all tasks. Note that the pre-training corpus of HPT-S does not include [6]. and simulation tasks have a focus on language conditioning and do not expose proprioception inputs, which is not suitable for HPT. To address these issues, we generate training data using the RT-1[6] the expert, and finetune HPT on the RT-IX supervised datasets with around 50 trajectories and the simulation protocol. We use HPT-base with language tokenizers as the backbone for this experiment. We use the results from the paper [43].
Additionally, under the LeRobot [8] implementation, we compare HPT with of-e-of-the-e art single- task policy architecture such as Diffusion Policy [13] and ACT [92]. In particular, we inherit simple version of these complex policies as the head architectures (Figure 14). We run full training runs with 50 evaluation trials every 20000 training steps, and use the maximum task success rates during the 100000 training steps for comparisons. with keypoint representations using the diffusion head and achieve similar success rates at 78%. We also compare with ACT on the Aloha Transfer Box task with the image representations using the transformer decoder head and achieve similar success rates at 60%. This showcases the flexibility of HPT architecture to work with state-of-the-arr policy architectures for gegh-freuuncy control in fine-grained tasks. Moreover, in the experiment with 50 episodes in total and HPT-base, we observe an improvement in sample efficiency with pre-trained models. We also see improvement in transfer learning loss with pre-trained models at inereasing scales.
Figure 17, we compare HPT with DP on the PushTT task
In Figure 10 (b), we ablate on the number of visual and proprioceptive tokens in the simulation transfer learning experiments. We observe that missing either information hurts the performance of the downstream policies. See Section A.4 for more implementation and experiment details.
B.2 Ablation Study on the Stem
In this experiment, we fix the number of datasets (27) in Open-X datasets and use a maximum of 1000 trajectories for each dataset. We consider several ablations the stem part of the HPTs.
22

---

Validation Loss (x0. (x0.001)
10
8.61
10
8.21
Volidation Loss [x0
10
6.73
6.73
6.73
6.84
6.80
6.68
Ve Valiiation (20,
5
w/o prop.
prop.
ResNet
DINO
CLIP
vir
(a)
(b)
(c)
Figure 18: Ablation Study on HPT Stem. We ablate the pre-training performance (a) proprioception,
(b) vision stems, and (c) vision encoders. Setting: HPT-S, batch 256, iterations 80000, 27 datasets with
maximum of 1000 trajectories for each dataset.
(a) Test Initial Conditions
(b) Failure Cases
Figure 19: (a) Initial Condition Overlay. We visualize different rollout initial conditions during test times. (b) Failure Cases the Learned Policy in the Real World. The robot sometimes has issues exccuting very precise manipulation.
Specifically, in Figure 18 (a, b), we observe an increase in validation losses when not using either the proprioception information the vision. Intuitively, using such proprioception vision information would make learning action predictions from heterogeneous datasets very challenging. This also implies that both information critical for policy pre-training scale.
We also conduct an ablation experiment over vision backbones on smaller subset of the training datasets among several popular vision encoders such ViT-base [24] and DiNO [59] (Figure 18). Further ablating on input image resolution or joint finetuning of the vision backbone on the downstream task success rates will be interesting future work. Although the default implementation focuses on single-view visual information, the stem can naturally extend to multiple views and other modalities such languages and action history.
B.3 Pre-training Ablation Study
In Figure 15, we conduct several ablations to pre-train the HPTs, such as using multiple views to provide 3D information implicitly. using languages for task guidance, and using previous action trajectories as additional ablations. We found these ablations to improve over the default HPT setting with single view and vision and proprioceptive inputs. These ablations are more pronounced in certain datasets such multiple views for insertion [68], and language modality for Language Table [48]. Using previous action trajectories helpful in providing additional context and embodiment information well. We believe that integrating multiple modalities and investigating how to handle missing modalities is an exciting future direction. We leave the exploration of these ablations to future work.
From the architecture perspective, we also ablate over the token sizes and observation history and do not find big effect on the averaged validation loss. We hypothesize there is trade-off between the information given to the policy and the desired generalization. In Figure 15(b.e), we have also experimented with discretized version [6] of the policy head and architecture that uses additionally learned position encodings as action tokens transformer. with conv ID head for action regression. We opt for regression on continuous values for its generality. An initial investigation of the attention map of the transformer blocks shows that there dense attention weight attributed to the proprioception and vision tokens.
23

---

C Failure Cases
In Figure 19, we show some failure cases of the learned HPT policies in the real world. One of the common issues is overshooting or undershooting. For example. the policies tend pour the water before it reaches the mug or pour the dog food a bit in front of the bowl. These issues could be due to the spatial precision of the policies and due to the data qualities failing to uncover the causal relationships. More targeted data recollection and better finetuning of the vision encoders might help address these issues.
D Discussion and Future Directions
We first discuss the metric used for measuring pre-training performance, intrinsic evaluation. Validation (or training) loss is a common metric to evaluate the progress of large-scale pre-training [34]. It is based on the goal of training a generalist model, where even fitting all the data can be challenge [58], as opposed to training a specialist model where overfitting is often appreciated. This also considered step towards more scientifically studying pre-training and scaling behavior in robotics [30]. Previously, people often rely on single binary metric for evaluating task success rates in the real world, with only some amount of test trials.
Admittedly, there are several caveats to this metric. From the evaluation perspective, the averaged validation loss depends the overall multitask losses of all datasets. For example, increasing the number of trajectories in one dataset might lower validation loss on the associated dataset, but may not lead to an overall loss decrease. In practice, the exact subset for evaluation and the number of training steps in each dataset also play a role in the averaged validation loss metric. For example, when evaluating whether additional datasets to the default setting contribute to the representation learning, selecting the default datasets allows us to compare on the same metric. But these datasets are inevitably trained less, if we fix the number of total training iterations. Moreover, the validation loss during pre-training and the downstream policy learning performance have an evaluation gap due to task differences, and downstream policy learning and task execution have another evaluation gap due to closed-loop execution.
Given the recent surge of scaled data, robot learning is still limited by its generality because of the heterogeneity, including different embodiments, tasks, and environments where the robots are operated. We propose HPT, novel architecture and framework to embrace this heterogeneity through pre-training. We align proprioception and vision information of different embodiments through a modular stem, shared scalable transformer trunk, and task-specific heads to actions. We explore and scale HPT with heterogeneous datasets to over 50 available datasets. The learned representation can be transferred and improved performance in both simulation and the real world. We hope this perspective will inspire future work in handling the heterogeneous nature of robotic data.
Since fine-tuning is still required for robotics generalist models, future research directions include exploring different algorithms including network architectures that incorporate embodiment-speciife structures, such as URDF, into network architecture well tokenization. One can also explore training objectives beyond supervised learning. As we move towards scaling robot pre-training, more high-quality diverse datasets with clear annotations would be crucial, including teleoperation data, simulation data, human videos, and deployed robot data. Understanding what kinds of mixture will contribute better representations is interesting future work.
Due to the complexity of real-world evaluation, large-scale unified simulation benchmarks with varying dexterity and generalization challenges would be very crucial to consistently compare among different models. For real-world policy performance, we believe that extending to longer-horizon fine manipulations with bimanual mobile setup would be interesting future works.
Future works can also study and extend scaling laws in policy learning, and explore representations from heterogeneous data in other domains beyond robotic policy learning. Finally, leveraging more modalities and domains in robotics such as 3D point clouds, tactile data, simulation domains, and human data, etc, is worth investigating.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to claborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we make the relevant claims on heteroge- neous pre- training, scaling behaviors, and transfer learning experiments as backed by our experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
In Section 6, we discuss the limitations and potential future work of this project.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical results associated with this work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation details are disclosed to full extent the appendix. The code are also open-source for checking experiments relevant to the main results.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open-source the code with detailed instructions to reproduce the main experimental results. The data can be generated on the fly as they come from online public datasets in Figure 13.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //n httllps.ccppblicc/eee hwwpollicggiiss////eee /CoeeeemomdSammii asi assionPoll onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
27

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section $4 and Section $A, we discuss all the relevant training and test detials, including dataset splits, training hyperparameters, type of optimizers. We will also open-source our code to follow open research.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in the appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Most training curve experiments (measured with validation losses) in Figure 8, 6, 5,7 are relatively stable with statistical significance. For simulation experiments in Figure 16 with task successes the metric,
Justification:
experiments and average across 5 trials, which results in stable and small variances. For real world task performance experiments in Figure 11 and 12 and 3, report the standard errors across different trials for each model for each task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eттor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
28

---

Answer: [Yes]
Justification: In Section A.3, we discuss the compute resources, including workers, memory, time of execution for experiments this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tw.risuu...pp/. wwttrrps.cccpppeeee cctppblic//aeeecccccccccc csGiideeleleesss
Answer:
[Yes]
Justification:
The code of ethics is strongly followed in this work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The project focuses on pre-training from heterogeneous robotic data, There no direct societal impacts associated with it.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
29

---

11. Safeguards
Question:
Does the paper describe safeguards that have been in place for the responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We released a series of robotic models that are trained action data and output action data. The data all have public licenses and do no contain any sensitive or risky information.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The released code, data, and models are under MIT license.
The code and models are free to use by most audience for the purpose of advancing the field.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes] The new released models and codebases are well documented for the released version.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
30

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: This project does not include crowdsourcing experiments with human subjects. The involved human video datasets are directly used from the dataset. The robot teleoperation is conducted by the authors.
Guidelines: [NA]
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: There is no human subjects with crowsourcing experiments in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "YBdix3s3I3", "reviewer": "Reviewer_Vi4L", "summary": "This paper presents an approach for training large transformer behavior models on diverse heterogenous data. This involves separate encoders for proprioception and image observations for each embodiment, as well as different action decoding heads. Experiments include validation loss comparisons.", "questions": "1. How does the presented approach perform when deployed on real/simulated robots in terms of success rate, and how does it compare to previous approaches [1,2,3] ? Is it important to have separate encoders/decoder for each observation/action modality?\n\n\n\n\n\n\n\n\n[1]: RT1-Robotics Transformer for Real-World Control at Scale, Brohan et al.\n\n[2]: Octo - Octo: An Open-Source Generalist Robot Policy, Octo Model Team\n\n[3]: OpenVLA: An Open-Source Vision-Language-Action Model, Kim et al.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "While training large models that can learn from diverse heterogenous datasets is an important problem of relevance to the community, the current paper has some key limitations, please see weaknesses.", "weaknesses": "After rebuttal - Concerns regarding validation loss vs success rate has been addressed with additional experiments, claims have been further strengthened with real world results\n\n########################\n\n1. Evaluation Metrics in the experiments\n\nFor training robot behavior models, the standard metric used in the community is the success rate of the trained model after being fine-tuned on a few demos for a new task [1,2,3]. The success rate measures the robot's ability to actually perform tasks of interest, using the large pre-trained weights. This is a more accurate metric than validation loss for how to judge if the large scale model training is making use of the diverse quantities of training data. Validation loss is not perfectly correlated with downstream robot performance. If it is difficult/cumbersome to do these in the real world, this analysis should atleast be done in simulation. The reason validation loss is a much weaker indicator in the robotics setting is because it doesn't measure how good the model is on the on-policy distribution, which can diverge wildly from the expert data distribution used for training. \n\n2. Design choice justification\n\nThe proposed approach has separate encoders and decoders for each observation and action modality. This seems like it could lead to less sharing of information across diverse datasets, leading to poorer transfer capabilities. Why should we not instead have a single action decoding head, as is done in [2]? What if the same image embedding backbone is used for all all observations - this would lead to the sharing of features potentially enabling better transfer. This choice of separate encoders/decoders for each modality is not sufficiently ablated and analyzed. \n\n\n\n\n\n\n\n[1]: RT1-Robotics Transformer for Real-World Control at Scale, Brohan et al.\n\n[2]: Octo - Octo: An Open-Source Generalist Robot Policy, Octo Model Team\n\n[3]: OpenVLA: An Open-Source Vision-Language-Action Model, Kim et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the extensive feedback and suggestions. We have revised the manuscript and **conducted new experiments in the attached PDF** and will address your questions below. \n\n> Evaluation Metrics in the experiments. Validation loss is not perfectly correlated with downstream robot performance. If it is difficult/cumbersome to do these in the real world, this analysis should at least be done in simulation.\n\nThanks for bringing this up and we are aware of the limitation on validation loss. Therefore, significant experiments have been done in simulation (**Figure 16 b**). We have shown that the more well-trained model HPT-XL can outperform HPT-base in the simulation benchmark. Moreover, we have added additional experiments to compare against several generalist policies in the recently released Simpler Benchmark. See **Figure 13(a)** for the result. While evaluation and metrics are still huge issues for robotics and generalist models, we believe that our simulation experiments are more extensive than previous works such as RT, Octo, and OpenVLA. Moreover, we kindly note that OpenVLA is released **after** the paper is submitted.\n\nWe also copy the general response here for completeness.\n\n- We highlight that the experiments in this work are three-fold: pre-training, evaluation in the simulation benchmark (**Figure 16**), and evaluation in the real world (**Figure 14, 15**). Only in the pre-training part, where we need to extensively measure **the scaling behaviors of the learned representations**, do we use the validation losses. It’s not an ideal choice, but it’s a common choice adopted by foundation model training in both LLM and multimodal foundation models. As an analogy, there are also potential issues raised on the validity of perplexity for a small dataset in language modeling (which can also have out-of-distribution issues), but it is generally acceptable that perplexity is a good metric for studying large-scale pre-training these days.\n\n- We note that the standard practice of robot learning (specialist model) is to overfit on ~100 trajectories with one day of training, in which cases validation loss might not be that indicative, and evaluation is not that difficult. But in our case, we are training with hundreds of GPUs and weeks of time with massive amounts of robot embodiments and trajectories (generalist model), where validation loss should be more representative of the learning progress.\n\n- Additionally, we have proposed a simple fix (using smooth L1 loss) to make it more balanced between the difficult and easy parts of the trajectories. This does not include the high costs of evaluating representations in robotics. Previously, validation losses were simply ignored by practitioners, and our work is one step towards that direction. We believe that validation loss can be a more continuous reflection of representation at a large scale, and we will further revisit and investigate the correlations at a large scale in future work.\n\n- Last but not least, we did many experiments in section 5 on transfer learning in simulation and the real world in the submitted version. The simulation experiment is more extensive than previous works with thousands of runs. These experiments have showcased the correlation between validation loss and transfer learning performance (**Table 4 and Figure 16b**), where larger and more well-trained models indeed can perform better. We wanted to highlight that previous works pre-trained on Open-X have been evaluated mostly using the training embodiments, which we do not have access to and we argue that measuring representation learning performance in a novel environment is also a valid choice. Our real-world evaluation tasks are also more challenging than previous works.\n\n\n\n> The proposed approach has separate encoders and decoders for each observation and action modality. This seems like it could lead to less sharing of information across diverse datasets, leading to poorer transfer capabilities. Why should we not instead have a single action decoding head, as is done in Octo?\n\nThe key motivation is that different observation and action spaces are very heterogeneous: they have different dimensions and live in different spaces. We want to separate out the part that can be potentially shared, which is the middle part of the network that contains more abstract representations. The proposed method is one architecture choice to handle such issues.\n\n**Encoder:** It is technically impractical to map different dimensions of tensors (in proprioception) to the same space with the same network, unless we have a unified tokenization method. Our proposed cross-attention (or resampler) is one method to unify different inputs. In Octo, they do not consider proprioception and therefore can use a domain-specific token to represent the heterogeneity. When they need to handle the proprioception, they still need to reinitialize a new encoder for such domain and input (see their Berkeley Insertion experiment).\n\n\n**Decoder**: We want to highlight that each of the action spaces can be drastically different from one another in robotics (such as pose and joint angles). Moreover, a finetuning procedure is still needed for any generalist model that we discuss in robotics. Octo trains a shared decoder based on transformer architecture and needs to reinitialize the task token during finetuning. In some cases, they still have to finetune the decoder head. The compute efficiency and the generalist design are not that different from our case. Indeed, in the early part of the project, we also experimented with Octo-like architecture with task-specific decoder tokens, but we did not find a big difference. We prefer to not expose the task information to the trunk to ensure it can learn task-agnostic representations. \n\n\n(continued)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDq3gZmHIg", "reviewer": "Reviewer_AQbN", "summary": "This paper aims to pretrain a policy representation across a variety of simulated and real robotics datasets such that it can be quickly adapted to new downstream problem instances (embodiment, environment, task). The key technical contribution is the design of a Transformer architecture that can consume a variety of robot data (i.e., observations and actions unique to each robot dataset) via a large sharable trunk + a number of small networks that translate dataset-specific information to this shared representation space. Experimental results indicate that the proposed architecture scales both in terms of dataset size / diversity and model capacity, and that it can be transferred to new real-world problems by finetuning on relatively few demonstrations (<=100).\n\n**Post-rebuttal update:** I believe that my concerns have been addressed by the author rebuttal. I was already leaning towards acceptance and have now updated my score from 6 -> 7.", "questions": "In addition to addressing my comments above, I would also like the authors to clarify the following:\n- L282 states 20-100 demos are used, whereas L300 states that as few as 10 demos are necessary. Can the authors point to such an experiment, if available? Or is this merely an observation made during preparation of the paper? I did not find any such experiments in the appendices.\n- Figure 10 (left) seemingly does not include the test loss of the larger model, whereas it is included in Figure 10 (right). Why is that?", "limitations": "I believe that limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Problem is interesting, timely, and likely to be of relevance to the NeurIPS community.\n- Paper is well written and easy to follow. I believe that there is sufficient discussion of related work for unfamiliar readers to appreciate the technical contributions.\n- The proposed architecture is fairly simple and intuitive. I consider this a strength. A series of ablation experiments validate the approach.\n- Limitations of the evaluation metric (validation loss) is clearly stated. I don't think anyone would disagree that this is a major limitation, but I appreciate the transparancy. Unfortunately the community does not have many viable alternatives at the moment. Transfer performance seem to correlate somewhat with the validation loss.", "weaknesses": "I'm overall quite pleased with the paper in its current form. However, there's a couple of thing that I'd like to point out:\n- Since the authors are conducting transfer learning experiments in simulation (in addition the real experiments), I would have liked to see more ablations in this setting. Transfer performance seems like a way stronger signal for ablations, e.g., validating that more data does indeed lead to better transfer, not just better validation loss.\n- It is not clear to me how the authors arrived at certain experimental setup choices. For example, the authors state on L282 that they use 20-100 demonstrations per task in experiments, but do not specify whether the model is transferred to each task *individually* or *jointly* (i.e., finetuning jointly on N*T demonstrations for T tasks and then evaluating vs. finetuning T models each on N demonstrations). It is also not clear to me which tasks require more demonstrations + whether this relates to the number of tasks considered in a given transfer experiment (e.g. 100 demos per task for Meta-World 5 vs. 20 demos per task for Meta-World 20 would be a reasonable comparison if the model is finetuned jointly on all tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the extensive feedback and suggestions. We have revised the manuscript and **conducted new experiments in the attached PDF**  and will address your questions below. \n\n> Since the authors conduct transfer learning experiments in simulation (in addition to real experiments), I would have liked to see more ablations in this setting. Transfer performance seems like a way stronger signal for ablations, e.g., validating that more data does indeed lead to better transfer, not just better validation loss. \n\nIn addition to the real-world experiments in **[Table 4]** where we show a better-trained trunk can achieve better performance and compare with several ablations. In **[Figure 16]**, we also show that a better-trained HPT-XL model can outperform an HPT-Base model across all the simulation benchmarks. We also have done additional simulation experiments **[Figure 13b]** on transferring with and without vision and proprioception tokens to illustrate their benefits. Hope this addresses your concerns.\n\nWe also copy the general response here for completeness. \n\nWe highlight that the experiments in this work are three-fold: pre-training, evaluation in the simulation benchmark (**Figure 16**), and evaluation in the real world (**Figure 14, 15**). Only in the pre-training part, where we need to extensively measure **the scaling behaviors of the learned representations**, do we use the validation losses. It’s not an ideal choice, but it’s a common choice adopted by foundation model training in both LLM and multimodal foundation models. We note that the standard practice of robot learning (specialist model) is to overfit on ~100 trajectories, in which cases validation loss might not be that indicative, and evaluation is not that difficult. But in our case, we are training representations on massive amounts of robot embodiments and trajectories (generalist model), where validation loss should be more representative of the learning progress.\n\nAdditionally, we have proposed a simple fix (using smooth L1 loss) to make it more balanced between the difficult and easy parts of the trajectories. This does not include the high costs of evaluating representations in robotics. Previously, validation losses were simply ignored by practitioners, and our work is one step towards that direction. We believe that validation loss can be a continuous metric of representation without running infinite physical experiments at a large scale, and we will further revisit and investigate the correlations at a large scale in future work.\n\nLast but not least, we did many experiments in two sections on transfer learning in simulation and the real world in the submitted version. The simulation experiment is more extensive than previous works. These preliminary experiments have showcased the correlation of validation loss and transfer learning performance (**Table 4 and Figure 16b**)We wanted to highlight that previous works pre-trained on Open-X have evaluated mostly using the training embodiments, which we do not have access to. Our real-world evaluation tasks are also more challenging than previous works.\n\n> For example, the authors state on L282 that they use 20-100 demonstrations per task in experiments, but do not specify whether the model is transferred to each task individually or jointly (i.e., finetuning jointly on N*T demonstrations for T tasks and then evaluating vs. finetuning T models each on N demonstrations).\n\nOther than the meta-world setup (where single-task evaluation becomes too easy), all experiments, including the real-world task, have training conducted on each task individually, as a standard setup. So we use “finetuning T models each on N demonstrations setup.”\n\n> It is also not clear to me which tasks require more demonstrations + whether this relates to the number of tasks considered in a given transfer experiment (e.g. 100 demos per task for Meta-World 5 vs. 20 demos per task for Meta-World 20 would be a reasonable comparison if the model is finetuned jointly on all tasks).\n\nThe only joint multitask experiment is meta-world. In that case, we try to use a demonstration number such that the trained policies performance is within a reasonable range, between 50 and 90 percent success rates. indeed, when we train with more tasks, we do not have to use the same number of trajectories for each task. There is no particular rule for which tasks require demonstrations in practice. It mostly depends on the learner's performance.\n\n> L300 states that as few as 10 demos are necessary. Can the authors point to such an experiment, if available? Or is this merely an observation made during preparation of the paper?\n\nThe 10 demo requirement is an observation that we made on the robomimic Lift task, which is arguably a simple simulation benchmark task. \n\n> Figure 10 (left) seemingly does not include the test loss of the larger model, whereas it is included in Figure 10 (right). Why is that?\n\nFor scaling ablation experiments, we mostly use the pre-training experiments as well as the task performance success rates as the metric, as the success rate has been brought up to be more important for robotics. We have completed a new set of experiments with different scales of pre-trained models. Please see **Figure 16(b)** for more details. \n\n\nThank you very much again for your constructive feedback. Please do not hesitate to let us know if you have any other questions and/or comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7orCIo7lRZ", "reviewer": "Reviewer_wUp5", "summary": "This paper presents Heterogeneous Pre-trained Transformers (HPT), a method for training robotic models that addresses the challenge of heterogeneity across different robot embodiments and tasks. HPT pre-trains a shared neural network trunk to create a universal representation, which is then fine-tuned for specific tasks, showing improved performance and scalability with diverse datasets as well as demonstrating effective generalization to new tasks in both simulation and real-world settings.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper shows that pre-training policies across heterogeneity can exhibit scaling behaviors of validation losses.\n\nThis paper is well-written and easy to understand.", "weaknesses": "Despite the difficulty of large-scale evaluation, there is no inevitable relationship between the loss values and the task success rates. For instance, the main challenge in some tasks lies in certain key frames, rather than the average error in lengthy trajectories.\n\nIn a sense, verifying the scaling behavior at the level of loss values is quite evident.\n\nIn terms of architectural design, this paper (stem, trunk and head) does not offer significant technical contributions.\n\nLacking comparisons with baselines (only some vision encoders on Sweep Leftover Task), such as RT-X and Octo.\n\nIn Figure 5, why is the loss lower without vision than without proprioceptive information? I am puzzled. Does \"no vision\" here mean that the policy has no visual information as input? Why can a reasonable loss value still be obtained without vision?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the extensive feedback and suggestions. We have revised the manuscript and **conducted new experiments in the attached PDF** and will address your questions below. \n\n> Despite the difficulty of large-scale evaluation, there is no inevitable relationship between the loss values and the task success rates. For instance, the main challenge in some tasks lies in certain key frames, rather than the average error in lengthy trajectories.\n\nThank you for bringing this up. Indeed, we use the smooth L1 loss to balance between the “difficult part” and the easy but lengthy part of the trajectories. I agree that it’s not optimal to use the validation loss and therefore we conduct many transfer learning experiments and only use the validation loss to evaluate the pre-training performance. Please see the paper for more discussions on using this metric. Since the submission, we have also conducted new experiments on a new real-world embodiment and did additional experiments in simulation benchmarks. \n\n**We also copy the general response here for completeness.**\n\nWe highlight that the experiments in this work are three-fold: pre-training, evaluation in the simulation benchmark (**Figure 16**), and evaluation in the real world (**Figure 14, 15**). Only in the pre-training part, where we need to extensively measure **the scaling behaviors of the learned representations**, do we use the validation losses. It’s not an ideal choice, but it’s a common choice adopted by foundation model training in both LLM and multimodal foundation models. We note that the standard practice of robot learning (specialist model) is to overfit on ~100 trajectories, in which cases validation loss might not be that indicative, and evaluation is not that difficult. But in our case, we are training representations on massive amounts of robot embodiments and trajectories (generalist model), in which faithful evaluation of such representation without expensive rollouts is still an open question. Compared to earlier works that ignore this metric, we believe validation loss is one step towards that goal and should be representative of the learning progress in this process than single-task training. We also investigate their correlations with transfer learning performance in later experiments, where we show that bigger and more well trained models indeed perform better.\n\nAdditionally, we have proposed a simple fix (using smooth L1 loss) to make it more balanced between the difficult and easy parts of the trajectories. This does not include the high costs of evaluating representations in robotics. Previously, validation losses were simply ignored by practitioners, and our work is one step towards that direction. How to evaluate robot representations without running a large amount of rollouts is still an open research question. We believe that validation loss can be a more continuous reflection of representation at a large scale, and we will further revisit and investigate the correlations at a large scale in future work.\n\nLast but not least, we did many experiments in the two sections on transfer learning in simulation and the real world. The simulation experiment is more extensive than previous works with thousands of runs. These preliminary experiments have also showcased the correlation of validation loss and transfer learning performance (**Table 4 and Figure 16b**). We want to highlight that previous works pre-trained on Open-X have evaluated mostly using the training embodiments, which we do not have access to. Our real-world evaluation tasks are also more challenging than previous works.\n\n> In a sense, verifying the scaling behavior at the level of loss values is quite evident.\n\nThank you for bringing this up. This scaling behavior analysis has become a norm in the LLM domain, but in robotics, there has not been any work investigating similar problems to the best of our knowledge. Under such heterogeneous training across diverse embodiments and tasks, it is unclear to us beforehand if the training will even converge or proceed without instability issues. We will highlight the part on model transfer learning in the revised version.\n\n> In terms of architectural design, this paper (stem, trunk, and head) does not offer significant technical contributions.\n\nThank you for bringing this up. Indeed, the architecture was designed from a simple and general mindset to tackle the problem of heterogeneous learning. It is different from a monolithic model such as Octo and OpenVLA, in order to handle the heterogeneity in embodiments, and is also aligned with the multimodal learning literature. Besides the heterogeneous pre-training and architecture contributions, our work also provides insights into the scaling behavior investigations and conduct evaluation in simulations. (continued)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ra75jsuiuD", "reviewer": "Reviewer_KFGf", "summary": "The paper introduces HPT a large-scale transformer model pretrained on multi embodiment robotic data. The main idea is to split the policy into three parts: a dataset specific token encoder for images and proprioceptive information called Stem. A shared Trunk, that processes all latent tokens and a dataset specific action head. By splitting the architecture into dataset specific encoders and dataset agnostic trunk the model is able to better process data from different robot embodiments and scale well using diverse data from Open-X-Embodiment Dataset (OXE). The proposed architecture is tested on pretraining on OXE and adapted for sim and real robot experiments.", "questions": "- Can you add adequate baselines such as Octo for the related experiments in sim and real to better justify the performance of HPT as mentioned in the weakness section. \n- I found the part about the image encoder a bit short of details. Is the ResNet pretrained on Image-Net in the final model or which pretrained image encoder are you using? Have you experimented with also fine-tunining it? Since ViT has a lower validation loss why did you decide to use ResNet as the default? Which pre-trained Vit are you using for the ablation? More details on this part would be helpful for the reader. \n- How fast is the inference time for the different models?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Simple but yet effective method that shows good scaling properties for a general multi-embodiment robot transformer. \n- Large scale pretraining study covering 50 datasets and many robot embodiments. Most diverse pretraining in the context of robotics that I am aware of. The reported results shows good potential for further exploring heterogeneous robot data sources\n- Insightful scaling analysis, which do not exist in the context of robotics yet.\n- Several additional ablation studies to further study the impact of future analysis with discrete action spaces more input data and first results for fleet learning experiments underline the potential of the architecture.\n- The paper is well structured and easy to follow.", "weaknesses": "- No adequate baselines in the real world: A comparison of a model pretrained on full robot trajectories with action prediction against a few pretrained vision encoders that are mostly trained on robotic-free first person human videos is not a fair or useful comparison and does not provide relevant insights for the readers. At least, I expect a comparison against Octo [1] as another generalist policy pretrained on OXE. Also the experiment is missing baselines trained from scratch such as Diffusion Policy or ACT. Further the experiments in the real world only cover a single robot embodiment. \n- For a method that is motivated by better learning from heterogeneous robot data, some additional results with other robots having different action spaces would be required to showcase the advantages of the proposed method\n- No adequate baselines in simulation: The experiments on robotic datasets such as Robomimic and the real world again do not contain adequate comparisons. Specifically in the Robomimic experiment there is no comparison against state-of-the-art in-domain Diffusion Policy or ACT trained from scratch. The are no insights if the proposed model pretrained on diverse data improves upon these models trained from scratch. While comparison against the same architecture with different ablations exist, comparisons against other architecture are missing. Experiments on the recently proposed SimlerEnv [2] could also be interesting to compare the performance of the method against other large-scale pretrained methods such as Octo and RT-2X.\n- Inference time: While the performance of larger variants of HPT is promising, I am not convinced by the idea that an $80$ layer transformer model is a suitable backbone for a policy used in a real robot context. \n- Several points about the image-encoder are not well described and miss relevant details.\n\n[1]: Team, Octo Model, et al. \"Octo: An open-source generalist robot policy.\" _arXiv preprint arXiv:2405.12213_ (2024).\n\n[2]: Li, Xuanlin, et al. \"Evaluating Real-World Robot Manipulation Policies in Simulation.\" _arXiv preprint arXiv:2405.05941_ (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the extensive feedback and suggestions. We have revised the manuscript and **conducted new experiments in the attached PDF**  and will address your questions below. \n\n> No adequate baselines in the real world .... I expect a comparison against Octo [1] as another generalist policy pretrained on OXE. Also the experiment is missing baselines trained from scratch such as Diffusion Policy or ACT. Further the experiments in the real world only cover a single robot embodiment.\n\nWe have added additional experiments to compare against several generalist policies in the recently released Simpler Benchmark. See Figure 13(a) for the result.  Specifically, we compare the models across 3 different tasks in the Google GDR embodiment with the visual matching setup. We have found HPT to perform quite well on these benchmarks compared to Octo for instance. For the real-world experiments, unfortunately, We have not been able to make Octo finetuning work well in our real-world setup despite weeks of effort, so its current finetuning performance does not match really well. It can be trained on the local machine but the trained policies are not doing anything meaningful. We hypothesize that there might be issues in the finetuning process.\n\n\nThe Fill-Water task in the real-world experiment in **Figure 14,15** is actually conducted with the **diffusion policy** architecture. Therefore the from-scratch baseline represents state-of-the-art single-task diffusion policies. We use the relative policy action output for the diffusion policies. We observe an improvement by using the pre-trained trunk compared to the train-from-scratch baselines. We highlight again that the ACT or diffusion policies are only considered heads of specialist policy in this framework, so it’s orthogonal to our contribution and is only used to showcase the flexibility of the pre-trained representations to improve model performance. In the Appendix, we also investigate the difference between using different heads in pre-training. We will clarify these in future versions of the writing.\n\n> For a method that is motivated by better learning from heterogeneous robot data, some additional results with other robots having different action spaces would be required to showcase the advantages of the proposed method.\n\nThanks for bringing this up. We have added additional experiments on a different embodiment conducting real-world insertion tasks. The action space of this robotic setting is absolute pose, rather than the relative pose used in the pet care setting. Please **Figure 14 and Figure 15 left** for more details and visualizations. We hope this answers your question.\n\n> Inference time: While the performance of larger variants of HPT is promising, I am not convinced by the idea that an 80 layer transformer model is a suitable backbone for a policy used in a real robot context. How fast is the inference time for the different models?\n\nThe inference time, which includes all the pre-processing and encoder time, on the local computer with an **ancient** NVIDIA RTX 3070 GPU is 39Hz for HPT-base and 33Hz for HPT-xlarge. It is expected to be 3-4 times faster on a more modern A100 used in OpenVLA. Indeed, the largest model can sometimes be overkill for local deployment of robot policies. But the HPT-XL model can easily run in real-time with our limited hardware in our real-world experiments.\n\n> Can you add adequate baselines such as Octo for the related experiments in sim and real to better justify the performance of HPT as mentioned in the weakness section.\n\nWe have added additional experiments to compare against several generalist policies in the recently released Simpler Benchmark. See Figure 13(a) for the result. Specifically, we compare the models across 3 different tasks in the Google GDR embodiment with the visual matching setup. We have found HPT to perform quite well on these benchmarks compared to Octo for instance. For the real-world experiments, unfortunately we have not been able to make Octo finetuning work well in our real-world setup despite weeks of effort, so its current finetuning performance does not match really well. It can be trained on the local machine but the trained policies are not doing anything meaningful. We hypothesize that there might be issues in the finetuning process. \n\n> I found the part about the image encoder a bit short of details ..... More details on this part would be helpful for the reader. Several points about the image-encoder are not well described and miss relevant details.\n\nThanks for bringing this up. We have added additional details to the network section in the appendix. We have copied it here for completeness. For vision inputs, we resize each image to the standard square size (224x224) before feeding into an Imagenet pre-trained ResNet18 into a 7x7 vision modality token, which are the features before the final pooling layer. If multiple views are available, we create individual projector MLP for each image view and then concatenate the vision tokens. We use ResNet mostly for its simplicity and legacy reasons (as the majority of the dataset is only processed once at the beginning of the project). \n\n\nDue to the computation resource limitation and other practical reasons, we did not finetune it as in other VLM works by default. We were using the ViT-base from the masked autoencoder and CLIP for vision encoder ablation. We hypothesize that the vision features for robotic control do not differ by that much across these features learned from the standard images found in vision benchmarks or the internet. We think exploring this further can be interesting future work.\n\nThank you very much again for your constructive feedback. Please do not hesitate to let us know if you have any other questions and/or comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable and constructive comments. Overall, we appreciate that most reviewers acknowledged the contributions, **“important problem of relevance to the community.”** (from Reviewer Vi4L), **“The proposed architecture is fairly simple and intuitive.”** (from Reviewer AQbN) and noted the paper shows **“pre-training policies across heterogeneity can exhibit scaling behaviors of validation losses.”** (from Reviewer wUp5) and is **“Insightful scaling analysis, which does not exist in the context of robotics yet”** (from Reviewer wUp5). 

In the two months since the submission, we have worked continuously on the paper to improve the quality. Many questions and concerns raised by the reviewers are answered through our additional experiments and we hope that reviewers  take the new experiments into consideration when making their final recommendations. We have attached a **rebuttal PDF summary** of Tables and Figures for the **additional experiments** (with highlights in blue) and will discuss them below. 

**Additional Experiments:**

**Simulation**: We redesigned **[Figure 16 (b)]** and added policy fine-tuning performances from larger HPT backbones. We conducted additional simulation experiments such as ablating the effects of vision and proprioception tokens in **[Figure 13 (b)]**. In **[Figure 13(a)]**, we discussed new experiments on the **Simpler [2] benchmark which demonstrated strong performance compared with other generalist models including RT-1, RT-2, and Octo.** In **Figure 17**, we highlight our extensive evaluation of the simulation across 4 different benchmarks and thousands of runs. [Reviewer Vi4L,wUp5,KFGf,AQbN]. Additionally, we have included a novel deployed robot dataset Frodobot [1]. We have revised the experiment in **[Figure 16 (a)]** and focused more on the effects of adding embodiment datasets to the pre-training performance.


 **Real-world**: We have added more sweep-food experiments with larger pre-trained trunks in **[Table 4]**. We ablate against pre-trained models without proprioception and compare with more baselines. Importantly, we also added additional experiments on the insertion task with a **new robot embodiment** with different observation and action spaces, shown in **Figure 14**. These new experiments further validate the performance of HPT pre-trained model. We have not been able to make Octo finetuning work well in our real-world setup despite weeks of effort, so its current finetuning performance does not match really well. OpenVLA is **released after our submission**, and its inference GPU requirement exceeds our real robot setup. [KFGf]



[1] “FrodoBots-2K Dataset”, FrodoBots Lab, 2024

[2] “Evaluating real-world robot manipulation policies in simulation”, Li et al., 2024.


Now we will summarize a few answers that hopefully address some of the common questions posed by reviewers and we will also briefly clarify a few points.

***Key Answers***



**1. Validation loss and evaluation metric [Vi4L,AQbN,wUp5]**: 

- We highlight that the experiments in this work are three-fold: pre-training, evaluation in the simulation benchmark (**Figure 16**), and evaluation in the real world (**Figure 14, 15**). Only in the pre-training part, where we need to extensively measure **the scaling behaviors of the learned representations**, do we use the validation losses. It’s not an ideal choice, but it’s a common choice adopted by foundation model training in both LLM and multimodal foundation models. As an analogy, there are also potential issues raised on the validity of perplexity for a small dataset in language modeling (which can also have out-of-distribution issues), but it is generally acceptable that perplexity is a good metric for studying large-scale pre-training these days.

- We note that the standard practice of robot learning (specialist model) is to overfit on ~100 trajectories with one day of training, in which cases validation loss might not be that indicative, and evaluation is not that difficult. But in our case, we are training with hundreds of GPUs and weeks of time with massive amounts of robot embodiments and trajectories (generalist model), where validation loss should be more representative of the learning progress.

- Additionally, we have proposed a simple fix (using smooth L1 loss) to make it more balanced between the difficult and easy parts of the trajectories. This does not include the high costs of evaluating representations in robotics. Previously, validation losses were simply ignored by practitioners, and our work is one step towards that direction. We believe that validation loss can be a more continuous reflection of representation at a large scale, and we will further revisit and investigate the correlations at a large scale in future work.

- Last but not least, we did many experiments in section 5 on transfer learning in simulation and the real world in the submitted version. The simulation experiment is more extensive than previous works with thousands of runs. These experiments have showcased the correlation between validation loss and transfer learning performance (**Table 4 and Figure 16b**), where larger and more well-trained models indeed can perform better. We wanted to highlight that previous works pre-trained on Open-X have been evaluated mostly using the training embodiments, which we do not have access to and we argue that measuring representation learning performance in a novel environment is also a valid choice. Our real-world evaluation tasks are also more challenging than previous works.

(continued)

Meta-review: This submission proposed a new method, called HPT, for quick (task, embodiment, and environment) adaptation by pretraining a policy representation from large datasets. In order to encourage shared learning across multi-modal data, the model divides the policy into three parts: a dataset-specific token encoder (Stem), a shared processing trunk, and dataset-specific action heads. The authors showed that this approach of dividing into three parts helps effective scaling and adaptability on both simulated and real-world tasks. The results are shown by finetuning with few demonstration in the Open-X dataset.

Overall, reviewers agreed that the method and the results are compelling and it should be shared with the community. There are several experiments and analysis reviewers asked for to further improve the submission. Please add them to your camera-ready copy.
Decision: Accept (spotlight)