Robot Learning with Sensorimotor Pre-training
Authors: Ilija Radosavovic
Baifeng Shi
Letian Fu
Ken Goldberg
Trevor Darrell
Jitendra Malik
Venue: CoRL 2023 Oral (2023)
Keywords: Robot Learning
Self-supervised
Sensorimotor
Pre-training

Abstract: We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. Given a sequence of camera images, proprioceptive robot states, and actions, we encode the sequence into tokens, mask out a subset, and train a model to predict the missing content from the rest. We hypothesize that if a robot can predict the masked-out content it will have acquired a good model of the physical world that can enable it to act. RPT is designed to operate on latent visual representations which makes prediction tractable, enables scaling to larger models, and allows fast inference on a real robot. To evaluate our approach, we collected a dataset of 20,000 real-world trajectories over 9 months using a combination of motion planning and grasping algorithms. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.

Robot Learning with Sensorimotor Pre-training
Ilija Radosavovic Baifeng Shi Letian Fu Ken Goldberg Trevor Darrell' Jitendra Malik
University of California, Berkeley
Abstract: We present self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. Given a sequence camera images, proprioceptive robot states, and actions, we encode the sequence into tokens, mask out subset, and train model to predict the missing content from the rest. We hypothesize that if robot can predict the masked-out content it will have acquired good model of the physical world that can enable it act. RPT is designed to operate on latent visual representations which makes prediction tractable, enables scaling to larger models, and allows fast inference on real robot. To evaluate our approach, we collected a dataset of 20,000 real-world trajectories over 9 months using com- bination of motion planning and grasping algorithms. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
Keywords: Robot Learning, Self-supervised, Sensorimotor, Pre-training
1
Introduction
Over the last couple of years, inspired by vision [1, 2, 4] and language [5, 6, 7], there has been an increased interest in pre-training for robotics. For example, we have promising results from self-supervised visual pre-training on large and diverse image collections [8]. However, robotic data contains rich sensory and motor information that is difficult to capture with visual pre-training alone. We ask: can we learn good sensorimotor representations from robotic trajectories?
In this paper, we propose a self-supervised sensorimotor pre-training approach for robotics. We for- mulate robotic pre-training this idea through a masked prediction task, similar to the counterparts in natural language process- ing and computer vision [6, 3, 4J. We hypothesize that if a robot can predict missing sensorimotor content it will have acquired good model of the physical world that can enable it to act.
general sensorimotor sequence prediction problem. We instantiate
Our model, called RPT, is a Transformer [9] that operates on sequences of sensorimotor tokens. Given input sequence of camera images, proprioceptive robot states, and actions, encode the interleaved sequence into tokens, mask out subset of the sequence, and predict the masked-out content from the rest. We perform masking across all modalities and time using high masking ratio, which encourages the model to learn cross-modal, spatio-temporal representations.
We encode camera images using a pre-trained vision encoder [8] and use latent visual representations for sensorimotor sequence learning. This enables us build on strong visual representations, trained on large and diverse image collections from the Internet. Compared to prediction in pixel space, performing prediction in the latent representation space makes the task more tractable. This design decouples the computational vision cost from the sensorimotor context length, making 10 Hz control with over 300M parameter models and large context lengths feasible on a physical robot.
Equua contribution. Videos are available on our project page.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Real-World Trajectories
Downstream Transfer
Sensorimotor Pre-training
images, propricception
Robot learning with sensorimotor pre-training. Left: We collect a large dataset of real-world trajectories that contain multiview RGB images, proprioceptive robot states, and actions to use for sensorimotor pre-training. Middle: Given a sensorimotor trajectory, we mask out a subset (shown by striped pattern) and train Transformer model predict the masked-out content from the rest. Right: We transfer the pre-trained representations to different downstream tasks and robots.
Figure 1:
To study our pre-trate vre-training approach, we collected a dataset of over 20,000 real-world trajectories over months using a combination of motion planning and model-based grasping algorithms. Each trajectory is a sequence of multi-view RGB camera images, proprioceptive robot states, and ac- tions. We collected trajectories for classic robotic tasks, namely, single object picking, bin picking, stacking, and destacking. All of the tasks include variations in object pose, shape, and appearance.
To understand the effect of pre-training, we perform a series of real-world experiments. We find that RPT consistently outperforms training from scratch and that the improvements are larger for harder tasks (up to for the block stacking task). We also find that our sensorimotor pre-training approach enables successful transfer across different tasks, lab environments, and robots. Moreover, approach has favorable scaling properties and benefits from better vision encoders, longer sorimotor context lengths, and larger pre-training datasets. Finally, find that masking across both modalities and time, with high masking ratio, is important for good performance. We encourage the readers to see the the extended version of this work on arXiv and also to check the project page.
2 Robot Learning with Sensorimotor Pre-training
Our approach consists of pre-training and a fine-tuning stage (Figure 1). We pre-train sensorimotor representations with masked prediction on sequences of camera images, proprioceptive states, and actions (Figure 2). After pre-training, we transfer representations downstream tasks (Figure 3).
2.1 Sensorimotor Pre-training
We begin by describing the pre-training stage of our approach. In the pre-training stage, given dataset D of sensorimotor trajectories T. Each sensorimotor trajectory is sequence of camera images, proprioceptive robot states, and actions: T (i1, $1 #i,..., ér, sr, ar). We assume no access to additional semantic information, like language instructions or task labels. We hypothesize that the unlabeled sensorimotor trajectories implicitly encode the structure the physical world and that we can use them learn general sensorimotor representations for downstream robotic tasks.
We formulate robotic pre-training all modalities and time. We instantiate this idea through a masked prediction task, similar to the counterparts in vision and language [6, 3, 4]. We mask out a subset of trajectory and train a model to predict the missing content. Specifically, given a sensorimotor sequence of L tokens, we sample a mask sub-sequence M C [1, L] and train model to minimize the mean squared error of the masked tokens TM conditioned on the observed tokens TI..IIMM The intuition is that if robot infill missing sensorimotor content it will have acquired good model of the physical world that enable it act. This general formulation enables us represent many different contextual prediction problems by simply using different masking patterns. We consider number of variants, including random masking at the modality, timestep, and token level, as well as causal masking.
general sensorimotor sequence prediction problem, across

---

Transformer
M
M
M
M
M
M
Figure 2: Sensorimotor pre-training. Our model is a Transformer that operates on interleaved sequence of camera images, proprioceptive robot states, and past robot actions. We encode sensori- motor inputs into tokens, mask out subset, and train model predict the missing content.
2.2 Architecture
Token masking. Given a sequence of sensorimotor tokens we mask out a subset. We represent each masked out imput using mask token that is learnt per modality and shared across time. We mask the input tokens independently without taking the time step or modality into consideration which encourages the model learn correlate information across both modalities and time.
Vision latents. Our sensorimotor model must process sequences of high- -igh-dimensooal images from multiple views, which is challenging from both the learning and the computational perspective. To overcome this, we use a pre-trained vision encoder to compute visual representations [8] and operate in the latent space. This enables us to build on strong visual representations, trained on large and diverse images collections from the Internet. Compared to prediction in pixel space, prediction in the latent space makes the task more tractable. This design also decouples the computational vision cost from the sensorimotor context length, making fast inference with large models feasible.
Token encoders. We use a separate linear encoder per modality. Since our modality encoders do not share weights, we do not use additional modality embeddings. To represent time, we add positional embeddings to each token. All tokens from single timestep share the positional embedding value.
Transformer model. The encoded and masked sequence of tokens is passed into Transformer model [9]. We follow the standard Transformer design consisting of a series of Transformer blocks with multi-head self-attention operations. Our model predicts latent representations for each input.
Prediction heads. We decode the hidden representations into prediction targets using linear project layers. Each latent is decoded from the hidden size back the original modality dimension. We make predictions in the original input space for joints and the visual latent space for images.
Masked objective. We compute the mean squared error reconstruction loss between the predictions and the ground truth input values. We apply the loss only to the predictions from the latent repre- sentations corresponding to the masked inputs. Predictions for the observed inputs incur no loss. To weight the importances and scale of different modalities we apply a per-modality loss weight.
2.3 Downstream Transfer
Our goal is to learn general sensorimotor representations that can be transferred to different down- stream tasks and robots. Inspired by vision and language [6, 3, 4], we explore two settings: (1) Fine-tuning. We fine-tune pre-trained model checkpoint on downstream task data. In this setting pre-training can be seen as providing good initialization for learning. (2) Linear probe. We use frozen pre-tree mreitraned model to extract sensorimotor features and train single linear layer predict actions. This enables us to evaluate the quality of the pre-trained representations alone.

---

a
Training
Frozen
M
(a) Finetune
(b) Linear Probe
Figure 3: Downstream transfer. We consider two different settings for evaluating representations on downstream tasks: (a) Fine-tuning: We fine-tune the entire pre-trained model on the downstream task data; (b) Linear Probe: We freeze the pre-trained model and train a linear action read layer.
3 Experimental Setup
Robot and tasks. We use 7-DoF Franka robot with the default 1-DoF parallel jaw gripper. We perform joint position control at 10 Hz. We include joint positions and the gripper state in the proprioceptive information. We use three RGB cameras, one attached to the robot hand and two on the sides (Figure 9). We consider four different tasks: Pick, Bin Pick, Destack, and Stack. To study the proposed approach, we collected a dataset of sensorimotor trajectories using a combination of motion planning and model-based grasping algorithms (see Appendix A for details).
Vision encoder. We use the Vision Transformer (ViT) architecture [10] to encode image inputs. Specifically, we use the pre-trained models from [8] which were trained via MAE [4] on a collection of 4.5M images from Ego4D [11]. Epic [12]. Something -oaitiinn-- [13], 100 Days of Hands [14], and ImageNet [15] images. We extract features from all three cameras using the same model. We use the mean pooling of output tokens from the vision encoder as the vision feature.
Sensorimotor transformer. Our sensorimotor model is an encoder-only Transformer with ~IM parameters. The transformer has a hidden dimension of 192 and four transformer blocks, each with heads and an MLP ratio of 2. The sensorimotor input contains multiple steps, each step including the visual features from 3 cameras views, the proprioceptive states and actions. Since the input from different modalities have different dimensions (e.g., 768 for images and 8 for proprioception and actions), we project each modality the same dimension of 192 using linear layer per modality.
Pre-training. During sensorimotor pre-training, we first randomly sample masking probability from a fixed range of masking ratios, and then independently mask each input token (visual, propri- oception, or action) with the same probability. The default masking ratio range is [0.7, 0.9]. which we find to work the best empirically. We also study other masking strategies such as masking all the tokens from modality or timestep. Please see ablations on masking strategy and masking ratio in Section 4.5. The masked reconstruction loss for each modality is weighted together using uniform weight. All models preitrrined for 300 epochs with batch size of 4096 and 50 warm-up epochs. We use the AdamW optimizer with learning rate of 4 x 10-2 and weight decay of 0.01.
Fine-tuning. We initialize the sensorimotor model with the pre-trained weights and fine-tune it with behavior cloning on the downstream task. At fine-tuning time, the model takes in a sequence of the same length as in pre-training with the action at the last time step replaced with mask token. To predict the action, other predictions. We train linear layer to predict next 16 actions from the last mask token. We fine-tune the model for 900 epochs for the task of Stack and 300 epochs for other tasks. Other hyperparameters such as learning rate and batch size stay the same as in pre-training.
the output token that corresponds to the mask input token and discard the
Inference. When testing the fine-tuned model, we feed the past sensorimotor trajectory as input and the model predicts the actions for the next 16 steps. The predicted actions are passed the robot controller which executes the actions at 10Hz. We experimented with executing one action the time and re- re-predicting but did not observe significant difference in performance. After each action is executed, the visual observations and the achieved state are recorded and, alongside the executed action, are fed back as the sensorimotor inputs for the next prediction in the autoregressive fashion.

---

100
Stack
RPT, pre-trained
Success (%)
80
MVP, scratch
60
40
Pick
Destack
20
RPT, pre-trained
RPT. pre-trained
MVP, scratch
MVP, scratch
50
100
150
200
250
100
200
300
400
500
100
200
300
400
500
Num. Trajectories
Num. Trajectories
Num. Trajectories
Figure 4: Sample complexity, fine-tuning. We study the effect of sensorimotor pre-training the amount of fine-tuning data increases. We find that sensorimotor pre-training (RPT) brings consistent improvements over training the sensorimotor model from scratch (MVP) and that the gains larger for the harder tasks (Stack). Note that the vision encoders are pre-trained and frozen for both [8].
4 Experimental Results
We perform evaluations in the real-world and study sample complexity on different tasks, transfer across tasks and robots, scaling properties, and different design decisions. For all experiments, report success rates across 16 real-world trials with variations in object positions and orientations.
4.1 Sample Complexity
We begin by studying the effect of sensorimotor pre-training by comparing its fine-tuning perfor- mances to training form scratch. As our seratch baseline we use an improved version of MVP [8], where the single-step MLP policy is replaced by a multi-step Transformer policy. For fair compar- isons, we use the same pre-trained vision encoders, sensorimotor architectures, and optimize the learning rate, batch size, and the number of epochs per model. We consider three downstream tasks of increasing difficulty: picking, destacking, and stacking. We pre-train using different subset of the data from the same task as in fine-tuning and hold out unseen set for evaluation. In Figure we show the performance as the number of fine-tuning demonstrations increases. We observe that pre-training leads to consistent improvements over training from scratch, across different tasks and data regimes. Moreover, the improvements are the largest for the hardest block stacking task.
4.2 Transfer Across Tasks
In the previous section used the data from the task for pre-training and fine-tuning. To evaluate if our pre-training approach can learn general sensorimotor rep- resentation instead of representation for a specific task, we study pre-training and fine-tuning across different tasks. Specifically, we first pre-train a model on data from different tasks: stacking, picking, successful bin picking trajectories, and bin picking trajectories that include fail- cases. We then fine-tune and evaluate the models stacking. We report the results in Figure 5. We observe that pre-training on all of stacking, picking, or bin pick- ing lead to similar downstream performance on stack- ing, which suggests that our sensorimotor pre- pre-traiee gre-trannin ap- proach can learn transferable representations across tasks. We also see lower performance when pre-training on all of bin picking data that includes failed trajectories, which highlights the importance of pre-training data quality.
Stack
80
Success (%)
Stack
Pick Bin pick
Bin all
Figure 5: Transfer across tasks. We compare pre-training different tasks and fine-tuning on stacking. We see that pre-training can lead to strong down- stream performance even across tasks.
5

---

Vision Encoder
Context Length
Pre-training Data
60
Success (%)
50
20
100
200
300 0
40
80
500
1000
1500
2000
Num. Parameters (M)
Num. Tokens
Num. Trajectories
Figure 6: Scaling studies. We find that our approach benefits from better vision encoders (left),
larger context lengths (middle), and more pre-training data (right). Evaluated on block stacking.
4.3 Scaling Studies
Vision encoder. We study the performance of our pre-training approach as the size of the vision encoder increases. In all cases, consider three ViT variants of increasing size: ViT-S, ViT-B, and ViT-L. We evaluate perfor- mance on the block stacking task which requires precise spatial localization. The results are shown in Figure 6, left. We observe that the performance improves significantly with better vision models. We note that our model is still capable of 10 Hz inference even when using the ViT-L vision encoder.
a pre-trained and frozen vision encoder from [8] Specifically,
Context length. We compare sensorimotor pre-training with varying context lengths. Namely, we consider context lengths of 8, and 16 timesteps. Note that each timestep contains tokens. In Figure 6, middle, observe that pre-training larger contexts leads to consistent improvements, which may suggest that longer contexts may facilitate richer sensorimotor pre-training problems.
Pre-training data, We study scaling of our approach as the amount of pre-training data increases. We consider pre-training on 480, 960, and 1920 trajectories and evaluate downstream performance on block stacking, In Figure 6, right, we observe that our approach benefits from more pre-training data which is promising signal for scaling sensorimotor pre- pre-training to larger trajectory collections.
4.4 Transfer Across Robots
Cross-lab transfer. We evaluate transfer across different robots. We first consider transfer to different instance of the same robot type. The downstream robot is in a dif- ferent lab that has differences in the environmental condi- tions, varying camera placement, background, and light- ning. We compare pre-training We report the results in Figure la and observe that pre- training outperforms training from scratch considerably.
pre-train
fine-tune
success (%)
Franka B
25.0
Franka
Franka B
68.8
(a) Franka + Franka. Effective transfer from a franka in one lab to another franka in different Iab, with differences in era positions, background, and lightning.
training from scratch.
pre-train
fine-tune success (%)
Cross-robot transfer. Next, we push this setting further and evaluate transfer across different robot types. Specifi- cally, we pre-train on xArm data from [8] which includes 640 trajectories collected via teleoperation across 8 dif- ferent tasks. Note that the gap here is quite large, with differences in the robot type, camera type and placement, tasks, background, lightning, data frequency, and collec- tion strategy. We compare pre-training on xArm to (a) no pre-training and (b) pre-training on the same amount of original Franka data. In Table 1b, we that pre-training on xArm outperforms training from scratch considerably and comes very close to pre- tre-nrainin the same robot.
Franka 25.0
xArm
Franka 50.0
Franka
Franka
56.3
(b) xArm - Franka. We find that senso- rimotor pre-training he effective even across different robot types come very close to pre-training on the same robot.
Table 1: Transfer across robots. We pre-train on data from one robot and evaluate downstream transfer to dif- ferent robot. We experiment with both cross-lab and cross-robot transfer.

---

Partial Fine-tuning
Masking Type
Masking Ratio
Success (%)
80
60
20
Lin. probe
Full fine-tune
1
2
Time
Modality
Token
Low
Medium
High
Num. Blocks Fine-tuned
Figure 7: Ablation studies. We find that while linear probing achieves non-trivial performance, fine-tuning leads to considerably better results and that fine-tuning a larger portion of the model leads to higher success (left). We observe that masking across all tokens works considerably better than masking all tokens from a timestep or all tokens from a modality at a time (middle). Moreover, using a high masking ratio essential for learning good sensorimotor representations (right).
4.5 Ablation Studies
Linear probe. We evaluate pre-trained sensorimotor representations via linear probing. Namely, we extract representations using a frozen sensorimotor model and train linear layer on top using 120 trajectories for picking. We report the results in Figure 7, left. We observe that linear probing reaches a success rate of 43.753.. 75% which is non-trivial but considerably lower than 93.8% with fine-tuning.
Partial fine-tuning. In fine-tuning evaluations fine-tune the full model by default. Here, study the impact of fine-tuning an increasing number of Transformer blocks. We report the results in Figure 7, left. Note that fine-tuning zero blocks and four blocks is equivalent to linear probing and full fine-tuning, respectively. We observe that the performance increases with the number of blocks fine-tuned and that achieving best performance requires fine-tuning majority of the blocks.
Masking type. We experiment with different masking strategies (see also Section 4.8). Specifically, consider time-step masking which masks all tokens from time step, modality masking which masks all of the tokens from modality, and token masking which masks across all tokens indepen- dently. In Figure 7, middle, we see that token masking outperforms the alternatives considerably.
Masking ratio. We ablate different values of the masking ratio. shown in Figure right, there a significant performance drop when masking ratio low (0.1,..9 or medium [0.4, 0.9] compared high masking ratio 0.9). This suggests a high redundancy between different time steps and modalities in the input sensorimotor sequences and high masking ratio is crucial for lcarning useful representations for downstream tasks. This is consistent with prior work on visual pre-training [4].
4.6 Inference Speed
Our sensorimotor model designed to operate on latent visual representations that are computed per image, which decouples the vision model from the sensorimotor model and makes the vision computation cost independent of the context length. This enables us to use large vision models (up to ViT-I with 307M parameters) with system level inference speed of 10Hz 2080 Ti GPU.
4.7 Emergent Self-Correction
We observe that some of our models that are pre-trained with sensorimotor prediction can exhibit an emergent self-correction behavior at test time. For example, if the robot fails to grasp an object initially, it would move back, and proceed to grasp the object successfully. Please see the project page for videos. Since these models were pre-trained and fine-tuned only with successful trajecto- ries, this type of behaviors have not been seen during training. Moreover, some of the states were present in the data at all (e.g., starting grasp or moving up with fully closed gripper).
7

---

4.8 Causal Masking
We experiment with a variant of our sensorimotor pre- training approach that uses causal masking instead of ran- dom masking, like in the experiments so far. We make minimal change and simply quences. Note that the self-attention is still bidirectional
100
Masking
Random
Success (%)
Causal
the sampled mask se-
60
rather than causal. The pre-training prediction problem is causal however. We consider two settings: pre-training the same task (pick to pick) and pre-training on different task (pick to stack). We report the results in Figure We find that causal pre-training leads to better performance on the same task and worse transfer across different tasks.
40
Pick + Pick Pick + Stack
Figure Causal masking.
Related Work
Self-supervised learning in robotics. There is rich body of work on self-supervised learning in robotics. [16, 17] learn grasping policies from large-scale self-supervision; [18] learns visuomotor policies with self-supervised visual correspondence; [19] calibrates learned grasping policies for particular objects based on grasp success feedback; [20] simultaneously collects robot trajectories for multiple tasks and learns corresponding policies; [21] uses an auxiliary contrastive learning task on human demonstrations; [22] learns goal-conditioned robot policies from teleoperation data; [23] uses multi-view reconstruction for model learning. Overall, prior work has largely focused on using self-supervision for learning a particular robotic task. In contrast, we use self-supervision for pre- training general sensorimotor representations that can be transferred to different downstream tasks.
Multi-fask and large models for robotics. A number of works have explored learning multi-task and large models for robotics. BC-Z [24] performs large-scale vision-based imitation learning: [25] trains policies with cross-domain datasets; [26] learns multi-task Transformer policies for manip- ulation with language; [27] pre-trains with offline reinforcement learning. [28] trains a generalist agent on trajectories from different tasks jointly. In contrast, we use masked objective, operate on latent visual representations, and focus real-world trajectories. [29] trains language-conditiong Transformer policies from human expert demonstrations. Likewise, we leverage Transformer models but focus on self-supervised learning from sensorimotor trajectories. [30] grounds language models with visual inputs via an embodied visual question answering model. Overall, we share the goal of multi task robot learning and training large models for robotics but propose a general self-supervised pre-training approach that learns from real- weal-world sensorimotor trajectories alone.
Discussion
Limitations. We note that our work has several important limitations. First, our dataset is collected using a single robot in single lab, which limits the diversity and realism of the data. Scaling to more diverse environments and robots remains important area for future research. Second, the tasks we consider, in both pre-training and fine-tuning, are relatively simple variations of pick and place. It would be good to explore more dexterous tasks with complex dynamics and contacts. Next, our model is pre-trained with the MSE loss and may struggle with multimodality. This is partially alleviated with conditional prediction over large context lengths. Nevertheless, it would be good to explore generative model variant. Finally, please see the project page for videos of failure cases.
Conclusion.
We describe approach for robot learning with sensorimotor pre-training. Our model is Transformer that operates on sequences of sensorimotor tokens. We pre-train our model by masked prediction. We find that pre-training on this data consistently outperforms training from scratch, leads to 2x improvements the block stacking task, and has favorable scaling properties. Finally, we demonstrate successful transfer across different tasks, lab environments, and robots.

---

Workspace Setup
Trajectory Counts
8000
6931
6000
1000
3228
3217
2000
Pick
Pick
Stack
Destack
Pick One Object
Bin Picking
Stack & Destack
Figure 9: Dataset. We collected dataset of over 20,000 real-world trajectories using Franka robotic arm. Each trajectory is sequence of high-quality images from three cameras (one attached the hand and two on the sides), proprioceptive robot states, and actions. We consider picking, bin bicking, destacking, and stacking tasks, with variations in object position, shape, and appearance.
Appendix A: Data Collection
Hardware. We used Franka robot with a 7-DOF arm and a parallel jaw gripper (Figure 9, top left). We recorded proprioceptive information in the form of joint positions and velocities. The workspace equipped with three high-quality Logitech Brio color cameras. One egocentric camera is attached the robot hand, and two exocentric cameras are attached to the left and the right side of the robot. We synchronized the data streams from the cameras and the robot 60 Hz and saved data at 30 Hz.
(I) Pick: The task is to grasp and lift one out of K cubes from the table. We script data generation scheme where the robot takes in K cubes at randomly generated 3 DoF poses in robot frame, that are not in collision, and generates sequences of DoF picks-and-placee poses for each the cubes. Then scripted policy grasps the cube. After grasping, the robot places the grasped cube in the next randomly generated pose, moves back the start joint configuration, and starts the next grasp.
The robot picks an object from a bin filled with randomly-placed, both soft and rigid, objects (see Figure 9). We use [31] to generate grasps from depth images, captured by an overhead depth camera. Before each grasp, the robot moves to joint confguration that does not occlude the camera view of the bin. After we obtain a grasp pose, we ise scripted policy to pick the object. the gripper is not fully closed the end of the trajectory, we consider the trajectory successful.
(2) Bin Pick:
(3) Stack and (4) Destack: Starting with two cubes of different colors resting on the table, the robot is tasked to first move one of the cubes onto the other cube, and then move the stacked cube back onto the table. The cube poses are generated in a similar fashion as (1). For stacking, the height of the place pose is offset by the height of the cube; similarly. for destacking, the pick pose height is offset by height the cube. scripted policy then executes the pick-and-place trajectory.
Statistics.
We collected ~20,000 real-world trajectories the course of 9 months (Figure 9). The dataset includes 77000 trajectories for both single object and bin picking, as well as ~3,000 trajec- tories for stacking and destacking each. The average length of picking and destacking trajectories ~300 while the stacking trajectories are longer at ~600 steps. The dataset contains variations in object poses, shape, and appearance. We show frames from example trajectories Figure 9.

---

Acknowledgments
We thank the anonymous reviewers for helpful feedback and suggestions during the review pro- This work supported in part by DARPA Machine Common Sense program, ONR MURI program (N00014-21-1-2801), NVIDIA, Autodesk, and BAIR's industrial alliance programs.
References
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012.
[2] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[3] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, 2020.
[4] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.0637 2021.
[5] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding
by generative pre-training, 2018.
[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HCT, 2019.
[7] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[8] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre- pre-training. arXiv:2210.03109, 2022. 2022.
[9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurlPS, 2017.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. ICLR, 2020.
[11] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv:21 10. 07058, 2021.
[12] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, Ma, E. Kazakos, D. Moltisanti, Munro, T. Perrett, W. Price, and M. Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. 1JCV, 2021.
[13] R. Goyal, S. Ebrahimi Kahou, Michalski, Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[14] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands in contact internet scale. In CVPR, 2020.
[15] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In CVPR, 2009.
[16] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv:1806.10201 2018.
10

---

[17] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. ICRA, 2016.
[18) Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. RA-L, 2019.
[19] M. Danielezuk, A. Balakrishna, D. S. Brown, S. Devgon, and Goldberg. Exploratory grasping: Asymptotically optimal algorithms for grasping challenging polyhedral objects. arXiv:2011.05632, 2020.
[20] D. Kalashnkov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
[21] A. Zhan, R. Zhao, L. Pinto, P. Abbeel, and M. Laskin. Learning visual robotic control effi- ciently with contrastive pre-training and data augmentation. In IROS, 2022.
[22] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, et al. From play to policy: Conditional behavior generation from uncurated robot data. 1.XXv:2210.1004:: 2022.
[23] Y. Seo, Kim, James, K. Lee, Shin, and Abbeel. Multi-view masked world models for visual robotic manipulation. arXiv:2302.02400 2023.
[24] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, 2022.
[25] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization robotic skills with cross-domain datasets. arXiv:2109.1333 2021.
[26] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[27] A. Kumar, A. Singh, F. Ebert, Y. Yang, Finn, and S. Levine. Pre-training for robots: Offline rl enables learning tasks from handful of trials. arXiv:2210.051/7. 2022.
[28] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth- M. Gimenez, Y. Sulsky, J. Kay, Springenberg, et al. generalist agent. arXiv:2205.06/75, 2022.
[29] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.
drXiv:2212.068117.200 2022.
[30] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. a3X3v:2303.03378. 2023.
[31] V. Satish, Mahler, and K. Goldberg. On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks. RAL, 2019.


Reviews:
[{"review_id": "jEhmgjly1V", "reviewer": "Reviewer_oCTA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for handling heterogenous data where each modality may not exist at every time step. It hypothesizes that if a robot can predict masked-out regions from a trajectory, it has acquired a good representation of the physical world that can be used downstream.\n\nMethodology consists of doing the following to a demonstration: getting vision latents from the images from a pretrained model; generating an input sequence of these latents proprioception, and action; encoding each modality separately; masking out a subset of input modalities randomly at each time step; and training a model to predict the masked inputs. \n\nThe experiment section compares this method to pretraining from scratch in terms of performance and efficiency. Then, the model is used on other data to show transfer capabilities of its learned representations. ", "summary_of_recommendation": "The novel and intuitive idea is interesting to explore. The transfer results are especially promising, which is what pushes this to an accept. More data would be useful, as well as more detailed and informative statistics and analyses.\n\nClarity needs various work - see review.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Why is it exactly that the vision latents from the pretrained vision model allow us to leverage *more* data? Is it because the latents make for a lower dimensional prediction problem and therefore less training time? Is there evidence for this? \n- When you say - pretrain the model on real-world robot trajectories \"with masked prediction\", what do you mean by saying \"pretrain on data with masked prediction\"? Should this say \"masking\", or did I misunderstand?\n- What's the reason for the double-randomness of randomly sampling a masking probability, then masking with that probability? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality\n#### Strengths\n- Interesting idea of random maskouts of various modalities\n- Experimental setup is thorough - transfer experiments are well-designed given the paper's central claims, and ablations provide insights into the nature of the method\n- Experimental results validate the approach.\n\n#### Weaknesses\n- Not clear how much this helps with real data, which is limited to the image modality (as pointed out by the paper). We see ablation results showing that masking out all of one modality is very harmful. We don't see an ablation with large stretches of one or two modalities masked out, but I suspect it would behave more like a total lack of that modality (at least at times) than like the random-t random-modality masking - interpolation would be much harder than with the nicely distributed masking.\n- Some results, such as the unsuccessful trajectories, would be useful to see\n- In general, more results would be appreciated\n\n### Clarity\n\n#### Strengths\n- Figure 4: interesting and useful! \n- Figures are super stylish\n- Writing is overall fine, just verbose\n\n#### Weaknesses\n- Intro is difficult to understand - it is heavy in prose, and goes into detail about visual representations, the dataset, and the experiments in uninterrupted paragraph form. This does give *some* sense of what to pay attention to, but readers may benefit more from a clear contribution list\n- Figure 1 is a good high-level description, but the middle panel isn't very effective - it's hard to see and the message is nonobvious. After several looks, it seems to be communicating that the input sequence is heterogenous in terms of which modalities appear, but the reconstructions are homogenous and complete in terms of modalities. This can probably be communicated in a simpler way\n- Figure 2 doesn't contain as much information as it could. What is the output of the transformer? We know it's a representation, but it may as well be labeled. Furthermore, the mask (a single concept) is indicated three ways - a fade, an \"M\", and a lighter blue. Not only is this poor data visualization, but the fade is hard to see as a clearly distinguishing factor in the bottom row. The fade also makes it harder to realize that each timestep *has* all three modalities but also may have some masked out. \n- Figure 3: low-info and redundant. Feel free to cut.\n\n### Significance\n- Transfer performance is promising, even though more results would be helpful. If successful, this capability seems to have robustness potential, making it a significant idea.\n\n### Originality\n- I am not aware of any other paper that takes this approach to masking data and reconstructing, then using the learned representations for transfer.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "XpO-XdIXO0", "reviewer": "Reviewer_Q5ex", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to use masked prediction objectives to learn good representation from images, robot states and past actions. The paper demonstrates that doing so leads to representation that supports more efficient fine-tuning, especially on harder tasks.", "summary_of_recommendation": "The paper is interesting, well written and timely. I am recommending accept because I think the contribution is relevant to robot learning and I do not find any obvious issue with the paper. I am happy to raise my scores if the authors provide more analysis and open-source data and code.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Can you also please illustrate why fine tuning is better than linear probe. I understand that this is a phenomena that is also observed in using masked prediction objectives in other research areas, but I would like to understand why this is the case in robotic manipulation.\n\nIt will be quite beneficial to the community if the authors release the code and the data sets that accompany the paper. I am happy to raise my score if the authors do so.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interesting, well written and timely. I enjoy the premise of the paper, which is to see whether good representation for robotics can be learned from sensory trajectories alone. \n\nI believe the various designs in the paper is or will become standard practices in robot learning, such as using a pre-trained vision encoder and using the latent visual representation for sequence learning, mask reconstruction task as an auxiliary loss. The experimental results are also well thought out. I particularly enjoy the study presented in section 6.2 about the generalizability to different training data. I also enjoy the ablation study on masking type. It is interesting that both timestamp and modality masking are important to obtain good results. \n\nI am recommending acceptance because I think the contribution is relevant to robot learning and I do not find any obvious issue with the paper.\n\nI would like to suggest a small minor correction about the infront speed comparison with RT1. The three hurts in front speed describe when the RT1 paper is system level latency which takes into account camera and action execution. Is the number 10 herd set the author referring to system latency or model inference latency?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "NvDpZzOWLQ", "reviewer": "Reviewer_6uLm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a representation learning scheme (model architecture, training recipe, objective function) for robot manipulation tasks. The model, referred to as RPT (Robotic Pre-trained Transformer), takes as input sequences of images, robot actions, and robot states. At training time, some of these inputs are masked out, and the RPT model is supervised (via a mean-squared error objective) to predict these masked inputs. Once trained, the model may be finetuned for a particlar task, to output actions given an image and robot state. RPT is demonstrated on four manipulation tasks (pick, bin pick, stack, destack) -- evaluated on 16 real-world trials per task (per each experimental study).", "summary_of_recommendation": "I feel this paper moves towards a new and exciting direction in terms of representation learning for robotics. My major concern at this point is that, the paper has a number of claims that are not well-substantiated. Addressing this will need multiple additional experiments and analyses that require a major revision. I am open to revising my score based on the author response and discussion, but based on the current version of the manuscript alone (and the fact that there weren't additional supplemental materials to refer to for clarification), I will for now score this a \"weak reject\".", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I would like to see weaknesses [W1], [W2], [W3] (from above) discussed during the author response and revision phase.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\nThis paper investigates an interesting research question; one that is extremely relevant to the robot learning community -- \"can representations for robotic manipulation be learned from sensorimotor pretraining alone\"?\n\nThe paper is quite easy to follow and understand (for the most part -- see \"weaknesses\" and \"minor remarks\" below for more). The motivation behind proposing the RPT scheme is clearly established; the model architecture and training scheme are explained in a very accessible manner.\n\nThe collected dataset, with its ~20K robot trajectories, is perhaps the key to making RPT work. (If this can be released, it would add value to the paper)\n\nThere are some ablation analyses to substantiate the design choices in RPT.\n\nOverall, this work is heading in an interesting direction towards learning general-purpose sensorimotor representations for robotics.\n\n\n\n## Weaknesses\n\nIn its current form, I have several concerns with the manuscript that I hope to see addressed over the author response phase. My support for this paper would substantially improve if some of these issues are addressed (listed in a partially-ordered sequence of decreasing importance).\n\n\n[W1] **Comparisons with prior art**: The comparisons made in this work are solely across variants of model architectures, training data, and masking strategies proposed in the paper (i.e., across ViT-B and ViT-L, varying context lengths, eval modes, pretraining tasks).\n\n* However, it is unclear from this evaluation alone as to how RPT compares with other alternative model architectural proposals that have been around for the past year. In terms of masking pre-training objectives for multimodal data, approaches such as MultiMAE (ECCV 2022) [Ref 1] and pre-cursors [Ref 2, Ref 3]. The premise for my assessment is that architectures and masked objectives for modeling multimodal data are neither new, nor the core focus of this paper. And I'd deem it completely acceptable to skip direct evaluation wrt [Ref 1, Ref 2, Ref 3] (I point these out here to encourage a discussion of the nuances of the proposed approach w.r.t these alternative ones).\n\n* In terms of quantitative comparisons, however, the closest set of baselines in the robotics community would be the (extremely broad) class of vision-based imitation learning algorithms. Particularly, for the task-based version of RPT, these baselines (minimally, behavior cloning (BC) ones) need to be evaluated against; to justify the impact of the proposed pretraining architecture. The proposed approach looks like it could exhibit similar shortcomings to vanilla BC approaches, in that once an output action steers the system towards an unknown state, the probability of recovering reduces exponentially for every subsequent action taken.\n\n* Another aspect with the manuscript in its current form is it feels incomplete, as the claims of data-scale, context length etc (Fig. 7) don't yet seem to be saturated. If larger ViT backbones, such as ViT-H / ViT-G, were to be evaluated; the claims would be better substantiated.\n\n\n[W2] **Generalization**: One of the key claims in this paper is that representations learned by RPT generalize better -- however, I feel like the parameters along with generalization are measured in the current experiment set are somewhat restrictive. I would like a clarification during the author response phase; but I presume all of the data currently collected is from a single Franka Panda arm in a tabletop-style environment? Meaning the extroceptive (images) and proprioceptive inputs (robot states) will implicitly be constrained to work within this robot-environment combination? The representation learning scheme used in RPT does not take a downstream task into consideration (when training on data from across multiple tasks), which means RPT will require to be finetuned when being applied to any downstream task for the first time. I have a number of follow-up questions:\n\n* I see one potential confounding factor in the experimental setup for Fig. 6, which evaluates generalization across various pre-training data subsets. The claim here seems to be that, regardless of the pretraining task (stack, pick, bin pick, bin all), the performance on the downstream task (stack) is fairly consistent (at least for the stack, pick, and bin pick training sub sets). However, one key detail is that regardless of the pretraining data subset, all these models must be finetuned for stacking, using trajectories collected for stacking. This makes it very difficult to assess whether it is the representation itself, or the finetuning phase, which results in performance being on par with a model trained for stacking. (also, there seems to be no clear way to even measure cross-task generalization in the current RPT train-eval framework).\n\n* What happens if either the robot or one of the cameras is moved from the training to evaluation phase (assuming the eventual hand-eye calibration is still accurately available)? (as an aside, it could be interesting for future editions of this work to explore equivariant representation learning)\n\n* Would RPT generalize to a different robot, for the same task? (at least, different instances of the same robot; i.e., another Franka)? Would RPT generalize if one of the joints were constrained to be in a specific pose (i.e., same robot -- but a different set of pose constraints at inference time)? Would RPT work if the environment undergoes major visual changes (while preserving the robot and camera poses)?\n\n\n[W3] **Missing details; statistical significance**: All experimental studies have been conducted only on 16 trials, which leads to questions about the statistical significance of the presented results. This would, in part, be alleviated if more details about the training and test setup and data became available -- i.e., how different are the train/test tasks in terms of complexity, visual similarity, successful trajectories? There are a number of other detail that I could not find in the paper, such as the dimensionality of the state/action space, the type of normalization or preprocessing applied to the input proprioceptive and controls, any attempts to aid in-/equi-variance of the learned representations. More importantly, it was unclear from the paper as to how RPT is applied at inference time. My understanding was that, upon task-specific finetuning, RPT is run on the input images and robot states to produce robot controls (all other outputs are discarded). However, it is unclear how these outputs interface with the robot controller, workspace constraints; and the rate at which these actions are executed (and, if at all, whether these actions are fed back into the model). This needs to be clearly explained in a revision (and I'd prefer finding space for this in the paper, opposed to the supplementary materials).\n\n\n### References\n----------\n\n[Ref 1] MultiMAE: Multi-modal Multi-task Masked Autoencoders. ECCV 2022.\n\n[Ref 2] LXMERT: Learning Cross-Modality Encoder Representations from Transformers. EMNLP 2019.\n\n[Ref 3] VL-BERT: Pre-training of Generic Visual-Linguistic Representations. ICLR 2020.\n\n\n\n## Minor remarks\n\nThese comments do not affect my final score; as I believe they are easily addressed in a minor revision. The authors needn't respond to these.\n\n* Line 16: \"few year\" -> \"few years\"\n* Line 66: incomplete/grammatically-inconsistent sentence\n* Line 76: \"perspective\" -> \"perspectives\"\n* Lines 80-81: The terms \"vision cost\" and \"sensorimotor cost\" are not well-defined, and are ambiguious. It would be nicer if they are replaced with terms that are precise. (i.e., \"cost\" in terms of what? parameteric complexity? space/time complexity? sample complexity?)\n* Line 228: sentence needs to be revised for language use / grammar\n\n* I could not make sense of Figure 2 until after I read through sections 3 and 4. I wonder if the figure design may be modified so it is self-contained. (This isn't necessary; but will make the exposition more accessible).\n* There is a possibly unintended overloading of the term \"fine-tuning\" across the paper. The term fine-tuning seems to refer to either (a) training a network from scratch vs fine-tuning a pretrained encoder (e.g., a ViT backbone) for a task (as in Fig. 5), or (b) fine-tuning vs linear-probing the embeddings of a trained/fine-tuned encoder. It would be nice if these two usages had different terms altogether.\n\n\n\n# Post-rebuttal update\n\nBased on the convincing rebuttal (see disucssion and comments for more info), the paper has improved significantly. I revise my initial rating of a \"weak reject\" to a \"strong accept\". Many thanks to the authors for carrying out these additional experiments to address all reviewer concerns.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-_yfydtHoB", "reviewer": "Reviewer_8qUw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a network that is transformer based on robotics data. They show that pretraining improves performance over non-pretraining. Then the paper has many ablations on design choices: masking during training, pretraining on specific tasks, context length, etc. The network itself is a pretrained ViT + sensorimotor network.", "summary_of_recommendation": "Overall, the paper needs some work, in explaining the network, tokenization schemes, and speed related claims. The results are not necessarily very surprising. It would be better understood when contrasted against previous work like RT-1 / PerAct. Limitations and conclusion is written lazily. The main utility of the paper is not well explained or is not readily apparent. However, though, the paper does put in a lot of work into collecting data and experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. There is barely any limitations or conclusion. Please write this sincerely.\n2. About speed of 10hz. VIT is large, so how is the image network and sensorimotor model decoupled? The decoupling leading to speed increase is not explained well\n3. Training procedure is not fully explained. What is the exact loss function used for reconstruction?\n4. How many tokens are there per time step? What fraction is the image?\n5. Tokenization scheme for non image inputs is not explained", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Some results are straightforward:\n1. Pretraining overall helps.\n2. Masking tokens randomly rather than removing a modality / timestep is better\n3. Pretraining on negatives makes the model worse\n4. Performance improves with number of demos\n5. Larger network better\n\nSome results are weird:\n1. They show that pretraining on stacking and finetuning on stacking has the same performance on stacking as pretraining on picking/bin-picking and fine tuning on stacking. In fact the former is lower than with pick pretraining. This shows that pretraining is not very relevant, because one would've expected that pretraining on the domain that is closest would've a higher success rate. This discrepancy is neither addressed nor evaluated. \n2. Linear probing, ie keeping the pretrained network frozen and with linear action read out gets 40% success. This is surprisingly high and looks fishy. \n\nSome results need more explanation:\n1. About speed of 10hz. VIT is large, so how is the image network and sensorimotor model decoupled?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the original submission, the rebuttal by the authors and the discussion that followed, the reviewers agree that this paper should be accepted to CoRL. 
To the authors: great job on writing a strong rebuttal and engaging with the reviewers during the discussion period! This was one of the strongest rebuttals in my batch and I'm really pleased to see the reviewers and the authors engage in discussion that resulted in a much better submission. Please address the remaining comments in the camera-ready version of the paper.
Decision: Accept (Oral)