Real-World Robot Learning with Masked Visual Pre-training
Authors: Ilija Radosavovic
Tete Xiao
Stephen James
Pieter Abbeel
Jitendra Malik
Trevor Darrell
Venue: CoRL 2022 Oral (2022)
Keywords: Self-Supervised Learning
Visual Representations
Robot Learning

Abstract: In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75\%), supervised ImageNet pre-training (up to 81\%), and training from scratch (up to 81\%). Finally, we train a 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.

Real-World Robot Learning with Masked Visual Pre-training
Ilija Radosavovic" Tete Xiao"
Stephen James Pieter Abbeel Jitendra Maliki Trevor Darrell
University of Califormia, Berkeley
Abstract: this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via masked autoencoder (MAE), frozen, and then passed into learnable control module. Unlike prior work, show that the pre-trained representations are effective across range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81 8)).. Finally, we train 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
Keywords: Self-supervised Learning, Visual Representations, Robot Learning
Introduction
Learning representations with large neural networks is the workhorse of modern deep learning. This has enabled impressive results in computer vision [1, 2], natural language processing [3, 4, 5], and audio generation [6, 7]. How can we transfer the success stories of representation learning to robotics? We can approach this from two ends: shared representations on the perception side or shared representations the action side. Our focus in this paper is shared visual representations.
Of course, the devil is in the details. Recent developments in the field of visual learning have made this more feasible: (1) the use of diverse, real-world data from the Internet and egocentric videos, (2) self-supervised objectives that do not overly rely on data augmentations or other forms of strong human-designed priors, (3) scalable and high-capacity transformer models [8, 9], and (4) training of control policies on top of frozen visual representations. In our recent work [10], we have shown that this recipe for supervised visual pre-training is effective for motor control in simulation.
In this paper, we show that this framework is effective for real-world robotic tasks well (Figure 1). We build on our prior work, but make significant advances in terms of data scale and diversity (7x larger), model size (15x x bigger), and real-world experiments (extensive real robot evaluations).
In particular, we train self-supervised visual representations on real-world images and videos from the Internet [11, 12, 13] and egocentric video datasets [14, 15]. We leverage the masked autoen- coders [16] that learn visual representations by masked prediction. The hope is that, by learning to predict the missing content in real-world images, the model will learn useful properties of the visual world that will enable it to learn to perform real-world robotic tasks. Given the pre-trained vision encoder, we freeze the encoder and learn control policies on top. The same visual representations are used for downstream robotic tasks and embodiments. We focus on efficient real-world learning through behavior cloning with a handful numan-provides demonstrations per task (20 80).
* Equal contribution. Code, pre-trained models, and videos are available on our project page.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

In-the-Wild Data
Masked Autoencoder
Real-Worke Robotic Tasks
Over 4.5 miltionn images Five diverse data sources
(a) Masking
(b) Autoencoder
Two robots (xArm, Allegro hand) Eight tasks (seenes, objects)
Decoder
Encoder
Figure 1: Real-world robot learning with masked visual pre-training. We learn visual represen- tations from a massive collection of Internet and egocentric data. We pre-train representations with masked image modeling, freeze the encoder, and learn control policies for robotic tasks on top.
We evaluate our approach in extensive real-world study and report results from 981 real-world experiments. We consider basic motor control tasks (reach, push, pick), as well as tasks with varia- tions in scenes and objects (Figure 1, right). We find that our approach achieves considerably higher performance than CLIP (up to 75%), supervised pre-training (up to %)), and training from scratch (up to 81 3). Furthermore, we observe that our representations lead to large improvements in sample complexity, reaching the strongest baseline performance with half the number of demonstrations.
In addition, we demonstrate the benefits of scaling visual pre-training for robotics by training 307M parameter transformer [9] massive collection of 4.5M images from ImageNet [1 1], Epic Kitchens [17], Something Something [12], 100 Days of Hands [13], and Ego4D [15] datasets. Im- portantly, we observe that it is not sufficient to scale the model alone and that larger models require bigger datasets. To the best of our knowledge, ours is the largest vision model deployed for robotics, and demonstrates clearly the benefits of visual pre- tre-training scale for robot learning. We encourage the readers see the the extended version of this work on arxiv and also to check the project page.
Related Work
End-to-end control is concerned with learning to predict robot actions (e.g., joint velocities, end- effector poses, etc) directly from observations [18, 19, 20], without the need to perform explicit 3D pose estimation [21], grasp planning [22], and motion planning [23]. However, these end-to-end approaches tend be too sample inefficient for real-world training. Some works have tried to find
a balance between these explicitly pipelined approaches and end-to-end approaches [24, 25, 26].
Supervised pre-training for robotics learns one more pretext tasks through strong supervision and then transfers the representations to downstream robotic tasks. Lin et al. [27] shows that rep- resentations learned from semantic tasks such as detection and segmentation correlate with affor- dance maps for object manipulation. Shridhar et al. [28] use maaguage-supervssee CLIP model [29] for learning language-conditionee imitation policy. In concurrent work, Nair al. [30] explore pre-training visual representations using time contrastive learning and language descriptions from human annotators. These methods all require expert labels or cross-domain supervision.
Self-supervised learning in robotics has been explored in number of settings: learning a dy- namic models [31]; learning visual representations from interaction with the environment [32]; learning vision-based grasping policies [33, 34]; learning visual autoencoders [35]; learning spa- tiotemporal representations through videos [36, 37]; learning visual correspondence [38]; utilizing non-parametric nearest-neighhor retrieval [39]: and conducting visual self-supervised learning on pre-collected demonstrations [40]. These methods require in-domain data collection, and thus may be difficult to extend beyond the training environment and task. In contrast, approach uses large-scale and diverse collection of real-world images and videos, making it more generalizable.

---

reach
π pick
T fridge
fruit
HT sink
7 tip
Pre-trained Vision Encoder
Figure 2: One encoder for all robots and tasks. We train control policies per task, on top of the frozen encoder. The same vision encoder is used for all downstream robotic tasks and embodiments.
Robot Learning with Masked Visual Pre-training
3.1 Masked Visual Pre-training
Data collection. We first compile large-scale dataset for learning visual representations. We primarily use Ego4D [15], massive scale, egocentric dataset from nine countries recorded via portable devices, covering over 3,670 hours of daily-life activities. We combine the Ego4D data with the ImageNet [11], as well as the Hand-object Interaction (Hol) data used in [10]. which comprises of the egocentric Epic Kitchens [17] dataset, the YouTube 100 Days of Hands dataset [13], and the crowd-soureds Something-Something dataset [12]. Our training data totals 4.5 million images, 6.5x of the Hol data. We find that a sufficiently large and diverse pre-training dataset to perform the mask image modeling self sele-suurr task is critical scale up the vision backbone for real robot tasks.
Self-supervised objective. At the core of our self-supervised representation learning approach is masked image modeling via the masked autoencoders (MAE) [16]. MAE masks out random patches in an image and reconstructs the missing content with vision transformer (VIT) [9]. A high masking ratio, e.g., 75%, and asymmetrical heavy-encoder light-decoder design, are important for learning good visual representations efficiently. Simple and free from dataset or task-specific augmenta- tions [41], MAE is the state-of-the-ar self-supervised framework in computer vision [42, 43, 44, 45], and has been demonstrated work well for motor control tasks in simulation well [10].
Architecture. We use the ViT models as our vision encoders. While the MAE-trained VIT models yield improving performance in vision tasks as model sizes grow [9, 16, 46], previous work [10] does not show improvement from switching a ViT-Small model to the ViT-Base counterpart of 4x as many parameters. In this work, we scale the model up to the ViT-Large and deploy it on the real robot. The model contains 307M parameters and runs at -64 gigaflops at input size 224 224, approximately 15x many as the commonly adopted ResNet-50 [47], the largest vision model deployed for robotics. As will show in the experiments, scaling model sizes while training sufficiently large data leads to consistent performance improvement on downstream robotic tasks.
3.2 Real-World Robot Learning
We learn to perform real-robot tasks through behavior cloning (BC). We collect demonstrations containing trajectories of RGB images from wrist-mounted camera and the robot's joint state at each time step. For most of the tasks, we use the motion-tracked HTC Vive VR system to control the end-effector. For some tasks that are difficult to demonstrate via the motion controller, egg., closing fridge door, we use kinesthetic teaching. Given the recorded demonstrations, we train a control policy that takes in the input image features and proprioceptive states (joint positions) at time step t and predicts the action time step t 1. We perform joint position control; we do use any end-effector information (e.g., the 6-DoF pose). We build on our MVP pipeline [10] and freeze the image encoder throughout the policy learning, which prevents large pre-trained encoders from overfitting to specific setting or task, and greatly reduces GPU memory footprint and training time.

---

Figure 3: Real-world robotic tasks. We perform extensive real robot evaluations using a 7 DoF robot arm with parallel jaw gripper. Our tasks include basic motor control skills (reaching red block, pushing a wooden cube, and picking yellow cube), variations in scenes (closing a fridge), objects (picking fruits), and scenes and objects (picking detergent bottle from cluttered sink).
4 Experimental Setup
In this section we provide implementation details for our approach and describe our evaluation setup.
Data. We extract frames from Ego4D, Epic Kitchens, and Something-Something at 0.2 fps, 1 and 0.3fps, respectively. We then combine the Ego4d with ImageNet and the YouTube 100 Days of Hands dataset. This process yields 2.6M frames from Ego4D, 1.2M images from ImageNet, and 700k Hol images from the rest, total of 4.5M images. We term the combined dataset "Ego" for abbreviation. Note that [10] only uses the 700k Hol images, excluding the Ego4D and ImageNet.
Encoders. We use the standard Vision Transformer (ViT) architecture as the image encoder. We use three models of various sizes: ViT-Small [48], ViT-Base, and ViT-Large models, of 22M, 86M, and 307M parameters, respectively. The ViT-Small model is approximately the same size as the ResNet- 50 model, while the ViT-Large model has 155 many parameters as the ResNet-50 model.
Pre-training. We pre-train the models via the MAE framework [16]. The training recipe closely follows [16], with dataset specific settings from [10]. We use the auxiliary dummy classification token for transferring to downstream robotics tasks. We train the MAE models for 400 epochs for the combined Ego dataset; 1600 epochs for the HOI dataset; and 1600 epochs for ImageNet dataset. We use the pre-training recipe in [49] for the study that involves ImageNet supervised models.
Controllers. The controller takes in both image features and the robot's proprioceptive state. We use joint positions as the proprioceptive state without explicitly appending the end-effector pose to the state. We do not use velocity in the state as our low-cost arm does not support true velocity sensing. The controller outputs delta joint angles. The controller's design closely follows [50], i.e. a four-layer MLP with a SeLU [51] activation following each hidden layer. The hidden size is [256, 128, 64] for most tasks and [512, 256, 128] for the PickSink task. We linearly project the image features and the proprioceptive states to a joint embedding space as the controller's input.
Robot and robotic tasks. We use the low-cost UFACTORY xArm7 robot (a 7-DoF arm) and a 1- DoF parallel jaw gripper. We use the arm's 's maximum control frequency of Hz for both collecting demonstrations and control. We use a first-person wrist-mounted RealSense camera for all tasks. We do not use depth information from the camera. We consider basic motor control tasks, i.e. ReachBlock, PushCube, and Pi PiCkbe,, and more challenging in-context tasks, i.e. CloseFridge, PickFruit, and PickSink. The tasks are shown in Figure (more details in the next section).
Demonstrations. We collect 80 demonstrations per task. We use the motion-tracked HTC Vive VR system for most tasks, except for CloseFridge we use kinematics teaching. We use trajectory replay on the robot for trajectory pruning. We do not use the key-frame information for the learning.
Evaluation. We systematically sweep across 16 variations of the environment, e.g., shifting the target object. For consistency and reliability of the study, we use the same 16 variations for all models in individual task, and evaluate models sequentially each variation. in order for similar lighting conditions, robot conditions, precise object initial locations, etc (see also the arXiv).

---

ReachBlock
PushCube
PickCube
100
Success (%)
60
20
Ours
CLIP
Sup.
Scratch
Ours
CLIP
Sup.
Scratch
Ours
CLIP
Sup.
Scratch
Figure 4: Comparison to vision encoders. We compare our approach to visual encoders trained with CLIP, supervised learning on the ImageNet, and from scratch on the task at hand. all cases, we observe that our approach consistently outperforms the baselines by a considerable margin.
100
Success (%)
80
ReachBlock
PushCube
PickCube
Ours
Ours
Ours
CLIP
CLIP
20
60
20
60
20
60
Num. Demos
Num. Demos
Num. Demos
Figure 5: Sample complexity. We show the performance our approach as the number demon- strations varies from 20 to 80. CLIP performance 80 demonstrations is shown with dashed lined for reference. Our approach is comparable to CLIP using only half the number of demonstrations.
5 Experimental Results
We perform extensive evaluations across range of visual backbones, real-world robotic tasks, ob- jects, and environments (Figure 3). In total, we report results from 981 real-world experiments.
5.1 Basic Motor Control
We evaluate three basic motor control tasks in visually simple contexts: reaching a red block, push- ing wooden cube, and picking yellow cube (see Figure 3 for task visualization). These tasks serve stepping stones for more complex tasks, and the visual representations that potentially be fundamental for robotics should learn these tasks efficiently. In all cases, we randomize the initial object and robot positions (see arXiv for details about the learning and task setup).
Comparison to various vision encoders. In Figure we compare our approach set of state- of-the-art vision backbones: CLIP [3] trained on 400M text-image pairs, supervised model trained on the ImageNet, and a model trained from scratch with in-domain demonstration data. For fair the ViT-Base [9] vision encoder for all methods. We empirically observe that the CL.IP encoder performs the best among the baselines, and the ranking order is consistent across the benchmark tasks. Our approach consistently outperforms the baselines by considerable margin.
comparisons,
Sample complexity. In Figure 5 we study the performance our approach as the number demon- strations varies from 20 to 80. For reference. we show the performance the most competitive baseline, CLIP, trained with 80 demonstrations (dashed horizontal line). In aggregate, we observe that approach reaches CLIP performance while using 50% fewer demonstrations. This result is a promising signal for using our models for learning more complex robotic tasks, as discussed next.

---

CloseFridge
PickFruit
PickSink
100
Success (%)
60
20
Ours
CLIP
Sup.
Scratch
Ours
CLIP
Sup.
Scratch
Ours
CLIP
Scratch Sup.
Figure 6: Variations in scenes and objects. We compare our approach to CLIP on tasks with varia- tions scenes (closing the fridge), objects (picking fruits), and scenes and objects (picking an object from a cluttered sink). The models are ViT-Base. Our approach considerably outperforms CLIP and the gap is larger than in simpler settings (see Figure 4). This may suggest that our representations capture more precise spatial structure that is helpful for robotic tasks in more realistic contexts.
5.2 Visually Diverse Scenes and Objects
We have previously shown that our approach can learn basic motor control tasks, such as reaching, pushing, and picking, at higher success rate and better sample complexity than the bascline vision backbones. To focus the motor control aspect, we used visually simple environment and basic objects. However, of the main potential benefits of our approach that learning visual represen- tations from diverse, real-world images may enable to solve robotic tasks that involve the interaction with everyday objects in more visually complex environments. In this subsection, we evaluate our visual representations on robotic tasks with variations in scenes and objects in more realistic setups.
Scene context. We first evaluate our approach in more realistic scene by considering the task of closing the door of toy fridge that is left open (termed CloseFridge). We randomize the location of the fridge, which side of the door to close, and initial angle of the door. As shown in Figure 6, bottom-left, the initial configuration of the fridge and the robot vary considerably, which is quite common in everyday settings. Figure 6, top-left, shows that our approach outperforms all baselines.
Different objects. Next, we evaluate on a task with variety of objects. The goal to pick up eight different fruits that vary color, shape, and size (termed PickFruit). In each trial, a fruit selected at random, and both the fruit and the robot's positions randomized. The number of training demonstrations remain unchanged, i.e., provide 10 demonstrations for each fruit. In Figure 6, middle, we show the evaluation results (top) and starting configuration samples (bottom). We see that baselines struggles in this setting while our approach achieves nearly perfect score.
Objects in context. Finally, we evaluate our approach on a task that features interacting with objects in everyday contexts. We task the robot with picking a detergent bottle from cluttered sink (termed PickSink). The task is challenging as the visual configuration of the scene, such as the toy plates, mug cups, and silverware, vary in unlimited ways, shown in Figure 6 right We observe that our approach considerably outperforms baseline approaches using the same ViT-B encoder.

---

PickFruit
PickFruit
PickSink
100
100
100
Model Data
VIT-L
Success (%)
80
Model
80
ViT-B
80
VIT-L
60
60
VIT-S
60
VIT-B
40
20
20
20
VIT-S
VIT-S Hol
VIT-B Hol
VIT-B Ego
100 200
300
100
200
300
Parameters (M)
Parameters (M)
Figure 7: Scaling model and data. We study the scaling properties of our approach. We observe that scaling the model size alone from ViT-S to ViT-B while keeping the dataset fixed (Hol image collection; see text for details) does not improve the performance and even hurts (left). However, when we scale both the model and data (our massive Ego image collection; see text for details) we see clear benefits from a larger model. The trend continues when going further from the 86M ViT-B to the 307M VIT-L model (middle & right). Moreover, the gains are larger for harder tasks (right).
5.3 Scaling Model and Data Size
Importantly, our visual pre-training approach uses self-supervised objective [16] that makes few assumptions about the data distribution, and does not rely human-designed pretext tasks such data augmentations. Therefore, the framework is well-suited for pre-training from massive collection of unlabeled and in-the-wild visual data. Here we study scaling model and data size.
We first consider increasing the model capacity. In Figure 7, left, we see that increasing the model size (~4.5x) from ViT-S to VIT-B, while keeping the data size fixed (HOI image collection [10],, does not increase performance and even hurts. This is consistent with the in-simulation results reported in [10]. However, if also scale the data size from HOI our massive Ego data collection, VIT-B yields better results. These results suggests that we must scale both the model and the data.
In Figure 7, middle & right, we show the performance as a function of model size. Additionally increasing the model size from the 86M parameter ViT-B to the 307M parameter ViT-L leads to further improvements. The gain is larger for visually more challenging task (PickSink). To the best of our knowledge, our work the largest vision model deployed to real robot tasks, which clearly demonstrates the benefits of scaling visual pre-training for real-world robot learning.
5.4 Comparison to Concurrent Work
We compare our approach to concurrent work [30]. submitted to the same conference (CoRL 2022). Similarly, it pre-trains visual representations on in-
params (M)
success
supervision
(%)
R3M
video-text
23
31.3
the-wild video data from Ego4D [15]. However, relies on paired language-video annotations that available as part of Ego4D. In contrast, our approach is fully self-supervised and makes minimal assump- tions about the data distribution, enabling us to lever- age massive collections of uncurated data (e.g., from the Internet). In Table 1, we compare our models to strongest available R3M ResNet-50 model. We observe that our medium-sized ViT-B model outper- forms R3M ResNet-50 by large margin (93.8% vs. 31.3%). We also that smallest ViT-S model from [10] outperforms it as well (68.8% 31.3%).
CLIP
image-text
86
18.8
22
68.8
Ours
image-only
93.8
307
100.0
Table 1: Comparison to concurrent work.
All of our vision models, trained with image- only self seff-supervision considerably outper- form the strongest available R3M [30] model trained on paired video-language labels from Ego4D [15]. The gains are larger for larger models. Evaluated on the PickFruit task.

---

Reach Seen
Reach Unseen
Flip Cube
Figure 8: Multi-finger hand. We show that our framework readily generalizes to a different robot morphology. We experiment with finger reaching, using seen and unseen objects, and cube flipping.
5.5 Case Study: Multi-finger Hand
Our approach makes no assumptions about the downstream robotic tasks or embodiments. In partic- ular, our policies take pixel images as input and predict joint position angles actions (rather than, e.g,, end-effector pose). In this subsection, we test the generality of approach by applying it for downstream tasks with multi-finger Allegro hand (see arXiv for more details the setup).
Visual reaching. We design a reaching task in which the hand learns to reach the top of an object with the tip of the index finger. The object's ' position is randomized across the palm. We provide 10 demonstrations for each the 8 different objects. At test time, we evaluate the trained policy on the seen objects as well as 45 unseen objects (see Figure 8). The success rate is ~50% across both.
Visual flipping. Next, consider cube flipping task in which the goal is flip rubber cube that is placed in the palm. The position of the cube is randomized across the palm. Thus, the policy must rely on visual cues to accomplish the task. In aggregate, we observe success rate of 50% across 30 trials. See Figure 8 for example key frames and also check out the videos on the project page.
Understanding vision through action. Training policies on top of frozen visual representations enables perform studies understand what the pre-trained visual representations utilize for downstream tasks. Here first train policy to reach yellow cube, but test with objects of different shape and color. First, we find that when given the same shape of different color (wooden cube) or same color and different shape (yellow ball), the policy reaches for the object. Next, when given both the yellow cube and a distractor it reaches for the yellow cube. Finally, when given an object different shape and color (blue cube) the hand stays still. See project page for videos.
6 Discussion
Limitations. While the scenes and objects used in our study are more realistic than in simpler robotic benchmarks, we still mostly use toy objects relatively clean lab environments, rather than real objects real-world scene contexts. Our tasks involve single object, and do not require the robot learn feedback control, to learn multi-step planning. Factors such as robot up-time may influence the results well. Overcoming these limitations is essential as move toward develop- ing, benchmarking, and widely deploying pre-trained models for real-world robotic applications.
Conclusion. We explore learning visual representations from a massive collection of real-world data and using them for downstream robotic tasks. We pre-train representations with masked modeling. freeze the encoder, and learn control policies on top. We perform extensive evaluations in the real world and show that, across various robotic tasks, our approach leads to higher success rate and better sample complexity than CLIP, supervised ImageNet pre-traie pre-training, and training from scratch. We further demonstrate the benefits of scaling the model and data size for real world robot learning.

---

Acknowledgments
We gratefully acknowledge the following colleagues for valuable discussions and support of our project: Shankar Sastry for Allegro hand access, Ken Goldherg for feedback and suggestions, Adam Curtis for help with wiring, Kevin Hu for help with Allegro hand retargeting, Erik Rogers for ROS suggestions, Lerrel Pinto for Allegro hand discussions, Raven Huang and Justin Kerr for help with 3D printing, and William Peebles for discussions. This work was supported in part by DARPA Ma- chine Common Sense, LwLL, and RACER programs; ONR MURI program (N00014-21-1-280) Hong Kong Centre for Logistics Robotics, BMW, as well as BAIR's industrial alliance programs.
References
[1] A. Krizhevsky, L. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[2] R. Girshick, J. Donahue, Darrell, and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[3] A. Radford, K. Narasimhan, Salimans, and Sutskever. Improving language understanding by generative pre-training. 2018.
[4] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HCT, 2019.
[5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[6] A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan. O. Vinyals, A. Graves, N. Kalchbrenner,
A. W. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 2016.
[7] Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and Sutskever. Jukebox: A generative model for music. arXiv:2005.003: 2020.
[8] A. Vaswani, N. Shazeer, Parmar, J. Uszkoreit, L. Jones, N. Gomez, t... Kaiser, and Polo- sukhin. Attention is you need. In NeurIPS, 2017.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- for image recognition at scale. In ICLR, 2020.
[10] T. Xiao, I. Radosavovie, T. Darrell, and J. Malik. Masked visual pre-training for motor control. 2rXiv:2003.06173, 2022.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A scale hierar- chical image database. In CVPR, 2009.
[12] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. ICCV, 2017.
[13] D. Shan, J. Geng, M. Shu, and F. Fouhey. Understanding human hands contact at internet. scale. In CVPR, 2020.
[14] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.

---

[15] K. Grauman, A. Westbury, E. Bymne Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv22110.07058, 2021.
[16] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 2021.
[17] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Rescaling egocentric vision: Collection, pipeline and challenges for spic-kitchens-100 IJCV, 2021.
[18] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[19] S. James, A. J. Davison, and E. Johns. Transferring end-to-end visuomotor control from sim- ulation to real world for a multi-stage task. CoRL, 2017.
[20] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, 2018.
[21] OpenAI, M. Andrychowicz. Baker, M. Chociej, R. Józefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, Welinder,
L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. IJRR, 2020.
[22] A. T. Miller, S. Knoop, H. Christensen, and K. Allen, Automatic grasp planning using
shape primitives. In ICRA, 2003.
[23] J. J. Kuffner and S. M. Valale Rrt-connect: An efficient approach to single-query path planning. In ICRA, 2000.
[24] A. Zeng, S. Song. S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In IROS, 2018.
[25] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. CoRL, 2020.
[26] S. James and A. J. Davison. Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation. RA-L, 2022.
[27] L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre- pre-training for manipulation. In ICRA, 2020.
[28] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In CoRI, 2021.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In /CML, 2021.
[30] Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv:2203. 12601, 2022.
[31] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. NeurlPS, 2016.
[32] L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
10

---

[33] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA, 2016.
[34] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. IJRR, 2018.
[35] C. Finn, X. Y. Tan, Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In ICRA, 2016.
[36] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. In RSS, 2017.
[37] P. Sermanet, C. Lyneh, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, Time- contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[38] Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. RA-L, 2019.
[39] J. Pari, N. Mubammad, S. P. Arunachalam, L. Pinto, et al. The surprising effectiveness of representation learning for visual imitation. arXiv:2112.01511, 2021.
[40] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. arXiv:2012.07975, 2020.
[41] T. Xiao, X. Wang, A. A. Efros, and T. Darrell, What should not be contrastive in contrastive learning. In ICLR, 2021.
[42] Y. Li, S. Xie, X. Chen, P. Dollar, K. He, and R. Girshick. Benchmarking detection transfer learning with vision transformers. arXiv:2111.11429, 2021.
[43] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders data-efficient learners for self- sulf-super video pre-training. arXiv:2203.12602, 2022.
[44] C. Feichtenhofer, H. Fan, Li, and He. Masked autoencoders as spatiotemporal learners. arXiv:2205.09113, 2022.
[45] R. Bachmann. D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders. arXiv:2204.0167,, 2022.
[46] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In CVPR, 2022.
[47] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[48] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data- efficient image transformers & distillation through attention. In ICML, 2021.
[49] T. Xiao, P. Dollar, M. Singh, E. Mintun, T. Darrell, and R. Girshick. Early convolutions help transformers see better. In NeurlPS, 2021.
[50] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, Hoeller, Rudin,
A. Allshire, A. Handa, et al. Isaae gym: High performance gpu-based physics simulation for robot learning. In NeurIPS, 2021.
[51] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. In NewrIPS, 2017.


Reviews:
[{"review_id": "pxgbrV74idV", "reviewer": "Reviewer_ZgtC", "summary": "This paper scales up vision models and data in order to train better vision backbones for visuomotor policies using behavior cloning. They aggregate several existing datasets in order to generate a large number of frames to pretrain a large high quality vision encoder (VIT using MAE). The authors conduct extensive experiments to show that scaling both the model and data substantially improves performancce on these tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\nThe paper is well presented and makes its points clealy.\nThe authors conduct many real world experiments to demonstrate the effectiveness of their approach.\nTheir new model performs very well against reasonable baselines.\n\nWeaknesses:\nThe only real weakness here is that there is very little technical novelty. At this point everybody knows scaling up to larger models with more data is a good way to go.\n\nMore Strengths:\nDespite the weakness above, this paper is still a valuable contribution to the community. While the large ideas here are all well known, the specific recipes are useful and will provide others with practical technical guidance.\nIt is also useful to see that so much of the performance deficit on these problems can be solved by simply improving the vision model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The lack of technical novelty prevents me from giving this a strong accept, however this paper is still a very useful resource for practioners and those trying to improve robotic capabilities. For \"Impact\" I wish there was another option: \"3.5: The work is incremental and does not contain many new interesting ideas, but will still have significant impact due to its strong experiments and useful practical guidance.\" Ultimately though, this paper passes an important test: if I were starting a new project in this area tomorrow, this is a paper I would want to read beforehand.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I don't have any major issues. Minor complaints: I would replace \"demos\" with \"demonstrations\" everywhere. The graphs could use better color choices with more contrast.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "WPi4-RYf0Qh", "reviewer": "Reviewer_yV1f", "summary": "This paper evaluates vision encoders pretrained using masked autoencoders on large scale datasets. Pretrained vision encoders are evaluated for real world robotics tasks, with variations of everyday scenes, as well as novel objects.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Evaluating performance for real world policies by pretraining on scalable datasets is an important problem that this paper addresses.\n* The real world experiments are good and contain some variety of tasks, demonstrating that pretraining using masked autoencoders works better than a CLIP baseline for several tasks.\n* Ablations comparing different camera viewpoints, model and data size.\n\nWeaknesses\n* The baselines compare to only ViT architectures. However these are generally harder to train than simpler conv architectures like ResNet. I would be interested in comparing to a from-scratch resnet50 (or smaller) which is easier to train, as another comparison point. Authors could also compare to the ResNet CLIP model. Comparing to visual encoders that are finetuned would also be valuable, since simpler architectures may improve with finetuning when training the policies on demonstration data.\n* Other comparisons and ablations would be useful for comparing effectiveness of method - for example: 1) evaluating the effectiveness fine-tuning the vision encoder during the policy learning phase,  2) comparing directly to R3M loss formulation [Nair et. al.]\n* Experiments in sim could allow more comprehensive and scalable evaluations.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper is clear and demonstrates an interesting result for an important problem - that using masked autoencoders to learn visual representations from scalable datasets is effective for robotics policies in the real world.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* Current results only show graphs, however tables showing more precise numbers would be easier to read\n* Authors should consider adding more comparisons/ablations as mentioned in weaknesses above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "OMrARPdkywj", "reviewer": "Reviewer_tTKM", "summary": "This paper shows a way to use large scale visual self supervision of in-the-wild, massive datasets, and use it to improve behavior cloning on robots. Specifically they use the Masked Autoencoder (MAE) framework of He et al, on a massive dataset of {ImageNet, Ego4D, etc.} image frames, to pretrain an encoder which shows improvements for training in-hand-camera-based BC policies. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nStrong accept.\n\n- While considerable work has investigated visual pre training to improve robot policy learning and for example behavior cloning, the key thing about this work compared to prior works is that it can benefit from pre training on just a big pile of images, with no assumptions about them other than that they cover a wide, diverse set of images. \n- It is indeed also notable that they show improvements of using ViT-Large, a 307M parameter model. For those that believe in the “just scale things up” hypothesis of robotics, this indicates a potentially very compelling route forward. \n- The experiments seem pretty good too. I think all the experiments only work for in-hand cameras, and all the tasks are basically just some form of a “reaching” task without any dexterous closed loop feedback required, but they are still good enough to interestingly explore some difficulties in this domain, such as the cluttered “sink” environment.\n- I appreciate the different types of scaling model size and/or data size experiments. \n- Although [9] is closely related and probably from the same set of authors, the present work is very interestingly different. For one, [9] used RL, and required millions of environment interactions, and was only shown in simulation. Instead, the present work shows real robot results and instead uses behavior cloning. It would be interesting to see RL on the real robots too, but of course it’s easier to deploy BC instead of on policy RL in practice. Also, the present work shows further scaling of vision model size than was shown in [9].\n\n# Weaknesses\n\nTo further improve the paper, the authors may consider addressing:\n\n- There are no videos of the Allegro experiments. It’s hard to evaluate these without videos. Currently they feel like kind of throwaway experiments. \n- How was imitation learning actually done? I assume it’s just M.S.E loss behavior cloning, but it’s not actually specified?\n- As mentioned above the tasks are interesting in some ways (clutter) but pretty simplistic in other ways. No feedback or multi-object-conditioning is required. Harder tasks would be interesting in future. \n- The discussion for related work could further be improved. Within self supervised vision for robotics, there have been many approaches that have shown improvements for policy learning. See for example https://arxiv.org/pdf/1909.06933.pdf. But I think the key thing is that the present paper shows using a significantly more massive dataset for pre training and with no assumptions about the data.\n- “In recent work, Xiao et al. [9] have shown that self-supervised visual pre-training is effective for learning motor control tasks in simulation.” Respectfully there is nothing in this statement that is unique to Xiao et al, it would probably be better to either more specifically call out the MAE method used in Xiao et al, and/or refer instead to the concept that it used self supervision on internet data, or instead make this a more generic statement about the field rather than specific to Xiao et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Great strengths outweighs minor weaknesses. Strong accept. Largest scale of vision data in vision-for-robots work I know of, and compelling results. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses discussion.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "MR7dmJC60Rf", "reviewer": "Reviewer_oQo2", "summary": "- This paper leverages the recent breakthrough of Masked AutoEncoders (MAE) for the domain of robotics.\n- It shows that visual knowledge can be transferred from different domains such as ego4d into specific robotics domains thanks to MAE pretraining.\n- The paper finds that MAE is superior to CLIP pretraining.\n- It provides some analysis of data and model scales and finds that large data and large model works the best.\n- This is evaluated on 4 real but toy scenes with picking, reaching and pushing skills, with high accuracy results for the MAE pretraining.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n- good to see that somewhat different domain data such as ego4d can be useful for this robotics setup\n- scaling to large models with large data improves performance, similar to trends seen in foundation models in other domains, this is a good because it means it can leverage large datasets from non robotics sources.\n- the advantages of MAE are kept here: minimal assumptions about data and not relying on specific data augmentation\n- the analysis of data and model sizes is useful information for the robotics domain (as opposed to just the vision domain from the MAE paper)\n- approach generalizes to different morphologies without finetuning\n\nweaknesses\n- no quantitative results or videos for hand experiment\n- not yet fully real scenes, setups are still somewhat simplistic with uniform backgrounds, sink and fridge scenes are a bit more complex but -not yet real:\n  - for the fridge door, the arm starts quite close to the door and only has to decide if it pushes the left side or the right side.\n  - for bucket scenes, objects are always at the same ground distance relative to the robot, this might be the reason why depth was not required to solve these tasks, but this will be different in real scenes where the robot body can be at any distance to the objects.\n- a pretty direct application of MAE, while it’s great to see it in a robotics setting, the innovation is pretty incremental\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "A decent paper, the robotics experiments are somewhat simple and limited compared to a truly realistic domain, a decent analysis and description of the model. It's somewhat of an incremental paper given that it's an application of recent breakthrough in self-supervision vision with MAE, however bringing that breakthrough to the domain of robotics might be a pivotal moment where robotics finally is able to leverage massive vision datasets. This paper is one example of that being possible, for that I think it deserves an accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- while the scenes are more realistic than some simpler robotics benchmarks, calling them “realistic” is also a little bit of a stretch, the scenes are not every day scenes with real objects, these are toy objects. see comments above."}}]

Author rebuttal:

Decision: Accept (oral)