SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arjun Majumdar $^{1*}$ , Karmesh Yadav $^{1*}$ , Sergio Arnaud $^{2*}$   
Yecheng Jason Ma $^{3}$ , Claire Chen $^{4}$ , Sneha Silwal $^{2}$ , Aryan Jain $^{5}$ , Vincent-Pierre Berges $^{2}$   
Tingfan Wu $^{2}$ , Jay Vakil $^{2}$ , Pieter Abbeel $^{5}$ , Jitendra Malik $^{2,5}$ , Dhruv Batra $^{1,2}$   
Yixin Lin $^{2\dagger}$ , Oleksandr Maksymets $^{2\dagger}$ , Aravind Rajeswaran $^{2\dagger}$ , Franziska Meier $^{2\dagger}$ $^{1}$ Georgia Tech,  $^{2}$ Meta AI,  $^{2}$ UPenn,  $^{2}$ Stanford University,  $^{2}$ UC Berkeley  
https://eai-vc.github.io
We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CORTEXBENCH, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CORTEXBENCH. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.
The visual cortex is a region of an organism's brain, which together with the motor cortex, enables sight to be converted into movement. In this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago - how do we design an artificial visual cortex, the module in a computational system that (together with a policy) enables an agent to convert camera input into actions? In contemporary AI, this question has been operationalized as the design of pre-trained visual representations (PVRs) or visual 'foundation models' for embodied AI (EAI). Indeed, recent work has shown that PVRs can substantially improve performance and learning efficiency for navigation [3-5] and manipulation tasks [6-9]. Unfortunately, prior studies are incommensurable - using different self-supervised learning (SSL) algorithms on different pre-training datasets, designed
3We use embodied AI (EAI) as an umbrella term for all communities studying visuomotor control such as robot learning, vision-based reinforcement learning, egocentric computer vision, etc.
for, and evaluated on different downstream EAI tasks. Naturally, one might ask: Does an artificial visual cortex already exist?4
To answer this question, we conduct the most comprehensive empirical study of visual foundation models for EAI to-date. We curate CORTEXBENCH, a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multifinger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15]. The visual environments span from flat infinite planes to table-top settings to photorealistic 3D scans of real-world indoor spaces. The agent embodiments vary from stationary arms to dexterous hands to articulated mobile manipulators. The learning conditions vary from few-shot imitation learning to large-scale reinforcement learning. The exhaustiveness of this study enables us to draw conclusions with unprecedented scope and confidence.
Our first finding is a negative result. We discover that while existing PVRs generally outperform learning-from-scratch baselines, no single PVR is universally dominant. Instead, we find that PVRs tend to work best in the domains (navigation, manipulation etc.) they were originally designed for. We note that no claims of universality were made in prior
Figure 1: An artificial visual cortex for embodied intelligence must support a diverse range of sensorimotor skills, environments, and embodiments; we curate CORTEXBENCH to systematically measure progress towards this ambitious goal. Our strongest model, denoted VC-1 (adapted), is competitive with or outperforms the best prior results (success rates) on all benchmarks in CORTEXBENCH. This comparison is particularly unforgiving because best prior results are benchmark-specific and not constrained to share any aspect of their design.
work, so this finding is illustrative rather than refutative, but nonetheless a significant finding that was not known a priori. Overall, serendipity did not come to pass – an artificial visual cortex does not already exist. $^{2}$  However, curiously, the kinds of PVRs that are locally-dominant in CORTEXBENCH differ significantly in the size and type of pre-training datasets: CLIP [16] used 400M image-text pairs from the web; MVP [8] used 4.5M frames from web-images and many egocentric-video datasets – yet, each performs best on some subset of tasks in CORTEXBENCH. This leads to a natural question: how does scaling model size, dataset size, or diversity affect performance on CORTEXBENCH? Can we use scaling as a means to learn a single PVR that works for all of the diverse tasks in CORTEXBENCH?
To study these questions, we combine over 4,000 hours of egocentric videos from 7 sources containing humans manipulating objects and navigating indoor spaces, together with ImageNet. From this union, we create 4 pre-training datasets of varying size and diversity, with the largest containing over 5.6M images. We train vision transformers (ViT-B and ViT-L) [17] on these 4 datasets using Masked Auto-Encoding (MAE) [18], and systematically analyze their performance on CORTEXBENCH. To benefit the EAI community, we will open-source these models, which required over 10,000 GPU hours to train.
We do find evidence supporting the scaling hypothesis, but the picture that emerges is more nuanced than what a superficial reading might suggest. Our largest model trained on all data, named VC-1, outperforms the best existing PVR by  $1.2\%$  on average. However, VC-1 does not universally dominate either – i.e., there are PVRs trained on smaller amounts of data that outperform it on specific tasks. A similar trend emerges for data diversity – more is better on average, but not universally. For instance, the best performance on the Mobile-Pick task from Habitat 2.0 [15] is achieved by pre-training on the subset of video data focused on manipulation; presumably because the mobility involved in the task is fairly limited. Thus, our second key finding is: Naively scaling dataset size and diversity does
To the degree of our ability to measure it. Moreover, we make no biological plausibility claims in this work. We are simply motivated by the broad generalization capabilities of a biological visual cortex.
not improve performance uniformly across benchmarks. We note that this broad evaluation refutes a naive extrapolation of the positive scaling trends observed in prior work on robot manipulation [8].
Our findings reveal a challenge and opportunity for the community – the search for a PVR that is universally dominant (or “foundational”) for EAI calls for innovations in architecture, learning paradigm, data engineering, and more. As a first step towards this open problem, we study adapting VC-1 with either task-specific training losses or datasets (via MAE [18]) to specialize VC-1 for each domain. We find that adapting VC-1 results in it becoming competitive with or outperforming the best prior results on all of the benchmarks in CORTEXBENCH. We highlight that this comparison is particularly unforgiving, since best prior results are highly domain-specific and are not constrained to share any aspect of their design. To our knowledge, VC-1 (adapted) is the first PVR that is competitive with (or outperforms) state-of-art results on such a diverse set of EAI tasks (Figure 1).
Finally, we conduct proof-of-concept hardware experiments using VC-1 in a few-shot imitation learning setting with two platforms: a TriFinger robot and a Franka Emika Panda arm. In this real-world setting, we find that VC-1 and VC-1 (adapted) substantially outperform pre-existing PVRs like MVP [8]. We will release code for CORTEXBENCH to enable the EAI, robotics, and CV communities to benchmark their own models, and share our pre-trained models (including VC-1) that we believe can serve as a starting point for all visuomotor tasks of interest today.
Pre-trained visual representations (PVRs). The last few years have seen increasing interest in the self-supervised learning (SSL) of visual representations [18-22]. These algorithms use contrastive [21, 22], distillation-based [19, 20], or reconstructive [18, 23] objectives for training. Recently, a flurry of works have proposed using the vision transformers (ViTs) [24] with masked image modeling [18, 25, 26], which among other benefits reduces the computation time required for pre-training. In this work, we use one such pre-training algorithm (MAE [18]) to explore scaling and adapting pre-trained visual representations.
PVRs for embodied AI. Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents [3-9]. Specifically, [6] evaluate several PVRs trained with supervised or self-supervised learning on a range of EAI tasks, demonstrating promising results under a few-shot imitation learning evaluation protocol. [7-9] introduce new methods for pre-training visual representations using egocentric video data, targeting robot manipulation tasks. Similarly, [3-5] use pre-trained visual representations to improve performance on multiple visual navigation tasks. Closely related, [8] demonstrate that MAE pre-training on internet-scale video and image data can produce effective visual representations for robot manipulation tasks. In contrast, our work studies a larger range of embodied AI tasks (collected in CORTEXBENCH) to understand how PVRs can provide a general-purpose foundation for embodied agents and explores in-domain model adaptation for various tasks.
Language guided foundation models in EAI. There has also been some recent works in the area of language-guided representation learning for control. [27] trains a ViT with a masked encoding objective on pairs of image frames and text. [28] focuses on self-supervised representation learning for goal-conditioned value functions using language-aligned videos. Additionally, [29, 30] employ open-vocabulary detectors and vision-language models to detect objects in tabletop views. These detections, along with the image and vision-language model instructions, are then used to train a policy. In [31], a multimodal transformer is pretrained on web-scale image-and-text data, and then used with a transformer-based policy for table-top manipulation tasks.
Scaling model and dataset size. Several works show that scaling model and dataset size improves performance on vision tasks like image classification [32-34]. In EAI, Radosavovic et al. [8] find that scaling model and data sizes consistently improves downstream policy performance for robot manipulation tasks. Our work is the first to study this question of scaling on a broad range of EAI tasks and refutes a naive extrapolation of the positive scaling trends observed in [8].
Adapting PVRs. When and how to adapt PVRs for downstream applications remains an open research question [35-39]. In the context of EAI, [6] and [40] show that naively fine-tuning PVRs with behavior cloning can reduce performance in simulation, and [8] observe minimal gains in real-world manipulation tasks. In large-scale RL settings, [4, 5] show that end-to-end finetuning considerably improves performance for indoor visual navigation. By comparison, [41] find simple
Figure 2: CORTEXBENCH: We systematically evaluate pre-trained visual representations by varying datasets and representation learning algorithms, coupled with reinforcement or imitation learning on diverse EAI tasks.
$k$ -nearest-neighbor adaptation works well for real-world visual imitation tasks. Our work neither aims nor expects to be the final word on this fertile topic.
3 Benchmarking Progress Towards an Artificial Visual Cortex
We curate CORTEXBENCH (as shown in Figure 1) to evaluate the ability of pre-trained visual representations (PVRs) to support a wide variety of EAI applications. Specifically, we chose 17 diverse tasks drawn from 7 existing EAI benchmarks that have been deemed important by the EAI community. For each task, we delineate a downstream policy learning paradigm (e.g., few-shot imitation learning) and evaluation protocol that follows community standards in each domain (Appendix A.2). By fixing the tasks and downstream learning methods (Figure 2), we are able to focus evaluations on the contribution of PVRs, which in turn allows measuring progress towards the development of an artificial visual cortex for embodied intelligence. We use CORTEXBENCH to conduct the largest and most comprehensive empirical study to-date.
We recommend two evaluation metrics: Mean Success and Mean Rank. Mean Success: the average success rate across all benchmarks. Mean Rank: for each benchmark, we rank PVRs based on their success rate; then we average these rankings across all benchmarks.
3.1 Embodied AI Tasks in CORTEXBENCH
CORTEXBENCH includes the tasks listed in Table 1, illustrated in Figure 1, and described here:
Adroit (AD) [12] is a suite of dexterous manipulation tasks in which an agent must control a 28-DoF anthropomorphic hand to perform a variety of tasks. We study the two hardest tasks from Adroit: Relocate and Reorient-Pen. In these tasks, an agent must manipulate an object into a goal position and orientation, where the goal must be inferred from the scene.
Table 1: CORTEXBENCH includes tasks from 7 diverse benchmarks with different combinations of observations, actions, goals, and standard policy learning paradigms.
Benchmark Suite	Observation Space	Action Space	Goal Specification	Policy Learning
Adroit (AD)	RGB + proprio.	Continuous	-	IL
Metaworld (MW)	RGB + proprio.	Continuous	-	IL
DMControl (DMC)	RGB + proprio.	Continuous	-	IL
TriFinger (TF)	RGB + proprio.	Continuous	Goal Image/Position	IL
ObjectNav (ON)	RGB + proprio.	Discrete	Object Category	IL
ImageNav (IN)	RGB	Discrete	Goal Image	RL
MobilePick (MP)	RGB + proprio.	Continuous	Goal Position	RL
MetaWorld (MW) [11] is a collection of tasks
in which agents command a Sawyer robot arm to manipulate objects in a tabletop environment. We consider five tasks from MetaWorld: Assembly, Bin-Picking, Button-Press, Printer-Open, and Hammer, following the evaluations in [7].
DeepMind Control (DMC) [10] is a widely studied image-based continuous control benchmark, in which agents perform locomotion and object manipulation tasks. We consider five DMC tasks: Finger-Spin, Reacher-Hard, Cheetah-Run, Walker-Stand, and Walker-Walk, following [6].
TriFinger (TF) is a robot, introduced in [13], composed of a three-finger hand with 3-DoF per finger. We focus on the Push-Cube task, which was a part of the Real Robot Challenge 2020 [42]. In this tasks, the agent may use all three fingers to push the cube and move it to a goal location (Push-Cube). Additionally, we consider the easier Reach-Cube task, which was also studied in [43].
Habitat [14] is a simulation platform that includes several visual navigation tasks in which agents explore highly photo-realistic unseen 3D environments. We consider two tasks in Habitat: image-goal navigation (ImageNav) [44] and object-goal navigation (ObjectNav) [45]. In both, the agent starts at a random location in an unknown 3D environment and must find a goal location – specified with an image from the goal location in ImageNav or the name of an object (e.g., 'chair') in ObjectNav.
Habitat 2.0 [15] includes mobile manipulation tasks in which agents control a Fetch robot with a 7-DoF arm, mobile base [46], and suction gripper to rearrange objects in apartment scenes. We consider a challenging version of the Mobile-Pick (MP) task from Habitat 2.0, in which an agent must pick up a target object from a cluttered receptacle (e.g., a counter) while starting from a position in which the object is outside of the robot's reach (thus, requiring navigation), using a relaxed dense goal specification (described in Appendix A.3).
Discussion CORTEXBENCH encompasses a wide range of tasks, from navigation to manipulation and locomotion. These tasks employ different policy learning methods, including low-level imitation learning (MW, DMC, and TF) and large-scale reinforcement learning (ImageNav, ObjectNav, and Habitat 2.0). Some tasks require a high-level understanding of semantic scene information (ImageNav and ObjectNav), while others focus on minor changes in the agent's pose for low-level control (DMC). This diversity in CORTEXBENCH allows us to draw generalized new conclusions about existing and new PVRs.
4 Do we already have a visual foundation model for EAI?
First, we evaluate several existing PVRs on CORTEXBENCH to study whether existing open-sourced visual backbones can consistently perform well across all tasks. For all evaluations preceding Section 6, we consider frozen visual representations to disentangle the effect of learned representations from downstream task learning. Specifically, we include the following models:
- CLIP [16] Contrastive image-language pre-training objective; Trained on 400M images-text pairs from the internet (WIT); ViT-B backbone.
- R3M [7] Time-Contrastive video-language alignment pre-training objective; Trained on 5M images from a subset of Ego4D; ResNet-50 backbone.
- MVP [8]. Masked Auto Encoding (MAE) pre-training objective; Trained on 4.5M images from egocentric videos and ImageNet; ViT-B and ViT-L backbones.
- VIP [9]. Goal-conditioned value function pre-training objective; Trained on 5M images from a subset of Ego4D; ResNet-50 backbone.
These models form a representative set for comparisons, spanning different architectures, pre-training objectives and datasets. Additionally, we include randomly initialized ViTs with frozen- and fine-tuned weights to assess the necessity of pre-training and the limitations of pure in-domain learning.
Table 2: Performance of frozen pre-trained visual representations (PVRs) on CORTEXBENCH. Best prior results are the best reported in literature prior to this work. Overall, we find that no single PVR consistently performs the best across all benchmarks. However, we find that several of these pre-trained models often outperform a random training from scratch baseline. Best prior results sources (row 1): Adroit and MetaWorld approximated from [7], DMControl from [6], ImageNav from [4], ObjectNav from [47]. Frozen PVR Sources (row 2): Adroit, MetaWorld, and DMControl are the same as SOTA, ImageNav from [4], ObjectNav from [48].
#	Model	Imitation Learning	Reinforcement Learning	Mean
Adroit	MetaWorld	DMControl	TriFinger	ObjectNav	ImageNav	Mobile Pick	Rank	Success
1	Best prior result (any setting)	75	80	77	-	70.4	82.0	-
2	Best prior result (Frozen PVR)	75	80	77	-	54.4	61.8	-
3	Random (ViT-B) Frozen	2.0 ± 2.0	0.5 ± 0.5	10.1 ± 0.6	57.8 ± 0.5	19.2 ± 0.9	42.1 ± 0.8	10.8 ± 1.4	7.2	20.4
4	Random (ViT-L) Frozen	2.7 ± 1.8	0.5 ± 0.5	9.1 ± 0.2	57.2 ± 0.9	19.3 ± 0.9	45.2 ± 0.8	20.6 ± 1.8	6.9	22.1
5	Random (ViT-B) Fine-tuned	44.0 ± 2.0	49.9 ± 7.3	43.5 ± 2.4	56.1 ± 1.3	28.5 ± 1.0	62.5 ± 0.7	47.6 ± 2.2	5.3	47.4
6	MVP (ViT-B)	48.0 ± 3.3	91.2 ± 2.9	65.9 ± 2.4	59.7 ± 0.3	51.2 ± 1.1	64.7 ± 0.7	56.0 ± 2.2	3.1	62.4
7	MVP (ViT-L)	53.3 ± 4.1	87.5 ± 3.4	69.2 ± 1.5	74.1 ± 0.3	55.0 ± 1.1	68.1 ± 0.7	65.4 ± 2.1	2.1	67.5
8	CLIP (ViT-B)	47.3 ± 3.0	75.5 ± 3.4	55.5 ± 1.4	62.0 ± 0.5	56.6 ± 1.1	52.2 ± 0.8	49.8 ± 2.2	3.9	57.0
9	VIP (RN-50)	54.0 ± 4.8	90.1 ± 2.2	72.5 ± 2.7	66.7 ± 0.2	26.4 ± 1.0	48.8 ± 0.8	7.2 ± 1.2	4.0	52.3
10	R3M (RN-50)	73.3 ± 2.0	96.0 ± 1.1	81.1 ± 0.7	69.2 ± 0.8	22.7 ± 0.9	30.6 ± 0.7	33.2 ± 2.1	3.4	58.0
Table 2 shows the evaluation results aggregated by benchmark; no single model excels in all cases. Among all of the models evaluated, R3M performs the best on Adroit, MetaWorld, and DMControl. While MVP (ViT-L) performs best on TriFinger, ImageNav, and Mobile Pick. CLIP, on the other hand, achieves the best results on ObjectNav. The variance in performance of existing PVRs on CORTEXBENCH is further illustrated in Figure 5 in Appendix A.4 and highlights that we do not yet have a single, strong performing artificial visual cortex for EAI.
5 Analyzing the Scaling Hypothesis for EAI
The previous section investigated models pre-trained on datasets of varying size and diversity. Interestingly, while the model pre-trained on the largest dataset (CLIP) performs well on one benchmark (ObjectNav) it does not perform well across all tasks. We now ask: how much does the relevance and
(a) scaling model size
(b) dataset diversity effect
(c) ranking of all models
Figure 3: Visualization of scaling hypothesis model performance averaged over CORTEXBENCH. We see modest but positive scaling trends in both (a) scaling model size and (b) dataset diversity. (c) Average ranking of existing PVRs (Table 2) and scaling models (Table 4); VC-1: Ego4D+MNI (ViT-L) has the highest average rank.
diversity of the pre-training dataset and the model size matter? To study this, we fix the pre-training objective (MAE [18]) and vary the composition of the pre-training dataset and the size of the visual backbone (ViT-B with 86M and ViT-L with 307M parameters). We measure the corresponding changes in performance on CORTEXBENCH. MAE is selected for these experiments due to the strong performance on CORTEXBENCH of MVP [8] (Table 2), which uses the MAE pre-training objective.
5.1 Constructing a Pre-training Dataset for EAI
To evaluate the impact of dataset size and diversity on CORTEXBENCH, which involve navigation and manipulation tasks, we employ a combination of 7 datasets. One cluster of datasets - Ego4D [49], 100 Days of Hands (100DOH) [50], Something-Something v2 (SS-V2) [51], and Epic Kitchens [52] - contains videos of people manipulating objects and is comparable to the datasets used in MVP [8]. A second cluster consists of egocentric indoor navigation datasets: the Real Estate 10K dataset [53] and the Open
Table 3: Datasets assembled to study effects of pretraining dataset size, diversity, and relevance - the largest (Ego4D+MNI) has 5.6M frames. More details in A.5.
Name	Frames Used
Ego4D	2,790,520
Ego4D+M (Manip)	3,538,291
Ego4D+N (Nav)	3,593,049
Ego4D+MN (Manip, Nav)	4,340,820
Ego4D+MNI (Manip, Nav, ImageNet)	5,621,987
House24 dataset (described in Appendix A.5.1). Finally, we include ImageNet [54]. We strategically select dataset combinations (shown in Table 3) to answer the following questions:
- What is the impact of scaling dataset size and diversity?
- How do less-relevant datasets influence the performance of PVRs on EAI tasks?
Ego4D [49] (our base dataset) includes a wide range of egocentric videos consisting of daily life activities such as home, leisure, transportation, and workplace activities.
Ego4D+M extends Ego4D with videos from 100DOH, SS-v2, and Epic Kitchens, resulting in 3.5M frames and making this dataset primarily focused on object manipulation scenarios.
Ego4D+N extends Ego4D with two indoor navigation datasets: OpenHouse24 and RealEstate10K. This dataset is similar in size to Ego4D+M (3.5M frames) but is more diverse because it contains a larger proportion of navigation data than the manipulation-centric datasets Ego4D and Ego4D+M.