Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Authors: Arjun Majumdar
Karmesh Yadav
Sergio Arnaud
Yecheng Jason Ma
Claire Chen
Sneha Silwal
Aryan Jain
Vincent-Pierre Berges
Tingfan Wu
Jay Vakil
Pieter Abbeel
Jitendra Malik
Dhruv Batra
Yixin Lin
Oleksandr Maksymets
Aravind Rajeswaran
Franziska Meier
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
pre-training
foundation models
embodied AI
reinforcement learning
imitation learning
TL;DR: We present the largest and most comprehensive empirical study of visual foundation models for Embodied AI (EAI).

Abstract: We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual ‘foundation models’ for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Arjun Majumdar', Karmesh Yadav Sergio Arnaud
Yecheng Jason Ma", Claire Chen', Sneha Silwal Aryan Jain', Vincent-Pierre Berges" Tingfan Wu?, Jay Vakil Pieter Abbeel', Jitendraa Malik2.5, Dhruv Batra 1.2 Yixin Lin i Oleksandr Maksymets Aravind Rajeswaran" Franziska Meier Georgia Tech, Meta AI, UUenn, Stanford University, PCC Berkeley https: hea//eaiveee tttps///ea-vc..ituub.... github.io
Abstract
We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CORTEXBENCH, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none universally dominant. To study the effect of pre-training data size and diversity, we combine over 4, 4,000 hours of egocentric videos from different sources (over 4.3M images) and ImageNet to train different- sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-speciff adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive superior performance than the best known results all of the benchmarks in CORTEXBENCH, Finally, we present real-world hardware experiments, which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours train) for the benefit of the research community.
Introduction
The visual cortex is region of an organism's brain, which together with the motor cortex, enables sight to be converted into movement. this work, we ask the same question that Fukushima [1, 2] asked nearly 50 years ago how do we design an artificial visual cortex, the module in computational system that (together with a policy) enables an agent to convert camera input into actions? In contemporary AI, this question has been operationalized as the design of pre-trained visual representations (PVRs) or visual 'foundation models' for embodied AI (EAI). Indeed, recent work has shown that PVRs can substantially improve performance and learning efficiency for navigation [3-5] and manipulation tasks [6-9]. Unfortunately, prior studies are incommensurable- using different self-supervised learning (SSL) algorithms different pre-training datasets, designed
Equual Contribution
Equal Contribution
'We use embodied AT (EAT) umbrella term for all communities studying visuomotor control such robot learning, vision-based reinforcement learning, egocentric computer vision, etc.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

for, and evaluated on different downstream EAI tasks. Naturally, one might ask: Does an artificial visual cortex already exist?"
To answer this question, we conduct the most comprehensive empirical study of visual foun- dation models for EAI to-date. We curate CORTEXBENCH, a benchmark for evaluating PVRs, consisting of 17 tasks spanning low- level locomotion [10], table-top manipula- tion [11], dexterous manipulation [12], multi- finger coordinated manipulation [13]. indoor visual navigation [14], and mobile manipu- lation [15]. The visual environments span from flat infinite planes to table-top settings to photorealistic 3D scans of real-world in- door spaces. The agent embodiments vary from stationary arms to dexterous hands to articulated mobile manipulators. The learn- ing conditions vary from few-shot imitation learning to large-scale reinforcement learning. The exhaustiveness of this study enables us to draw conclusions with unprecedented scope and confidence.
Meta Work
Control
Tillnger
Adron
Mabi to
92
Des Prio fieaut
vCl (Acopnod)
Imegekav
ObjectNov
CortexBench
Figure An artificial visual cortex for embodied in- telligence must support a diverse range of sensorimotor skills, environments, and embodiments; TEXBENCH systematically measure progress towards this ambitious goal. Our strongest model, denoted VC-1 (adapted). is competitive with or outperforms the best prior results (success rates) on all benchmarks in COR- TEXBENCH. This comparison particularly unforgiving because best prior results are benchmark specific and not constrained share any aspect of their design.
Our first finding is a negative result. We dis- cover that while existing PVRs generally out- perform
curate COR-
frcamsctttch baselines, no single PVR is universally dominant. Instead, we find that PVRs tend to work best in the domains (navigation, manipulation etc.) they were originally designed for. We note that no claims of universality were made in prior work, so this finding is illustrative rather than refutative, but nonetheless a significant finding that was not known a priori. Overall, serendipity did not come to pass an artificial visual cortex does not already exist. However, curiously, the kinds of PVRs that are docally-dominant differ significantly in the size and type of pre-training datasets: CLIP [16] used 400M image-text pairs from the web; MVP [8] used 4.5M frames from web-images and many egocentric-video datasets yet, each performs best on some subset of tasks in CORTEXBENCII This leads to a natural question: how does scaling model size, dataset size, or diversity affect performance on CORTEXBENCH? Can we use scaling as means to learn a single PVR that works for all of the diverse tasks in CORTEXBENCH?
in CORTEX BENCH
To study these questions, we combine over 4,000 hours of egocentric videos from sources containing humans manipulating objects and navigating indoor spaces, together with ImageNet. From this union, we create 4 pre-training datasets of varying size and diversity, with the largest containing over 5.6M images. We train vision transformers (ViT-B and ViT-L) [17] on these datasets using Masked Auto-Encoding (MAE) [18], and systematically analyze their performance on CORTEX To benefit the EAI community, we will open-source these models, which required over 10,000 GPU hours to train.
We do find evidence supporting the scaling hypothesis, but the picture that emerges is more nuanced than what a superficial reading might suggest. Our largest model trained on all data, named VC-1, outperforms the best existing PVR by 1.2% average. However, VC-1 does not universally dominate either i.e., i.e., there are PVRs trained on smaller amounts of data that outperform it on specific tasks. A similar trend emerges for data diversity- more is better average, but not universally. For instance, the best performance on the Mobile Pick task from Habitat 2.0 [15] is achieved by pre-training on the subset of video data focused on manipulation; presumably because the mobility involved in the task is fairly limited. Thus, our second key finding is: Naively scaling dataset size and diversity does
'To the degree of our ability to measure it. Moreover, we make no biological plausibility claims in this work. We simply motivated by the broad generalization capabilities of a biological visual cortex.

---

not improve performance uniformly across benchmarks. We note that this broad evaluation refutes a naive extrapolation of the positive scaling trends observed in prior work on robot manipulation [8].
Our findings reveal a challenge and opportunity for the community - the search for a PVR that is universally dominant (or "foundational") for EAI calls for innovations in architecture, learning paradigm, data engineering, and more. As a first step towards this open problem, we study adapting VC-1 with either task-specific training losses or datasets (via MAE (18]) to specialize VC-1 for each domain. We find that adapting VC-1 results in it becoming competitive with or outperforming the best prior results on all of the benchmarks in CORTEX We highlight that this comparison is particularly unforgiving, since best prior results are highly domain-specific and are not constrained to share any aspect of their design. To our knowledge. VC-1 (adapted) is the first PVR that is competitive with (or outperforms) state-of-ar art results on such diverse set of EAI tasks (Figure 1).
Finally, conduct proof-of-concee hardware experiments using VC-1 in few-shot imitation learning setting with two platforms: a TriFinger robot and a Franka Emika Panda arm. In this real-world setting, we find that VC-1 and VC-1 (adapted) substantially outperform pre-existing PVRs like MVP [8]. We will release code for CORTEXBENCII to enable the EAI, robotics, and CV communities benchmark their own models, and share our pre-trained models (including VC-1) that we believe can serve as a starting point for all visuomotor tasks of interest today.
Related Work
Pre-trained visual representations (PVRs). The last few years have seen increasing interest in the self-supervised learning (SSL) of visual representations [18-22]. These algorithms use contrastive [21, 22]. distillation-based [19, 20], or reconstructive [18, 23] objectives for training. Recently, flurry of works have proposed using the vision transformers (ViTs) [24] with masked image modeling [18, 25, 26), which among other benefits reduces the computation time required for pre-training. In this work, we use one such pre-training algorithm (MAE (18)) to explore scaling and adapting pre-trained visual representations.
PVRs for embodied AI. Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents [3-9]. Specifically, [6] evaluate several PVRs trained with supervised range of EAI tasks, demonstrating promising results under a few-shot imitation learning evaluation protocol. [7-9] introduce new methods for pre-training visual representations using egocentric video data, targeting robot manipulation tasks. Similarly, [3-5] use pre-trained visual representations to improve performance on multiple visual navigation tasks. Closely related, [8] demonstrate that MAE pre- pre-training on internet-scale video and image data can produce effective visual representations for robot manipulation tasks. In contrast, our work studies larger range of embodied Al tasks (collected in CORTEX BENCH) to understand how PVRs can provide general-purpose foundation for embodied agents and explores in-domain model adaptation for various tasks.
self-supervised learning on
Language guided foundation models in EAL There has also been some recent works in the area of ainuagg--gided representation learning for control. [27] trains a ViT with masked encoding objective on pairs of image frames and text. [28] focuses on self-supervised representation learning for codditondi value functions using language-alige aligned videos. Additionally, [29, 30] employ open- veen-vocabblary detectors and vision-lane languae models to detect objects in tabletop views. These detections, along with the image and dision-language model instructions, are then used to train policy. In [31], a multimodal transformer's is pretrained on web-scale image-and-text data, and then used with cransformer-based policy for table-top manipulation tasks.
Scaling model and dataset size. Several works show that scaling model and datasct size improves performance vision tasks like image classification [32-34]. In EAI, Radosavovic et al. [8] find that scaling model and data sizes consistently improves downstream policy performance for robot manipulation tasks. Our work is the first study this question of scaling on broad range of EAI tasks and refutes a naive extrapolation of the positive scaling trends observed in [8].
Adapting PVRs. When and how to adapt PVRs for downstream applications remains an open research question [35-39]. the context of EAI, [6] and [40] show that naively fine-tuning PVRs with behavior cloning can reduce performance in simulation, and [8] observe minimal gains in real-world manipulation tasks. large-scale RI. settings, [4, 5] show that end-to-end finetuning considerably improves performance for indoor visual navigation. By comparison, [41] find simple
3

---

detasadx
methods
Ego4D
Behavier Policy triniing
Evaluction on Adroh Dexterous
Assembled
Datasel
Cortex
Evalustion
RealEstate1OK
Visual Representation
Learning
AL Policy training
Evaluation on Habitat Imagehay
Figure 2: CORTEX BENCH: We systematically evaluate pre-trained visual representations by varying datasets and representation learning algorithms, coupled with reinforcement or imitation learning on diverse EAI tasks.
n-niahost-neighhor adaptation works well for real-world visual imitation tasks. aims nor expects to be the final word on this fertile topic.
Our work neither
Benchmarking Progress Towards an Artificial Visual Cortex
We curate CONTEXBENCH (as shown in Figure 1) to evaluate the ability of pre-trained visual representations (PVRs) support wide variety of EAI applications. Specifically, we chose 17 diverse tasks drawn from existing EAI benchmarks that have been deemed important by the EAI community. For each task, we delineate downstream policy learning paradigm (c.g.. few- shot imitation learning) and evaluation protocol that follows community standards in each domain (Appendix A.2). By fixing the tasks and downstream learning methods (Figure 2), we are able to focus evaluations on the contribution of PVRs, which in turn allows measuring progress towards the development of an artificial visual cortex for embodied intelligence. We use CORTEXBENCH to conduct the largest and most comprehensive empirical study to-date.
We recommend two evaluation metrics: Mean Success and Mean Rank. Mean Success: the average success rate across all benchmarks. Mean Rank: for each benchmark, we rank PVRs based on their success rate; then we average these rankings across all benchmarks.
3.1 Embodied AI Tasks in CORTEXBENCH
CORTEXBENCH includes the tasks listed in Table 1, illustrated in Figure 1, and described here:
Adroit (AD) [12] is suite of dexterous ma- nipulation tasks in which an agent must control 28-DoF anthropomorphic hand to perform variety tasks. We study the two hardest tasks from Adroit: Relocate and Reorient- -Pn. In these tasks, an agent must manipulate an object into a goal position and orientation, where the goal must be inferred from the scene.
Table I: CORTEX includes tasks from 7 diverse benchmarks with different combinations of observations, actions, goals, and standard policy learning paradigms.
Bendhmark
Oboarvation
Action
Gual
Pulicy
Sulte
Space
Space
Speciticttin
Leamingg
AdreR (AD)
RGB peopsio.
Continouss
Miteworld
peopris.
Continans
DMContol (DNCC)
peopeio.
Contmoous
peopeio
OtjectNas (ON)
RGBB
Discrete
Object Cargonyy
ImogeNas (IN)
RGB
Decrnt
Geal Imoge
Meta World (MW) [11] is collection of tasks in which agents command Sawyer robot arm manipulate objects in a tabletop environment. We consider five tasks from MetaWorld: Assembly, Bin-Pickinn Button-Press Drawer-Open
MobilePick (MP)
RGB peopeio.
Continauus
Goal Position
and Hammer, following the evaluations in [7].
Deep.ind Control (DMC) [10] is a widely studied image-based continuous control benchmark, in which agents perform locomotion and object manipulation tasks. We consider five DMC tasks: Finger-Spin, Reacher-Hard, Cheetah-Run Walker-Stan -Stand, and Walker-Walk following [6].
TriFinger (TF) is robot, introduced in [13], composed of three-finger hand with 3-DoF per finger. We focus on the Push tasks, the agent may use all three fingers to push the cube and move it to goal location Additionally, we consider the easier Reach-Cube task, which was also studied in [43].
task, which was a part of the Real Robot Challenge 2020 [42]. this
Habitat [14] is a simulation platform that includes several visual navigation tasks in which agents explore highly photo-realistic unseen 3D environments. We consider two tasks in Habitat: image-goal navigation (ImageNav) [44] and object-goa navigation (ObjectNav) [45]. In both, the agent starts random location in an unknown 3D environment and must find a goal location specified with an image from the goal location in ImageNav or the name of an object (c.g., 'chair") in ObjectNav.

---

Habitat 2.0 [15] includes mobile manipulation tasks in which agents control Fetch robot with 7-DoF arm, mobile base [46], and suction gripper to rearrange objects in apartment scenes. We consider a challenging version of the Mobile-Pick (MP) task from Habitat 2.0, in which an agent must pick up a target object from a cluttered receptacle (e.g., a counter) while starting from a position in which the object is outside of the robot's s reach (thus, requiring navigation), using relaxed dense goal specification (described in Appendix A.3).
Discussion CORTEXBENCI encompasses wide range of tasks, from navigation to manipulation and locomotion. These tasks employ different policy learning methods, including low-level imitation learning (MW, DMC, and TF) and large-scale reinforcement learning (ImageNav, ObjectNNvv, and Habitat 2.0). Some tasks require a high-level understanding of semantic scene information (ImageNav and ObjectNary while others focus on minor changes in the agent's 's pose for low-level control (DMC). This diversity in CENCH allows us to draw generalized new conclusions about existing and new PVRs.
4 Do we already have a visual foundation model for EAI?
First, we evaluate several existing PVRs on CORTEXBENCH to study whether existing open-soureed visual backbones can consistently perform well across all tasks. For all evaluations preceding Section 6, we consider frozen visual representations to disentangle the effect of learned representations from downstream task learning. Specifically, we include the following models:
CLIP [16] Contrastive image-language pre-training objective; Trained on 400M images-text pairs from the internet (WIT); ViT-B backbone.
R3M [7] Cone-Cotrasttive video-language alignment pre-training objective; Trained on 5M images
from subset of Ego4D; ResNet-50 backbone.
MMP [8]. Masked Auto Encoding (MAE) pre-training objective; Trained on 4.5M images from egocentric videos and ImageNet: ViT-B and ViT-L backbones.
VIP [9]. Goal-conditioned value function pre-training objective; Trained on 5M images from a subset of Ego4D; ResNet-50 backbone.
These models form representative set for comparisons, spanning different architectures, pre-training objectives and datasets. Additionally, we include randomly initialized ViTs with frozen- and fine- tuned weights assess the necessity of pre-training and the limitations of pure in-domain learning.
Table 2: Performance of frozen pre-trained visual representations (PVRs) on CORTEXBENCH. Best prior results are the best reported in literature prior to this work. Overall, we find that no single PVR consistently performs the best across all benchmarks. However, random training from scratch baseline. Best prior results sources (row 1): Adroit and MetaWorld approximated from [7]. DMControl from [6]. ImageNav from [4], ObjectNav from [47]. Frozen PVR Sources (row 2): Adroit, MetaWorld, and DMControl are the same as SOTA, ImageNav from [4], ObjectNav av from [48].
find that several of these pre-trained models often outperform
Imitation Leurning
Reinforcement Learning
Mean
Model
Adroit
MetaWorid
DMCoorrl
Inkinger
ObjectNav
ImageNav Mobile Pick
Rink
Success
Best poior result Cany setting) prior result
15
80
70.4
$2.0 61.8
PVR)
75
80
54.4
Kundom (VIT-B) Frozen
10.1
57.8
19.2
10.5
7.2
Random (ViTLL) Frozen
27 1.8
0.5 0.5
9.1102
57.2 1 0.9
18.3
1 0.9
45.2 0.8
20.5 1
6.9
22.1
Random (VIT-B) Fine-tuned
440- 2.0
49.9 7.3
43.5
2.4
56.1
+ 1.3
28.5
1.0
62.5 + 0.7
47.6 +
5.3
47.4
MVP (VIT B) MVP (VT--))
3.3
9.
65.9
59.7
0.3
1..2111.
61.7 0.7
56.0 122
3.1
62.4
53.3114.1
87.5 3.4
692 15
74,1
0.
55.0 1.1
68.1 0.7
65.4 1.1
2.1
67.5
CLIP (VIT-B)
4..3
3.44
15
62.0
0.5
1.1
52.2 à: 0.8
49.8 22
3.9
57.0
VIP (RN-50)
540--488
4-2.
26.4
48.8+ 0.8
7.2-11.
4.0
10
R3M (XN-50)
73.3=20
9.1
81.1
69.2
22.7
30.6 0.7
33.2++21
3.4
58.0
Table 2 shows the evaluation results aggregated by benchmark; no single model excels in all cases. Among all of the models evaluated, R3M performs the best on Adroit, MetaWorkd, and DMControl. While MVP (ViT-L) performs best on TriFinger, ImageNav, and Mobile Pick. CLIP, on the other hand, achieves the best results on ObjectNav. The variance performance of existing PVRs on CONCEXBENCH is further illustrated in Figure 5 in Appendix A.4 and highlights that we do not yet have a single, strong performing artificial visual cortex for EAI
Analyzing the Scaling Hypothesis for EAI
The previous section investigated models pre-trained on datasets of varying size and diversity. Inter- estingly, while the model pre-trained on the largest dataset (CLIP) performs well on one benchmark (ObjectNav) it does not perform well across all tasks. We now ask: how much does the relevance and
5

---

101
Averngg Success
65
Auenng Strrwann
80
60
NAT Egp-t 181
MAT Egg-th-W
WAT Top-t--1t
Wot tge4 9hh
MAd cp-4D-N
MAT Typ
HMA
ROL Ege4
MAX Egrtt
NAT Egg
122 Eg
8
RANDOO
5
5
175
(a) scaling model size
(h) dataset diversity effect
(c) ranking of all models
Figure 3: Visualization of scaling hypothesis model performance averaged over CORTEXBENCH. We modest
but positive scaling trends in both (a) scaling model size and (b) dataset diversity. (c) Average ranking of existing PVRs (Table 2) and scaling models (Table 4); VC-1: Ego4D+MNI (ViT-L) has the highest average rank.
diversity of the pre-training dataset and the model size matter? To study this, we fix the pre- pre-training objective (MAE [18]) and vary the composition of the pre-training dataset and the size of the visual backbone (ViT-B with 86M and ViT-L. with 307M parameters). We measure the corresponding changes performance on CENCHXBENCH. MAE is selected for these experiments due to the strong performance on CORTEXBENCH of MVP [8] (Table 2), which uses the MAE pre-training objective.
5.1 Constructing a Pre-training Dataset for EAI
To evaluate the impact of dataset size and diver- sity on FORTEXBENCEE which involve naviga- tion and manipulation tasks, we employ com- bination of 7 datasets. One cluster of datasets Ego4D [49], 100 Days of Hands (100DOH) [50], SomnetinggSSomethigg
Table 3: Datasets assembled to study effects of pre- training dataset size, diversity, and relevance the largest (Ego4D+MNI) has 5.6M frames. More details A.5.
Name
Frames Used
v2 (SS-V2) [51], and Epic Kitchens [52] - contains videos of people
Ego4D
2,790,520
Ego4D+M (Manip)
3,538,291
manipulating objects and is comparable to the
Ego4D4N (Nav)
3.593,049
datasets used in MVP [8]. A second cluster con- sists of egocentric indoor navigation datasets: the Real Estate 10K dataset [53] and the Open-
Ego4D+MN (Manip, Nav)
4,340,820
Ego4D+MNI (Manip. Nav, ImageNet)
5,621,987
House24 dataset (described in Appendix A.5.1). Finally, we include ImageNet [54]. We strategically
select dataset combinations (shown in Table 3) to answer the following questions:
What is the impact of scaling dataset size and diversity?
How do less-relevant datasets influence the performance of PVRs on EAT tasks?
Ego4D [49] (our base dataset) includes a wide range of egocentric videos consisting of daily life activities such as home, leisure, transportation, and workplace activities.
Ego4D+M extends Ego4D with videos from 100DOH, SS-v2, and Epic Kitchens, resulting in 3.5M frames and making this dataset primarily focused on object manipulation scenarios."
Ego4D+N extends Ego4D with two indoor navigation datasets: OpenHouse24 and RealEstate 10K. This dataset is similar in size to Ego4D+M (3.5M frames) but is more diverse because it contains a larger proportion of navigation data than the manipulation-centric datasets Ego4D and Ego4D+M.
Ego4D+MN combines Ego4D with the three object manipulation-centric datasets and two indoor navigation dataset, resulting dataset with 4.3M frames. While larger than Ego4D+M and Ego4D+N, it does not include any new types of data beyond the manipulation and navigation videos in the previous subsets. Thus, it is more diverse than Ego4D+N (which includes both types of data).
Ego4D+MNI includes Ego4D, all manipulation and navigation datasets, and ImageNet for a total of 5.6M frames. This dataset is used to study the role of internet images for our benchmark tasks.
5.2 Scaling Hypothesis Findings
We now turn to analyzing the effect of increasing model size, dataset size, and dataset diversity. The full set of results is shown in Figure 3 and Table 4. The key takeaways are:
Model Size. We find that increasing model size positively impacts performance on CORTEXBENCH. Specifically, in Figure 3a, we find that with all pre-training datasets, switching from ViT-B to VIT-L
Whiie Ego4D does contain navigation data, it is heavily skewed towards object manipulation.

---

Table 4: Performance of scaling hypothesis models on CORTEXBENCH. We find that on average the VC-1 EG04D+MNT (VIT-L.) model performs best, but is not the best for each benchmark.
Model
Adroit
Met-Workd
DMCosatee
TriFinger
OhjectNav
Imagehive
Mobile Pick Mean Rank
Mean Sureses
Bost pelce rosllt Ianyy setting)
75
70.4
$2.0
Rand (VTT-)) finc-uned
44.0
40.9 96.0
14.2
$5.0 74.1
28.5
65.0
47.6 65.4
Red roult Table (Frosss PVR)
75.3
$1.1
56.6
68.1
48.7 = 1.3
86.1 + 2.1
64.1 +.2
68.3 1.1
46.8
11
640--07
574 2.2
8.6
62.3
Fgo4D( (VTEL.)
50.0 + 1.3
92.0 + 2.4
60.5+11
697
42.6
11
558 +58+008
67.6
2.1
5.9
61.5
RgolD+N (VTE-8)
50.0 = 24
8644
2.9
59.5 2.4
678 1,3
54.7
1.1
68.7+0.77
504
2.2
7.2
6388
EgoID+N (VIT-L)
54.0 1:0110 122
89.1
1.29
66.4
1 1.7
669
574
L1
70. 1.7
65.2
2.1
3.5
67.1
RgodD-M
$1.3 24
85.5
2:
64.3 18
1:04
47.
11
658
598
222
7.0
630
RgodD-MM (VIT-L)
$3.0 1.3
88.1 + 1.2
64.7+224
647++0
47.3
11
655+007
68.6
2.1
5.0
64.4
10
EgoID+MN (VITB)
48.7 24
85.3
1:52
642 19
70.3
528
1.1
58.6 22
6.9
68.1
11
EgoID+MN (VIT
52.7 42
86.7 1.99
69.7 1:3
105
58,4
1.1
69.1
612
22
3.1
67.2
12 EgoiDDMMNI (VIT-D) D)
54.0
8,0
89.6 3.9
2.7
12.
58.4
111
679.0.7
000 22
4.4
66.2
11 VC-E Ego4D (VI
59.3
88.8.1.22
669 1.4
7.7
00.5 1.1
703
63.2 22
24
68.7
improves average performance on CORTEXBENCIL However, in Table 4, we find exceptions where this general trend does not hold. For instance, when pre-trained on Ego4D+MNI, the ViT-B model outperforms the VIT-L model on MetaWorld and TriFinger.
Dataset Size and Diversity. Figure 3b shows that, in general, increasing dataset size and diversity leads to improved performance. Models are are ordered from right to left by increasing size and the diversity of their pre-training dataset, and we mostly see improvements for both ViT-B and ViT-L., For instance, Ego4D+M slightly improves upon Ego4D by 0.6 and 0.9 points (62.2 -> 62.8 and 63.5 - 64.44) in the case of VIT-B and ViT-L,, respectively. The gains with Ego4D+N are larger and it outperforms Ego4D by 1.6 points using ViT-B (62.2 + 63.8) and by 3.6 points for ViT-L (63.5 - 67.1). It is interesting to note that Ego4D+N has larger improvement over the base Ego4D dataset than Ego4D+M, even though Ego4D+N and Ego4D+M dataset are similar size. In these results, we find that increasing diversity by adding indoor navigation data improves performance more than adding additional manipulation data to Ego4D.
Additionally, we find that pre-training on Ego4D+MN is roughly on par with pre- pre-training on Ego4D+N. We see a 0.3 and 0.1 point difference (63.8 -> 64.1 and 67.1 + 67.2) for ViT-B and ViT-L., respectively, even though Ego4D+MN has about 800K more training frames. Together with the results from above this demonstrates that increasing data diversity seems to matter more than simply increasing dataset size.
Next, we find that adding ImageNet positively impacts average performance on CORTEX For example, models pre-trained on Ego4D+MNI outperform those pre-trained on Ego4D+MN by 1.9 points (64.1 + 66.2) for ViT-B and 1.5 points (67.2 68.7) for ViT-L. Interestingly. these results demonstrate that including static internet images can significantly boost performance on EAI tasks. This finding further highlights the importance of seeking data diversity to build better representations.
Finally, our largest model (ViT-L) pre-trained on all datasets (Ego4D+MND), achieves the best rank when averaged across all benchmark tasks (Table 4 row 11), with mean rank of 2.4. We call this model VC-1, and will open-source it. VC-1 is superior to the second-best model (Ego4D+MN ViT-L, Table 4 row 9), which has an average rank of 3.1.
However, upon further dis-aggre,, we observe we find that while VC-1 performs best on average, it is not the best for each benchmark. For example, the best model for Mobile Pick, mobile manipulation task, is a ViT-L trained on Ego4D+M and the best model for ImageNav, an indoor navigation task, is the ViT-L trained on Ego4D+N. These findings suggest that task-specific pre-training datasets could enhance the performance of models on individual tasks. However, is important note that this approach would lead to multiple pre-trained models, each tailored to specific task, and not a unified visual foundation model.
5.3 How does VC-1 compare to existing PVRs?
We now compare VC-1 to PVRs from Section 4. On average, VC-1 has the best rank across all benchmarks (Figure 3c). In terms of mean success, VC-1 (Table 4 row II) outperforms MVP (ViT-L) by +1.2 points (67.5 - 68.7), R3M by + +10.7 (58.0 -> 68.7), CLIP by +11.7 (57.0 + 68.7), and end-to-end fine-tuning from scratch +1 +19.6 (49.1 68.7).
Impressively, VC-1 outperforms CLIP on every benchmark (Figure 4), despite training on a 70X smaller dataset, emphasizing the importance of egocentric interaction datasets. VC-1 also outperforms fine-tuning from scratch on every benchmark, indicating that PVRs trained with oft-on-domain data can outperform in in-domain, end-to-end learning.
7

---

Magiena
Nasletes
DrGaeesl trajetts
bricetall inajette
Wecelral insdetar
Checcbr
(LT
Kan (4T2)
13N (WH50)
fardam (155)) MCP Fe M5S NIT1)
MaK HUTTL
NOW 40 2A
Figure Comparison VC-1 with existing PVRs. VC-1 matches or exceeds existing PVRs on benchmarks except R3M on AD, MW, and DMC, indicating an opportunity for model adaptation.
The MVP model the most similar in terms results, architecture, and pre- trainiiinng objective to VC-1, with the main difference being the addition of a convolutional stem in MVP. VC-1 outperforms MVP VIT-L by 1.3 points on mean success and performs better on four out of seven benchmarks diverse dataset.
(Figure 4), likely due to the of
When compared to R3M, VC-1 demonstrates superior performance both on average and in 4 out of 7 benchmarks (Figure 4). However, R3M outperforms VC-1 Adroit, MetaWorld and DMControl benchmarks. The cause of this gap is unclear could be due to differences in pre-training objectives, datasets, or the backbone architecture. That said, the fact that R3M, which uses ResNet-basee architecture, performs better on some tasks than VC-1 which employs a Transformer based backbone, suggests the potential value of exploring approaches inductive biases and scalability. Taken together, these observations highlight the need for more robust and standardized evaluations benchmarks like CORTEXBENCH
architectures that integrate the strengths of both
Overall, VC-1 is effective across broad set of tasks and thus a reasonable starting point for novel EAI problems. However, it is not always the best model for specific task. This leads to theorize that there is a domain gap that might be bridged with dataset engineering or adaptation of the PVR.
6 Adapting VC-1
In prior sections, we focused on evaluating VC-1 as frozen PVR. We now study if adapting VC-1 improve results in downstream tasks. We use broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some all weights of the backbone [5, 57].
In context of PVRs for EAL, adaptation can serve at least two purposes. The first is task- specialization in the feature extraction stage. Since VC-1 was trained with MAE [18], it captures fea- tures that are generally useful for reconstructing images (see VC-1 attention maps in Appendix 10). Adaptation can specialize the visual backbone to extract features required for performing specific EAI tasks. Secondly, adaptation can also help mitigate domain-ged gap that might exist between pre- training and evaluation settings. In general, domain-gan gap arise for several reasons such poor coverage in pre-training datasets or deployment in novel conditions (e.g., ., on robots) not seen in the pre-training data (e.g., in human-centric video datasets). Domain gap is naturally instantiated in setup, since VC-1 was pre- pre-trained on real-world, human video data while our downstream evaluation in CORTEX uses simulated EAI domains with different visual characteristics.
In this work, we explore two methods for adaptation: end-to-end (E2E) fine-tuning and MAE adaptation. While we do not explore wompping-basee adaptation, because our visual encoders were not originally designed to utilize prompts, it may be an interesting direction for future work.
End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature [18, 20, 58, 59]. To study E2E fine-tum fine-tuning of VC-1, we use the same policy learning methods described in Appendix A.2, except we allow updates the VC-1 weights. In the CORTEXBENCH results in Table 5, we find an interesting mixed result. In domains that involve large-scale IL or RL (ObjectNav, ImageNav, and Mobile Pick), the strategy proposed in [5] of adapting VC-1 with E2E fine-tuning significantly improves performance over using a frozen VC-1 backbone. Specifically, we see an improvement in ObjectNav success rate (SR) of +7.4 (60.3 -> 67.7), ImageNav SR of +11.3 (70.3 + 81.6), and Mobile Pick SR of +10.8 (63.2 - 74.0). These results suggest that E2E

---

Table 5: Adapting VC-1 with end-to end tne-tuning or self superviee learning (MAE) on task-specific data leads to substantial gains. Best prior results are across any setting (frozen/finttned), Best result (our exp.) are from Section 5. indicates that, hardware, we only evaluated MVP (ViT-L., the closest competitor to VC-1. The TF Push-Cube results are averaged across 12 trials with randomized start/goal configurations. The Franka results are averaged across tasks and 10 trials per task (details in Appendix A.12).
ComedBench TriFinge
Hardware
Molhad
Admul
MenWWeld
DMCsstol
OlgasiNav
InageNav
Mibile
Push-Cube
Franka
Best
80
70.4
H2.0
Best result In domsin MAL (baseline)
2 47.3
96.0 1:11
2..7 77.6
74.1 1.0.
60.3
10341.
205.100 0.7
58.6 28.62:2.
3.9 119.440 4.3" 22,9
55.0 2 10.4 35.0 13,9
83.4
1..32
39,9 1 1.09
43.6 a
51. 2.25
VC-I
99.5
88,8 122
669 1.4
71. 0.
60.3
70.5
63.2 3.22.22.
70. 10.9
VC-I C2E Eine taning
15.9
22.7
1.7
6.7
709 1.0.
67.7 105
81.6
74.0 1.96
$2.7 9.2
67.5 13,4
VC-I MAE edaptation
72,0
96.0
188
809 a 188
80.61.0.22
57.4
62.4
4.
1..0
fine-tuning of VC-1 can achieve the benefits of both sask-specialization and domain adaptation. A comparison of VC-1 attention maps before and after adaptation is provided in Appendix A.
However, in few-shot IL domains (Adroit, MetaWorld, DMC, and TriFinger), E2E fine-tuning does not result in improvements. In fact, in these domains, it leads to drop in performance, finding consistent with prior work [6, 40]. We hypothesize that the poor performance of E2E fine-tuning in fow-shot IL domains is caused by overfitting, due to fine-tuning large model with 307M parameters on small dataset (S 50K frames).
MAE adaptation to mitigate domain-gap. As an alternative to E2E fine-tuning, we explore adapting VC-1 with self-supervised learning (SSL). Specifically, in MAE adaptation we continue training the backbone network with the MAE [18] pre- pre-training objective on task-specific data. Then, we freeze these adapted representations and use them to learn task-specific policies. We note that in MAE adaptation, the backbone is adapted using the same data that is used for training the policy (e.g., frames from expert demonstrations), and no additional in-domain datasets are used. While this adaptation strategy cannot address task- speccialiiiziiinnn it may serve to mitigate domain gap.
For MAE adaptation, we initialize with VC-1 weights, and then train with MAE for 100 epochs. In domains where expert demonstrations are available (i.e., Adroit, MetaWorld, DMControl, TriFinger, and ObjectNav), we use the RGB frames from these demonstrations for adaptation. In the remaining two benchmarks (ImageNav and Mobile Pick) we sample frames from training environments to create adaptation datasets. Finally, to isolate the importance of initializing with VC-1 weights, we train in-domain MAE baselines by starting from a random initialization and then following the same approach used for MAE adaptation.
In the CORTEXBENCH results in Table 5, we observe MAE adaptation substantially improves per- formance in few-shot learning domains. Specifically, on Adroit performance improves by +12.7 (59.3 + 72.0). MetaWorld by +7.2 (88.8- + 96.0), DMC by +14.0 (66.9 + 80.9), TriFinger by +8.9 (71.7 + 80.6). Interestingly, in DMC and TriFinger, the in-domain MAE baseline (row 3) performs surprisingly well, highlighting the importance of in-domain data for representation learning. Finally, in large-scale IL or RL domains (ObjectNav, ImageNav, and Mobile Pick), we find MAE adaptation results in small reductions in performance from VC-1 (row vs. 6). In these domains, where sub- stantial amounts of data is available for task-specific training (large-scale IL or RL), we find that E2E fine-tuning is the superior approach for adaptation. In aggregate, these results suggests that MAE adap- tation can be explored as a powerful alternative in few-shot domains or where E2E fine-tuning fails.
Overall, we find that adapting the VC-1 model leads to performance improvement in all the bench- mark domains. Furthermore, on MetaWorid, DMControl, and TriFinger, VC-1 with MAE adaptation (row 6) comparable with the best known results (SoTA) and the best results from previous sections (rows I and 2). Similarly, on ImageNav and Mobile Pick, VC-1 with E2E fine-tuning (row 5) matches or exceeds the best results. Together, these results demonstrate that adaptation of PVRs can be a powerful paradigm for EAI, especially when compared to training representations from scratch.
7 Proof-of-Concept Hardware Experiments
In addition to simulation experiments with CONCEXBENCH, we also explore proof-of-concept hardware experiments utilizing VC-1 as a backbone PVR for training policies with IL. Our hardware evaluation spans two platforms: TriFinger (1 task) and a Franka-Emika Panda arm (4 tasks). Setup details are provided in Appendix A. and 1. We follow similar experiment protocol to the simulated counterparts by studying few-shot imitation learning, where the demonstrations are collected directly in the real-world via tele-operation hand-designed controllers.

---

We study the cases when using VC-1 as a frozen PVR, VC-1 with MAE adaptation and VC-1 with E2E adaptation (as specified in Section 6), and the MVP model as baseline (best PVR from Section 4). The results are summarized in the hardware section of Table 5. Overall, we observe similar trends to our findings in Section
We observe that in frozen mode (row 4), VC-1 substantially outperforms both MVP (row 2) and in-domain MAE (row 3) in both robot setups. We also find that adaptation via E2E fine-tuning improves real-world TriFinger performance and that MAE adaptation leads to large improvements TriFinger performance, this is likely due to the mere 300 real robot images available for adaptation. Together, these results suggest that learning task-specific features (via fine-tuning) was more important than closing any domain gap (via MAE adaptation) for TriFinger. Overall, these results demonstrate that VC-1 can effectively function prior PVRs that have shown success on hardware, such as MVP. Furthermore, it reinforces finding from Section 6 that adapting VC-1 with SSL objectives (MAE) can improve the performance.
Franka tasks. We note that while MAE adaptation does not help
PVR for multiple hardware platforms, and outperform
8 Discussion
This work introduced CORTEXBENCH, which comprises 17 different EAI task spanning locomotion, indoor navigation, and dexterous and mobile manipulation: and conducted the most comprehensive study to-date of PVRs (or visual foundation models) for EAL We find that (1) despite significant progress in a number of narrow domains, we do not yet have universal visual backbone for all EAT tasks of interest, (2) naively scaling model size and pre-training data diversity does not improve performance universally across all tasks, but does on average, (3) adapting our largest pre-trained model (VC-1) results in performance that is competitive with or ouiperforms the best known results all benchmarks in CORTEXBENCE of-concept generalization to hardware. Our study is an attempt to unify various EAI areas using the perception module as comeestone.. The study and development of visual representations for sensorimotor control today appears splintered across different cub-communities studying egocentric computer vision, locomotion, navigation, dexterous and mobile manipulation. However, contend that this cannot be the final solution. Biological organisms have one visual cortex, not one per 'task'. Analogously, it must be possible for embodied Al agent to have universal artificial visual cortex supporting a diverse range of sensorimotor skills, environments, and embodiments. We speculate that learning visual representation using temporal signals, 3D spatial priors, objectness will help make progress towards this goal. Our final contention is that in order for the research community develop such model, we need to create benchmarks that test broad generalization capabilities; we hope CORTEXBENCI will help the community make progress towards that.
and (4) VC-1 and adaptation show proof-
Acknowledgements
The Georgia Tech effort was supported in part by ONR YIP and ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implicd, of the U.S. Govermment, or any sponsor.
References
[1] Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics, 20(3): Sep 1975. ISSN 1432-0770. doi: 10.1007/BF003422333. URL https://doi. org/100 10. 1007 (007/BF00342633.
[2] Kumihiko Fukushima. Neocognitron: self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 193-)22 Apr 1980. ISSN 1432-0770. doi: 10. 1007/B/000344251. URL. httph::/ http::::///oi..o/////// wtt.:/....rr..............0..... BF00344251. 1
[3] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In CVPR, 2022. 1,3
[4] Karmesh Yaday, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for embodied navigation. In arXiv preprint arXiv:2204.13220 2022. 5.5
10

---

[5] Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. Ovrl-v2: simple state-of-art baseline for imagenav and objectnav. arXiv preprint arXiv:2303.07798, 2023. 3,
[6] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The Unsurprising Effectiveness of Pre- Trained Vision Models for Control. ICML, 2022. 1,3, 3,,4, 5,
16
[7] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A Universal Visual Representation for Robot Manipulation. CoRL,, 2022. 3, 4,5, 16
[8] Iliaa Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real world robot learning with masked visual pre-training. In 6th Annual Conference Robot Learning, 2022. 2, 3, 5, 6,
[9] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000.33 2022. 3,
[10] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. DeepMind Control Suite. arXiv: 1.Ki::100100000 100600, 2018. 2,4
[11] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1094-1100 PMLR, 2020. 2,
[12] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations, Proceedings of Robotics: Science and Systems (R:SS), 2018. 2, 4
[13] Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, Julian Viereck, Maximilien Naveau, Ludovic Righetti, Bernhard Schölkopf, and Stefan Bauer. Trifin- ger: An open-source robot for learning dexterity. CoRR, abs/2008.03594 2020. URL
https://arxiv: org/ abs /tt.///00.... 03596. 2, 4
[14] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In International Conference on Computer Vision (ICCV), 2019. 4, 16
[15] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurlPS), 2021. 2, 16, 17
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICMI.), 2021. 5
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.1 19299 2020. 2
[18) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:211 1.02777.. 2021. 3, 6, 8, 18
11

---

[19] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020. 3
[20] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprini urXiv:2202.033555 2022. 3,
[21] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A Simple Frame- work for Contrastive Learning of Visual Representations. arXiv:2002.05709, 2020.
[22] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 3
[23] Hangbo Bao. Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ArXiv, abs/2106.08254, 2021.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICIR), 2021. 3
[25] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. arXiv preprint arXiv:2212.07555,, 2022. 3
[26] Yuchong Yao, Nandakishor Desai, and Marimuthu Palaniswami. Masked contrastive represen- tation learning. arXiv preprint arXiv:2211.008 1.06012, 2022. 3
[27] Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302. 12766, 2023. 3
[28] Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image image representations and rewards for robotic control. arXiv preprint arXiv:2306. 00958,, 2023. 3
[29] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023. 3
[30] Jiange Yang, Wenhui Tan, Chuhao Jin, Bei Liu, Jianlong Fu, Ruihua Song, and Limin Wang. Pave the way to grasp anything: Transferring foundation models for universal pick-place robots. arXiv preprin! arXi::2306.057//1 16, 2023. 3
[31] H Liu, L Lee, K Lee, and P Abbeel. cntructiio-followinng agents with multimodal transformer. arXiv preprint 2rXiv:2210.13431, 202.. 3
[32] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12104-12113, 2022. 3
[33] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10063-10074, 2021.
[34] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Man- nat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.0/99 2021. 3
[35] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform duito--distrbuution. arXiv preprint 22X2 2022. 3,8
12

---

[36] Erik Wijmans, Irfan Essa, and Dhruv Batra. VER: Scaling on on-policy rl leads to the emergence of navigation in embodied rearrangement. In Conference on Neural Information Processing Systems (NeurIPS), 2022. 16
[37] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint AXiv:2204.029377. 2022.
[38] Yoonho Lee, Annie Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210. 1466, 2022.
[39] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghumathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint W.N::2222.00666 2022. 3
[40] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran. Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline. arXiv preprint arXiv:2212.05749 2022. 3,9
[41] Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint 12.00:21 2021.
[42] Real Robot Challenge 2020. Real robot challenge 2020. https://real-rooo-chhae com/2020, 2020.
4
[43] Andrea Dittadi, Frederik Träuble, Manuel Wüthrich, Felix Widmaier, Peter Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. The role of pretrained representations for the ood generalization of reinforcement learning agents, 2021.
URL https: /ttp/v/rxvvvv/vv///////eee/aaava org/ abs/aabs//..//////////////// 05686. 4, 21
[44] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Kumar Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. ICRA, 2017. 4
[45] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating objects. arXiv preprint arXiv:2006. 1311700006.1,1,, 2020. 4, 17
[46] Jiayuan Gu, Devendra Singh Chaplot, Hao Su, and Jitendra Malik. Multi-skill mobile manipula- tion for object rearrangement. arXiv preprint arXiv:2209.02778 2022. 5, 17
[47] Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imitation and rl finetuning for objectnav. arXiv preprint arXiv:2301.07330, 2023. 5
[48] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprint arXiv:2206.0665 06994, 2022. 5
[49] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18995-19012. 2022.
[50] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVE conference on computer vision and pultern recognition, pages 9869-9878, 2020. 6
[51] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevie. The "something something" video database for learning and evaluating visual common sense, 2017. 6
13

---

[52] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler. Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018. 6
[53] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint A.K::18000098111 2018. 6
[54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages
leee, 2009. 6
[55]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creet, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumhouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff' Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng. Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2 100.2008.72588 2021. 8
[56] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 1903, 2022. 8
[57] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran Modem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint, 2022.
[58] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 8
[59] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), 2021. 8
[60] Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari. Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D' Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, An- thony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martin-Martin, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenen- baum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, and Jiajun Wu. Retro- spectives on the embodied ai workshop. arXiv preprint arXiv:2210.06845 2022. 16
[61] Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, et al. Habitat- matterport 3d semantics dataset. arXiv preprint arXiv:2210.0566. 2022. 16, 17
14

---

[62] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learn- ing embodied object-search from human demonstrations scale. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. 16
[63] Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICLR), 2020. 16
[64] Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-augmented reinforcement learning for image- goal navigation. arXiv preprint arXiv:2101.0518/ 2021. 16
[65] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9339-9341 2019. 16, 17
[66] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-workd world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9068-9079, 2018. 16
[67] Ziad AI-Halah, Santhosh K Ramakrishnan, and Kristen Grauman. Zero experience re- quired: Plug & play modular transfer learning for semantic visual navigation. arXiv preprint arXiv:2202.022 02440, 2022. 17
[68] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg. John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. matitattmatterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 17
[69] Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang. Alexander Clegg, Manolis Savva, et al. Habitat challenge 2022, 2022. 17
[70] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018. 17
[71] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web. Learning embodied object-search strategies from human demonstrations scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173-5183, 2022. 17
[72] Zhao Mandi, Homanga Bharadhwaj. Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. ArXiv, abs/2212.05711 2022.
[73] Robohive - unified framework for robot learning. https: cot.sttss.googgg.....ooiiee////e com/view/ robohive, 2020. URL https://sites: google. htt.e////c/iiie///oooieeeeee 22
15

---

A Appendix
A.1 Limitations
This study presents thorough examination of visual foundation models but has several limitations. Firstl, in proposing a benchmark, we sought to find a balance between task diversity and the computational resources required for evaluation. However, new and challenging benchmarks in embodied AI, such as those presented in [60], continue to emerge and may merit inclusion in future studies track progress in this field. Additionally, while we have focused on masked outo-encoders (MAE) as the pre- pre-training objective and ViTs as the architecture in our study, there may be other SSL algorithms that exhibit different scaling behaviors or superior performance on the proposed tasks in our benchmark. Lastly, the adaptation procedures studied in this work necessitate separate training on in-domain datasets, as well as careful tuning of hyperparameters such as the number of training epochs and sampling ratio of the dataset. This results in a significant effort to produce a separate adapted PVR model for each benchmark evaluated on our benchmark, and the overall effort increases proportionately with the number of benchmarks included in the study.
In conclusion, it is important to note that although we utilize real-world images and videos for pre- trainingn our visual representation models (PVRs), the evaluation benchmarks used in this study serve as proxies for actual robotic tasks, and thus, the performance of the PVR models on real robots may differ from the rankings established in this study. Further research is necessary to fully evaluate the effectiveness of these models in real-world scenarios.
A.2 Overview of Downstream Policy Learning in CORTEXBENCH
Given a frozen PVR, an agent needs learn a policy for each task. The EAI community has developed a range of policy learning algorithms from few-shot imitation learning (IL) to large-scale reinforcement learning (RL). For each task in CORTEXBENCH, we conform to the community standard for achieving state-of-art performance in that domain.
"MuJoCo Tasks" On the tasks from the Adroit, MetaWorld, and DMC suites train policies using behavior cloning on a small number of expert demonstrations (100 for Adroit and DMC and 25 for Meta World, which follows [6, 7]. Specifically, we train policies for 100 epochs and report the average rollout performance on the test set for the best intermediate policy during training. For all tasks, we use frame-stacking and 3-layer MIP policy network. When using vision transformers (ViT) based PVRs, we use the [CLS] token as input to the policy, and with ResNets we use features from the final convolutional layer after global average pooling. These design choices follow prior work such as [7, 8].
"TriFinger Tasks" For TriFinger, train policies using behavior cloning 100 demonstrations per task. Specifically, we train a policy network composed of a 3-layer MLP for 100 epochs for Reach-Cube and 1,000 epochs for Push-Cube. We report the average score for the best checkpoint over the course of training. The input to the policy is the [CLS] token for ViT-based PVRs and average pooled features from the last convolutional layer for ResNet-based models.
"Habitat Tasks" We train ObjectNav policies with behavior cloning on 77k human demonstra- tions [61] collected by [62], totaling 360M environment steps. For ImageNav and Mobile-Pick we use RL for 500M environment steps with DD-PPO [63] and VER [36]. We use patch representations for ViT-based PVRs and grid-features from last convolutional layer for ResNet models, passed through a compression layer [14] for a lower dimensional representation for use by the policy layers, which is 2-layer LSTM for navigation and 2-layer GRU for manipulation.
A.3 Additional Details of Tasks and Downstream Learning in CORTEXBENCH
In this section, we discuss further details for a subset of downstream tasks in CONTEXBENCIL.
ImageNav. Our study conducts ImageNav experiments using the standard dataset presented in [64]. This benchmark utilizes the Habitat simulator [15, 65] and is situated within the Gibson [66] envi- ronments, which comprise 72 training scenes and 14 validation scenes. The validation set includes 300 episodes for each scene, for a total of 4,200 episodes. In this benchmark, agents are modeled as cylinders with height of 1.5m, radius of 1.,, and sensors located 1.25m above the center of the
16

---

base The RGB camera has resolution of 128 x 128 and 90º field-of-view. Agent is able to take up 1000 steps within the environment and are deemed successful they reach a location within Im of the goal position and call STOPACTION.
To train the agents within the Gibson environments, utilize 500M timesteps (25k updates) with 320 environments running in parallel. Each environment collects up to 64 frames of experience, which is followed by 2 PPO epochs utilizing 2 mini-batches. Unless otherwise specified, we use a learning rate of 2.5 X 10 for training the agents and update the parameters using the AdamW optimizer with a weight decay of 10-6. We train agents with the reward functions presented in [67] utilizing the following settings: success weighting C₂ 5.0, angle success weighting Cu = 5.0, goal radius rg 1.0, angle threshold 8,- = 25,, and slack penalty y 0.01. We evaluate performance every 25M steps of training and report metrics based on the highest success rate (SR) achieved on the validation set.
ObjectNav. We present an evaluation of object navigation (ObjectNav) using the HM3D-SEM dataset [61]. The dataset is comprised of 80 training. 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments. Our results are reported on the v0.1 HM3D-SEN VAL split, which was used in the 2022 Habitat Challenge [69] ObjectNav benchmark. The agent in this evaluation is modeled after the LocoBot [70] with a height of 0.88m, radius of 0.18m, and sensors placed at the top of the agent's head. The RGB camera has 640 480 resolution and 79 horizontal field of view. The task for the agent is to locate objects from one of categories: 'chair', 'bed', 'plant', 'toiler', 'tvmonitor', and 'sofa' within 500 steps. Successful episodes are determined by the agent stopping within 0. m. of a viewpoint that is (a) within Im of any instance of the target object and (b) from which the object is visible, as outlined in the evaluation protocol of [45].
We utilize a dataset of human demonstrations for training our imitation learning agent in the task of ObjectNav. The dataset was collected using Habitat-Web [61, 71] and Amazon Mechanical Turk, and consists of 77k demonstrations for 80 scenes from the LMM3D-SEA dataset [69]. Each scene contains approximately 158 episodes, each with a unique goal object category and a randomly set start location, resulting in approximately 950 demonstrations per scene. The dataset includes a total of 12.1 million steps of experience, with an average of ~55 steps per episode. By leveraging this human demonstration data, our imitation learning agent is able to learn more effective policy for navigating to objects in complex environments.
We trained object navigation (ObjectNav) agent in the HM3D environment for an approximate total of 400 million steps, utilizing 25,000 updates and 512 parallel environments. Similar our previous image-based navigation (ImageNav) experiments, we employed a weight decay of 10-6 and utilized different learning rates for the visual encoder and other elements of the model. Specifically, we used a learning rate of 10-4 for the visual encoder and 10 for all other elements, with the AdamW optimizer. To ensure the quality of our trained models, we evaluated checkpoints after every 10M steps and only reported metrics for the checkpoints with the highest validation success rate.
Mobile Pick. We investigate the Habitat 2.0 Rearrangement task proposed by [15]. This task involves a mobile manipulation scenario in which a Fetch robot navigates an ReplicaCAD apartment to pick up a target object from cluttered receptacle using mobile base [46]. The robot starts from a non- coivirivial position and must utilize a variety of sensors, including an egocentric RGB camera, proprioceptive joint sensing, and an object grasping indicator. The action space for the robot includes continuous control over the robot's 7-DOF arm, base movement, and suction gripper. We relax the dense goal specification, where the relative position between the end-effector and the target object must be updated at each step, to a sparse goal specification, where this information is only provided at the start of the episode. This relaxation places greater emphasis on visual input and makes the task significantly more challenging.
TriFinger Tasks. The TriFinger tasks are implemented in Pybullet. For Reach-Cube, the state for the BC policy is [ar[ ( 2t], where z/" is the current fingertip position and o, is the latent visual state vector, obtained by passing the current image observation through the PVR. The success metric captures how close the fingertip is to the optimal distance from the center of the cube, accounting for
the half=width of the cube. For Push-Cube, the state for the BC policy is [at 21, Arg). where Arg is the goal position for the cube, specified as displacement from its initial position. Hére the success
17

---

is the distance of the center of the cube to the target goal position. We train policy network with hidden layers of size 2000 and learning rate 10 for up 100 epochs for the reach task and 1000 epochs for the Push-Cube task.
A.4 Additional Analysis of Existing Pre-Trained Visual Representations (PVRs)
The rank distribution for existing PVRs (shown in Figure 5) demonstrates that there high variability in the performance of the models (from prior work) across the benchmarks in CORTEXBENCH.
MVP (MIT)L)
R3M IRN 50)
CLIP (VIT-8)
(RN-50)
Random finetune (VIT-8)
(VIT-L)
Random (VIT-B)
Rank
Figure 5: Rank distribution of existing pre-trained visual representations. For every model, we compute the ranks it achieved on each of the 7 benchmarks. We visualize them vertical lines, where each rank number receives tick if that model achieved such rank For instance, MVP (ViT-L) achieves ranks 1.1.1.2.3.3,4 across the benchmarks. Significant variability exists performance PVRs across benchmarks.
A.5
Scaling Hypothesis Datasets
The datasets composed for scaling hypothesis experiments are detailed in Table 6.
A.5.1
OpenHouse22 Dataset
The OpenHouse24 (OH24) dataset is a collection of video walk-throught of furnished residential real estate properties. Over 1600 homes are represented in the dataset, totaling 139 hours of video footage. Each home is traversed in a continuous shot with a stable HD RGB camera by an operator that efficiently visits each room. The dataset represents a diverse set of properties, including (but not limited to) small and large suburban homes, high-rise apartments, ranch homes, and condos. The ensuing walk-throughs range from under a minute to 14 minutes in length, with the average taking 5 minutes and 12 seconds. The dataset will be open-sourced by a separate research project.
A.6 Scaling Hypothesis Pretraining Details
To train the MAE models, we use the official codebase released by the authors on GitHub [18] and use the default hyperparamettrs provided by the repo train the ViT-B and ViT-L models. We found the default values worked well on the CORTEXBENCH, However, we do vary the number of epochs we use to train the different models in Section given the different dataset sizes. We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset. We provide details about the dataset sizes and the epochs calculated for the different runs in Table 7.
A.7 Additional Analysis of Scaling Hypothesis Results
Figure (left) provides additional evidence of the significance of pre-training dataset diversity. We observe that while the VIT-I. models trained in Ego4D+M and Ego4D+N datasets, achieve the best result in one of the benchmarks, they perform the worst and second-wors, in other benchmarks. However, by adding diversity, the Ego4D+MN and Ego4D+MNI models have decreased variance
18

---

Table Datasets used for scaling hypothesis experiments, containing up 5.6M frames.
Name
Contains
Total Frames
Frames used
Ego4D
Ego4D
418,578,043
2,790,520
Ego4D
418,578,043
2,790,520
100DOH
99,899
99,899
Ego4D+M (Manipulation)
SS-v2
25,209,271
315,115
Epic Kitchens
19,965,439
332,757
Total
3,538,291
Ego4D
418,578,043
2,790,520
Ego4D+N (Navigation)
OpenHouse24
27,806,971
499,442
RealEstate 10K
10,000,000
303,087
Total
3,289,962
Ego4D+M
3,538,291
3,538,291
Ego4D+MN (Manipulation, Navigation)
OpenHouse24
27,806,971
499,442
RealEstate
10,000,000
303,087
Total
4,340,820
Ego4D+MN ImageNet
4,340,820
4,340,820
Ego4D+MNI (Manipulation, Navigation, ImageNet)
1,281,167
1,281,167
Total
5,621,987
Table 7: Experiment Details of Training PVRs.
Dataset Name
Epochs
Frames used
Ego4D+N (VTT-B)
289
3,538,291
Ego4D+N (VIT-L)
289
3,538,291
Ego4D+M (VIT-B)
414
3,289,962
Ego4D+M (VIT-L))
414
3,289,962
Ego4D+MN (VTT-B)
236
4,340,820
Ego4D+MN (VIT-L)
236
4,340,820
Ego4D+MNI (VIT-B)
182
5,621,987
VC-1 (Ego4D+MNT (VIT-L))
182
5,621,987
in their rank distributions. Notably, the Ego4D+MNI model consistently performs well across all benchmarks, and ranks among the top models.
A.8 Additional Analysis of All Models Evaluated CORTEXBENCH
Figure 6 (right) provides complete picture of the rank distribution for all of the models evaluated in this study. Similarly, Table provides results for models evaluated in this study collected.
A.9 Additional Analysis of Scaling Model Size
Figure illustrates that scaling model size has positive effect on every benchmark and on fifteen out of the seventeen tasks.
19

---

NAV Fart MIT
MAZ Eg MIT
Spoit (08.00
WAd Ego TUTTR
Has Eopitee mur
Spent MEVITL
Toe Spoez-i (VED) vr mw 501
MAC SgeiD-Me ToTT
MAl Eg (ITRR
IMe 6y viil)
OPP (/70)
Random
NAE tyo40 (0
PLandor (IT8)
5.0
15.0
175
Pankppr Tock
Bunk per Toxt
Figure 6: (left) Rank distribution per model - scaling hypothesis. (right) Rank distribution per model existing PVRs and scaling hypothesis models
Table 8: The success rate for each task and each model we evaluate during the study before being aggregated by benchmark.
YASK
MAR V11.81
70.3
18.5
15.3
$6.
34.0
12.6
71.
12.1
14.3
30.3
MAR
110.
15411
$1.6
06.3
63.4
50.1
MAR CoothoM
111
66.3
41.3
760
33.1
25.5
363
1880
MAR 12--D-A (VITLE
307
181
1980
6622
62.6
76-7
30.4
15.1
530
364
SEAR CU-4D-ANNE (VITA)
880
560
910
12.
1880
720
1030
113
633
60.1
60.3
100
13.3
85.0
16.1
133
137
SAA Eg-ID-N (VTEL)
06.3
75.
893
333
1880
713
65.3
715
653
133
21.8
760
81.1
13.0
13.4
967
1100
SEVP-VVELLI
1837
15.5
16.7
83.
160
545
181
83.3
993
RUM
1000
513
363
09
134
28
421
413
22.
362
163
00
955
153
136
755
719
11
361
911
760
553
106
76.
813
817
866
611
Adrri
MatslePak
WerdWorld
Imagelav
Diveenrrol
WIB-B
Dijectt
Tritrger
WTB
VT-L
(a) scaling model size per benchmark
(b) scaling model size task
Figure 7: Scaling model size has a positive effect on (a) every henchmark and on (b) fifteen out of the seventeen tasks.
A.10 Attention Visualizations of VC-1
To visualize the attention apply pooling operation the attention matrices of the VIT encoder's s final layer during inference for downstream tasks. The resulting values are then overlaid onto the image.
start by noticing the effect MAE pre-training: frozen VC-1 attention maps appear to focus on the contours and general features of the image. We hypothesize that this results from the MAE
20

---

deconstruction-baseee training objective, as contours provide essential information for reconstructing images.
Additionally, we study the attention maps after end-to-end fine-tuning of VC-1 on the downstream tasks. The attention appears to focus regions of the image that are important for the task (e.g.. the objects being manipulated), Thus, through adaptation (via E2E fine-tuning), the model learns to drop attention on areas irrelevant to the specific task.
Random ViT-L
VC-1
Adapted VC-1
Figure 8: Attention Visualization: (left) Random ViT-L; (middle) VC-1 frozen; (right) VC-1 E2E finetuned. We overlay the mean attention matrix in the last layer of the ViT encoder in one of our tasks -MobilePick-. We notice the effect of MAE pre pre-trainnn training VC-1: The attention focuses in general features of the image; and of task-adaptation: the attention concentrates in task-specific regions of the image
A.11 TriFinger Hardware Experiment Setup
We carried out experiments on the real TriFinger robot (shown in Figure 9) for the Push-Cube task, after training model using behavior cloning on 30 real-world demonstrations. The success for Push-Cube defined as the distance which the cube is able to travel relative its starting distance to the goal, similar to [43]. The episode length for this task is 20 steps. For each model, we chose seed and ran it 12 different start and goal configurations, mostly centered in the arena.
The action space for this environment consists of end-effector displacements for the three fingers. The motors are controlled at frequency of 1kHz and the action sent to the robot is 9 dimensional vector specifying the joint torques. We use an Intel RealSense camera, positioned next to the table overlooking the scene, similar to how our simulation image observations are captured. To detect the cube, we make sure that the green side is always facing upwards and detect the center of that face, and use that to track the cube's position in the bowl.
21

---

Figure TriFinger Push Cube task
A.12 Franka Hardware Experiment Setup
We conduct behavior cloning experiments on manipulation platform comprising a Franka arm and Robotiq gripper fitted with Festo Adaptive Gripper Finger. Figure 10 shows the task setups:
The reaching task requires controlling the end-effector to reach an randomly selected point between (-0.3, 0.4, 0.2) and (0.3, 0.4, 0.2) in the robot frame within 5cm error given goal image prompt.
The bottle pickup task requires reaching and picking up a bottle with the gripper. The bottle is randomly positioned on a 10 cm line, same orientation).
The close-drawer and toaster plunge tasks are similar to the Cacti setup [72].
For all the tasks, the observations are single RGB camera image (420x224 resolution) and pro- prioceptive signals (7 dfgree-of-freeddn joint angles and 1 degree-of-freedde freeedooooooo gripper width). A 3-hidden layer, each with 256 hidden nodes, multi-layer perceptron policy takes PVR encoded images concatenated with scaled proprioceptive signals input and predicts the desired absolute joint angles and gripper width as actions. A low-level IkHz real-time joint-space PD controller follows the policy-gence generated desired trajectory. We specifically keep the gains low (half of the factory setting) so that the robot will not break anything even if it were commanded to hit the table. This results in poor trajectory tracking performance, yet both human teleoperators and learned policy can control the robot to complete the tasks. We use Robohive [73] as the middleware to interface with all sensors and robots.
Demonstrations are collected from human teleoperation using a Quest controller. For PVR (frozen encoders), we use Adam optimizer with learning rate 10 to train the policies. For fine-tuning we use the same learning rate for policies but lower learning rate (10-5) for the visual encoders. Table 9 shows the behavior cloning success rates.
Reaching Ran-
(b) Bottle Pickup
(c) Toaster Plunge task on kitchen table-top setup
(d) Open Drawer task on kitchen table-top setup
dom Point
Figure 10: Franka manipulation tasks
22

---

Table Franka manipulation task success
Reaching
Bottle Pickup
Open Drawer
Plunge Toaster
Success Count
Success %
Demos
250
250
Evals
VC-1 frozen
80
100
60
40
28
70.0
VC-1 E2E fine-tuning
50
80
50
27
67.5
VC-1 MAE adaption
34
MAE baseline
30
50
50
14
35.0
MVP
70
100
30
20
22
55.0
23


Reviews:
[{"review_id": "YRnGLCL7nN", "reviewer": "Reviewer_nrzP", "summary": "This paper evaluates different pre-trained visual representations w.r.t. their capability to serve as foundation models for Embodied AI. To do so, they compile a dataset (CortexBench), with 17 simulated EAI tasks. The intuition here is that a single pre-trained visual representation which would perform well across all tasks could be considered as an Artificial Visual Cortex. The paper collects several egocentric video datasets starting with Ego4D,  and sequentially adding datasets to create different variations trained with different training datasets. Their proposed artificial visual cortex (VC-1) is trained on all datasets (ViT-L trained with Ego4D + MNI). They also propose adaptations to VC-1 to improve its capability to perform well across different datasets. Finally a proof of concept hardware experiment using VC-1 to learn policies with imitation learning for few shot manipulation tasks.", "questions": "Please refer to weaknesses above for questions.", "limitations": "N/A", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Exhaustive training with lots of variations with architectures and increasingly growing dataset.\n\n2. Evaluation conducted on a large scale dataset with a diverse set of EAI tasks.\n\n3. Writing: The paper is easy to read and follow. The figures do a good job of illustrating the dataset and the experiments. Also, the. ", "weaknesses": "1. No data contribution: CortexBench is a compilation of existing datasets, and it's not clear what new it brings to the table beyond what's already out here. From the introduction, it initially seemed as thought it's newly proposed benchmark in this paper. However, it seems to be a set of already existing datasets. Why is it being claimed/renamed as a new dataset?\n\n2. No new findings: \n\n- The first negative result is that specialized pre-trained models are better - that is, in-distribution models perform better than out-of-distribution ones (lines 42-45). This is well known in the machine learning community, and thus, is not a new finding. \n- Lines 70-85 can be summarized as saying: More data is better, but OOD data is a problem. This too is a well known property of deep learning vision models.\n- Naively scaling size and diversity: The paper does not study data diversity explicitly, but rather only \n\n3. Data diversity and Dataset size are not controlled: When the new datasets are added to the foundation models, are the number of images held constant? I didn't see this detail in the paper. Thus, it is unclear what helped - dataset size, or diversity? To control data diversity independently, a control would be to reduce number of images per dataset as # datasets is increased to isolate the effect of size and diversity. \n\n4. The adaptive VC-1 does not really work well: of the datasets with prior results in Table 5, VC-1 does better for only 2 task classes. Without adaptation too, it could be trained to do well on one of these task classes alone. Thus, the adaptation took it to performing well from 1 task to 2 task classes. Furthermore, there are no error bars in these results which makes it hard to evaluate the reliability of these findings given the stochasticity of the models.\n\n5. Claim too grandiose: Calling a pre-trained visual recognition model an artificial cortex is a very far fetched claim on several levels of abstraction. Firstly, the cortex does much more than EAI: so testing it only on a small set of functionalities of the cortex means there's a mismatch at a behavioral level between the cortex and the artificial cortex. Secondly, by not comparing with any neural data, there is a mismatch at the mechanistic level as well. A more objective representation of the work done here is that this paper investigates --- Can a single pre-trained representation perform well across a diverse set of EAI tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking time to review our paper. We believe there may be some major misunderstandings about our submission, and we would like to take this opportunity to clarify. \n\n> No data contribution. From the introduction, it initially seemed as thought it's newly proposed benchmark in this paper. Why is it being claimed/renamed as a new dataset?\n\nWe do not claim a data contribution. As stated in the intro, \"we *curate* CortexBench\" and we cite the sources in L34-35; section 3 describes details. A name is used to succinctly refer to the curation in the exposition and to the associated software. \n\nOur contributions are stated upfront in abstract L16-20: \"Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.\"\n\nFurthermore, we agree with reviewer aWtL: \"Both the evaluation benchmark and released checkpoints (as promised by the authors) will be extremely beneficial for embodied AI research.\"\n\n> specialized pre-trained models are better - that is, in-distribution models perform better than out-of-distribution ones (lines 42-45). \n\nThis interpretation is not inline with our findings. None of the PVRs considered in this work (except in Section 6) were trained on images from the evaluation tasks or simulators, and thus are all out-of-domain by definition. What we're seeing is that previous work had specialized on evaluating PVRs on a small set of tasks and shown strong results, but those results do not carry over to a wide distribution of tasks, which is a new finding not known in prior work. There is a broader question here of “what exactly is a domain”, but that is beyond the scope of this work (we use standard definitions). \n\n> Lines 70-85 can be summarized as saying: More data is better, but OOD data is a problem.\n\nNot quite. As pointed out above, all training data we use is OOD, so what this finding is saying is that it is difficult to know a priori which data should be included or excluded from the pre-training set; and without our experiments, we would not know about the existence of this problem. As we say in L84-85: \"[our] broad evaluation refutes a naive extrapolation of the positive scaling trends observed in prior work on robot manipulation [8].\" \n\n> Data diversity and Dataset size are not controlled: When the new datasets are added to the foundation models, are the number of images held constant? I didn't see this detail in the paper.\n\nYes, the training dataset size is held constant to support claims about data diversity. The number of images per dataset is provided in Table 3 on page 5 and a more detailed breakdown is provided in Table 6 in the Appendix.\n\nSpecifically, comparisons between the **Ego4D+M** and **Ego4D+N** training datasets – both containing 3.5M frames – support the dataset diversity claims. As stated on L288 “[Ego4D+N] is similar in size to Ego4D+M (3.5M frames) but is more diverse because it contains a larger proportion of navigation data than the manipulation-centric datasets Ego4D and Ego4D+M.”\n\n> of the datasets with prior results in Table 5, VC-1 [adapted] does better for only 2 task classes.\n\nIt is not clear why nearly half the experiments in Table 5 are being ignored in making this claim. To recap, Table 5 presents results on 7 simulation benchmarks (\"task classes\") and 2 hardware tasks. Prior published results exist for 5 out of 9 benchmarks because as we describe in the abstract, intro, and experiments, prior work has focused on narrow task domains; our contribution is to systematically benchmark existing PVRs on this broad set of tasks (which is what Table 2 and row 2 in Table 5 provide). \n\nWhen all tasks are considered, VC-1 adapted (via MAE adaptation in low-shot IL or via fine-tuning in large-scale RL) outperforms VC-1 (without adaptation) on all 7 benchmarks. Additionally, as stated on L351-354, “Furthermore, on MetaWorld, DMControl, and TriFinger, VC-1 with MAE adaptation (row 6 [in Table 5]) is comparable with the best known results (SoTA) and the best results from previous sections (rows 1 and 2 [in Table 5]). Similarly, on ImageNav and Mobile Pick, VC-1 with E2E fine-tuning (row 5 [in Table 5]) matches or exceeds the best results.”\n\n> No error bars in Table 5.\n\nThank you for this suggestion. We have added error bars to Table 5. We find that the error bars for the adapted versions of VC-1 (Table 5 rows 5 and 6) are very similar to the error bars we report for VC-1 in Table 4 row 11. Accordingly, all of the conclusions drawn in the paper remain unchanged."}, {"review_id": "a88bBEMb9E", "reviewer": "Reviewer_aWtL", "summary": "This paper focuses on the analysis of large-scale pre-trained vision models for interactive tasks. The authors curate CortexBench which includes tasks covering dexterous manipulation, navigation, and also scene-level interaction as the testing benchmark. Further experiments were made on testing existing pre-trained vision models on this task. The authors also control the dataset type, the scale used in pertaining and proposed their VC-1 model pre-trained in an MAE fashion which shows superior performance compared to baselines. Additional analysis were made showing the effectiveness of egocentric images for interaction and also gap between existing pre-trained vision models and the desired \"visual cortex\" representation.", "questions": "Please see the weakness section for questions.", "limitations": "The authors have properly addressed the limitations of this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper conducts solid experiments on testing existing pre-trained visual representations (PVR) on their own curated interaction benchmark CortexBench. Both the evaluation benchmark and released checkpoints (as promised by the authors) will be extremely beneficial for embodied AI research. The overall analysis is technically sound with proper conclusions made, pointing to future directions for PVR studies.", "weaknesses": "One concern of this paper is that except for the conclusion pre-training with egocentric frames is beneficial for interactive EAI tasks, I do not see signature conclusions that could be drawn from the current analysis. After reading this manuscript, I do not feel that the question in the paper title has been properly addressed:\n- First, there is no task analysis for the curated tasks in CortexBench, for a proper visual cortex, to what extent should we expect with \"visual cortex\"? are these tasks equally important? are there any relationships between these tasks?\n\n- Second, since the current VC-1 model still does not universally dominate either task, what is missing on the training/model side? is it missing for example value condition in VIP or some embodied experience?\n\n- Third, as a pre-trained vision representation, what should we expect from VC-1 in addition to its capabilities in CortexBench? Should we expect the model to at least have some sort of objectness as DINO? Can this pre-trained model be used in more general vision settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking time to review our paper and sharing your thoughts. We have addressed your concerns below. Please let us know if any additional clarifications are needed.\n\n> First, there is no task analysis for the curated tasks in CortexBench.\n\n\nBecause CortexBench is curated from existing benchmarks (Sec. 3.1) a detailed analysis of each task can be found in the respective papers. The tasks span a wide spectrum of EAI. They vary in visual complexity, task complexity, task domain, skills required, agent embodiments, and how policies are trained. In that sense we have a good coverage of various EAI problems.\n\n> Are these tasks equally important?\n\nIn this work we do not take a position on which tasks are more or less “important” – that is for the community to decide. However, all of the tasks are widely studied in the community as evidence by the citation counts for each benchmark: Adroit [12] (801 citations), MetaWorld [11] (648 citations), DeepMind Control [10] (420 citations), TriFinger [13] (43 citations), Habitat [14] (931 citations), and Habitat 2.0 [15] (277 citations).\n\n> Are there any relationships between these tasks?\n\nThe overarching relationship is that these tasks have been deemed important and interesting by the EAI community as evidenced by the body of work that has gone into studying each of the tasks separately. In this work, we use the collection to draw new conclusions about existing and new PVRs.\n\nAt a lower level, there are relationships in the way goals are specified and the policy learning algorithms that have led to SoTA performance in prior work as illustrated in Table 1. Finally, there are commonalities in the behaviors required. For example, ObjectNav, ImageNav, and Mobile Pick all require navigation. We will add this to the discussion.\n\n> VC-1 does not universally dominate; what is missing? Is it missing for example value condition in VIP or some embodied experience?\n\nGreat question! Value conditioning or embodied experience may help and would be quite interesting to explore.\n\nAdditionally, the visual embedding in VC-1 is for a single frame, so most of the downstream policies offload the environmental representation construction to a generic RNN or frame stacking. There are several avenues to consider for improving representations including using videos/temporal axis, 3D spatial priors, objectness, etc. We will add these future directions to the discussion section.\n\n> What should we expect from VC-1 in addition to its capabilities in CortexBench? Should we expect the model to at least have some sort of objectness as DINO? Can this pre-trained model be used in more general vision settings?\n\nWe expect that VC-1 will be useful in other EAI tasks because of its effectiveness on the diverse range of tasks in CortexBench. Additionally, we provide some preliminary analysis of the visual features in appendix A.10, where we find, for example, that after adaptation the visual attention focuses more on task-relevant parts of the image such as task-relevant objects. In general, we are excited for the community to use our open-sourced models on additional tasks and settings."}, {"review_id": "N33HgHYgm8", "reviewer": "Reviewer_5TRU", "summary": "•\tThis paper presents the largest and most comprehensive empirical study of pre-trained visual representations for Embodied AI based on proposed CORTEXBENCH.\n\n•\tThis paper focuses on studying the effect of pre-training data size and diversity for PVRs, further proposing VC-1.\n\n•\tThis paper shows that task- or domain-specific adaptation of VC-1 leads to substantial gain by E2E fine-tuning or MAE adaptation.\n", "questions": "•\tI'm curious how you choose specific tasks under each benchmark. Does this follow some basis?\n\n•\tI'm curious why you did not model the temporal information in policy model. For example, adding temporal attention in ViT, even if utilizing a large amount of video data for pre-training.\n", "limitations": "•\tAlthough the evaluation of this work is already very comprehensive, I believe that 17 tasks are not sufficient. I found that even for the same benchmark, different tasks exhibit significant differences.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "•\tThis paper proposes a comprehensive benchmark CORTEXBENCH for pre-trained visual representation evaluation of embodied intelligence, which is very valuable and meaningful.\n\n•\tThis paper studies the effect of pre-training data size and diversity for PVRs, which is novel and insightful.\n", "weaknesses": "•\tThere may be some typos in experiments, such as ‘Trifinger by+7.4 (72.7 → 80.1)’ in the 343 line, and according to the table, it should be Trifinger (71.7 → 80.6). Please carefully check.\n\n•\tAlthough I think this work has been done well enough. However, there are still many questions that can be explored: based on ViT and the same data, which is better for MIM and time-contrastive video-language alignment. Is ResNet better than ViT when few-shot imitation learning?\n\n•\tThis paper only attempts to adapt VC-1, but there are no corresponding experiments for R3M and CLIP.\n\n•\tI strongly recommend including a concise introduction to multimodal-based pre-training visual representation in the related works section, citing relevant sources such as [1-5]. This is crucial as multimodal presentation learning now plays a pivotal role in numerous vision tasks.\n[1] Instruction-Following Agents with Multimodal Transformer. [2] Language-Driven Representation Learning for Robotics. [3] Open-World Object Manipulation using Pre-trained Vision-Language Models. [4] Pave the Way to Grasp Anything: Transferring Foundation Models for Universal Pick-Place Robots. [5] LIV: Language-Image Representations and Rewards for Robotic Control. Among them, [3] and [4] explicitly adapt the middle-level output of the pre-trained model through larger policy models, which also utilize the raw image as input.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking time to review our paper and sharing your thoughts. Please find responses to your questions/suggestions below. Please let us know if any additional clarifications are needed.\n\n> Add a concise introduction to multimodal-based pre-training visual representation in the related works section, citing relevant sources such as [1-5].\n\nThanks! We would be happy to add a discussion of these works to the paper.\n\n> Although I think this work has been done well enough. However, there are still many questions that can be explored: based on ViT and the same data, which is better for MIM and time-contrastive video-language alignment. Is ResNet better than ViT when few-shot imitation learning?\n\nThese are good questions that we are also interested in. However, given the scale of experiments required to answer the research questions posed in sections 4, 5, and 6, these additional questions fell beyond the scope of this work. We speculate that using temporal information for learning visual representations (as done in time-contrastive learning) will be useful, and might be a good direction to pursue in future work.\n\n> Although the evaluation of this work is already very comprehensive, I believe that 17 tasks are not sufficient.\n\nThe tasks included in CortexBench were chosen to cover a wide range of embodied AI (EAI) applications, and are of substantial interest to the EAI community as evidenced by the body of work that uses and reports on these tasks. That said, we agree that additional tasks could be added in future studies. As discussed in L633-L636 in the Appendix, \"in proposing a benchmark, we sought to find a balance between task diversity and the computational resources required for evaluation. However, new and challenging benchmarks in embodied AI, such as those presented in [55], continue to emerge and may merit inclusion in future studies to track progress in this field.”\n\n> I'm curious how you choose specific tasks under each benchmark. Does this follow some basis?\n\nYes, we selected tasks following the precedent and trends seen in recent works that have studied each of the individual benchmarks. For example, the MetaWorld tasks were selected “following the evaluations in [7]” (L157), the DMC tasks “following [6]” (L160), and in Adroit “we study the two hardest tasks” (L149-150).\n\n> I'm curious why you did not model the temporal information in policy model.\n\nMost of the downstream policies do model temporal information either through frame stacking (Adroit, MetaWorld, DMC) or by using a recurrent neural network (ObjectNav, ImageNav, Mobile Pick). We will make this more clear in the paper.\n\n> Typos in experiments\n\nThanks for pointing this out. We will correct these typos."}, {"review_id": "o9tMbsHvRE", "reviewer": "Reviewer_on5Q", "summary": "The paper introduces a study of pre-trained visual encoders for Embodied AI, and produces, among others, a new pre-trained ViT-L encoder (VC-1) using Masked Auto-Encoding on a dataset combining an expanded collection of egocentric video and ImageNet. VC-1, when adapted for agents solving 17 task types covering dexterous manipulation, locomotion, mobile manipulation, and navigation, performs best in average. The curated set of 17 task types is presented as a new benchmark named CortexBench. The results show that no visual encoder is universally dominant, and that just scaling the pre-training dataset size and diversity does not imply uniform improvements across benchmarks, refuting previous extrapolated results. Further experiments on real hardware are also included, showing the improvement in comparison with the best performing existing Masked Visual Pre-training visual encoder.", "questions": "- Line 370 states \"MAE adaptation does not help TriFinger performance, this is likely due to the mere 300 real robot images available for adaptation.\". However, fine-tuning seems to do a good job in this case, contradicting the general consensus derived in Section 6. Is there an intuitive explanation?\n- Why is prompting/conditioning VC-1 not considered as an adaptation method in the study? It might be a viable not-so-costly alternative to fine-tuning.\n- Is the advantage of R3M indicating that known architectures might be far from optimal for the goal of having a single visual encoder able to feed all EAI tasks or is the scale of the pre-training datasets considered in the study the most likely reason?", "limitations": "The paper discusses adequately the limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Thorough study of pre-trained visual encoders for Embodied AI tasks across different pre-training schemes.\n- Provides a reference model with high performance (especially after adaptation) across all considered task types.\n- Thorough experimental section and introduction of the very relevant CortexBench benchmark.\n- The paper reads really well.", "weaknesses": "- In an ideal world, it would have been wonderful to have found a single pre-trained model that could be used without adaptation with high performance across all tasks, but the actual outcome is thoroughly justified in the paper. One open question is whether, with larger and more diverse datasets, a single pre-trained model could eventually be the absolute winner before adaptation, but I agree this is out of scope for this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking time to review our paper and sharing your thoughts. Please find responses to your questions/suggestions below. Please let us know if any additional clarifications are needed.\n\n> Line 370 states \"MAE adaptation does not help TriFinger performance, this is likely due to the mere 300 real robot images available for adaptation.\". However, fine-tuning seems to do a good job in this case, contradicting the general consensus derived in Section 6. Is there an intuitive explanation?\n\nThere are broadly two reasons why a frozen visual encoder may improve via fine-tuning on task data: 1) closing the domain gap, 2) moving from task-independent features to task-specific features. MAE adaptation does the former, BC fine-tuning does the latter. Which among the two is more important could be task-dependent. Our results suggest that the task-specific fine-tuning was more important than closing any domain gap for TriFinger. We will add this discussion to the paper.\n\n> Why is prompting/conditioning VC-1 not considered as an adaptation method in the study? It might be a viable not-so-costly alternative to fine-tuning.\n\nIt is unclear to us how to produce prompts for the tasks in CortexBench, as there is no standard approach that we are aware of. However, we’d be happy to add more comments if the reviewer could provide more details about the prompts they had in mind.\n\n> Is the advantage of R3M indicating that known architectures might be far from optimal for the goal of having a single visual encoder able to feed all EAI tasks or is the scale of the pre-training datasets considered in the study the most likely reason?\n\nWe're not sure we fully understand the question. We believe you're asking what are important ingredients to training a general purpose visual encoder for EAI (architecture, pre-training objective, dataset diversity, dataset size, etc.). The short answer is that we believe all of these matter, and finding the best combination is an open research problem. Our work shows that pre-training dataset diversity definitely matters, and that (thus far) a ViT backbone slightly outperforms other backbones (on average).\n"}]

Author rebuttal:
We thank the reviewers for their thoughtful comments. We want to begin with a few high-level observations.

There appears to be consensus that our work presents “a large scale dataset with a diverse set of EAI tasks” (nrzP) – i.e., CortexBench. Several reviewers believe CortexBench will be “extremely beneficial for embodied AI research” (aWtL), is “valuable and meaningful” (5TRU), and is “very relevant” (on5Q). Similarly, there appears to be agreement that we perform a “thorough study” (on5Q) with “exhaustive training with lots of variations” (nrzP).

However, one reviewer disagrees with the others on the novelty and the degree of surprise in the findings from our study.

On one hand, several reviewers find that we present experiments that are “novel and insightful” (5TRU) and “technically sound with proper conclusions made, pointing to future directions for PVR studies.” (aWtL). On the other hand, one reviewer expresses that the findings are “well known in the machine learning community” (nrZP). The disagreement here lies in the perceived newness of our findings rather than the technical substance of our research.

We contend that these are precisely the kinds of papers that should be presented at a conference like NeurIPS because they provoke thought, stimulate discussion, and provide rigorous benchmarking that forms the foundation for future work. In our case, we present a rigorous empirical study that pushes the boundaries of understanding in the field of Embodied AI.

If one of the reviewers is skeptical about certain aspects of our work, that's part of the scientific discourse—we can let the broader community and future research be the judge of the long-term impact.

Below, we address specific concerns raised by each reviewer to provide further clarity and address any misunderstandings.


Meta-review: This work received 2 positive, 1 borderline positive and 1 strongly negative review. 

The positive reviewers appreciated
- exhaustive evaluation of pretrained visual encoders across a variety of Embodied AI tasks
- study of the effect of diversity and scale
- release of checkpoints (as promised by authors)

However, one reviewer (nrzP) expressed strong concerns about:
1. whether CortexBench may be considered a contribution since it is a combination of existing benchmarks?
2. are there really any new findings in this work?
3. while studying the effect of dataset size and diversity, the two factors are entangled. For instance, to study the effect of diversity, the reviewer suggests keeping the total number of images the same while varying the proportion of different tasks in the mixture. The diversity question really is - *Given a fixed budget of training images, is it better to have fewer images per task but a lot of tasks or a huge number of images per task from fewer tasks*?
4. Grandiose claims - is the pretrained model in this work any more a visual cortex than any other self-supervised pretrained model?

The AC finds reviewer nrzP’s concerns reasonable and commends them for their dedication towards a thorough review. Having read all reviews and the rebuttal, here’s the AC’s take on each of these 4 issues:
1. Even if CortexBench is constituted entirely of existing benchmarks, unifying several benchmarks and making it easier for other researchers to evaluate their models and representations could be considered a contribution. However, the AC believes the main merit of this work is not in the benchmark itself but in running a *large scale empirical study* focused on the question - *could scaling MAE across diverse benchmarks lead to good universal visual representations for EAI tasks*? 
2. The AC somewhat agrees that the work doesn’t present a completely unexpected or novel finding and some of the results are indeed mixed. However, the AC believes the results shown in this paper could serve as a useful reference point for future studies. 
3. The reviewer does have a valid point here, and the AC believes the authors didn’t adequately address this concern. 
4. The AC somewhat agrees with this sentiment, but while the title and introduction might appear a bit grandiose, the technical portions of the paper seem well-grounded. Some readers might enjoy an imaginative introduction while others don’t - a matter of taste. 

Overall, weighing the pros and the cons, the AC believes this work is **worth accepting** as a useful reference point for future research on building universal visual representations for generalist Embodied agents. Besides the empirical study as presented in the paper, the evaluation benchmark and checkpoints could also be of value. 

The AC would encourage the authors to incorporate reviewers’ suggestions and discussion to increase the impact of their work.
Decision: Accept (poster)