SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ego4D+MN combines Ego4D with the three object manipulation-centric datasets and two indoor navigation dataset, resulting a dataset with 4.3M frames. While larger than Ego4D+M and Ego4D+N, it does not include any new types of data beyond the manipulation and navigation videos in the previous subsets. Thus, it is no more diverse than Ego4D+N (which includes both types of data).
Ego4D+MNI includes Ego4D, all manipulation and navigation datasets, and ImageNet for a total of 5.6M frames. This dataset is used to study the role of internet images for our benchmark tasks.
5.2 Scaling Hypothesis Findings
We now turn to analyzing the effect of increasing model size, dataset size, and dataset diversity. The full set of results is shown in Figure 3 and Table 4. The key takeaways are:
Model Size. We find that increasing model size positively impacts performance on CORTEXBENCH. Specifically, in Figure 3a, we find that with all pre-training datasets, switching from ViT-B to ViT-L
While Ego4D does contain navigation data, it is heavily skewed towards object manipulation.
Table 4: Performance of scaling hypothesis models on CORTEXBENCH. We find that on average the VC-1 Ego4D+MNI (VIT-L) model performs best, but is not the best for each benchmark.
#	Model	Adroit	Meta-World	DMControl	TriFinger	ObjectNav	ImageNav	Mobile Pick	Mean Rank	Mean Success
1	Best prior result (any setting)	75	80	77	-	70.4	82.0	-
2	Rand (ViT-B) fine-tuned	44.0	49.9	34.2	55.0	28.5	65.0	47.6
3	Best result Table 2 (Frozen PVR)	73.3	96.0	81.1	74.1	56.6	68.1	65.4
4	Ego4D (ViT-B)	48.7 ± 1.3	86.1 ± 2.1	64.1 ± 2.3	68.3 ± 1.1	46.8 ± 1.1	64.0 ± 0.7	57.4 ± 2.2	8.6	62.2
5	Ego4D (ViT-L)	50.0 ± 1.2	92.9 ± 2.4	60.8 ± 3.3	69.7 ± 0.5	47.6 ± 1.1	55.8 ± 0.8	67.6 ± 2.1	5.9	63.5
6	Ego4D+N (ViT-B)	50.0 ± 2.4	86.4 ± 2.9	59.5 ± 2.4	67.8 ± 1.3	54.7 ± 1.1	68.7 ± 0.7	59.4 ± 2.2	7.2	63.8
7	Ego4D+N (ViT-L)	54.0 ± 1.2	89.1 ± 2.9	66.4 ± 1.7	66.9 ± 0.4	57.4 ± 1.1	70.5 ± 0.7	65.2 ± 2.1	3.5	67.1
8	Ego4D+M (ViT-B)	51.3 ± 2.4	83.5 ± 2.6	64.3 ± 1.8	69.1 ± 0.4	47.3 ± 1.1	65.8 ± 0.7	59.8 ± 2.2	7.0	63.0
9	Ego4D+M (ViT-L)	52.0 ± 1.3	88.3 ± 3.2	64.7 ± 2.4	64.7 ± 0.9	47.3 ± 1.1	65.5 ± 0.7	68.6 ± 2.1	6.0	64.4
10	Ego4D+MN (ViT-B)	48.7 ± 2.4	85.3 ± 5.2	64.2 ± 1.9	70.3 ± 0.5	52.8 ± 1.1	68.9 ± 0.7	58.6 ± 2.2	6.9	64.1
11	Ego4D+MN (ViT-L)	52.7 ± 4.2	86.7 ± 3.9	69.7 ± 3.3	72.4 ± 0.5	58.4 ± 1.1	69.1 ± 0.7	61.2 ± 2.2	3.1	67.2
12	Ego4D+MNI (ViT-B)	54.0 ± 4.0	89.6 ± 3.9	63.8 ± 2.7	72.2 ± 0.6	55.4 ± 1.1	67.9 ± 0.7	60.6 ± 2.2	4.4	66.2
11	VC-1: Ego4D + MNI (ViT-L)	59.3 ± 5.2	88.8 ± 2.2	66.9 ± 1.4	71.7 ± 0.4	60.3 ± 1.1	70.3 ± 0.7	63.2 ± 2.2	2.4	68.7
improves average performance on CORTEXBENCH. However, in Table 4, we find exceptions where this general trend does not hold. For instance, when pre-trained on Ego4D+MNI, the ViT-B model outperforms the ViT-L model on MetaWorld and TriFinger.
Dataset Size and Diversity. Figure 3b shows that, in general, increasing dataset size and diversity leads to improved performance. Models are ordered from right to left by increasing size and the diversity of their pre-training dataset, and we mostly see improvements for both ViT-B and ViT-L. For instance, Ego4D+M slightly improves upon Ego4D by 0.6 and 0.9 points (62.2 → 62.8 and 63.5 → 64.4) in the case of ViT-B and ViT-L, respectively. The gains with Ego4D+N are larger and it outperforms Ego4D by 1.6 points using ViT-B (62.2 → 63.8) and by 3.6 points for ViT-L (63.5 → 67.1). It is interesting to note that Ego4D+N has a larger improvement over the base Ego4D dataset than Ego4D+M, even though Ego4D+N and Ego4D+M dataset are similar in size. In these results, we find that increasing diversity by adding indoor navigation data improves performance more than adding additional manipulation data to Ego4D.
Additionally, we find that pre-training on  $\mathbf{Ego4D + MN}$  is roughly on par with pre-training on  $\mathbf{Ego4D + N}$ . We see a 0.3 and 0.1 point difference (63.8 → 64.1 and 67.1 → 67.2) for ViT-B and ViT-L, respectively, even though  $\mathbf{Ego4D + MN}$  has about 800K more training frames. Together with the results from above this demonstrates that increasing data diversity seems to matter more than simply increasing dataset size.
Next, we find that adding ImageNet positively impacts average performance on CORTEXBENCH. For example, models pre-trained on Ego4D+MNI outperform those pre-trained on Ego4D+MN by 1.9 points  $(64.1\rightarrow 66.2)$  for ViT-B and 1.5 points  $(67.2\rightarrow 68.7)$  for ViT-L. Interestingly, these results demonstrate that including static internet images can significantly boost performance on EAI tasks. This finding further highlights the importance of seeking data diversity to build better representations.
Finally, our largest model (ViT-L) pre-trained on all datasets  $(\mathbf{Ego4D} + \mathbf{MNI})$ , achieves the best rank when averaged across all benchmark tasks (Table 4 row 11), with a mean rank of 2.4. We call this model VC-1, and will open-source it. VC-1 is superior to the second-best model  $(\mathbf{Ego4D} + \mathbf{MN}$  ViT-L, Table 4 row 9), which has an average rank of 3.1.
However, upon further dis-aggregation, we observe we find that while VC-1 performs best on average, it is not the best for each benchmark. For example, the best model for Mobile Pick, a mobile manipulation task, is a ViT-L trained on Ego4D+M and the best model for ImageNav, an indoor navigation task, is the ViT-L trained on Ego4D+N. These findings suggest that task-specific pre-training datasets could enhance the performance of models on individual tasks. However, it is important to note that this approach would lead to multiple pre-trained models, each tailored to a specific task, and not a unified visual foundation model.
5.3 How does VC-1 compare to existing PVRs?
We now compare VC-1 to PVRs from Section 4. On average, VC-1 has the best rank across all benchmarks (Figure 3c). In terms of mean success, VC-1 (Table 4 row 11) outperforms MVP (ViT-L) by +1.2 points  $(67.5\rightarrow 68.7)$ , R3M by +10.7  $(58.0\rightarrow 68.7)$ , CLIP by +11.7  $(57.0\rightarrow 68.7)$ , and end-to-end fine-tuning from scratch +19.6  $(49.1\rightarrow 68.7)$ .
Impressively, VC-1 outperforms CLIP on every benchmark (Figure 4), despite training on a 70X smaller dataset, emphasizing the importance of egocentric interaction datasets. VC-1 also outperforms fine-tuning from scratch on every benchmark, indicating that PVRs trained with out-of-domain data can outperform in-domain, end-to-end learning.
Figure 4: Comparison of VC-1 with existing PVRs. VC-1 matches or exceeds existing PVRs on all benchmarks except R3M on AD, MW, and DMC, indicating an opportunity for model adaptation.
The MVP model is the most similar in terms of results, architecture, and pre-training objective to VC-1, with the main difference being the addition of a convolutional stem in MVP. VC-1 outperforms MVP VIT-L by 1.3 points on mean success and performs better on four out of seven benchmarks (Figure 4), likely due to the use of a more diverse dataset.
When compared to R3M, VC-1 demonstrates superior performance both on average and in 4 out of 7 benchmarks (Figure 4). However, R3M outperforms VC-1 on Adroit, MetaWorld and DMControl benchmarks. The cause of this gap is unclear - it could be due to differences in pre-training objectives, datasets, or the backbone architecture. That said, the fact that R3M, which uses a ResNet-based architecture, performs better on some tasks than VC-1 which employs a Transformer-based backbone, suggests the potential value of exploring new architectures that integrate the strengths of both approaches - inductive biases and scalability. Taken together, these observations highlight the need for more robust and standardized evaluations on benchmarks like CORTEXBENCH.
Overall, VC-1 is effective across a broad set of tasks and thus a reasonable starting point for novel EAI problems. However, it is not always the best model for a specific task. This leads us to theorize that there is a domain gap that might be bridged with dataset engineering or adaptation of the PVR.
6 Adapting VC-1
In prior sections, we focused on evaluating VC-1 as a frozen PVR. We now study if adapting VC-1 can improve results in downstream tasks. We use a broad definition of adaptation [55], which, in the context of large pre-trained foundation models, can take several forms from simple prompting [56], to selectively updating some or all weights of the backbone [5, 35, 57].
In the context of PVRs for EAI, adaptation can serve at least two purposes. The first is task-specialization in the feature extraction stage. Since VC-1 was trained with MAE [18], it captures features that are generally useful for reconstructing images (see VC-1 attention maps in Appendix A.10). Adaptation can specialize the visual backbone to extract features required for performing specific EAI tasks. Secondly, adaptation can also help mitigate domain-gap that might exist between pretraining and evaluation settings. In general, domain-gap can arise for several reasons such as poor coverage in pre-training datasets or deployment in novel conditions (e.g., on robots) not seen in the pre-training data (e.g., in human-centric video datasets). Domain gap is naturally instantiated in our setup, since VC-1 was pre-trained on real-world, human video data while our downstream evaluation in CORTEXBENCH uses simulated EAI domains with different visual characteristics.
In this work, we explore two methods for adaptation: end-to-end (E2E) fine-tuning and MAE adaptation. While we do not explore prompting-based adaptation, because our visual encoders were not originally designed to utilize prompts, it may be an interesting direction for future work.
End-to-end (E2E) fine-tuning with a task-specific loss function can in-principle capture both of the aforementioned benefits of adaptation, and is widely used in computer vision literature [18, 20, 58, 59]. To study E2E fine-tuning of VC-1, we use the same policy learning methods described in Appendix A.2, except we allow updates to the VC-1 weights. In the CORTEXBENCH results in Table 5, we find an interesting mixed result. In domains that involve large-scale IL or RL (ObjectNav, ImageNav, and Mobile Pick), the strategy proposed in [5] of adapting VC-1 with E2E fine-tuning significantly improves performance over using a frozen VC-1 backbone. Specifically, we see an improvement in ObjectNav success rate (SR) of  $+7.4$  ( $60.3 \rightarrow 67.7$ ), ImageNav SR of  $+11.3$  ( $70.3 \rightarrow 81.6$ ), and Mobile Pick SR of  $+10.8$  ( $63.2 \rightarrow 74.0$ ). These results suggest that E2E
Table 5: Adapting VC-1 with end-to-end fine-tuning or self-supervised learning (MAE) on task-specific data leads to substantial gains. Best prior results are across any setting (frozen/finetuned). Best result (our exp.) are from Section 5. ** indicates that, on hardware, we only evaluated MVP (ViT-L), the closest competitor to VC-1. The TF Push-Cube results are averaged across 12 trials with randomized start/goal configurations. The Franka results are averaged across 4 tasks and 10 trials per task (details in Appendix A.12).
#	Method	CortexBench	Hardware
Adroit	MetaWorld	DMControl	TriFinger	ObjectNav	ImageNav	Mobile Pick	TF Push-Cube	Franka
1	Best prior result	75	80	77	-	70.4	82.0	-	-	-
2	Best result (our exp.)	73.3 ± 2.0	96.0 ± 1.1	81.1 ± 0.7	74.1 ± 0.3	60.3 ± 1.1	70.5 ± 0.7	68.6 ± 2.1	31.9 ± 4.3**	55.0 ± 10.4**
3	In-domain MAE (baseline)	47.3	83.4	77.6	80.4 ± 0.32	39.9 ± 1.09	47.6 ± 0.77	51.6 ± 2.23	22.9 ± 5.4	35.0 ± 13.9
4	VC-1	59.3 ± 5.2	88.8 ± 2.2	66.9 ± 1.4	71.7 ± 0.4	60.3 ± 1.1	70.3 ± 0.7	63.2 ± 2.2	45.8 ± 6.5	70.0 ± 10.9
5	VC-1 E2E fine-tuning	15.9 ± 9.8	22.7 ± 9.7	6.7 ± 3.8	70.9 ± 0.5	67.7 ± 1.05	81.6 ± 0.6	74.0 ± 1.96	52.7 ± 9.2	67.5 ± 13.4
6	VC-1 MAE adaptation	72.0 ± 3.3	96.0 ± 1.8	80.9 ± 1.8	80.6 ± 0.25	57.4 ± 1.11	67.0 ± 0.73	62.4 ± 2.16	43.5 ± 6.7	85.0 ± 9.1
fine-tuning of VC-1 can achieve the benefits of both task-specialization and domain adaptation. A comparison of VC-1 attention maps before and after adaptation is provided in Appendix A.10.
However, in few-shot IL domains (Adroit, MetaWorld, DMC, and TriFinger), E2E fine-tuning does not result in improvements. In fact, in these domains, it leads to a drop in performance, a finding consistent with prior work [6, 40]. We hypothesize that the poor performance of E2E fine-tuning in few-shot IL domains is caused by overfitting, due to fine-tuning a large model with 307M parameters on a small dataset ( $\leq 50K$  frames).
MAE adaptation to mitigate domain-gap. As an alternative to E2E fine-tuning, we explore adapting VC-1 with self-supervised learning (SSL). Specifically, in MAE adaptation we continue training the backbone network with the MAE [18] pre-training objective on task-specific data. Then, we freeze these adapted representations and use them to learn task-specific policies. We note that in MAE adaptation, the backbone is adapted using the same data that is used for training the policy (e.g., frames from expert demonstrations), and no additional in-domain datasets are used. While this adaptation strategy cannot address task-specialization, it may serve to mitigate domain gap.
For MAE adaptation, we initialize with VC-1 weights, and then train with MAE for 100 epochs. In domains where expert demonstrations are available (i.e., Adroit, MetaWorld, DMControl, TriFinger, and ObjectNav), we use the RGB frames from these demonstrations for adaptation. In the remaining two benchmarks (ImageNav and Mobile Pick) we sample frames from training environments to create adaptation datasets. Finally, to isolate the importance of initializing with VC-1 weights, we train in-domain MAE baselines by starting from a random initialization and then following the same approach used for MAE adaptation.
In the CORTEXBENCH results in Table 5, we observe MAE adaptation substantially improves performance in few-shot learning domains. Specifically, on Adroit performance improves by  $+12.7$ $(59.3\to 72.0)$ , MetaWorld by  $+7.2$ $(88.8\to 96.0)$ , DMC by  $+14.0$ $(66.9\to 80.9)$ , TriFinger by  $+8.9$ $(71.7\to 80.6)$ . Interestingly, in DMC and TriFinger, the in-domain MAE baseline (row 3) performs surprisingly well, highlighting the importance of in-domain data for representation learning. Finally, in large-scale IL or RL domains (ObjectNav, ImageNav, and Mobile Pick), we find MAE adaptation results in small reductions in performance from VC-1 (row 4 vs. 6). In these domains, where substantial amounts of data is available for task-specific training (large-scale IL or RL), we find that E2E fine-tuning is the superior approach for adaptation. In aggregate, these results suggest that MAE adaptation can be explored as a powerful alternative in few-shot domains or where E2E fine-tuning fails.
Overall, we find that adapting the VC-1 model leads to performance improvement in all the benchmark domains. Furthermore, on MetaWorld, DMControl, and TriFinger, VC-1 with MAE adaptation (row 6) is comparable with the best known results (SoTA) and the best results from previous sections (rows 1 and 2). Similarly, on ImageNav and Mobile Pick, VC-1 with E2E fine-tuning (row 5) matches or exceeds the best results. Together, these results demonstrate that adaptation of PVRs can be a powerful paradigm for EAI, especially when compared to training representations from scratch.
7 Proof-of-Concept Hardware Experiments
In addition to simulation experiments with CORTEXBENCH, we also explore proof-of-concept hardware experiments utilizing VC-1 as a backbone PVR for training policies with IL. Our hardware evaluation spans two platforms: TriFinger (1 task) and a Franka-Emika Panda arm (4 tasks). Setup details are provided in Appendix A.11 and A.12. We follow a similar experiment protocol to the simulated counterparts by studying few-shot imitation learning, where the demonstrations are collected directly in the real-world via tele-operation or hand-designed controllers.
We study the cases when using VC-1 as a frozen PVR, VC-1 with MAE adaptation and VC-1 with E2E adaptation (as specified in Section 6), and the MVP model as a baseline (best PVR from Section 4). The results are summarized in the hardware section of Table 5. Overall, we observe similar trends to our findings in Section 6. We observe that in frozen mode (row 4), VC-1 substantially outperforms both MVP (row 2) and in-domain MAE (row 3) in both robot setups. We also find that adaptation via E2E fine-tuning improves real-world TriFinger performance and that MAE adaptation leads to large improvements on Franka tasks. We note that while MAE adaptation does not help TriFinger performance, this is likely due to the mere 300 real robot images available for adaptation. Together, these results suggest that learning task-specific features (via fine-tuning) was more important than closing any domain gap (via MAE adaptation) for TriFinger. Overall, these results demonstrate that VC-1 can effectively function as a PVR for multiple hardware platforms, and can outperform prior PVRs that have shown success on hardware, such as MVP. Furthermore, it reinforces finding from Section 6 that adapting VC-1 with SSL objectives (MAE) can improve the performance.
This work introduced CORTEXBENCH, which comprises 17 different EAI task spanning locomotion, indoor navigation, and dexterous and mobile manipulation; and conducted the most comprehensive study to-date of PVRs (or visual foundation models) for EAI. We find that (1) despite significant progress in a number of narrow domains, we do not yet have a universal visual backbone for all EAI tasks of interest, (2) naively scaling model size and pre-training data diversity does not improve performance universally across all tasks, but does so on average, (3) adapting our largest pre-trained model (VC-1) results in performance that is competitive with or outperforms the best known results on all benchmarks in CORTEXBENCH, and (4) VC-1 and adaptation show proof-of-concept generalization to hardware. Our study is an attempt to unify various EAI areas using the perception module as a cornerstone. The study and development of visual representations for sensorimotor control today appears splintered across different sub-communities studying egocentric computer vision, locomotion, navigation, dexterous and mobile manipulation. However, we contend that this cannot be the final solution. Biological organisms have one visual cortex, not one per 'task'. Analogously, it must be possible for an embodied AI agent to have one universal artificial visual cortex supporting a diverse range of sensorimotor skills, environments, and embodiments. We speculate that learning visual representation using temporal signals, 3D spatial priors, or objectness will help make progress towards this goal. Our final contention is that in order for the research community to develop such a model, we need to create benchmarks that test broad generalization capabilities; we hope CORTEXBENCH will help the community make progress towards that.
The Georgia Tech effort was supported in part by ONR YIP and ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
[1] Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics, 20(3):121–136, Sep 1975. ISSN 1432-0770. doi: 10.1007/BF00342633. URL https://doi.org/10.1007/BF00342633.1
[2] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193-202, Apr 1980. ISSN 1432-0770. doi: 10.1007/BF00344251. URL https://doi.org/10.1007/BF00344251. 1
[3] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In CVPR, 2022. 1, 3
[4] Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for embodied navigation. In arXiv preprint arXiv:2204.13226, 2022. 3, 5
[62] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search from human demonstrations at scale. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 16
[63] Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICLR), 2020. 16
[64] Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-augmented reinforcement learning for imagegoal navigation. arXiv preprint arXiv:2101.05181, 2021. 16
[65] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9339–9347, 2019. 16, 17
[66] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9068-9079, 2018. 16
[67] Ziad Al-Halah, Santhosh K Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. arXiv preprint arXiv:2202.02440, 2022. 17
[68] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 17
[69] Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang, Alexander Clegg, Manolis Savva, et al. Habitat challenge 2022, 2022. 17
[70] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018. 17
[71] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173-5183, 2022. 17
[72] Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. *ArXiv*, abs/2212.05711, 2022. 22
[73] Robohive - a unified framework for robot learning. https://sites.google.com/view/robohive, 2020. URL https://sites.google.com/view/robohive.22