SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This study presents a thorough examination of visual foundation models but has several limitations. Firstly, in proposing a benchmark, we sought to find a balance between task diversity and the computational resources required for evaluation. However, new and challenging benchmarks in embodied AI, such as those presented in [60], continue to emerge and may merit inclusion in future studies to track progress in this field. Additionally, while we have focused on masked auto-encoders (MAE) as the pre-training objective and ViTs as the architecture in our study, there may be other SSL algorithms that exhibit different scaling behaviors or superior performance on the proposed tasks in our benchmark. Lastly, the adaptation procedures studied in this work necessitate separate training on in-domain datasets, as well as careful tuning of hyperparameters such as the number of training epochs and sampling ratio of the dataset. This results in a significant effort to produce a separate adapted PVR model for each benchmark evaluated on our benchmark, and the overall effort increases proportionately with the number of benchmarks included in the study.
In conclusion, it is important to note that although we utilize real-world images and videos for pre-training our visual representation models (PVRs), the evaluation benchmarks used in this study serve as proxies for actual robotic tasks, and thus, the performance of the PVR models on real robots may differ from the rankings established in this study. Further research is necessary to fully evaluate the effectiveness of these models in real-world scenarios.
A.2 Overview of Downstream Policy Learning in CORTEXBENCH
Given a frozen PVR, an agent needs to learn a policy for each task. The EAI community has developed a range of policy learning algorithms from few-shot imitation learning (IL) to large-scale reinforcement learning (RL). For each task in CORTEXBENCH, we conform to the community standard for achieving state-of-art performance in that domain.
"MuJoCo Tasks" On the tasks from the Adroit, MetaWorld, and DMC suites we train policies using behavior cloning on a small number of expert demonstrations (100 for Adroit and DMC and 25 for MetaWorld), which follows [6, 7]. Specifically, we train policies for 100 epochs and report the average rollout performance on the test set for the best intermediate policy during training. For all tasks, we use frame-stacking and a 3-layer MLP policy network. When using vision transformers (ViT) based PVRs, we use the [CLS] token as input to the policy, and with ResNets we use features from the final convolutional layer after global average pooling. These design choices follow prior work such as [7, 8].
"TriFinger Tasks" For TriFinger, we train policies using behavior cloning on 100 demonstrations per task. Specifically, we train a policy network composed of a 3-layer MLP for 100 epochs for Reach-Cube and 1,000 epochs for Push-Cube. We report the average score for the best checkpoint over the course of training. The input to the policy is the [CLS] token for ViT-based PVRs and average pooled features from the last convolutional layer for ResNet-based models.
"Habitat Tasks" We train ObjectNav policies with behavior cloning on 77k human demonstrations [61] collected by [62], totaling 360M environment steps. For ImageNav and Mobile-Pick, we use RL for 500M environment steps with DD-PPO [63] and VER [36]. We use patch representations for ViT-based PVRs and grid-features from last convolutional layer for ResNet models, passed through a compression layer [14] for a lower dimensional representation for use by the policy layers, which is a 2-layer LSTM for navigation and a 2-layer GRU for manipulation.
A.3 Additional Details of Tasks and Downstream Learning in CORTEXBENCH
In this section, we discuss further details for a subset of downstream tasks in CORTEXBENCH.
ImageNav. Our study conducts ImageNav experiments using the standard dataset presented in [64]. This benchmark utilizes the Habitat simulator [15, 65] and is situated within the Gibson [66] environments, which comprise 72 training scenes and 14 validation scenes. The validation set includes 300 episodes for each scene, for a total of 4,200 episodes. In this benchmark, agents are modeled as cylinders with a height of  $1.5\mathrm{m}$ , radius of  $0.1\mathrm{m}$ , and sensors located  $1.25\mathrm{m}$  above the center of the
base. The RGB camera has a resolution of  $128 \times 128$  and a  $90^{\circ}$  field-of-view. Agent is able to take up to 1000 steps within the environment and are deemed successful if they reach a location within  $1\mathrm{m}$  of the goal position and call STOPACTION.
To train the agents within the Gibson environments, we utilize 500M timesteps (25k updates) with 320 environments running in parallel. Each environment collects up to 64 frames of experience, which is followed by 2 PPO epochs utilizing 2 mini-batches. Unless otherwise specified, we use a learning rate of  $2.5 \times 10^{-4}$  for training the agents and update the parameters using the AdamW optimizer with a weight decay of  $10^{-6}$ . We train agents with the reward functions presented in [67] utilizing the following settings: success weighting  $c_{s} = 5.0$ , angle success weighting  $c_{a} = 5.0$ , goal radius  $r_{g} = 1.0$ , angle threshold  $\theta_{g} = 25^{\circ}$ , and slack penalty  $\gamma = 0.01$ . We evaluate performance every 25M steps of training and report metrics based on the highest success rate (SR) achieved on the validation set.
ObjectNav. We present an evaluation of object navigation (ObjectNav) using the HM3D-SEM dataset [61]. The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments. Our results are reported on the v0.1 HM3D-SEM VAL split, which was used in the 2022 Habitat Challenge [69] ObjectNav benchmark. The agent in this evaluation is modeled after the LocoBot [70] with a height of  $0.88\mathrm{m}$ , radius of  $0.18\mathrm{m}$ , and sensors placed at the top of the agent's head. The RGB camera has a  $640\times 480$  resolution and a  $79^{\circ}$  horizontal field of view. The task for the agent is to locate objects from one of 6 categories: 'chair', 'bed', 'plant', 'toilet', 'tv/monitor', and 'sofa' within 500 steps. Successful episodes are determined by the agent stopping within  $0.1\mathrm{m}$  of a viewpoint that is (a) within  $1\mathrm{m}$  of any instance of the target object and (b) from which the object is visible, as outlined in the evaluation protocol of [45].
We utilize a dataset of human demonstrations for training our imitation learning agent in the task of ObjectNav. The dataset was collected using Habitat-Web [61, 71] and Amazon Mechanical Turk, and consists of  $77k$  demonstrations for 80 scenes from the HM3D-SEM dataset [69]. Each scene contains approximately 158 episodes, each with a unique goal object category and a randomly set start location, resulting in approximately 950 demonstrations per scene. The dataset includes a total of  $\sim 12.1$  million steps of experience, with an average of  $\sim 159$  steps per episode. By leveraging this human demonstration data, our imitation learning agent is able to learn a more effective policy for navigating to objects in complex environments.
We trained object navigation (ObjectNav) agent in the HM3D environment for an approximate total of 400 million steps, utilizing 25,000 updates and 512 parallel environments. Similar to our previous image-based navigation (ImageNav) experiments, we employed a weight decay of  $10^{-6}$  and utilized different learning rates for the visual encoder and other elements of the model. Specifically, we used a learning rate of  $10^{-4}$  for the visual encoder and  $10^{-3}$  for all other elements, with the AdamW optimizer. To ensure the quality of our trained models, we evaluated checkpoints after every  $10M$  steps and only reported metrics for the checkpoints with the highest validation success rate.
Mobile Pick. We investigate the Habitat 2.0 Rearrangement task proposed by [15]. This task involves a mobile manipulation scenario in which a Fetch robot navigates an ReplicaCAD apartment to pick up a target object from a cluttered receptacle using a mobile base [46]. The robot starts from a non-trivial position and must utilize a variety of sensors, including an egocentric RGB camera, proprioceptive joint sensing, and an object grasping indicator. The action space for the robot includes continuous control over the robot's 7-DOF arm, base movement, and suction gripper. We relax the dense goal specification, where the relative position between the end-effector and the target object must be updated at each step, to a sparse goal specification, where this information is only provided at the start of the episode. This relaxation places greater emphasis on visual input and makes the task significantly more challenging.
TriFinger Tasks. The TriFinger tasks are implemented in Pybullet. For Reach-Cube, the state for the BC policy is  $[x_t^{ft}, z_t]$ , where  $x_t^{ft}$  is the current fingertip position and  $z_t$  is the latent visual state vector, obtained by passing the current image observation through the PVR. The success metric captures how close the fingertip is to the optimal distance from the center of the cube, accounting for the half-width of the cube. For Push-Cube, the state for the BC policy is  $[x_t^{ft}, z_t, \Delta x_g^c]$ , where  $\Delta x_g^c$  is the goal position for the cube, specified as a displacement from its initial position. Here the success
is the distance of the center of the cube to the target goal position. We train a policy network with hidden layers of size 2000 and learning rate  $10^{-4}$  for up to 100 epochs for the reach task and 1000 epochs for the Push-Cube task.
A.4 Additional Analysis of Existing Pre-Trained Visual Representations (PVRs)
The rank distribution for existing PVRs (shown in Figure 5) demonstrates that there is high variability in the performance of the models (from prior work) across the benchmarks in CORTEXBENCH.
Figure 5: Rank distribution of existing pre-trained visual representations. For every model, we compute the ranks it achieved on each of the 7 benchmarks. We visualize them as vertical lines, where each rank number  $x$  receives a tick if that model achieved such rank  $x$ . For instance, MVP (ViT-L) achieves ranks 1,1,1,2,3,3,4 across the 7 benchmarks. Significant variability exists in the performance of PVRs across benchmarks.
A.5 Scaling Hypothesis Datasets
The datasets composed for scaling hypothesis experiments are detailed in Table 6.
A.5.1 OpenHouse24 Dataset
The OpenHouse24 (OH24) dataset is a collection of video walk-throughs of furnished residential real estate properties. Over 1600 homes are represented in the dataset, totaling 139 hours of video footage. Each home is traversed in a continuous shot with a stable HD RGB camera by an operator that efficiently visits each room. The dataset represents a diverse set of properties, including (but not limited to) small and large suburban homes, high-rise apartments, ranch homes, and condos. The ensuing walk-throughs range from under a minute to 14 minutes in length, with the average taking 5 minutes and 12 seconds. The dataset will be open-sourced by a separate research project.
A.6 Scaling Hypothesis Pretraining Details
To train the MAE models, we use the official codebase released by the authors on GitHub [18] and use the default hyperparameters provided by the repo to train the ViT-B and ViT-L models. We found the default values worked well on the CORTEXBENCH. However, we do vary the number of epochs we use to train the different models in Section 5 given the different dataset sizes. We choose the number of epochs per run such that the number of model updates remain constant across all runs and match the number of model updates taken by MAE on the ImageNet dataset. We provide details about the dataset sizes and the epochs calculated for the different runs in Table 7.
A.7 Additional Analysis of Scaling Hypothesis Results
Figure 6 (left) provides additional evidence of the significance of pre-training dataset diversity. We observe that while the ViT-L models trained in  $\mathbf{Ego4D + M}$  and  $\mathbf{Ego4D + N}$  datasets, achieve the best result in one of the benchmarks, they perform the worst and second-worst in other benchmarks. However, by adding diversity, the  $\mathbf{Ego4D + MN}$  and  $\mathbf{Ego4D + MNI}$  models have decreased variance
Table 6: Datasets used for our scaling hypothesis experiments, containing up to 5.6M frames.
Name	Contains	Total Frames	Frames used
Ego4D	Ego4D	418,578,043	2,790,520
Ego4D+M (Manipulation)	Ego4D	418,578,043	2,790,520
100DOH	99,899	99,899
SS-v2	25,209,271	315,115
Epic Kitchens	19,965,439	332,757
Total	3,538,291
Ego4D+N (Navigation)	Ego4D	418,578,043	2,790,520
OpenHouse24	27,806,971	499,442
RealEstate10K	10,000,000	303,087
Total	3,289,962
Ego4D+MN (Manipulation, Navigation)	Ego4D+M	3,538,291	3,538,291
OpenHouse24	27,806,971	499,442
RealEstate10K	10,000,000	303,087
Total	4,340,820
Ego4D+MNI (Manipulation, Navigation, ImageNet)	Ego4D+MN	4,340,820	4,340,820
ImageNet	1,281,167	1,281,167
Total	5,621,987
Table 7: Experiment Details of Training PVRs.
Dataset Name	Epochs	Frames used
Ego4D+N (VIT-B)	289	3,538,291
Ego4D+N (VIT-L)	289	3,538,291
Ego4D+M (VIT-B)	414	3,289,962
Ego4D+M (VIT-L))	414	3,289,962
Ego4D+MN (VIT-B)	236	4,340,820
Ego4D+MN (VIT-L)	236	4,340,820
Ego4D+MNI (VIT-B)	182	5,621,987
VC-1 (Ego4D+MNI (VIT-L))	182	5,621,987
in their rank distributions. Notably, the Ego4D+MNI model consistently performs well across all benchmarks, and ranks among the top models.
A.8 Additional Analysis of All Models Evaluated on CORTEXBENCH
Figure 6 (right) provides a complete picture of the rank distribution for all of the models evaluated in this study. Similarly, Table 8 provides results for all models evaluated in this study is collected.
A.9 Additional Analysis of Scaling Model Size
Figure 7 illustrates that scaling model size has a positive effect on every benchmark and on fifteen out of the seventeen tasks.
Figure 6: (left) Rank distribution per model - scaling hypothesis. (right) Rank distribution per model - existing PVRs and scaling hypothesis models
Table 8: The success rate for each task and each model we evaluate during the study before being aggregated by benchmark.
model	TASK
assembly	bin_picking	button_press	cheetah_run	drawer_open	finger_spin	hammer	imagenav	mobile_pick	move_cube	objectnav	pen	reach_cube	reach_reacher	relocate	walker_stand	walker.walk
CLIP (VIT-B)	70.7	68.0	48.0	22.7	100.0	74.6	90.7	52.2	49.8	40.1	56.6	72.0	83.8	89.9	22.7	64.9	25.4
MAE Ego4D (VIT-B)	81.3	76.0	80.0	29.1	100.0	76.9	93.3	64.0	57.4	54.0	46.8	74.7	82.6	79.8	22.7	84.3	50.5
MAE Ego4D (VIT-L)	98.0	84.0	84.0	20.7	100.0	76.5	98.7	55.8	67.6	57.0	47.6	76.0	82.4	71.9	24.0	78.4	56.3
MAE Ego4D+M (VIT-B)	76.0	58.7	84.0	31.9	100.0	75.5	98.7	65.8	59.8	57.5	47.4	77.3	80.7	89.3	25.3	80.7	44.3
MAE Ego4D+M (VIT-L)	89.3	73.3	84.0	33.5	100.0	75.6	94.7	65.5	68.6	47.2	47.3	74.7	82.1	85.8	29.3	76.2	52.3
MAE Ego4D+MN (VIT-B)	82.7	74.7	77.3	32.0	100.0	77.5	92.0	68.9	58.6	62.1	52.8	73.3	78.5	85.6	24.0	84.1	41.8
MAE Ego4D+MN (VIT-L)	83.3	70.7	74.7	38.1	100.0	77.0	94.7	69.1	61.2	62.4	58.4	78.7	82.4	91.7	26.7	83.0	58.9
MAE Ego4D+MN (VIT-L)	88.0	78.7	82.7	32.3	100.0	78.0	98.7	69.9	60.6	60.6	55.7	76.0	83.9	83.9	24.0	84.4	44.7
MAE Ego4D+MN (VIT-L)	88.0	84.0	80.0	32.8	100.0	76.8	92.0	70.3	63.2	60.2	60.3	80.0	83.3	88.0	38.7	83.3	53.7
MAE Ego4D+N (VIT-B)	86.7	76.0	73.3	28.1	100.0	75.8	96.0	68.7	59.7	54.7	54.4	77.3	81.6	78.7	22.7	72.4	42.6
MAE Ego4D+N (VIT-B)	89.3	73.3	89.3	33.3	100.0	76.2	93.3	70.5	65.2	52.7	57.4	76.0	81.1	88.6	32.0	83.4	50.7
MVP (VIT-B)	92.0	73.3	92.0	33.9	100.0	76.9	98.7	64.7	56.0	44.3	51.2	69.3	75.0	86.3	26.7	84.7	47.9
MVP (VIT-L)	89.3	78.7	70.7	36.9	100.0	76.4	98.7	68.1	65.4	63.4	55.0	76.0	84.8	90.2	30.7	83.2	59.3
R3M (RN-50)	97.3	93.3	89.3	66.1	100.0	77.1	100.0	30.6	33.2	51.9	22.6	81.3	86.5	98.4	65.3	93.8	70.1
Random (VIT-B)	0.0	0.0	2.7	0.4	0.0	0.0	0.0	42.1	10.8	41.3	19.2	4.0	74.3	23.4	0.0	22.7	4.0
Random (VIT-L)	0.0	0.0	0.0	0.5	0.0	0.2	2.7	45.2	20.6	39.4	19.3	5.3	74.9	19.9	0.0	20.1	4.6
Random finetune (VIT-B)	61.3	34.7	20.0	10.2	40.0	48.6	93.3	62.5	47.6	37.6	28.5	73.3	74.5	26.8	14.7	73.6	58.1
VIP (RN-50)	93.3	76.0	88.0	53.2	100.0	76.1	93.3	48.8	7.2	47.2	26.4	81.3	86.2	83.2	26.7	86.6	63.4
(a) scaling model size per benchmark
(b) scaling model size per task
Figure 7: Scaling model size has a positive effect on (a) every benchmark and on (b) fifteen out of the seventeen tasks.
A.10 Attention Visualizations of VC-1
To visualize the attention we apply a mean pooling operation to the attention matrices of the ViT encoder's final layer during inference for downstream tasks. The resulting values are then overlaid onto the image.
We start by noticing the effect of MAE pre-training; frozen VC-1 attention maps appear to focus on the contours and general features of the image. We hypothesize that this results from the MAE
reconstruction-based training objective, as contours provide essential information for reconstructing images.
Additionally, we study the attention maps after end-to-end fine-tuning of VC-1 on the downstream tasks. The attention appears to focus on regions of the image that are important for the task (e.g., the objects being manipulated). Thus, through adaptation (via E2E fine-tuning), the model learns to drop attention on areas irrelevant to the specific task.
Figure 8: Attention Visualization: (left) Random ViT-L; (middle) VC-1 frozen; (right) VC-1 E2E finetuned. We overlay the mean attention matrix in the last layer of the ViT encoder in one of our tasks -MobilePick-. We notice the effect of MAE pre-training on VC-1: The attention focuses in general features of the image; and of task-adaptation: the attention concentrates in task-specific regions of the image
A.11 TriFinger Hardware Experiment Setup
We carried out experiments on the real TriFinger robot (shown in Figure 9) for the Push-Cube task, after training a model using behavior cloning on 30 real-world demonstrations. The success for Push-Cube is defined as the distance which the cube is able to travel relative to its starting distance to the goal, similar to [43]. The episode length for this task is 20 steps. For each model, we chose 1 seed and ran it on 12 different start and goal configurations, mostly centered in the arena.
The action space for this environment consists of end-effector displacements for the three fingers. The motors are controlled at a frequency of  $1\mathrm{kHz}$  and the action sent to the robot is a 9 dimensional vector specifying the joint torques. We use an Intel RealSense camera, positioned next to the table overlooking the scene, similar to how our simulation image observations are captured. To detect the cube, we make sure that the green side is always facing upwards and detect the center of that face, and use that to track the cube's position in the bowl.
Figure 9: TriFinger Push Cube task
A.12 Franka Hardware Experiment Setup
We conduct behavior cloning experiments on a manipulation platform comprising a Franka arm and a Robotiq gripper fitted with Festo Adaptive Gripper Finger. Figure 10 shows the task setups:
- The reaching task requires controlling the end-effector to reach an randomly selected point between  $(-0.3, 0.4, 0.2)$  and  $(0.3, 0.4, 0.2)$  in the robot frame within  $5\mathrm{cm}$  error given a goal image prompt.
- The bottle pickup task requires reaching and picking up a bottle with the gripper. The bottle is randomly positioned on a  $10\mathrm{cm}$  line, same orientation).
- The close-drawer and toaster plunge tasks are similar to the Cacti setup [72].
For all the tasks, the observations are single RGB camera image (420x224 resolution) and proprioceptive signals (7 degree-of-freedom joint angles and 1 degree-of-freedom gripper width). A 3-hidden layer, each with 256 hidden nodes, multi-layer perceptron policy takes PVR encoded images concatenated with scaled proprioceptive signals as input and predicts the desired absolute joint angles and gripper width as actions. A low-level  $1\mathrm{kHz}$  real-time joint-space PD controller follows the policy-generated desired trajectory. We specifically keep the gains low (half of the factory setting) so that the robot will not break anything even if it were commanded to hit the table. This results in poor trajectory tracking performance, yet both human teleoperators and learned policy can control the robot to complete the tasks. We use Robohive [73] as the middleware to interface with all sensors and robots.
Demonstrations are collected from human teleoperation using a Quest 2 controller. For PVR (frozen encoders), we use Adam optimizer with a learning rate  $10^{-3}$  to train the policies. For fine-tuning, we use the same learning rate for policies but a lower learning rate  $(10^{-5})$  for the visual encoders. Table 9 shows the behavior cloning success rates.
(a) Reaching Random Point
(b) Bottle Pickup
(c) Toaster Plunge task on a kitchen table-top setup
(d) OpenDrawer task on a kitchen table-top setup
Figure 10: Franka manipulation tasks
Table 9: Franka manipulation task success rates.
Reaching	Bottle Pickup	OpenDrawer	Plunge Toaster	Success Count	Success %
Demos	30	50	250	250
Evals	10	10	10	10
VC-1 frozen	80	100	60	40	28	70.0
VC-1 E2E fine-tuning	50	90	80	50	27	67.5
VC-1 MAE adaption	90	100	80	70	34	85.0
MAE baseline	30	10	50	50	14	35.0
MVP	70	100	30	20	22	55.0