What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Authors: Kaylee Burns
Zach Witzel
Jubayer Ibn Hamid
Tianhe Yu
Chelsea Finn
Karol Hausman
Venue: CoRL 2024 (2024)
Keywords: representation learning
manipulation
visual features

Abstract: Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevant features. Although this approach improves the efficiency of policy learning, it remains unclear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual representations designed for control tasks do not necessarily generalize under subtle changes in lighting and scene texture or the introduction of distractor objects. To understand what properties _do_ lead to robust representations, we compare the performance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on a suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.

What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Kaylee Burns
Zach Witzel
Jubayer Ihn Hamid Karol Hausman 1.2 GGoggl DeepMind
Tianhe Yu²
Chelsea Finn
Stanford University
Abstract: Inspired by the success of transfer learning in computer vision, roboti- cists have investigated visual pre- pre-training as a means to improve the learning ef- ficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevan features. Although this approach improves the efficiency of policy learning, it remains clear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual rep- resentations designed for control tasks do not necessarily generalize under subtle changes lighting and scene texture or the introduction of distractor objects. To understand what properties do lead to robust representations, we compare the per- formance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.
Keywords: representation learning, manipulation, visual features
Introduction
In spite of vast progress in computer vision, the question of how learn good visual representation for robotics remains open [1]. Elsewhere in computer vision, internet datasets retrofit to new tasks with transfer learning, which promises both generalization and fast adaptation to downstream tasks in exchange for large-scale pre-training. But in the field of robotics, this promise has yet to be fulfilled even though policies learned from pixels struggle substantially with data efficiency [2) and especially generalization under visual changes in a scene [3].
Recent work [4, 5] posits that the missing piece is large pre- pre-training dataset of object interactions across diverse environments - the ImageNet [6] or CommonCrawl [7] of manipulation. That is, if we want to improve the visual generalization ability of pre-trained models we simply need to collect datasets of this kind at scale. Indeed, training on large datasets of first-person human interaction data increases policy performance and learning efficiency downstream [8, 9], but these evaluations occur in environments that are very similar to those used for policy learning. Robotic applications commonly contain environments with varying lighting conditions, scene textures, and background objects, and want pre-trained representations to allow the robot to handle such variability. Yet have few concrete measures of how well pre-traines ire-traned representations generalize out-of -ifoof--dttiitiinnn To take step towards understanding these problems, our goal in this paper is to thoroughly answer the questions "which models generalize?" and "how can we predict generalization ability?"
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

With Segmenting Features accuracy drops 37% with distractors
Our first key finding is that, when evaluated under visual distribution shifts, models that are designed for manipulation and control do not outperform stan- dard visual pre-training methods. This finding vio- lates our intuitions about what is needed to scale up robot learning and brings into question what consti- tutes relevant data, how to quantify useful features, and the importance of design choices such as model architecture. In other words, we need more guid- ing principles to understand what representations are good for manipulation and make the problem of iter- ating on pre-training strategies much more straight- forward. Currently, evaluating pre-trained policy requires training and rolling out downstream policies across multiple environments and experimental con- ditions. Instead, we can take inspiration from com- puter vision, which has developed proxies for robust performance on out-of-distribution datasets [10].
Without Sagmenting-F Features acouracy drops 85% distractors
find the emergent segmenta- tion ability of ViT attention heads (measured by Jaccard index) predicts performance under visual distribution shift. We refer to models with this property as having "segmenting-featte features." Notice how the attention MVP shifts towards the sugar box distractor object in the bottom right image. The impact of this factor overshadows other de- sign choices such as data relevance.
Figure 1:
Our second key finding is that the emergent seg- mentation ability of a ViT model is a strong pre- dictor of out-of-distribution generalization perfor- mance. We visualize this phenomenon, which we refer to as "xegmenting-fcattred in Figure 1. Other metrics of model quality, such as linear probes on ImageNet [11], and out-of-distributie per- formance, such as in-domain accuracy [12] and shape-bias [13], are not predictive for this model class, despite their predictive power in other commonly-studice domains like image classification. This hints at the possibility that the transfer setting of manipulation differs from computer vision tasks typically studied within the robustness literature.
To reach the conclusions above, we run 9,000 different simulated evaluations. Our simulated en- vironments are adapted from two different existing visual distribution shift benchmarks [14, 15] to capture the shifts that arise commonly in robotics applications: changes in lighting, background and object texture, and the appearance of distractors. More specifically, we train policies on top of 15 pre-trained models, including 4 models designed for manipulation or control: R3M [8], two MVP variants [9, 16]. and VIP [17). We further validate these findings by comparing model designed for manipulation against model with similar parameter count on a real-world screwdriver pick- up task using the ACT training framework [18]. Through these experiments, we make two striking findings: (1) pre-trained visual models designed for control do not necessarily generalize better than models pre-trained on more standard vision datasets and (2) the emergent segmentation performance of ViT model is strong predictor of the out-of-distribution generalization of a down-stream policy.
2 Related Work
Representation learning for manipulation. The correct approach to visual representation learning for robotics still an open question. There is evidence that separating visual representation learn- ing from policy learning can further improve performance [19, 20]. Recent works have shown that models pre-trained on large manipulation-rellvvnn datasets [21, 4, 22, 5] or learned with visual affor- dances from RGBD data [23] can improve the efficiency and performance of policy learning [24] in comparison standard vision datasets such as ImageNet [6]. but they do not focus on performance under visual distribution shift. We evaluate the performance of R3M [8], MVP [9, 16]. and VIP [17]. Other work has studied generalization of pre-trained representations to new reinforcement learning tasks for manipulation [17] and navigation [25] where the agent is able to train on visual data from the new environment. Our objects study are models that map RGB image data into single-vecto feature representations. This excludes models such as LIV [26] and SUGAR [27], which operate on different data modalities, and Seyment-Anything [28]. which produces 3-dimensional feature maps and not compressed representations. Separate from the question of pre-training visual representa-

---

Policy Training
Zero-Shot Evaluation
Trann Dist.
Dett
stractors
lighting
Mocel @
proprio
lle-all all
Pro Trained Model
bnvionmert ngl). Dunsi
Figure 2:
Evaluation Scheme. We begin our evaluation procedure by training policy with behavior cloning on top of frozen features. In every experimental setting, we ablate the image observation encoder. The learned policy is then evaluated in each of the shift environments to attain a zero-shot success value.
tions is the question of how to best train policies top of pixel observations [29, 30J. Majumdar et al. [31] benchmarks the performance of pre-trained visual representations handful of ma- nipulation environments, but they focus in-domain performance and also investigate navigation environments. Hu et al. [32] shows that model performance is highly sensitive to evaluation. We use imitation learning for our evaluation protocol, which they find to be a more stable measure of performance. Concurrently with our work, [33] demonstrates that the importance of proper data balancing supersedes the content of any one pre-training dataset. We focus on benchmarking visual generalization specifically and focus on advancing metries that are predictive of generalization.
Robustness in computer vision. There is extensive work studying the impact of design choices, such as architecture, loss, and data, on the performance of visual models under distribution shift. See Geirhos et al. [10] for a comprehensive comparison. Most relevant to our paper are studies of shape-bias and architecture. While shape-biased models tend to be more robust than texture-biased ones [13], the impact of architecture on robustness is less straightforward. For example, vision trans- formers exhibit better robustness universal adversarial attacks [34], but they are more susceptible to patch-level attacks [35]. When compared natural distribution shifts [36, 37, 38], vision trans- formers and convolutional networks achieve comparable performance when provided with enough data [39]. But for occlusions specifically, vision transformers appear to have an edge [40]. Miller et al. [12] studies the predictive power of in-domain performance for out-of-distributtin generaliza- tion. Unlike
these prior works, we focus on how pre-trained representations affect robustness in downstream robotics tasks, instead of downstream vision tasks.
Learning robust policies. Unlike work that focuses changes in dynamics initial state distri- bution [41, 42, 43, 44, 45, 46), we focus exclusively on the setting of visual distribution shifts. Kirk et al. [47] and Zhao et al. [48] provide a comprehensive survey on non-visual distribution shifts in decision making problems. Policy adaptation approaches enable visual robustness specifically by leveraging insights from domain adaptation during policy training [49, 50, 51] or during deploy- ment [52]. In the special case of closing the sim-to-real domain gap. a popular approach is add randomized textures while training in simulation [53, 54, 55, 56). By contrast, work is inter- ested in explaining properties of robust visual model for control. Consequently, our insights can be leveraged with or without any task specific data.
3 Environments, Evaluation Protocol, and Pre-Trained Models
Our goal is to understand how robust existing representations for manipulation are to visual distri- bution shifts that are realistic in robotic applications. To that end, we learn policies parameterized by multi-layer perceptrons (MLPs) on top of frozen, pre-trained encoders and then evaluate these policies zero-shot under changes in lighting, object and scene texture, and the presence distrac- tors. We opt for ML.--aased policy evaluations on frozen backbones because they the standard for probing visual representations [1, 11, 57], have strong theoretical underpinnings [58], and the de facto benchmark for benchmarking representations for control [9, 8, 17]. The shifts tested are visualized in Appendix Figure 7 and high level summary of our evaluation procedure is visualized
3

---

FrankaKitchen
2001
1:3
VIT-NSUP
VIP
MAKE-W, IN, VITSS
VIT-OINO
10
Dod.
Teat (lightirg))
3ad (ecture)
(Cltrrstoon)
MAgg!
Meta-World
32
70
50-
8
1/
10
5
Tain DIC
Tet (Lighang)
Test (Tsturr)
Text (Dtoractorr)
Rss. (Aug)
Figure 3: Visual Generalization Performance. Models trained with supervision ImageNet shades of in red. Models trained explicitly for control tasks
blue. Models trained with sulf-supervision ImageNet are orange. Dotted hars denote ResNets and slashed hars denote ViTs. Surprisingly, the best performing models are not necessarily designed for manipulation. Each bar is an average over 30 experimental conditions.
in Figure 2. In this section, describe the specifics of the manipulation environments, distribution shifts, and policy training setups.
Environments and tasks. We study ten tasks across two simulated manipulation environments, which are selected based on their popularity studying learning-based approaches to manipulation. Within FrankaKitchen [59] we evaluate performance on opening a microwave, sliding cabinet door open, pulling cabinet open, turning knob, and turning light. Within Meta-World [60] study assembling ring onto peg, placing object between two bins, pushing button, opening a drawer, and hammering nail.
Distribution shifts. We develop a benchmark for out-of-distributicn generalization within FrankaK- itchen and Meta-World. Within FrankaKitchen, we reimplement the texture and lighting changes from KitchenShift [14]. Within World we use texture changes from Xie* et al. [15] and reim- plement the same lighting changes in FrankaKitchen. In both environments levels of distractors: one, three, and nine YCB objects [61]. More details about the implementation and parameterization of the distribution shifts are provided in Section A.3.
include three
Policy training. Policy training done in the same manner as R3M [8]. summary of the eval- uation scheme is provided Figure 2. We train an MLP on top the pre-trained embedding with imitation learning (IL), which, given actions sampled from expert trajectories, mizes the mean squared error objective, ||a al12. Here à denotes the action predicted from given policy. Details of the training procedure are provided in Section A.4. The embedding weights are frozen during policy learning, so the pre-trained models receive no task data. We train different seeds within each task for each of two different camera angles. In total, we learn 60 policies for each model and perform evaluations per policy, including on the train distribution.
Dirain, mini-
Formally, for pre- pre-trained representation & learn policies, Ta, each trained with different seed, camera angle, and task. We average the performance of TH along each experimental condition and compute the mean performance and error across seeds.
Pre-trained Visual Representations. We categorize pre-trained models by loss type and data source: supervised ImageNet models, self-supervised ImageNet models, and models trained for manipulation and control tasks. Model specifics are provided in Appendix Section A.1.
Generalization of Models Pre-Trained for Manipulation
One factor motivating work in learning-based robotics is the hypothesis of scale: if we collect more high-quality manipulation data, should improvements in policy generalization. However, our understanding of what high-quality data looks like for manipulation and control tasks is still

---

imprecise. Past work on pre-training visual representations for manipulation and control tasks has focused on collecting large object interaction datasets and developing manipulation-relewe losses. But the generalization ability of such models in comparison to standard pre-training methods is still unknown. The goal of this section is to ask: which models generalize?
To focus analysis, compare models pre- trained for manipulation to two self-supervised ImageNet models and two supervised Ima- geNet models. Our main result is presented in Figure where we plot the average success rate of the learned policies in the training environ- ment distribution, within each class of visual shift, and across all types of visual shifts.
Train Dist.
60
40
20
0
oo DINO
Anpo VIF- (HOI)
MAE IN, VitsS
CuP virerLs
w
SN-SUP
Dmo. vyr
Moto. aN
MoCo vo, VIT
VI-DINO
MoCo vira
Models pre-trained for manipulation. Past work has trained visual representations for manipulation in two ways: by training with manipulation-speciff losses or on data of human-object interactions. We focus on three recently introduced pre-trained models for ma- nipulation that use different combinations of these approaches: Masked Visual Pretrain- ing (MVP) 19), Reusable Representations for Robot Manipulation (R3M) [8], and Value- Implicit Pre- Trainingn (VIP) [17]. We include important characteristies of these models, in- cluding dataset sizes, architecture sizes, and augmentations in Section A.I and Table 1.
RESUP
Test (Avg)
20
15
14
15
151
10-
5
mye. ViT' (HO))
Kaw
MAEE an Vit-s
Cue VT-8/L
We
Dion: v2, wy
Mote AN
Mace vs, Ni
Mode v3, a
Sup
Figure 4: Average success rates for training and test
These models perform strongly within the train- ing distribution: R3M and VIP in particular comfortably beat standard pre-training base- lines. This is expected, especially for R3M which was evaluated on the same training environment. However, under subtle distribution shifts, models designed for manipulation struggle to generalize as well as supervised or self- self-supervised training with ImageNet. This is surprising for a few reasons. First, each manipulation model is trained on a larger dataset than the pre-trained baselines. Ego4D alone is 4.5M frames while Im- ageNet is only 1.2M. By parameter count, MVP is also larger than the ViT-S baselines. Finally, we expect human-object interaction datasets such as Ego4D to be more similar to the distribution of images observed when training manipulation policy. The viewpoints scenes less curated than ImageNet. Although we expect this to improve the generalization of the learned policy, these results show that other factors may supersede the impact of data relevance or scale alone.
distribution across both environments for every model our evaluation suite. The best best-performin model that was designed for manipulation ranks seventh out of all models evaluated.
more varied and the
Supervised ImageNet models. Supervised training on ImageNet has long been a baseline for vi- sual pre-training Past work has found that features learned with supervised learning on ImageNet also strong baseline for control: even frozen features are competitive with ground-truth state information on variety of simulated control tasks [20]. However, Parisi et al. [20] also find that self self-supervised learning outperforms supervised learning. Our results contradict this finding. Fig- ure shows that supervised training on Stylized ImageNet achieves a higher success rate in the training distribution than self-supervised training on ImageNet with a masked auto-encoding loss. These models maintain the same rank out-of-domain as well. Even without stylization, in-domain performance of supervised ImageNet models are competitive with models trained with MAE on FrankaKitchen. From these results, conclude that the presence of supervision is not predictive of in-domain or out-of-domain performance as other factors. We also find that supervised ImageNet training is still strong baseline for model generalization: VIT-INSUP outperforms R3M and MVP
5

---

va OOU (Avg)
Imagetket 000 (Pug)
10
Acurag 0g0 Succiss
1.
12
12
10
ViT' rhow0.73)
(1P-0.077, dhoe0.211
Resfttts m0300 00000.00
Resliets (R'=0.00, mhom-0 50)
Thin Dist.
thagellet
Shape Bies va OOD (Avg)
Jeccard va OGD (Avg))
20
?
20
18
48
Arsnage O2D Succoss
Avanaga
VITs rhom-0.18)
Restlets (R'-0.95 99, rha-1.00)
VITs (R"-0.65, rha-0.90)
Shape Blas
jeccart Index
WIEDING
CUP. VIFICC
Dino
MAB-N, VIT.S
McCo.
VIT
v2.
Figure 5: We plot the relationship between different metrics and out-of cittof-diitribution (OOD) generalization. There is a promising correlation between shape-bias and OOD performance for ResNets, but not ViTs. Instead, OOD performance ViTs is strongly correlated with Jaccard index.
Self-Supervised ImageNet Models. In Figure 3 include two self-supervised VIT-S models. Un- der visual distribution shifts, the model trained with the DINO objective outperforms all three mod- els that are designed for manipulation. Moreover, this trend holds for every distribution shift except Meta-Words with distractors. The distractors evaluation suite averages over different levels of dis- tractions and therefore favors models with a high performance training. In Appendix Section A.9 we plot model performance across different levels distractors and find that several self-supervised ViTs experience smaller drop in performance as more distractors are added compared ResNet based pre-trate pre-trained manipulation models like R3M and VIP
Training with masked autoencoding performs well under distribution shifts in Meta-Worlds World, but is less strong under distribution shifts within FrankaKitchen. In Figure 4, we see that MoCo. v3, ViT-B also performs strongly out-of-distribution. When we compare MoCo and DINO against MAE-style training we see that MoCo and DINO use more extensive set of augmentations, Taking this into account alongside the observation that ViT trained with supervision on Stylized ImageNet per- forms well out-of-distribution conclude that choice of augmentations outweighs the importance of supervision. This extends the findings of Geirhos et al. [10] to the setting of robust manipulation.
VITs vs ResNets. One important design choice when selecting a pre-trained model is the choice of architecture. In all of our experiments, we use ResNet-50 [62] to consistent with past work on visual pre-training [20, 8, 17]. Vision transformers (ViTs) [63] have seen widespread adoption within computer vision [64], but have only recently been used for learning representations for control [9]. We find that, average, ViTs have slight edge on out-of-distributin generalization compared to equivalently trained ResNets. In Figure 6, out of the seven pre-trained models that perform best out-of distribution six are ViTs. Ablating architecture alone while holding dataset, training augmentations, and parameter count constant, we can compare the model pairs "MoCo. v3, RN and "MoCo. v3, VIT", "RN-DINO" and "ViT-DINO", and "RN-INSUP" and "VIT-INSUP" In the latter two pairs, the ViT variant is much stronger out-o--distributtinn than the ResNet variant. For MoCo, the variants achieve similar performance out-of-distribution.
Properties of Robust Visual Representations for Manipulation
Summary. This section identified which pre-trained models generalize, with several interesting findings. First, models designed for manipulaiton do not necessarily perform well under subtle distribution shifts in comparison to more standard pre-training methods. Second, the presence or

---

absence of supervision does not matter as much as other factors on both in- and out-of of-distribution generalization. Finally, ViT' have slight edge over ResNets in -it-o--dittribution generalization.
Our findings in the last section are both surprising and somewhat unsatisfying because they contra- dict many of our intuitions about scale and generalization. In our evaluation suite, saw that better generalization is not cleanly explained by more data, bigger models, more relevant data. The goal of this section is to identify the properties of pre-trained models that are predictive of general- ization. To that end, we correlate out-of distriibbttttoinnn performance with three metrics that have been previously connected to generalization in the machine learning and computer vision literature- in-domain performance. accuracy linear probe trained on ImageNet, and shape-bias. We also include fourth metrie, which is specific ViTs: the emergent segmentation accuracy of the output attention heads. We describe each metric in detail in Section 5.1, discuss our setup for correlating performance in Section 5.2, and analyze our results in Section 5.3.
5.1 Metrics
ID OOD. One of the goals of this paper is to understand how well the findings from existing evaluations of pre-trained models hold under the inevitable environment changes that we expect to see in real-world settings. If in-distribution performance is reasonably predictive of generalization, it is sufficient for researchers to continue developing pre-trained models with existing methods of ovaluation. Past work has also shown that the in-distribution performance of a pre-trained model positively correlated with outto--distributiin performance for a variety of computer vision tasks [12]. Concretely, we measure in-distribution performance the success rate of the policy within the training distribution.
Success with Increasing Distractors
Imagenet vs OOD. Training linear probes on Imagenet is a common protocol for evaluating the quality of learned representations [65, 11]. Hu et al. [32] make the related finding that the ImageNet k-NN accuracy of pre-trained model is predictive of performance on imitation learning with visual reward function. We evaluate ImageNet val- idation accuracy for all models with linear probes.
Avg Success
30
Jaccard Index Index
20
20
10
10
Shape-Bias vs OOD, Shape bias is the extent to which a model makes prediction decisions based on shape. We calculate shape bias the percent of shape (as opposed to texture) classification decisions on the validation set [13] using the same probes described above.
1
Level of Distractors
3
9
INSUP
Dino. v2, VIT
VIT-DINO
MVP, VIT-S (HOI) CLIP, VIT-B/16 MAE-IN, ViT-S
SIN-SUP
MVP
MoCo, v3, VIT
MoCo. v3, VIT-B
Jaccard vs OOD, Finally, for all of the ViT models, we look at the emergent segmentation performance. We eval- uate the Jaccard index of an interpolated attention map averaged across heads in the last attention block the [CLS) token.
Figure 6: What happens to models with high Jaccard index under object-level distribution shift? Surprisingly, the models with the highest Jaccard index maintain the highest performance as the number of dis- tractors increases.
5.2 Sctup
We measure the coefficient of determination (R²) and Spearman's 's rank correlation (p) for the corre- lation between the duitrf-distribution success rate and each metric described above. Our goal is to find a metric that will result in high correlation between the metric and the OOD success, iee. both coefficients being close to 1.0. We fit separate trend lines to ViT' and ResNets. Because of the lack of available probes, we exclude MVP, MVP VIT-S HOI, R3M, VIP, and MAE-IN ViT-S from the shape bias and ImageNet probe correlations. Each point represents one of the 15 pre-trained models we evaluated and represents the average of 6,000 evaluation runs.
5.3 Results
We visualize the correlation between each metric and the average out-of-distribution success rate in Figure 5, Although a positive relationship between in- and out-of distribution generalization, there are pre-trained models that notably deviate from this trend. Among VIT models one example is
7

---

MVP, VIT-S (HOI); the average success rate of this model drops to 6.63 from 39.86. By contrast, we find that ImageNet accuracy of a linear probe poorly predicts generalization performance for ViT's.
We also see little correlation between shape-bias and OOD performance for VIT models, but promisingly strong correlation the subset of ResNets evaluted. This is surprising because humans make highly shape-biased decisions and increasing shape-bias increases the robustness of imagenet trained CNNs [13, 10]. One explanation of this finding is that the VIT architecture obviates the need for shape- shape-biased biased features. For example, ResNet-50 trained with the DINO training scheme has strong shape-bias, but not the equivalent ViT model.
Finally, we visualize the relationship between the Jaccard index and OOD performance on all ViT models in Figure 5. There is strong positive correlation between Jaccard index and OOD perfor- mance both in terms of rank correlation and the coefficient of determination. These results suggest that while shape-bias may not be predictive of the OOD generalization ability of pre-trained ViT, the segmentation ability is a predictive alternative.
One counter-argument to the use of Jaccard index as a metric for for OOD performance is that would be less predictive for object-level distribution shift, which would occur any time a large distractor is placed in the background of the image. In Figure 6, plot the success rates of each VIT model as the number of objects increases and verify that the models with the higher Jaccard index actually maintain the highest performance as the number of distractors increases.
5.4 Validating in the real world
In this section, we validate our finding on a real-world generalization scenario by comparing a ViT-B model designed for control (MVP) against a model not designed for control but with a high emergent segmentation score (MoCo-v3).
Setup. We learn policies for picking up screwdriver on the ALOHA setup using the ACT train- ing framework [18]. We follow the standard ACT training paradigm with further details listed in Appendix Table 3. From the training data to the test runs there is a distribution shift in both the placement of the target object (the screwdriver) and in the direction of the lighting. We calculate success on screw pick ups averaged over 10 rollouts the test environment.
Results. We find that MoCo-v3, with a success rate of 40% is able outperform MVP, with success rate of 0%, even though it is not explicitly designed for manipulation. Qualitatively, the MVP model fails in localizing the object when attempting the grasp, whereas MoCo-v3 model reliably localizes the object, but experiences more failure in finding the right grasp point.
6 Conclusion
In this paper, we uncover a recipe for generalization: ViT models with a high emergent segmen- tation accuracy generalize well under visual distribution shifts. Emergent segmentation accuracy not only a stronger predictor of generalization than many other metries for robustness, but also requires no additional training to evaluate. This insight can guide the development of pre-trained vi- sion models in future work: preferring architecture training algorithms that lead strong emergent segmentation opposed to only training more camipulation-relevaae data.
Limitations and future work. One limitation of this work is that our metric for predicting the ro- bustness of visual representations is specific to VIT architectures. We believe that this an artifact of the fundamental differences in the way representations are learned VIT and ResNet architectures. One direction for future work is better understanding the conceptual connection between "shape- bias" in convolutional networks and the "segmenting features" in VIT models. The majority of our evaluation is in the theoretically well-understood, but still limited setting of training small multi- layer perceptrons top of frozen encoders. An important direction for future work is to expand these findings with more complex decoders and with fine-tuning.

---

References
[1] X. Chen*, S. Xie*, and K. He. An empirical study of training self-supervised vision trans- formers. arXiv preprint 1rXiv:210..02057, 2021.
[2] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. ArXiv, abs/1812.00231, 2018.
[3] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to bench- mark reinforcement learning. ArXiv, abs/1912.01588, 2019.
[4] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti,
J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
[5] K. Grauman, A. Westbury, E. Byme, Z. Q. Chavis, A. Furnari, R. Girdhar, J. Hamburger,
H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fue- gen, A. Gebreselasie, González, J. M. Hillis, X. Huang, Y. Huang, W. Jia, W. Y. H. Khoo, Kolár, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, Mangalam, R. Modhugu, Munro, T. Murrell, T. Nishiyasu, W. Price, P. Puentes, M. Ramazanova, Sari, K. K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbeláez, D. J. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawa- har, H. Joo, K. Kitani, H. Li, R. A. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the world in 3,000 hours of egocentric video. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18973-18990, 2022.
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchi- cal image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10::10.11
[7] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-6) 2020. URL http: org/ atg/papers/v21/ 20-074.htt
[8] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
[9] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv:2203.06173, 2022.
[10] R. Geirhos, K. Narayanappa, B. Mitzkus, T. Thieringer, M. Bethge, F. A. Wichmann, and W. Brendel. Partial success in closing the gap between human and machine vision. In Neural Information Processing Systems, 2021.
[11] T. Chen, S. Komblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations, ArXiv, abs/2002.05709, 2020.
[12] Miller, Taori, A. Raghunathan, Sagawa, W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy on the line: the strong correlation between out-of-distributtion and in-distribution generalization. ArXiv, abs/2107. 0os/2107.00449, 2021.
[13] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and ro- bustness. In International Conference Learning Representations, 2019. URL https:
// /ppenreview wwwt/eeee//teeeee nett/forrm/iddByyyyjo0oo

---

[14] E. Xing, A. Gupta, S. Powers", and V. Dean Kitchenshift: Evaluating zero-shot gener-
alization of imitation-based policy learning under domain shifts. In NeurIPS 202/ Work-
shop on Distribution Shifts: Connecting Methods and Applications, 2021. URL https:
//openreview.o wtt..e/e/ooruri.ddddgg.goooe.eee
[15] A. Xie*, L. Lee*, and C. Finn. Benchmarking enviromment generalization in robotic imitation learning, 2023. URL httppp///ggaaab....oaa/gg//////e cct.///LLggent/aacttroooooo
[16] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and 'T. Darrell. Real-world robot learning with masked visual pre-training. CoRL, 2022.
[17] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210000330, 2022.
[18] T. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine- fine-grained bimanual manipulation with low-cost hardware, 2023.
[19] J. Pari, N. M. M. Shafiullah, S. P. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. ArXiv, abs/2112.01511, 2022.
[20] S. Parisi, A. Rajeswaran. S. Purushwalkam, and A. K. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, 2022.
[21] R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fründ, N. Yianilos, M. Fueiter-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The "something something" video database for learning and evaluating visual common sense. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5843-5851, 2017.
[22] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands in contact internet scale. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9866-9875,
[23] L. Yen-Chen, A. Zeng. S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In JEEE International Conference on Robotics and Automation (ICRA), 2020. URL https: httss /tiia::s//yenchenltt in. melvissionnattion/
[24] S. Karameheti, S. Nair, A. Chen, Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for roboties. arXiv preprint 2rXiv:2302.127660 2023.
[25] A. Sax, B. Emi, A. R. Zamir, L. J. Guibas, S. Savarese, and J. Malik. Mid-level visual rep- resentations improve generalization and sample efficiency for learning visuomotor policies. 2018.
[26] Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and Jayaraman. Liv: Language-image representations and rewards for robotic control. arXiv preprint arXiv:2306.00958 2023.
[27] S. Chen, R. Garcia, I. Laptev, and C. Schmid. Sugar: Pre-training 3d visual representations for robotics. In CVPR, 2024.
[28] A. Kirillov, E. Mintun, N. Ravi, Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:230.002643, 2023.
[29] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for inforcement learning. Proceedings of the 37th International Conference Machine Learning, Vienna, Austria, PMLR 119, 2020. arXiv:2004.04136.
10

---

[30] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https: /ttp////pppe/eeee/////ee// met/forumiid=GiVV 6s.tffff..
[31] A. Majumdar, K. Yadav, S. Armaud, Y. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier. Where we in the search for an artificial visual cortex for embodied intelligence? 2023.
[32] Y. Hu, R. Wang, L. E. Li, and Y. Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal, 2023.
[33] Dasari, M. K. Srirama, Jain, and A. Gupta, An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning. PMLR, 2023.
[34] R. Shao, Z. Shi, J. Yi, P.-Y. Chen, and C.-.. Hsieh. On the adversarial robustness of vision transformers. Transactions Machine Learning Research, 2022. URL https: // //openreview net/forum?id-117Kan
[35] Y. Fu, S. Zhang, S. Wu, C. Wan, and Y. Lin. Patch-fool: Are vision transformers always robust against adversarial perturbations? In International Conference on Learning Representations, 2022. URL https: /ttp////eeeeeeeeee/eeeeeeee/eeee http://eeeeeeeeeeeeeooetaaaaaaaa new.:forun2.dd22119tt....
[36] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In International Conference Learning Representations, 2019. URL
[37] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A crit- ical analysis of out-of-distribution generalization. In ICCV, pages 8320-8329, 2021. URL https:/ htops://doi. org/10. 1109/ICCV48922. 2021.
[38] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CVPR, 2021.
[39] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterthiner, and A. Veit. Understanding robustness of transformers for image classification. In 202/ IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 1021 1-10221, 2021. doi: 109/ICCV4892..222... 01007.
[40] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. Khan, and M.-H. Yang. Intriguing properties of vision transformers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openrevion htt/forummdd==2mll
[41] B. Huang, F. Feng, C. Lu, S. Magliacane, and K. Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. ArXiv, abs/2107.02729, 2021.
[42] R. Raileanu, M. Goldstein, D. Yarats, I. Kostrikov, and R. Fergus. Automatic data augmenta- tion for generalization in deep reinforcement learning. ArXiv, Abs/2006.12862, 2020.
[43] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. /4990, 2020.
[44] K. Cobbe, C. Hesse, J. Hilton, and Schulman. Leveraging procedural generation to bench- mark reinforcement learning. arXiv preprint arXiv: 2019.
[45] C. Packer, K. Gao. J. Kos, P. Krähenbühl, V. Koltun, and D. X. Song, Assessing generalization in deep reinforcement learning, ArXiv, abs/1810, 122.22 2018.

---

[46] J. Farebrother, M. C. Machado, and M. H. Bowling. Generalization and regularization in dqn. ArXiv, abs/1810.00123, 2018.
[47] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktaschel. A survey of generalisation in deep reinforcement learning. ArXiv, abs/2111.09794, 2021.
[48] C. Zhao, O. Sigaud, F. Stulp, and T. M. Hospedales. Investigating generalisation in continuous deep reinforcement learning. ArXiv, abs/ 000010000,1 2019.
[49] N. Hansen and X. Wang. Generalization in reinforcement learning by soft data augmentation.
In International Conference on Robotics and Automation, 2021.
[50] L. Fan, G. Wang. D.-A. Huang, Z. Yu, L. Fei-Fei, Zhu, and A. Anandkumar. Secant:
Self-expert cloning for zero-shot generalization of visual policies. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3088-3099, PMLR, 18-24 Jul 2021. mlr.
URL https::/procceei///
[51] T. Yoneda, G. Yang, M. R. Walter, and B. Stadie. Invariance through latent alignment, 2021.
[52] N. Hansen, R. Jangir, Y. Sun, G. Alenyå, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang. Self- supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021.
[53] Sadeghi and S. Levine. Cad2rl: Real single-iman flight without a single real image. ArXiv, abs/161 1.04201, 2017.
[54] Tobin, R. Fong. A. Ray, J. Schneider, W. Zaremba, and Abbeel. Domain randomiza- tion for transferring deep neural networks from simulation the real world. 2017 IEEE/RS, International Conference on Intelligent Robots and Systems (IROS), pages 23-30, 2017.
[55] X. B. Peng, M. Andrychowiez, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamies randomization. 2018 IEEE International Conference Roboties and Automation (ICRA), pages 1-8, 2018.
[56] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canoiiee adaptation networks. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12619-12629, 2019.
[57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, 0os/2103.000200 2021. URL https://arxiv org/ 2bs0000000000.
[58] A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning distort pre- trained features and underperform dut-of-distiibution In International Conference on Learn- ing Representations, 2022. URI. /
[59] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, pages 1025-1037, PMLR, 2020.
[60] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https: //tpxa/aivv////////////a///aaai/ . org/ abs//910.. 10897.
[61] B. Calli, Singh, Walsman, Srinivasa, Abbeel, and A. M. Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference Advanced Robotics (ICAR), pages 510-517, 2015. doi: 109/ICA..205. 7251504.
12

---

[62] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[63] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con- ference on Learning Representations, 2021. URL https: httts://openreviowt wtt.://wwwee/fomm/iiiii YicbFaNTTy.
[64] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers vision: A survey. ACM Comput. Surv., 54(10s), sep 2022. ISSN 0360-0300. doi:10.1 1051335544/// URL https: /doii org/10 10. 1145/3505244.
[65] K. Hc, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conferencee on Computer Vision and Pattern Recognition (CVPR). pages 9726-9735, 2019.
[66] A. Paszke, Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, Gimelshein, Antiga, Desmaison, A. Kopf. E. Yang. De- Vito. M. Raison, Tejani, S. Chilamkurthy, B. Steiner, Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. URL http://papers. httpppppprrrr.rrs.s..eeo.eom 9015-pytorch- an- imperative-steee highle-hhigh-eefrraacccceee deeerre ceaemiinngibbbbrry library. pdf.
[67] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data- efficient image transformers; distillation through attention. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 Pro- ceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021. URL bttts://procestt
mlr. prress/v13//oovve/
html.
[68] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Confer- ence on Computer Vision (ICCV), 2021.
[69] Sermanet, C. Lynch, Chebotar, Hsu, E. Jang. S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. Proceedings of International Conference in Robotics and Automation (ICRA), 2018. URL http::/ wrgg abs
[70] K. He, X. Chen, S. Xie, Li, P. Doll'ar, and B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979 15988, 2021.
[71] O. Nachum, B. Dai, Kostrikov, Chow, L. Li, and Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02072 2019.
[72] M. Oquab, T. Darcet, Moutakanni, H. V. Vo, M. Szafraniec, Khalidov, P. Fernandez, D. Haziza, F. Massa, El-Nouby. Howes, P.-Y. Huang, H. Xu, Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. LLbatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[73] T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In Proc. CVPR, 2020.
[74] A. Gupta, Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horize tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11995 2019.
13

---

[75] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, F. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chat- terji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, Ermon, Etchemendy, K. Ethayarajh, L. Fei-Fei, Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, Grossman, N. Guha, Hashimoto, P. Henderson, Hewitt, E. Ho, Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchan- dani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan,
Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, Tramèr, R. E. Wang, W. Wang, B. Wu, Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang. L. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models, 2022.
14

---

A Appendix
A.1 Pre-Trained Model Details
RN-INSUP [62] is ResNet model trained on the ImageNet classificaiton task. We use the default weights and model provided by the Pytorch [66] library.
ViT-INSUP Vision Transformer [63] that has been distilled [67] from larger network that was trained on the ImageNet classification task. our experiments, we use the model weights and architecture provided in Naseer et al. [40] with a patch size of 16.
SIN-SUP [40] trains a vision transformer on Stylized Image-Net (SIN) [13]. The SIN dataset was constructed to increase the degree to which a model makes predictions on shape instead texture. Our model weights come from Naseer et al. [40] and we use the non-distilled DeiT [67] training variant.
VIT-DINO [68] is trained with extensive augmentations and a self-supervised, contrastive loss that together lead to emergent segmentation within the self-attention heads of the ViT model. We use the model and weights provided by Caron et al. [68]. Interestingly, we don't find the DINO objective to lead to a high shape-bias. This suggests that there are other metrics that measure the degree to which a model is object-centric other than shape-bias.
ResNet5O-DINO is learned with the same recipe VIT-DINO. We use the model and weights from Caron et al. [68].
MoCo. v3, RN [1] leverages a contrastive loss with momentum encoding [65] of positive targets. trained with the same recipe as MoCo. v3, ViT-B.
MoCo. v3, VIT-B [1] are trained in similar manner as the original MoCo [65], but with changes to improve the stability of training, which are specific to the VIT archiecture. We use the checkpoint after 300 epochs.
MoCo. v3, VIT-S [1] trained a similar manner as MoCo. v3, ViT-B. Even though the smaller model benefits from a longer training horizon, we use the checkpoint at 300 epochs for consistency.
MAE-IN, VIT-S follows the same training recipe as MVP, but on top of the ImageNet dataset. We use the weights provided by Radosavovic et al. [16].
R3M [8] trains a ResNet model with a combination of manipulation-sppeiiic losses-including a time-contrastive loss [69], video-language alignemnt loss, and LI-regularizzaioon-n the Ego4D [5] dataset.
MVP [16] trains a ViT-B for masked autoencoding (MAE) [70] on the Ego4D [5], Something- Something [21], YouTube 100 Days of Hands [22], EpicKitchens [4], and ImageNet [6] datasets.
Unlike R3M, the model is not designed to be exclusive to manipulation.
MVP, ViT-S (HOI) [9] is predecessor of the model described above that trains VIT-S/16 with an MAE objective
Sooeetiinnnn [21], YouTube 100 Days of Hands [22], EpicKitchens
[4], and ImageNet [6].
VIP [17] uses an action-free dual of the Algaedice [71] objective to learn representations that are useful for trajectory optimization or reinforcement learning of control tasks unseen during represen- tation pre-training. They train ResNet-50 on Ego4D with this objective.
CLIP, VIT-B/16 [57) uses contrastive language Language-image pre-training to learn visual representations trained on an extensive internet datsact. The learned models exhibit strong zero-shot performance for multiple tasks such as image classification.
DiNo v2, ViT [72] scales Caron al. [68] more parameters and larger dataset. The full model is IB parameter ViT trained on LVD-142M, which is 142M frame datasct composed of ImageNet- 1k, ImageNet-22k,, Google Landmarks [73], and a collection of other datasets spanning fine-grained classification, segmentation, depth estimation, and retrieval. The full model is distilled into smaller
15

---

models. We select the VIT-S distilled model for our experiments. In Table 1, we list the augmen- tations used on the teacher model. The training loop is only lightly modified during distillation.
Suprisingly, the v2 model sees worse in- and out-of-domain performance on our evaluation suite in spite of being distilled from a ladrger model trained on bigger dataset.
Name
Loss Function
Architecture
Datasels
Augmentations
BCE.Los
ResNet-50
ImageNet
Random crop.
(23M params)
(1.2M frames)
Horizontal flip
VIT INSUP
Loss
VIT-S/16
ImageNet
Random crop.
(22M params)
2M frames)
Horizontal flip
SIN SUP
BCE.
V/T
Stylized ImageNet
Random crop.
(22M params)
(1 (1.2M frames)
Horizontal flip
RrsNoCO-DINO
Distillation
ReaNet-50
ImageNer
Muli-erop,
(23M params)
(1.2M frames)
Color-jitering.
Gaussia bluz.
Solarization
VIEDINO
Distillation
VIF-S/16 (22M params)
ImageNet
Multi-rrpp.
(1.2M frames)
Color-jiteriig.
Gaussian blur,
Solarization
MoCo. V3,
Contrastive
ResNet5U (23M purams))
ImageNet (1.2M frames)
Resize,
Color Coler-jiltering
Horizontal fip.
Gmyscale,
Gaussian blur, Solarization
MoCo. v3, VITS
Contrastive
V/16 S/16 (22M params)
VIT
ImugeNet (1.2M frames)
Resize, Colar-jitering.
Horizontal flip. Grayacale.
Gaussian blur, Solarization
MoCo, WI-B
Contrastive
VIT-B/16 (8RM params)
ImageNer (1.2M framos)
Resize,
Color-jitcriin.
Horizonial llip. Grayscale,
Gaussian blur,
Solarization
MAE-IN, VII-S
Masked auto-cacoding
VIF-S (22M paramss)
ImageNNt (1.2M frames)
Random reaize,
Random crop
R3M
Time-contrastive,
ReaNet-50
Ego4D
Random crop
LI regularization,
(23M purams)
(4.3M frames)
Vidco-lang alignment
MVE WTT.S (HOI)
Masked muto-encoding
VTT-$ (22M params)
EpicKitchent
None
100 Days Hands, Something Someiin-n
(700k frames)
Masked auto-encoding
VIT-B
Ego-D, ImageNet
None
(88M params)
Epic Kitchens,
100 Days Hands,
Something Something
(4.5M frames)
Algaedice Dual
ReaNet-50
Ego-ID
crop
(23M params)
(4.3M frames)
CLIP, VIT-/16
Contrastive
VIT-B/16
Internet data
Random crop
(88M params)
(400M pairs)
DiNo v2, VIT
Disillation
VIT
L.D
Multi-ery Color-jinering, Grayscale, Gaussian blur. Solarization
(21M params)
(142M frames)
Table 1: List of pre-trained models with corresponding loss function, augmentations, and datasets used for pre- training. color code by the data loss type: ImageNet supervised, self superviedd, trained specifically for manipulation or control tasks, and other.
A.2 Details of the Environments
FrankaKitchen [74] is simulated kitchen environment with a 9-DoF Franka robot. There a mul- tiple household objects available for interaction. The environment is designed to compose tasks together hierarchically, but focus on learning policies to successfully complete single task. The episode length is 50 and inherit the randomization scheme used in R3M, which randomizes the position of the kitchen at the start of each episode.
16

---

(Lighting)
Test (Textures)
(Distractons)
Dark
Bright
Woced
Mosal, Slue
Medium
Hard
Figure 7: We visualize each distribution shift from the left camera angle on the FrankaKitchen (top) and Meta- World (bottom) environments.
Meta- Meta-World [60] is simulated manipulation environment that consists of various table-top ma- nipulation interactions. Unlike FrankaKitchen, the scene objects vary between different tasks. The positions of the objects are randomized at the start of each episode. The maximum episode length is 500.
A.3 Details of the Disribution Shifts
Each distribution shift visualized from the left camera angle in Figure 7, We don't use the MuJoCo scanned object dataset that is used in [15] because of imperfections in the coloring of the textures.
A.4 Policy Training Details
Hyperparameter
Value
Loss type
MSE
Learning rate
0.001
Batch size
32
Train steps
20,000
Optimizer
Adam
Table 2: Hyperparameters for IL Training
We learn 2-layer MLP on top of the pre-trained, frozen features with 10 demonstrations. We use the same expert demonstrations as in R3M. We train policies independently over the 'left_cap2' and 'right cap2' camera angles and show results averaged over both camera angles. We also provide proprioception to the policy. The final performance is averaged over the task settings for each seed. The hyperparamters for policy training are summarized in Table 2. Error bars are 95% confidence interval over seeds.
A.5 OOD Perf Details
To provide more granular understanding of how the complete set of models performs our evaluation suite, we break down performance by distribution shift type and environment in Figures 8 and 9.
A.6 ImageNet vs OOD Details
To evaluate ImageNet accuracy, we use publicly available probes that have been trained on top of the frozen model features and evaluate them on the ImageNet validation set. The models with available probes are INSIP RN-DINO, MoCo. v3 RN, VIT-INSUP, ViT-DINO, MoCo. v3 VIT, Dino v2 VIT, MoCo. v3 VIT, SIN-SUP, and CLIP V/T-B/16 and we use the probes that are provided in the implementations cited in Section A.I.
A.7 Shape-Bias Details
We evaluate shape-bias using the 'model-vs-human' evaluation framework from Geirhos et al. [10] and use the same probes from Section A.6 to get classification results the SIN validation dataset (Deue-comfice
True-come flict)...
17

---

Frankakitchen Train Dist.
Frankakitchen Test (Lighting)
40
20
139
830
0.45
0.3
RL. NTT
de 1922
So 101
WTT
FrankaKichen Test (Texture)
Frankakitchen Test (Distractors)
20
10
A11
BuN 422
2,60
2.57
MEC V982
us M059
Frankakitchen (Aug)
1ha au 1857 11 thes 18p5
2,24 aan
PaCU 43 10
4n Nit
Figure 8: Detailed OOD Performance on FrankaKitchen.
Meta- Wordd Train Dist:
Meto World Test (Lighting)
40
30
27
2017
40
70
4.4
wuth W0th meddo
101
(lexture)
Meta World Test (Distractors)
30.0
I5e t
1.5
5.0
20
2.5
0.5
1.29
212
MoCo va 2i
Mue WIT 2eot
World Test (Aug]
1a 2a 2617
wv vis mwen
Figure 9: Detailed OOD Performance on Meta-World.
Notably, Naseer et [40] find that vision transformers are more shape-biased when making classi- fication decisions than equivalently trained convolutional networks. our results, we don't find vi- sion transformers to be more strongly shape biased. Vision transformers and convolutional networks vary in how they handle spatial resolution: spatial resolution decreases in each layer of ResNet-50
18

---

but remains constant within VIT. This could explain why we see the VIT architecture somewhat obviating the need for shape-bias in our results.
A.8 Jaccard Index Metric Details
We denote this nonlinear, deterministic transform as M. Formally, we compute the Jaccard index by calculating the mloU on PASCAL VOC validation set, D Pascati
An B AUB
[z,, 22)) ED Pasca!
Where A is a shorthand for positive classification for the target class by M(b()) and B is short- hand for positive label for the target class. J is evaluated pixel-wise over image indices 2; and Ij-
A.9
Different Levels of Distractors
FrankaKitchen
Meta WoradWorld
Average OOD Success
Average OOD Success
30
30
20
20
10
10
0
0
1
3
9
1
3 Level of Distractors
Level of Distractors
RN-INSUP
SIN-SUP
MoCo. v3, ViT
CLIP, ViT-B/16
RN-DINO
VIP
MoCo. v3, RN
MAE-IN, ViT-S
VIT-INSUP
MVP
Dino. v2, VIT
MoCo. v3, ViT-B
VIT-DINO
R3M
MVP, VIT-S (HOI)
Figure 10: Different levels of distractors.
We extend Figure by including results for ResNets Figure 10. Models are color coded using the original color scheme in the paper.
A.10
Finetuning
FrankaKitchen (Fine-tuning)
63,50
MVP
R3M
VIT-DINO
60
Success Rate
40
37,94
34,00
2801
20000
20
JA.22
4:59
3.41
06t
Train Dist.
Test (Lighting)
Test (Texture) Test (Distractors)
Test (Avg)
Figure 11: Finetuning in FrankaKitchen.
Because the goal of this paper is to probe the quality of learned representations, follow the tradition of performing evaluation top of frozen model features. This evaluation is also consistent
19

---

with the increasing view of pre-trained visual representations as "foundation models" [75, 72] that can be deployed without any gradient updates. Nonetheless, even in the fine-tuning regime, in Figure 11 we still see stronger performance from models that are not designed for manipulation. In this setting, we increased the number of demonstrations to 25 allow for more data diversity when training the encoders.
A.11 Real-World Experiment Details
Our demonstration data contains two subtasks: an initial screwdriver pick-up and then a handover that happen in sequence. We only evaluate success on the subtask of picking up the screwdriver. The training dataset is comprised of 50 episodes collected by an expert human demonstrator, Images are collected from 4 camera view points (one each wrist, replace the standard encoder with ViT-B and change the initialization of the encoder based on the experimental condition (i.e., we select for different pre-trained model).
top camera, and one front camera). We
Hyperparameter
Value
Chunk Size
100
KL. Weight
10
Batch size
8
Epochs
10,000
Optimizer Adam
Learning Rate
le-5
Table 3: Hyperparameters for Policy Training
A.12 Additional Real-World Experiments
We also compare MVP and MoCo-v3 on additional three real-world tasks and evaluate ten roll- outs across three different visual distribution shifts. For each task, we collect 30 demonstrations. Our training protocol is identical to Appendix Section A.11. I. However because these tasks are more challenging than the screw pick-pu task, we train for 20,000 total epochs with a batch size of 16. The world tasks are:
Corn Plate. In this task, arm must pick up plastic corn and stack it top of plate the table.
Carrot on Plate. In this task, one arm must pick up plastic carrot and stack it on top of a plate on the table. This is slightly more difficult than the corn plate task because the plastic carrot comes to narrow point that is difficult to grasp.
Fork Hand-Over. This is a long horizon task in which the right arm of the ALOHA robot must pick up a fork, hand to the left arm, and the left arm must place the fork a cup on the table.
We simulate the three classes of distribution shift as follows:
Lighting We evaluate at a different time of day (evening) than the data was collected (day- time). This setting the closest to the training setting.
Texture. We cover the table with blue construction paper to simulate change in table texture.
We place single plastic piece of fruit to serve as a distracteor object in the center of the table.
Distractors.
To compute success, we assign reward value of one to each of three subtasks -ubtasks-localizingg the target object, completing successful grasp or handover (depending on the task), and placing the target object in the correct location-ann report the percent of reward achieved averaged over 10 trials.
Our results reported in Table 4. We find that the high-segmennation scoring model (MoCo-v3) performs more robustly than the low-segmentation scoring model (MVP) average. On casy tasks
20

---

Task/Distribution Shift
MVP
MoCo-v3
Corn on Plate (Lighting)
100%
100%
Corn on Plate (Distractor)
100%
100%
Corn on Plate (Blue Table)
96%
100%
Corn on Plate (Average)
99%
100%
Carrot on Plate (Lighting)
93%
90%
Carrot Plate (Distractor)
100%
97%
Carrot on Plate (Blue Table)
33%
50%
Carrot on Plate (Average)
75%
80%
Fork Hand-Over (Lighting)
20%
100%
Fork Hand-Ove (Distractor)
0%
96%
Fork Hand-Over (Blue Table)
0%
100%
Fork Hand-Over (Average)
7%
99%
All Tasks (Average)
60%
93%
Table Performance under real-world distribution shifts
and with small distribution shifts, the difference between the two models is slight. However, on the long-horizon task, MVP struggles maintain good performance. Interestingly, MoCo experiences larger drop the short-horizon Carrot on Plate task than the Fork Hand-Over task. We believe this is because slight displacements in the gripper position cause the plastic carrot to slip out from the grasp whereas the fork is more robust to grasp position.
21


Reviews:
[{"review_id": "GzYEd8jLPN", "reviewer": "Reviewer_hDPj", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the generalization capabilities of different visual encoders and pre-training methods for downstream policy learning to solve manipulation tasks. Furthermore, they evaluate different metrics to predict the policy success rates under visual distribution shifts. The authors compare 3 models that are pre-trained on manipulation-specific and 4 models that are trained on generic computer vision datasets. The pre-trained models are then frozen and the behavior cloning is trained on top of them using demonstrations from two common simulation benchmarks.", "summary_of_recommendation": "Due to my main concern relating to the simulation-real-world visual gap, I recommend rejecting the paper. If authors can show more results in the real world, I'm inclined to change my decision towards acceptance. The model selections and the metric analysis can potentially make this work very impactful for predicting the performance of policies in the real world.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "* The Jacard index has been shown to be a good indicator of policy performance, however, this metric is only applicable to visual transformers. The authors can also calculate the visual attention of the ResNet-based models using grad-cam methods that show the class activations. \n\n* Since segmentation features are shown to be the best prediction metric, it would make sense to add segmentation models (maybe distilled versions). I recommend adding backbones with trained Segment-Anything.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A4N7yo5jok", "reviewer": "Reviewer_nqHm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper compares the generalization ability of 15 pre-trained visual encoders in the visuomotor robot policies in the presence of distribution shifts of images. Their two key findings are: 1) models that are designed for manipulation and control do not outperform standard visual pre-training methods. 2) the emergent segmentation ability of a ViT model is a strong predictor of out-of-distribution generalization performance. They conduct systematic analysis of the relation between the OOD performance and other metrics, such as emergent segmentation ability, downstream ImageNet accuracy,in-domain accuracy, and shape-bias, and conclude that conclude that the emergent segmentation ability is the most relevant metric. In experimental evaluations, the authors show that pre-trained ViT model not designed for control has better generalization both in the simulation and the real world.", "summary_of_recommendation": "The paper is well motivated with a good set of experiments. Still experiments have room for improvement (see list of weaknesses).", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Will the two key findings in this paper remain the same when we replace the MLP action decoder with RNN or transformer action decoders? Could the author(s) provide experimental results or theoretical clarifications about this?\n\n- Could the author(s) provide real-world experiment results on harder manipulation tasks and more drastic distribution shifts to support their conclusion?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MsvadUNGKO", "reviewer": "Reviewer_cqr9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper conducts comprehensive experiments across a wide class of pretrained visual representations to investigate their effectivenes in learning robust manipulation policy through imitation learning. The methods used for pretrained visual representations include supervised training, self-supervised learning, as well as manipulation-driven pretraining. The pretraining architectures include resnet and ViT. Through these experiments, the authors establishes insights into what visual representation leads to improved performance in imitation learning manipulation policies in two classes of simulated environments, as well as a real-world robotics study using the ALOHA setup.", "summary_of_recommendation": "This reviewer believes the insights of this paper is useful for the community. The paper still bears room for improvment to make the conclusion more powerful.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Will the conclusion of this paper still hold in the fine-tuning scenarios? It would be great if the authors could extend the results presented in figure 12.\n\n- It would be interesting to include a baseline where the visual encoder is trained from scratch or can the author explain why it should be excluded? maybe the training cost will be too high compared to using pretrained representations. The current training cost is presumably very low, as the visual signals are encoded once and then only a 2-layer MLP is trained. The visual encoder can potentially be efficiently trained with parameter-efficient fine-tuning, or only fine-tune a subset of the encoder layers. The underlying question is, what are the use cases of pretrained visual representation in robotics learning? Is it mostly highly resource-constraint cases? From this reviewer's opinion in a lot of robotics learning scenarios data could be more constrained, and a bit more cost in fine-tuning the visual encoder could be a price that we are willing to pay. This question links to the first question -- in this reviewer's opinion, the key conclusion of this paper will be a lot more powerful if the results on fine-tuning are more comprehensive so that it can go beyond the frozen visual encoder.\n\n- Maybe the authors could also discuss the above questions in the limitation section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper asks a very relevant question and provides two interesting results: Models trained on manipulation-specific data do not perform better than models trained on vision-only data, and the strongest predictor of out-of-distribution performance is emergent segmentation capability. The study is carefully done and well presented. Nevertheless, it is still unclear how robust these findings are. As Reviewer hDPj notes, other work, not cited here, reaches contradictory conclusions. Findings mostly rely on non-photorealistic simulations; it is unclear how they translate to real robots. Also, results may differ with changes in the setup such as more powerful action decoders or fine-tuned encoders.

During the rebuttal the authors answered most questions convincingly. The one problem that still stands is the real-world significance of the main findings; the additional results are suggestive but far from exhaustive. The main conclusion might be summarized as "For vision-guided grasping tasks, prefer vision-only ViT over manipulation-specific ViT" (presumably because of the vast vision-only training data available). Framed this way it does not look overly surprising but at least it kicks off a discussion of practical value.
Decision: Accept