What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:28:20

Abstract: Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevant features. Although this approach improves the efficiency of policy learning, it remains unclear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual representations designed for control tasks do not necessarily generalize under subtle changes in lighting and scene texture or the introduction of distractor objects. To understand what properties _do_ lead to robust representations, we compare the performance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on a suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.

Main paper: 
 introduction: 
 In spite of vast progress in computer vision, the question of how to learn a good visual representation for robotics remains open (Chen* et al., 2021) . Elsewhere in computer vision, internet datasets are retrofit to new tasks with transfer learning, which promises both generalization and fast adaptation to downstream tasks in exchange for large-scale pre-training. But in the field of robotics, this promise has yet to be fulfilled even though policies learned from pixels struggle substantially with data efficiency (Cobbe et al., 2018) and especially generalization under visual changes in a scene (Cobbe et al., 2019a) .Recent work (Damen et al., 2018; Grauman et al., 2022) posits that the missing piece is a large pre-training dataset of object interactions across diverse environments -the ImageNet (Deng et al., 2009) or CommonCrawl (Raffel et al., 2020) of manipulation. That is, if we want to improve the visual generalization ability of pre-trained models we simply need to collect datasets of this kind at scale. Indeed, training on large datasets of first-person human interaction data increases policy performance and learning efficiency downstream (Nair et al., 2022; Xiao et al., 2022) , but these evaluations occur in environments that are very similar to those used for policy learning. Robotic applications commonly contain environments with varying lighting conditions, scene textures, and background objects, and we want pre-trained representations to allow the robot to handle such variability. Yet we have few concrete measures of how well pre-trained representations generalize outof-distribution. To take a step towards understanding these problems, our goal in this paper is to thoroughly answer the questions "which models generalize?" and "how can we predict how well a pre-trained model will generalize?"Our first key finding is that, when evaluated under visual distribution shifts, models that are designed for manipulation and control do not outperform standard visual pre-training methods. This finding violates our intuitions about what is needed to scale up robot learning and brings into question what constitutes relevant data, how to quantify useful features, and the importance of design choices such as model architecture. In other words, we need more guiding principles to help us understand what representations are good for manipulation and make the problem of iterating on pre-training strategies much more straightforward. Currently, evaluating a pre-trained policy requires training and rolling out downstream policies across multiple environments and experimental conditions. Instead, we can take inspiration from computer vision, which has developed proxies for robust performance on vast out-of-distribution datasets (Geirhos et al., 2021) .Without spatial features accuracy drops 85% with distractors Our second key finding is that the emergent segmentation ability of a ViT model is a strong predictor of out-of-distribution generalization performance. We visualize this phenomenon, which we refer to as "spatial features," in Figure 1 . Other metrics of model quality, such as linear probes on ImageNet (Chen et al., 2020) , and metrics of out-of-distribution performance, such as in-domain accuracy (Miller et al., 2021) and shape-bias (Geirhos et al., 2019) , are not predictive for this model class, despite their predictive power in other commonly-studied domains like image classification. This hints at the possibility that the transfer setting of manipulation differs from computer vision tasks typically studied within the robustness literature.To reach the conclusions above, we run 9,000 different simulated evaluations. Our simulated environments are adapted from two different existing visual distribution shift benchmarks (Xing et al., 2021; Xie* et al., 2023) to capture the shifts that arise commonly in robotics applications: changes in lighting, background and object texture, and the appearance of distractors. More specifically, we train policies on top of 15 pre-trained models, including 4 models designed for manipulation or control: R3M (Nair et al., 2022) , two MVP variants (Xiao et al., 2022; Radosavovic et al., 2022), and VIP (Ma et al., 2022) . We further validate these findings by comparing a model designed for manipulation against a model with a similar parameter count on a real-world screwdriver pick-up task using the ACT training framework (Zhao et al., 2023) .Through these experiments, we make two striking findings: (1) pre-trained visual models designed for control do not necessarily generalize better than models pre-trained on more standard computer vision datasets and (2) the emergent segmentation performance of a ViT model is a strong predictor of the out-of-distribution generalization of a down-stream policy. 
related work: 
 Representation learning for manipulation. The correct approach to visual representation learning for robotics is still an open question. There is evidence that separating visual representation learning from policy learning can further improve performance (Pari et al., 2022; Parisi et al., 2022) . Recent works have shown that models pre-trained on large manipulation-relevant datasets (Goyal et al., 2017; Damen et al., 2018; Shan et al., 2020; Grauman et al., 2022) from RGBD data (Yen-Chen et al., 2020) can improve the efficiency and performance of policy learning (Karamcheti et al., 2023) in comparison to standard vision datasets such as ImageNet (Deng et al., 2009) , but they do not focus on performance under visual distribution shift. We evaluate the performance of R3M (Nair et al., 2022) , MVP (Xiao et al., 2022; Radosavovic et al., 2022), and VIP (Ma et al., 2022) . Other work has studied generalization of pre-trained representations to new reinforcement learning tasks for manipulation (Ma et al., 2022) and navigation (Sax et al., 2018) where the agent is able to train on visual data from the new environment. Separate from the question of pre-training visual representations is the question of how to best train policies on top of pixel observations (Laskin et al., 2020b; Yarats et al., 2021) . Majumdar et al. (2023) benchmarks the performance of pre-trained visual representations on a handful of manipulation environments, but they focus on in-domain performance and also investigate navigation environments. Hu et al. (2023) shows that model performance is highly sensitive to downstream policy learning strategy. We use imitation learning for our evaluation protocol, which they find to be a more stable measure of performance.There is extensive work studying the impact of design choices, such as architecture, loss, and data, on the performance of visual models under distribution shift. See Geirhos et al. (2021) for a comprehensive comparison. Most relevant to our paper are studies of shape-bias and architecture. While shape-biased models tend to be more robust than texture-biased ones (Geirhos et al., 2019) , the impact of architecture on robustness is less straightforward. For example, vision transformers exhibit better robustness to universal adversarial attacks (Shao et al., 2022) , but they are more susceptible to patch-level attacks (Fu et al., 2022) . When compared on natural distribution shifts (Hendrycks & Dietterich, 2019; Hendrycks et al., 2021a; b) , vision transformers and convolutional networks achieve comparable performance when provided with enough data (Bhojanapalli et al., 2021) . But for occlusions specifically, vision transformers appear to have an edge (Naseer et al., 2021) . Miller et al. (2021) studies the predictive power of in-domain performance for out-of-distribution generalization. Unlike all of these prior works, we focus on how pre-trained representations affect robustness in downstream robotics tasks, instead of downstream vision tasks.Learning robust policies. Unlike work that focuses on changes in dynamics or initial state distribution (Huang et al., 2021; Raileanu et al., 2020; Laskin et al., 2020a; Cobbe et al., 2019b; Packer et al., 2018; Farebrother et al., 2018) , we focus exclusively on the setting of visual distribution shifts. Kirk et al. (2021) and Zhao et al. (2019) provide a comprehensive survey on non-visual distribution shifts in decision making problems. Policy adaptation approaches enable visual robustness specifically by leveraging insights from domain adaptation during policy training (Hansen & Wang, 2021; Fan et al., 2021; Yoneda et al., 2021) or during deployment (Hansen et al., 2021) . Other policy adaptation approaches blend pre-training together with reinforcement learning across diverse visual environments (Yuan et al., 2022) . In the special case of closing the sim-to-real domain gap, a popular approach is to add randomized textures while training in simulation (Sadeghi & Levine, 2017 ; Tobin et al., 2017; Peng et al., 2018; James et al., 2019) . By contrast, our work is interested in explaining properties of a robust visual model for control. Consequently, our insights can be leveraged with or without any task specific data. 
environments, evaluation protocol, and pre-trained models: 
 Our goal is to understand how robust existing representations for manipulation are to visual distribution shifts that are realistic in robotic applications. To that end, we learn policies on top of frozen, pre-trained encoders and then evaluate these policies zero-shot under changes in lighting, object and scene texture, and the presence of distractors. These shifts are visualized in Appendix Figure 8 and a high level summary of our evaluation procedure is visualized in Figure 2 . In this section, we describe the specifics of the manipulation environments, distribution shifts, and policy training setups.Environments and tasks. We study ten tasks across two simulated manipulation environments, which are selected based on their popularity in studying learning-based approaches to manipulation. Within FrankaKitchen (Gupta et al., 2020) we evaluate performance on opening a microwave, sliding a cabinet door open, pulling a cabinet open, turning a knob, and turning on a light. Within Meta-World (Yu et al., 2019) we study assembling a ring onto a peg, placing an object between two bins, pushing a button, opening a drawer, and hammering a nail.Distribution shifts. We construct environments to study out-of-distribution generalization within FrankaKitchen and Meta-World. Within FrankaKitchen, we reimplement the texture and lighting changes from KitchenShift (Xing et al., 2021) . Within Meta-World we use texture changes from Xie* et al. ( 2023) and reimplement the same lighting changes as in FrankaKitchen. In both environments we include three levels of distractors: one, three, and nine YCB objects (Calli et al., 2015) . We show average performance on each of these distributions shifts as well as performance on the original training distribution, which samples initial positions of the table and kitchen at random. More details about the implementation and parameterization of the distribution shifts are provided in Section A.3.Policy training is done in the same manner as R3M (Nair et al., 2022) . A summary of the evaluation scheme is provided in Figure 2 . We train an MLP on top of the pre-trained embedding with imitation learning (IL), which, given actions sampled from expert trajectories, a ∼ D train , Figure 5 : We plot the relationship between different metrics and out-of-distribution (OOD) generalization. There is a promising correlation between shape-bias and OOD performance for ResNets, but not ViTs. Instead, OOD performance for ViTs is strongly correlated with Jaccard index.minimizes the mean squared error objective, ||a -â|| 2 2 . Here â denotes the action predicted from a given policy. Details of the training procedure are provided in Section A.4. The embedding weights are frozen during policy learning, so the pre-trained models receive no task data. We train 3 different seeds within each task for each of two different camera angles. In total, we learn 60 policies for each model and perform 11 evaluations per policy, including on the train distribution.Formally, for a pre-trained representation ϕ we learn policies, π ϕ , each trained with a different seed, camera angle, and task. We average the performance of π ϕ along each experimental condition and compute the mean performance and error across seeds.Models. We categorize models by loss type and data source: supervised ImageNet models, selfsupervised ImageNet models, and models trained for manipulation and control tasks. 
generalization of models pre-trained for manipulation: 
 One factor motivating work in learning-based robotics is the hypothesis of scale: if we collect more high-quality manipulation data, we should see improvements in policy generalization. However, our understanding of what high-quality data looks like for manipulation and control tasks is still imprecise. Past work on pre-training visual representations for manipulation and control tasks has focused on collecting large object interaction datasets and developing manipulation-relevant losses. But the generalization ability of such models in comparison to standard pre-training methods is still unknown. The goal of this section is to ask: which models generalize?To focus our analysis, we compare models pre-trained for manipulation to two self-supervised Ima-geNet models and two supervised ImageNet models. Our main result is presented in Figure 3 where we plot the average success rate of the learned policies in the training environment distribution, within each class of visual shift, and across all types of visual shifts. All of the model names as well as the datasets, dataset sizes, model sizes, and loss functions are listed in Appendix Table 2.We recommend that readers visit this table to get a high level view of each model in our comparison suite.Average success rates for training and test distribution across both environments for every model in our evaluation suite. The bestperforming model that was designed for manipulation ranks seventh out of all models evaluated. Models pre-trained for manipulation. Past work has trained visual representations for manipulation in two ways: by training with manipulation-specific losses or on data of human-object interactions. We focus on three recently introduced pre-trained models for manipulation that use different combinations of these approaches: Masked Visual Pretraining (MVP) (Xiao et al., 2022) , Reusable Representations for Robot Manipulation (R3M) (Nair et al., 2022) , and Value-Implicit Pre-Training (VIP) (Ma et al., 2022) . We include important characteristics of these models, including dataset sizes, architecture sizes, and augmentations in Section A.1 and Table 2 .These models perform strongly within the training distribution: R3M and VIP in particular comfortably beat standard pre-training baselines. This is expected, especially for R3M which was evaluated on the same training environment. However, under subtle distribution shifts, models designed for manipulation struggle to generalize as well as supervised or selfsupervised training with ImageNet. This is surprising for a few reasons. First, each manipulation model is trained on a larger dataset than the pre-trained baselines. Ego4D alone is 4.5M frames while ImageNet is only 1.2M. By parameter count, MVP is also larger than the ViT-S baselines. Finally, we expect human-object interaction datasets such as Ego4D to be more similar to the distribution of images observed when training a manipulation policy. The viewpoints are more varied and the scenes are less curated than ImageNet. Although we expect this to improve the generalization of the learned policy, these results show that other factors may supersede the impact of data relevance or scale alone. Supervised ImageNet models. Supervised training on ImageNet has long been a baseline for visual pre-training. Past work has found that features learned with supervised learning on ImageNet are also a strong baseline for control: even frozen features are competitive with ground-truth state information on a variety of simulated control tasks (Parisi et al., 2022) . However, Parisi et al. (2022) also find that self-supervised learning outperforms supervised learning. Our results contradict this finding. Figure 4 shows that supervised training on Stylized ImageNet achieves a higher success rate in the training distribution than self-supervised training on ImageNet with a masked auto-encoding loss. These models maintain the same rank out-of-domain as well. Even without stylization, in-domain performance of supervised ImageNet models are competitive with models trained with MAE on FrankaKitchen. From these results, we conclude that the presence of supervision is not as predictive of in-domain or out-of-domain performance as other factors. We also find that supervised ImageNet training is still a strong baseline for model generalization: in both settings ViT-INSUP outperforms R3M and MVP.Self-Supervised ImageNet Models. In Figure 3 we include two self-supervised ViT-S models. Under visual distribution shifts, the model trained with the DINO objective outperforms all three models that are designed for manipulation. Moreover, this trend holds for every distribution shift except Meta-World with distractors. The distractors evaluation suite averages over different levels of distractions and therefore favors models with a high performance in training. In Appendix Section A.8 we plot model performance across different levels of distractors and find that several self-supervised ViTs experience a smaller drop in performance as more distractors are added compared to ResNet based pre-trained manipulation models like R3M and VIP.Figure 6 : What happens to models with a high Jaccard index under an object-level distribution shift? Surprisingly, the models with the highest Jaccard index maintain the highest performance as the number of distractors increases.Training with masked autoencoding performs well under distribution shifts in Meta-World, but is less strong under distribution shifts within FrankaKitchen. In Figure 4 , we see that MoCo. v3, ViT-B also performs strongly out-of-distribution. When we compare MoCo and DINO against MAE-style training we see that MoCo and DINO use a more extensive set of augmentations. Taking this into account alongside the observation that a ViT trained with supervision on Stylized Im-ageNet performs well out-of-distribution we conclude that choice of augmentations outweighs the importance of supervision. This extends the findings of Geirhos et al. (2021) to the setting of robust manipulation.ViTs vs ResNets. One important design choice when selecting a pre-trained model is the choice of architecture. We focus on ResNets and ViTs. In all of our experiments, we use ResNet-50 (He et al., 2016) to be consistent with past work on visual pre-training (Parisi et al., 2022; Nair et al., 2022; Ma et al., 2022) . Vision transformers (ViTs) (Dosovitskiy et al., 2021) have seen widespread adoption within computer vision (Khan et al., 2022) , but have only recently been used for learning representations for control (Xiao et al., 2022) . We find that, on average, ViTs have a slight edge on out-of-distribution generalization compared to equivalently trained ResNets. In Figure 4 , out of the seven pre-trained models that perform best out-of-distribution six are ViTs. Ablating architecture alone while holding dataset, training augmentations, and parameter count constant, we can compare the model pairs "MoCo. v3, RN" and "MoCo. v3, ViT", "RN-DINO" and "ViT-DINO", and "RN-INSUP" and "ViT-INSUP." In the latter two pairs, the ViT variant is much stronger out-of-distribution than the ResNet variant. For MoCo, the two variants achieve similar performance out-of-distribution.Summary. This section identified which pre-trained models generalize, with several interesting findings. First, models designed for manipulaiton do not necessarily perform well under subtle distribution shifts in comparison to more standard pre-training methods. Second, the presence or absence of supervision does not matter as much as other factors on both in-and out-of-distribution generalization. Finally, ViTs have a slight edge over ResNets in out-of-distribution generalization. 
properties of robust visual representations for manipulation: 
 Our findings in the last section are both surprising and somewhat unsatisfying because they contradict many of our intuitions about scale and generalization. In our evaluation suite, we saw that better generalization is not cleanly explained by more data, bigger models, or more relevant data. The goal of this section is to identify the properties of pre-trained models that are predictive of generalization. To that end, we correlate out-of-distribution performance with three metrics that have been previously connected to generalization in the machine learning and computer vision literaturein-domain performance, accuracy of a linear probe trained on ImageNet, and shape-bias. We also include a fourth metric, which is specific to ViTs: the emergent segmentation accuracy of the output attention heads. We describe each metric in detail in Section 5.1, discuss our setup for correlating performance in Section 5.2, and analyze our results in Section 5.3. ID vs OOD. One of the goals of this paper is to understand how well the findings from existing evaluations of pre-trained models hold under the inevitable environment changes that we expect to see in a real-world setting. If indistribution performance is reasonably predictive of generalization to our suite of distribution shifts, it is sufficient for researchers to continue developing pre-trained models with existing methods of evaluation. Past work has also shown that the in-distribution performance of a pre-trained model is positively correlated with out-of-distribution performance for a variety of computer vision tasks (Miller et al., 2021) . Concretely, we measure in-distribution performance as the success rate of the policy within the training distribution.Training linear probes on Imagenet is a common protocol for evaluating the quality of learned representations (He et al., 2019; Chen et al., 2020) . Hu et al. (2023) make the related finding that the ImageNet k-NN accuracy of a pre-trained model is predictive of performance on imitation learning with a visual reward function. We evaluate ImageNet validation set accuracy for all models with linear probes available.Shape-Bias vs OOD. Shape bias is the extent to which a model makes prediction decisions based on shape. We calculate shape bias as the percent of shape classification decisions out of the set of texture or shape classifications on the Stylized-ImageNet validation set (Geirhos et al., 2019) using the same probes described above.Jaccard vs OOD. Finally, for all of the ViT models, we look at the emergent segmentation performance. We denote this nonlinear, deterministic transform as M . Formally, we compute the Jaccard index by calculating the mIoU on the PASCAL VOC validation set, D P ascal :J(x i , x j ) = E D P ascal A ∩ B A ∪ BWhere A is a shorthand for positive classification for the target class by M (ϕ(•)) and B is a shorthand for positive label for the target class. J is evaluated pixel-wise over image indices x i and x j . We evaluate the Jaccard index of an interpolated attention map averaged across heads in the last attention block at the [CLS] token.We measure the coefficient of determination (R 2 ) and Spearman's rank correlation (ρ) for the correlation between the out-of-distribution success rate and each metric described above. Our goal is to find a metric that will result in high correlation between the metric and the OOD success, i.e. both coefficients being close to 1.0. We fit separate trend lines to ViTs and ResNets. Because of the lack of available probes, we exclude MVP, MVP ViT-S HOI, R3M, VIP, and MAE-IN ViT-S from the shape bias and ImageNet probe correlations. Each point represents one of the 15 pre-trained models we evaluated and represents the average of 6,000 evaluation runs.We visualize the correlation between each metric and the average out-of-distribution success rate in Figure 5 . Although we see a positive relationship between in-and out-of distribution generalization, there are pre-trained models that notably deviate from this trend. Among ViT models one example is MVP, ViT-S (HOI): the average success rate of this model drops to 6.63 from 39.86. By contrast, we find that ImageNet accuracy of a linear probe poorly predicts generalization performance for ViTs.We also see little correlation between shape-bias and OOD performance for ViT models, but a promisingly strong correlation on the subset of ResNets evaluted. This is surprising because humans make highly shape-biased decisions and increasing shape-bias increases the robustness of imagenet trained CNNs (Geirhos et al., 2019; 2021) . One explanation of this finding is that the ViT architecture obviates the need for shape-biased features. For example, a ResNet-50 trained with the DINO training scheme has a strong shape-bias, but not the equivalent ViT model.Model Success MVP 0% MoCo-v3 40%Table 1 : Success rates on the task of picking up the screwdriver.Finally, we visualize the relationship between the Jaccard index and OOD performance on all ViT models in Figure 5 . There is a strong positive correlation between Jaccard index and OOD performance both in terms of rank correlation and the coefficient of determination. These results suggest that while shape-bias may not be predictive of the OOD generalization ability of a pre-trained ViT, the segmentation ability is a predictive alternative.One counter-argument to the use of Jaccard index as a metric for for OOD performance is that it would be less predictive for object-level distribution shift, which would occur any time a large distractor is placed in the background of the image. In Figure 6 , we plot the success rates of each ViT model as the number of objects increases and verify that the models with the higher Jaccard index actually maintain the highest performance as the number of distractors increases.In this section, we validate our finding on a real-world generalization scenario by comparing a ViT-B model designed for control (MVP) against a model not designed for control but with a high emergent segmentation score (MoCo-v3).Setup. We learn policies for picking up a screwdriver on the ALOHA setup using the ACT training framework (Zhao et al., 2023) . The training dataset is comprised of 50 episodes collected by an expert human demonstrator. Images are collected from 4 camera view points (one on each wrist, one top camera, and one front camera). We replace the standard encoder with a ViT-B and change the initialization of the encoder based on the experimental condition (i.e., we select for a different pre-trained model). We follow the standard ACT training paradigm with the hyperparameters listed in Appendix Table 4 . From the training data to the test runs there is a distribution shift in both the placement of the target object (the screwdriver) and in the direction of the lighting. This is visualized in Figure 7 . We calculate success on screw pick ups averaged over 10 rollouts in the test environment.Results. We find that MoCo-v3 is stronger on this setting than MVP, even though it is not explicitly designed for manipulation. We find that the MoCo-v3 initialized encoder is able to achieve a success rate of 40% on this task while the MVP initialized encoder is not able to successfully grasp the target object. Qualitatively, the MVP model fails in localizing the object when attempting the grasp, whereas MoCo-v3 model reliably localizes the object, but experiences more failure in finding the right grasp point. 
conclusion: 
 Summary. In this paper, we make several surprising findings about the generalization ability of pre-trained visual representations for manipulation tasks. First, we find that, contrary to the current direction in the literature, models pre-trained on manipulation-relevant data do no necessarily generalize better than models trained on standard pre-training datasets (such as ImageNet MVP, ViT-S (HOI) (Xiao et al., 2022) is a predecessor of the model described above that trains a ViT-S/16 with an MAE objective on Something-Something (Goyal et al., 2017) , YouTube 100 Days of Hands (Shan et al., 2020) , EpicKitchens (Damen et al., 2018) , and ImageNet (Deng et al., 2009) .VIP (Ma et al., 2022) uses an action-free dual of the Algaedice (Nachum et al., 2019) objective to learn representations that are useful for trajectory optimization or reinforcement learning of control tasks unseen during representation pre-training. They train a ResNet-50 on Ego4D with this objective.CLIP, ViT-B/16 (Radford et al., 2021) uses contrastive language-image pre-training to learn visual representations trained on an extensive internet datsaet. The learned models exhibit strong zero-shot performance for multiple tasks such as image classification.DiNo v2, ViT (Oquab et al., 2023) scales Caron et al. (2021) to more parameters and a larger dataset. The full model is a 1B parameter ViT trained on LVD-142M, which is a 142M frame dataset composed of ImageNet-1k, ImageNet-22k, Google Landmarks (Weyand et al., 2020) , and a collection of other datasets spanning fine-grained classification, segmentation, depth estimation, and retrieval. The full model is distilled into smaller models. We select the ViT-S distilled model for our experiments. In Table 2 , we list the augmentations used on the teacher model. The training loop is only lightly modified during distillation. Suprisingly, the v2 model sees worse in-and out-ofdomain performance on our evaluation suite in spite of being distilled from a ladrger model trained on a bigger dataset. 
 
 Appendix: FrankaKitchen (Gupta et al., 2019) is a simulated kitchen environment with a 9-DoF Franka robot. There a multiple household objects available for interaction. The environment is designed to compose tasks together hierarchically, but we focus on learning policies to successfully complete a single task. The episode length is 50 and we inherit the randomization scheme used in R3M, which randomizes the position of the kitchen at the start of each episode.Meta-World (Yu et al., 2019) is a simulated manipulation environment that consists of various table-top manipulation interactions. Unlike FrankaKitchen, the scene objects vary between different tasks. The positions of the objects are randomized at the start of each episode. The maximum episode length is 500.Each distribution shift is visualized from the left camera angle in Figure 8 . We don't use the MuJoCo scanned object dataset that is used in (Xie* et al., 2023) We evaluate shape-bias using the 'model-vs-human' evaluation framework from Geirhos et al. (2021) and use the same probes from Section A.6 to get classification results on the cue-conflict validation dataset (D cue-conf lict ). The cue-conflict dataset contains images where the shape and texture cues are in conflict (e.g., a cat with the texture of the elephant). The shape bias of the model is the ratio of classification decisions made based on the shape cue (e.g., cat) vs the texture cue (e.g., elephant).  


Decision: This paper asks a very relevant question and provides two interesting results: Models trained on manipulation-specific data do not perform better than models trained on vision-only data, and the strongest predictor of out-of-distribution performance is emergent segmentation capability. The study is carefully done and well presented. Nevertheless, it is still unclear how robust these findings are. As Reviewer hDPj notes, other work, not cited here, reaches contradictory conclusions. Findings mostly rely on non-photorealistic simulations; it is unclear how they translate to real robots. Also, results may differ with changes in the setup such as more powerful action decoders or fine-tuned encoders.

During the rebuttal the authors answered most questions convincingly. The one problem that still stands is the real-world significance of the main findings; the additional results are suggestive but far from exhaustive. The main conclusion might be summarized as "For vision-guided grasping tasks, prefer vision-only ViT over manipulation-specific ViT" (presumably because of the vast vision-only training data available). Framed this way it does not look overly surprising but at least it kicks off a discussion of practical value.

Reviews: review: title: title: Official Review of Paper 593 by Reviewer hDPj
paper_summary: paper_summary: This paper studies the generalization capabilities of different visual encoders and pre-training methods for downstream policy learning to solve manipulation tasks. Furthermore, they evaluate different metrics to predict the policy success rates under visual distribution shifts. The authors compare 3 models that are pre-trained on manipulation-specific and 4 models that are trained on generic computer vision datasets. The pre-trained models are then frozen and the behavior cloning is trained on top of them using demonstrations from two common simulation benchmarks.
main_review: main_review: ### Strengths

* Approach: The authors carefully selected the most relevant and common encoder backbones for analyzing the out-of-distribution generalization for manipulation policy learning. Also, they found a commmon metrics used in computer vision are mostly not useful for predicting performance. 

* Clarity: The paper is written very well and the experiment results are discussed at length. I liked reading the paper since the motivation is very well motivated and the authors made the paper easy to follow. Enough details are given for the experiment design making this paper reproducible. 

### Weaknesses

* My main concern for the overall paper is that the results presented for studying visual domain shift are shown in the simulation while the pre-trained models are trained on real-world data. Especially, the paper claims that the models trained on generic vision datasets like ImageNet outperform the models trained with manipulation-specific datasets like Ego4D. This counter-intuitive result suggests that evaluating the policies in the simulation does not reflect the effectiveness of pre-training on manipulation-specific datasets. Other works [1], [2] not cited in this paper have also shown that training on manipulation-orientated datasets such as Ego4D or EpicKitchen improves real-world policy performance. While it's true that some of these findings will transfer to the real world, the results do not show conclusive results for real-world deployments. Even if the results were to be shown in the simulation, I would suggest using a simulation with photo-realistic simulation or showing a strong correlation between the success rate in the real world and simulation policy execution. However, there is only one real-world experiment which is not enough to show the visual gap between simulation and the real world is small.

* The pre-trained visual encoders are frozen during the policy learning phase. However, some work shows that fine-tuning the pre-trained policies can increase the downstream policy performance in out-of-distribution settings. The analysis in this paper should also consider this scenario. 

1. LIV: Language-image representations and rewards for robotic control

2. SUGAR: Pre-training 3D Visual Representations for Robotics
questions: questions: * The Jacard index has been shown to be a good indicator of policy performance, however, this metric is only applicable to visual transformers. The authors can also calculate the visual attention of the ResNet-based models using grad-cam methods that show the class activations. 

* Since segmentation features are shown to be the best prediction metric, it would make sense to add segmentation models (maybe distilled versions). I recommend adding backbones with trained Segment-Anything.
limitations: limitations: 1
review_summary: review_summary: Due to my main concern relating to the simulation-real-world visual gap, I recommend rejecting the paper. If authors can show more results in the real world, I'm inclined to change my decision towards acceptance. The model selections and the metric analysis can potentially make this work very impactful for predicting the performance of policies in the real world.
title: title: Official Review of Paper593 by Reviewer nqHm
paper_summary: paper_summary: This paper compares the generalization ability of 15 pre-trained visual encoders in the visuomotor robot policies in the presence of distribution shifts of images. Their two key findings are: 1) models that are designed for manipulation and control do not outperform standard visual pre-training methods. 2) the emergent segmentation ability of a ViT model is a strong predictor of out-of-distribution generalization performance. They conduct systematic analysis of the relation between the OOD performance and other metrics, such as emergent segmentation ability, downstream ImageNet accuracy,in-domain accuracy, and shape-bias, and conclude that conclude that the emergent segmentation ability is the most relevant metric. In experimental evaluations, the authors show that pre-trained ViT model not designed for control has better generalization both in the simulation and the real world.
main_review: main_review: Strength:

- Studies an important problem (generalization ability of pre-trained visual encoder) and introduction motivates the problem well.

- Thorough comparison of the current pre-trained visual encoders is very informative for researchers to choose the pre-trained model they need or develop new pre-training method.

- Visualization of the attention region in the observation reveals the mechanism of how the emergent segmentation ability influence the generalization ability.


Weakness:

- The downstream action decoders of all the visuomotor policies are limited to the simplest 2-layer MLP. For more difficult tasks, using an RNN decoder (such as the LSTM-GMM) or a transformer decoder will achieve higher performance. These more advanced action decoders should be the choice of other researchers, but we don’t see an analysis and comparison of them on top of the pre-trained models, which limits the usefulness of the results in this paper.

- The real-world experiments are too simplistic. Lifting up an object is a very simple manipulation task. Changing the direction of the lighting and the position of the target objects are also the easier options in all the possible distribution shifts. For the task, I would like to see harder tasks such as pick-and-place and cube-stacking tasks. For the distribution shifts I would like to see the changing of the background texture and adding more distractors.
questions: questions: - Will the two key findings in this paper remain the same when we replace the MLP action decoder with RNN or transformer action decoders? Could the author(s) provide experimental results or theoretical clarifications about this?

- Could the author(s) provide real-world experiment results on harder manipulation tasks and more drastic distribution shifts to support their conclusion?
limitations: limitations: 2
review_summary: review_summary: The paper is well motivated with a good set of experiments. Still experiments have room for improvement (see list of weaknesses).
title: title: Pre-trained visual representations for OOD manipulation tasks
paper_summary: paper_summary: This paper conducts comprehensive experiments across a wide class of pretrained visual representations to investigate their effectivenes in learning robust manipulation policy through imitation learning. The methods used for pretrained visual representations include supervised training, self-supervised learning, as well as manipulation-driven pretraining. The pretraining architectures include resnet and ViT. Through these experiments, the authors establishes insights into what visual representation leads to improved performance in imitation learning manipulation policies in two classes of simulated environments, as well as a real-world robotics study using the ALOHA setup.
main_review: main_review: This paper aims to answer an important question in identifying suitable pretrained visual representations for manipulation tasks. This reviewer believes the work is original, and clearly presented, and the conclusions are significant.

Strengths:

- This paper contains many interesting and significant insights into the correlation between different visual representation learning metrics and performance in downstream robotics manipulation tasks. In particular, the Jaccard index correlation figure is very conclusive.
- The experiments are comprehensive over a wide class of pre-trained visual representations. 
- The exposition of the paper is clear.
- This reviewer believes the findings are highly original and are helpful for the community to adopt pretrained visual representations for robotics tasks.

Weakness:

-  The insights of this paper regarding the quality of the pre-trained visual representation are interesting and conclusive. However, as the authors also noted, in practical scenarios, when trained visual representations are adopted for downstream robotics tasks, allowing fine-tuning of the encoder is also an important scenario that is of high interest to practitioners. Current results on fine-tuning are limited (in the appendix) and not as conclusive as those presented in the main paper.
 
- A baseline with a from-scratch visual encoder might further elucidate the quality of pre-trained visual representations.
questions: questions: - Will the conclusion of this paper still hold in the fine-tuning scenarios? It would be great if the authors could extend the results presented in figure 12.

- It would be interesting to include a baseline where the visual encoder is trained from scratch or can the author explain why it should be excluded? maybe the training cost will be too high compared to using pretrained representations. The current training cost is presumably very low, as the visual signals are encoded once and then only a 2-layer MLP is trained. The visual encoder can potentially be efficiently trained with parameter-efficient fine-tuning, or only fine-tune a subset of the encoder layers. The underlying question is, what are the use cases of pretrained visual representation in robotics learning? Is it mostly highly resource-constraint cases? From this reviewer's opinion in a lot of robotics learning scenarios data could be more constrained, and a bit more cost in fine-tuning the visual encoder could be a price that we are willing to pay. This question links to the first question -- in this reviewer's opinion, the key conclusion of this paper will be a lot more powerful if the results on fine-tuning are more comprehensive so that it can go beyond the frozen visual encoder.

- Maybe the authors could also discuss the above questions in the limitation section.
limitations: limitations: 2
review_summary: review_summary: This reviewer believes the insights of this paper is useful for the community. The paper still bears room for improvment to make the conclusion more powerful.