The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
Authors: Moritz Schneider
Robert Krug
Narunas Vaskevicius
Luigi Palmieri
Joschka Boedecker
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
rl
model-based reinforcement learning
representation learning
pvr
visual representations
Primary area: reinforcement_learning

Abstract: Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution (OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors to OOD generalization performance.

The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
Moritz Schneider 123* Robert Krug 12 Narunas Vaskevicius 12
Luigi Palmieri 12 Joschka Boedecker 34
Bosch Center for Artificial Intelligence Bosch Corporate Research
University of Freiburg BoainLiins-BBannToolss
Abstract
Visual Reinforcement Learning (RL) methods often require extensive amounts data. As opposed to model-free RL, model-based RL (MBRL) offers a po- tential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied the context of model-free RL, their potential MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better out-of-distributee
(OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors OOD generalization performance.
Introduction
Reinforcement Learning (RL) provides an elegant alternative to classic planning and control schemes, as it allows for complex behaviors to emerge by just specifying a reward, rather than hand-modelling and tuning environments and agents. Despite their success, most methods need extensive data and can be used only on their respective task, lacking the generalization capabilities needed handle the complexity of real world tasks. On hardware, RL is costly in terms time and wear, therefore model-based approaches are attractive as they promise to improve sample efficiency. For many real-life problems, vision is an invaluable source of state information, but due to its high-dimensional nature it is challenging incorporate it in RL algorithms. Therefore, the use of pre-trained visual representations (PVRs) is attractive as, intuitively, it promises to improve sample efficiency and generalization. Most existing approaches use investigate PVRs in the context of model-fres RL. For example, CLIP I is already widely used as pre-trained vision model for model-free robotic RL tasks 2 4 One would assume that the benefits such representations yield for model-free settings equally apply to madel methods. In model-based reinforcement learning (MBRL), features of convolutional neural networks (CNNs) usually used visual state representations, whereas other representation types such as keypoints, surface normals, depth, segmentation and pre-trained
"oorreppondence moritz. scceteeeecccceeeeddddd........h com Project website: https:://cchnee http://hhhoiimmcccoo//rrrrrr wwm/pvr4mbrll
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

representations are often ignored. Moreover, model-based methods are usually trained under an objective mismatch as the training process is required optimize the accuracy of the dynamics model and the overall performance of the agent at the same time [5] Naturally, this makes the training procedure different and more difficult than in model-free settings.
In this work, we focus on model- based RL and benchmark a set of representative PVRs on a set of challenging control tasks. To this end we want answer the following question:
i Is MBRL more sample efficient when using PVRs in contrast to learning representation from scratch?
Furthermore, PVRs nowadays increasingly often trained general datasets (e.g. ImageNet [6), Ego4D 77. etc.) and should be reusable in a wide variety of downstream tasks without further fine- tuning on in-domain data. We would like to empower downstream RL algorithms with corresponding generalization capabilities. Most existing implementations only investigate the distribution shift for the PVRs, but not for the downstream RI algorithm. This leads to the additional questions:
ii Can model-based agents generalize better to (i.e. can PVRs pass on their generalization capabilities to model-based agents)? ii How important are different training properties like data diversity and network architecture of PVR for model-based agents on downstream RI. task?
(OOD) settings with PVRs
Compared with the model-free RL approach, MBRL methods learn accurate models of the environ- ment for efficient learning and planning. Therefore, additionally we want to investigate the final question:
iv How does the quality of the learned dynamics models terms of accumulation errors and prediction quality differ between models trained from scratch and those based on PVRs?
Contributions.
The key contributions of this paper are summarized as follows:
Benchmarking PVRs for MBRL. Using PVRs trained on diverse and general data, study the generalization capabilities to out-of-distribution (OOD) settings of model-based agents utilizing these PVRs. To the best of our knowledge, perform the first such comparison for MBRL.
OOD Evaluation for MBRL and PvRs. Most other benchmarks only look into the case that PVRs should facilitate better training performance, We additionally look into the case of shifting the distribution also for the underlying MBRL agent, i.e., we have large differences between training and evaluation set. In this way, investigate what extent PVRs transfer their generalization capabilities to downstream RL agents. Our experiments reveal that PVRs are often ineffective for the MBRL agents. Furthermore, agents using representations learned from scratch outperform the PVR- agents most of the time.
Important OOD Properties of PVRs. We investigate and discuss important properties of PVRs for generalization in downstream control tasks in a MBRL context. We find that data diversity and network architecture are the most important contributors to OOD generalization performance.
Analysis of Model Quality. To explain our results more in depth, we analyze the quality of the trained world models and find that those which use representations learned from scratch are in general more accurate and have less accumulation errors regarding predicted rewards than the ones using PVRs.
2 Related Work
Model-Based Reinforcement Learning (MBRL) combines planning and learning for sequential decision making by using a (learned) predictive model of the environment, a learned value function and/or policy. These methods have shown impressive results in various domains [8 g 10 11] However, MBRL often applied to problems featuring complete state information derived from proprioception [2] [3), which may not always be available in practical scenarios like robotics. Also.

---

because of the data efficiency of those methods, MBRL on images has been explored extensively [14
16 17 TO 18. These algorithms typically learn representations from scratch and while there
has been research on the influence of reward and image prediction in MBRL. 19] we currently do
not know any method or review that combines PVRs with MBRL. The only recent trend in a similar direction is action-free free pre-training of world models for MBRL 21 and offline MBRL [22 23 24].
Both directions pre-train more or all parts of a MBRL algorithm instead of the representation only.
Learning for Control. Representation learning is a performance bottleneck in visual reinforcement learning. Pre-training methods and methods using auxiliary tasks offer the potential for greater sample efficiency. As result, visual representation learning for control especially has received increasing attention recently. In this line of work, RAD [25] enhances the generalization performance and sample efficiency of RL algorithmss using visual representations by simply integrating different data augmentations. Concurrently, DrQ (26 and DrQ-v2 [2 integrate similar data augmentations for off-policy RI. Likewise, CURL. 28 applies data augmentations to reinforcement learning as well, but uses an additional contrastive auxiliary loss to train the representations. In contrast to the methods by Laskin et al. [25]. Yarats et al. 27 and Yarats et al. [26] which learn representations from scratch, we study the use of pre-trained representations on out-of-domain datasets. Pie-G [29] simply uses PVRs pre-trained on ImageNet [6] and shows that using early features of the PVR in combination with BatchNorm [30 results in larger performance gains than using the full PVR. Nair et al. [3]] train representations specifically for robotics using a combination of vido-language alignment and time- contrastve learning. Value- Implicit Pre- Training (VIP) 32 is method that combines unsupervised pre- pre-training and reinforcement learning, where a value function is learned implicitly through a self-supervised task, leading to improved performance in downstream reinforcement learning tasks. Radosavovic et al. (33. Xiao et al. [34]
Visual Representation
and Majumdar et al. (35) pre-train representations based on Masked Autoencoding [36]. While there are combinations which couple representation and reinforcement learning specifically focusing on we focus on general pre-trained representations.
generalization 25 28 26 27],
Benchmarking PVRs in RL and Robotics Incorporating PVRs into model-free RL agents has emerged as a pivotal strategy for enhancing the efficiency and generalization capabilities of au- tonomous agents in visually rich environments. Many studies already investigate different types of pre-trained visual representations for reinforcement learning, robotics or control in general: Sax et al. (37 and Chen et al. (38) benchmark mid-level representations like segmentation and depth estimation models for navigation and manipulation tasks. Wulfmeier et al. [39 analyse dimensional- ity, disentanglement and observability of a small amount of PVRs and their usefulness as auxiliary task generators in a robot learning setting. Majumdar et al. 35 evaluate a handful of PVRs on a large amount of tasks and introduce new PVR, VC-1, which we include in our evaluation. Parisi et al. [40] investigate the influence of PVRs on the performance of policies using imitation-learning in a variety of tasks. Similarly, Hu et al. 4il study PVRs using multiple policy learning methods like model-free RL, behavior cloning and imitation learning with visual reward functions. Tomar al. [42) study the influence of different components for visual RL including world models. They show that MBRL agents using image reconstruction losses fail in diverse environments. In contrast to our approach, they do not study the influence of PVRs on the performance of MBRL. Träuble et al. 43 Chen et al. (38] and Burns et al. 444) are the only works we are aware of that investigate robustness and performance of policies trained using PVRs in OOD settings. The work in Hansen et al. 45) is closest to ours since the authors present similar results in the imitation learning and model-free RL setting by using data augmentation. But of the papers mentioned previously, use either imitation learning or model-free RL. None of them specifically targets MBRL. In contrast to model-free RL or imitation learning methods, MBRL methods are usually trained under an objective mismatch [5). Due to this mismatch, we believe in the importance of focused study on the use of PVRs in MBRL.
3 Experiments Setup
3.1 Model-Based Reinforcement Learning
As downstream RL algorithms we utilize DreamerV3 II and TD-MPC2 46) which achieve state-
of-the-art performance in many benchmarks and are often used the field 47 48
49
18
50
51 52
An overview of the algorithms and how we integrated the PVRs is shown in Figure
3

---

0e0
cas
dec
CBC
(aton Mnd
Plul
selor)
gn(os)
avior)
Soloras
Dolosas)
Components of our PVR-based DreamerV3 (left) and TD-MPC2 (right) architec- tures. In DreamerV3, the output 2, of the frozen pre- trainrd vision module Je is given to the encoder onc(2,/z)) which maps its input to discrete latent variable 2e- In TD-MPC2 a stack 2x-3: of the last 3 PVR embeddings is given the encoder enc(x;-3:;) which maps the inputs to fixed-dimensional simplices. The encoder of DreamerV3 additionally requires the recurrent state hg as input. The rest of both algorithms remains unchanged. Adapted from Hafner al. and Hansen al. [46].
Figure 1:
In the original setup without PVRs, both algorithms receive (stack of m) (visual) observation(s) or E x= from the environment, map to a (discrete) latent variable 2, E TR using an encoder enc(o:). In both cases, can be unrolled into future states 4++1 using a latent dynamics model. Based on 2,, both methods utilize additional reward and value models for planning and learning. DreamerV3 additionally utilizes decoder project latent states back into the observation space. Both MBRL algorithms learn policies an actor-critic style. TD-MPC2 uses model -odee-ppredictve control (MPC) planner in combination with the policy. To study the different PVRs and retain as much as possible of the original algorithms, we replace the encoder enc( ot) partly with frozen PVR 9* (o.). To keep much information possible of the PVR encoding x, 9a (o())) R", we decided to use linear mapping for the encoder. This decision should reflect current promises of PVRs which state that methods using state-of-the-ard representation learning able solve downstream tasks using a single linear layer only [53). The linear mapping is trained jointly with the rest of the MBRL algorithm. Otherwise, the algorithms are kept unchanged and mostly the same hyperparameters as the original implementations. For more details we refer to Appendix A. and A.2
3.2 Pre-Trained Visual Representations
We refrain from utilizing proprioceptive information (such as end-effector poses and joint positions) to ensure fair comparison among vision models that solely rely visual observations. Generally we use the largest published model each PVR. For more details we refer to Appendix A..]
We chose to investigate a variety of PVRs, some of which are popular in the field of policy learning (like CLIP) whereas others are less considered in other works (e.g. mid-level representations). Most of them are trained on self-supervisee objectives and use either Vision Transformers (ViT) [54] or ResNets (55). Furthermore, all them are open-source and easily available. Specifically, we include the following models: CLIP D1, R3M (11, Taskonomy [56), VIP 32 DINOv2 (53) OpenCLIP 57
VC-1 35,, R2D2 [58). A more in-depth discussion and overview can be found in Appendix A.3
We additionally include our own pre- tre-trained autoencoders that are trained on task data. This allows us investigate the influence of the pre-training data on the performance of the PVRs. During pre-training, the autoencoders see the same distribution of data as the other approaches during the reinforcement learning procedure and thus they should have significant advantage the subsequent reinforcement learning phase (in which only the encoder is used).
In Appendix C we show that the performance differences between multiple nonlinear layers and a single linear layer are negligible. A single linear layer is therefore sufficient.

---

Figure Illustration of tasks ranging from DMC ManiSkil12 to Miniworld with randomizations. Note that while DMC and Miniworld task images show the perspective of the agents, agents in ManiSkill2 tasks utilize the perspective of a wrist-mounted camera.
3.3 Domains
We evaluate all representations across total of 10 diverse control tasks from 3 different domains: DeepMind Control Suite (DMC) [59], ManiSkil12 [60] and Miniworld [61]. All environment observations consist of 256 256 RGB images, which corresponds to the resolution used for all pre-trained vision models. Most PVRs crop those images down to 224 x 224. An overview of the environments and tasks is given in Figure
The agents are trained under a distribution of visual changes in the environment (which we refer to as I Distribution (ID) and are evaluated later under different distribution of unseen changes (OOD changes). ID training and OOD evaluation are implemented through randomizations of visual attributes in the environments by splitting all possible randomizations into ID training and OOD evaluation sets. We focus exclusively on the setting of visual distribution shifts.
We train instances of each agent with different random seeds each performing 12 evaluation rollouts in the training environment every 50000 environment steps during training. For ID and OOD evaluation we perform 200 episode rollouts for each instance after training, resulting in 1200 episodes for each representation per environment. We train DMC agents for 3 million and ManiSkill2 as well as Miniworld agents for 5 million environment steps. Furthermore, TD-MPC2 agents are trained with smaller set of PVRs on the DMC tasks due the high computational costs of the experiments and the algorithm.
DeepMind Control Suite (DMC) [59 is widely used and applied benchmark environment for continuous control based on the MuJoCo simulator [62]. It includes locomotion as well as object manipulation tasks in which the agent applies low-level control the environment. We consider five tasks from the suite: Walker-Walk.
Swingup, Cheetah-Run Run, Cup-Catch, Finger-Spin, and
To measure ID and OOD performance, we slightly adapt the original tasks from the DMControl Generalization Benchmark (33 by changing the background colors of the tasks. Furthermore, randomize all dimensions of the simulation geometries randomly around their initial values. More specifically, sample the size values uniformly in range of [0.7 long, 1.3 lorg] of the original simulation value lorg- The OOD evaluation set is a held-out set of 20% of all colors and sizes included in the DMControl Generalization Benchmark. The ID training set therefore consists of 80% of the colors and sizes.
Even though the Deepmind Control Suite represents standard benchmark in RL, none of the PVRs are trained on a visual data distribution similar to DMC providing an even stronger OOD generalization test.
ManiSkill2 [60] is suite of robotic manipulation tasks based on the Sapien simulator (64. The tasks are more challenging than those of DMC, due to their contact-rich nature. Many of the tasks include cluttered and diverse environments and thus are more suitable for testing the generalization capabilities of PVRs. Since many of the PVRs are trained on robotic manipulation data, those tasks represent visually casier shifts than DMC, as a distribution shift still exists but not as large as in aforementioned DMC tasks. Nevertheless, the tasks are still challenging due to the necessity of precise control skills. We consider four tasks from the suite: StackCube, PlugCharger, PickClutterYon and Assembl ingkits.
Similar to our DMC experiments, we randomize different aspects of the tasks to differentiate between ID and OOD. For StackCube we randomize size and color of the cubes but leave the semantic meaning of the colors to solve the task untouched (i.e. picking up red cube and placing it onto green one is still the task goal). For PlugCharger we randomize shape, size and color of the charger and wall. For both tasks we exclude 20% the possible randomizations from training and use them for OOD evaluation. PickClutterYCB and dasemblingkie diverse combinations
5

---

Combined Tasks
Combined Tasks (ManíSkill2)
PickupObjects-vo (Miniworld)
(DMC)
Retur (ID)
1.00
1.00
0.75
0.75
0.50
0.2
0.50
0.25
0.1
0.25
0.0
0.00
600K 1.2M 1.8M 2.4M 3M
1M
2M
3M
4M
5M
1M
2M
3M
4M
5M
Step
Step
Step
From Scratch [CNN]
DINOv2 [VTT-G]
Autoencoding (TKY) [RN-50)
In-Distribution Autoencoder [RN-50]
R2D2 (L2-Net]
3D Keypoint (TKY) [RN-50]
OpenCLIP [V/T-H]
R3M [RN-50]
Z-Depth (TKY) [RN-50]
CLIP [ViT-L]
VIP [RN-50]
2.5D Seg. (TKY) [RN-50]
[RN-50X64]
VC-1 [VIT-L]
Figure 3: Normalized ID performance and data-efficieney comparison on DMC, ManiSkill2 and Miniworld environments between the different representations. Each line represents the mean over all runs with a given representation, the shaded area represents the corresponding standard deviation. Solid lines represent DreamerV3 runs, whereas dashed lines indicate TD-MPC2 experiments. Espe- cially in the DMC experiments, representations trained from scratch outperform PVRs also in terms of data-efficiency. Curves of each environment individually can be found in Appendix D
of different objects and object positions and are therefore already diverse by themselves. Thus, we split all possible configurations of positions and objects into training and evaluation set without additional changes. We train on 80% of the configurations and evaluate OOD on the remaining 20% of configurations. As shown in Hsu et al. [65]. it is beneficial to use wrist-mounted cameras instead of third party views. Therefore, we use wrist-mounted camera observations for all ManiSkill2 tasks. For further information, we refer to Appendix[A.4
Miniworld 61 is a multi-room 3D world containing different objects. We consider the PickupObjecte-v0 task in which the agent is sparsly rewarded for picking up 5 different ob- jects. The location and orientation randomize the color of the target objects. During OOD evaluation the agent is confronted with unseen object colors. We selected the ccckpbbjeccs--- ManiSkill2 tasks, it utilizes discrete actions and the navigation-base based nature implies dynamically changing background.
both the agent and targets are randomized. In addition, we
environment because, ullike the DMC and
Results
Here present the results of our evaluation and answer the research questions outlined in Section We start with a general comparison of the data efficiency of PVRs in MBRL (Section[4.]), Afterwards, evaluate the OOD generalization of PVRs in MBRI. (Section 4.2) Furthermore, investigate which properties of PVRs are important for OOD generalization (Section 4.3p. Finally, we analyze the prediction quality of the world models in order to explain results before (Section (4.4). For better comparisons throughout domains, returns are normalized as recommended by Agarwal et al.
[66]. For further information, we refer to Appendix A.5
4.1 Data Efficiency
In the following we want to answer research question I and investigate the sample-efficiency of PVR-based MBRL agent against the same agent using visual representation learned from scratch. In general, one would expect that agents using PVRs are more data efficient than their counterparts with representations trained via reinforcement learning. The common assumption is that PVRs are able to capture relevant information the environment due to their pre- pre-training phase. Therefore, the downstream learning algorithm can focus learning the dynamics of the environment. This ought to be especially pronounced for visual foundation models, which promise to generalize to different domains (67) The results for our MBRL setting summarized in Figure 3
Surprisingly, representations learned from scratch are in most cases equally or even more data-efficient than the PVRs. We want to highlight that this is also true for autoencoders which are pre-trained

---

Combined Tasks (DMC)
Combined Tasks (ManiSkill2)
PickupObjects-vo (Miniworld)
JEN [-50
0.8
Return (OOD)
0.12
0.45
Aumeeccdiing mee (B
30 Keppints (180)) (TRO (RN50)
Desth 188 IRN (RN
2.50 Se ITSM I (0
0.08
Ausencdiin 17&1 I
30 K6 (180 (R0-50000000 50)
2 Desth 1T8 ISS LSN 50)
2.59 Sep. ITSII (00550)
0.6
neddinn (Ma (AN-50]
CCNN
LCWN
0.4
From Scratoh (CNN)
0.30
0.15
IRN (MTT-)]
awl (U2
(RNU50)
MIP ERN 50)
VCS (VITE)
(K3650044)
VIP 30)
0.04
re D (12-dee]
(LSM (R0N500)
Vif ViP (0)
VC-s (UTT
0.2
(AN-50)
VC- (VTL))
From
0.00
0.00
0.0
DreamerV3
TD-MPC2
Figure 4: Average normalized performance on DMC, ManiSkil12 and Miniworld tasks in the OOD setting. The baseline representation learned from scratch outperforms all PVRs, even in the OOD settings. Thin black lines denote the standard error.
in-distribution on our own task-specific data. This contradicts the general belief that PVRs accelerate the training of (MB)RL agents [38]. We hypothesize that this due to the objective mismatch in MBRL 5 The overall optimization is decoupled into two optimization procedures (one for the dynamics and one for the policy/reward/value) making harder to adapt to an existing representation instead of learning a new one from scratch.
4.2 Generalization to OOD Settings
Even if PVRs are not able to perform equally well as representations learned from scratch in the ID case, they might be able to perform better in OOD domains. This should be especially true for visual foundation models which are often trained on diverse data [I 68 [3 69 (7). Therefore, here we want answer research question and evaluate the OOD performance on both domains after training. The results are visualized Figure 4 With the exception of VC-1,
noticeable that no PVR performs good in both domains. Even autoencoders trained in-distribution on task-specific data perform worse compared to training an encoder from scratch. This is surprising, since some PVRs trained on large sets of diverse data and should therefore generalize well to OOD domains which, however, we find not to hold true when compared to agents with representations learned from scratch. This is especially evident for the DMC environments where the PVRs perform worse than training from scratch.
4.3 Properties of PVRs for Generalization
The results far show that PVRs perform not well in MBRL and for OOD generalization as they do in policy learning settings 40 35 41
Also, the results do not explain which properties of the different PVRs relevant for OOD generalization. This is the subject of our next research question Based on Table 2] we group the PVRs into categories and evaluate the ID and OOD performance of the PVRs in each category. The exact categorization be found in Appendix B The results depicted in Figure 5) and discussed below.
Language Conditioning. All CLIP-based PVRs as well as R3M are conditioned on language in their training procedure. However, the combined performance of these PVRs shown in Figure 5 indicates that language conditioning is not necessary for good OOD generalization. This is surprising, since language conditioning is a popular technique to improve capabilities vision- and is assumed to be strong bias for visual model as [31. Nevertheless, our results suggest that pre-training representations with language might not be helpful for control tasks might be for other direct downstream learning tasks.
based agents (2] 70 31 71
72 48
should provide semantically relevant features
Sequential Data. Reinforcement learning is sequential decision making problem. Thus, good representation should capture the sequential dynamics of the environment. This even more relevant for MBRL, where the representation is additionally used to predict the state evolution. Therefore,

---

Combined Tasks (DMC)
Combined Tasks (ManíSkill2)
PickupObjects-vo (Miniworld)
0.056
Return (OOD)
0.8
0.54
0.048
0.6
0.040
0.48
0.4
0.42
0.2
0.36
0.5
0.6
0.04
0.08
0.4
Return (ID)
Retum (ID)
Return (ID)
VIT Non-VIT
Sequential Data
Language Conditioned
Diverse Data Non-Diverse
Non-Sequential
Non-Language Conditioned
Figure 5: IQM return of the different categorizations. Each marker represents the interquartile- mean performance of an individual group. The x-axis shows the ID performance and the y-axis the OOD performance. Especially, VIT representations or representations trained on diverse data perform well in the OOD setting. Sequential data seem to help in ManiSkill2 and Miniworld but not in DMC. Categorization plots for each environment individually can found Appendix B
this category includes PVRs which are trained on sequential data. The results in Figure show that the sequential order of the data is somewhat beneficial (especially in the ManiSkil12 experiments).
Data Diversity. Foundation models well known be trained diverse data We define data diversity based on the number of datasets used to train PVR and/or based on the size of dataset (e.g. WIT supposed to diverse). If a PVR is trained on more than one dataset or data domain we assume the data to be diverse. The results indicate that data diversity is generally important for performance both task domains.
68 72].
Vision Transformer Architecture. In recent years, ViTs have gained attention as a powerful alternative ResNets. In contrast the other networks, ViTs show good performance on both domains which aligns with results from Burns et al. [44]. However, we find that the ViT architecture is the only relevant factor for good OOD generalization. For example, CLIP [ViT-L] and OpenCLIP are both based on the ViT architecture, but both perform on par with CLIP [RN50x64). This is suprising, since the ViT-based CLIPs are trained with another structure and the same loss, but with different data. This indicates that data diversity might be more important than the network architecture.
4.4 World Model Differences
According to research question iv investigate how the different visual representations influence the quality of the world model which is learned the downstream MBRL algorithm. To this end, we train model-based agents the Pendulum Swingup task of DMC with the same setup as described in Section 3.3) We then use a pre-collected dataset of 200 diverse trajectories to analyze the world models of the agents. We plot the error of the complete trajectory and a smaller window of the same trajectories with a horizon of 33 timesteps to show the differences between long-term and shorter predictions.
Dynamics Prediction Error. For planning purposes, MBRL algorithms employ a forward dynamics model of the environment. The model can either be given [8] 9 or learned [16] 73
74 46).
DreamerV3 as well as TD-MPC2 belong to the latter category and we can therefore analyza malyze the dynamics prediction eπrоr of the underlying models. The results shown in Figure The plots show that the state evolution prediction accuracy of -VR-based approaches is comparable to model- based agents using representations learned from scratch. Furthermore, we observe a slight negative correlation between the values presented in Figures 4 and b (r = -0.22, 0.4). This suggests that the quality in the dynamics prediction plays minor role in the performance since the models with the best task performance do not necessarily have the lowest dynamics prediction error. Thus the dynamics prediction enror not the only important factor for the performance of the agent.
Reward Prediction Error. The reward model is crucial part of the world model. Because state- of-the-art model-based algorithms are actor-critic based (as in case), they use learned value

---

Horizon=500
Horizon=33
Horizon=500
Horizon=33
Acc. Dynamics Pred. Error
20000
0.08
1200
1.00
15000
3D Keypoint ( labe (ide-3))
900
0.75
0.06
10000
From Seigich [OW]
OpunClip (UT++]
CLP (Aw 50x04)
DiNons wmm
Z-waahh (18O (nne-50)
2:50 Sep IDST (its
600
From Soptoo son [CNN]
Spancuue (UTH))
CLPP (ne 50x64)
Dinovo (MTE:)
RRD2 (L2 Nett
Anteencadiio (TKy)
2.0rpth (it
(RIE50)
0.50
0.04
CLIP IN1 L]
CEiP wis L]
Raa IRN--00 50)
MIP (RN550)
(WTTL
CLIP [vi 3
VIP JAN [0]
VC. [WTT
VIP IRN-30)
VC [MTU
5000
RRD2 (L2
LEN
300
0.25
0.02
From
0.00
0.00
DreamerV3
TD-MPC2
Figure 6: Average Accumulated Dynamics Prediction Errors on the Pendulum Swingup task for 200 trajectories. For DreamerV3 we average the forward and backward KL divergence between the prior and posterior distributions of the latent state 2;. For TD-MPC2 the MSE between the predicted latent state & of the dynamics model and the encoded latent state 2; is plotted. Thin black lines denote the standard error.
Horizon=500
Horizon=33
2.0
Acc. Reward Error
aumonncdinng (Ter) [RM-S0)
Bp Keypoint (TK)) JAN (RN-50)
Auttenncdinng (ner) [RN-50]
mp Koypoint (TR IRN 50)
450
Digph (ne9 IRN sQ]
Ség. (TSV)) (RM-50)
1.5
Dopth (TC)) IRN so]
7:50 Seg, (TV9) (RM-S0)
From Scratch (CNN]
rowN]
Frim Scratch [CW)
300
OpencLIP [MTH]
Cup [Viru]
CLP Lea (aa-50000 50x64]
Dilivv (Vira)]
rapa [12 (L2-Wett)
NEW (RW-50
WPP jswe
yc [VTT-]]
CUP LXmiy
WPP [RN-50]
VCC [VT--1
1.0
OpennCLP (MTE-1)))TTT)))) [WT-+1]
Cup [WFF)
CP [00-50064]]
DiNov2 (VT--)
pap2 (L 2.
RaM Bww300
50)
ve (VTFL)
frim Scratch
cup [Vmn]
VIP (R2) (R2-50)
vI [VTTA11
150
0.5
VIP
0.0
Dreamerv3
TD-MPC2
Figure 7: Average Accumulated Reward Errors the Pendulum Swingup task for 200 trajecto- rics. The error is calculated as the absolute difference between true and predicted reward [ee fzl- Thin black lines denote the standard error.
functions to update the policy. Since the value prediction depends upon the the reward prediction, the agent might not be able to learn a good policy if the reward prediction is inaccurate. Figure 7 shows analysis of the models reward prediction errors. The plots demonstrate that models which have a better task performance (e.g. from scratch or VC-1) generally produce more accurate reward predictions. Here we observe significantly stronger negative correlation between task performance and reward eтror compared to the dynamics prediction case (r = -0.66, = 0.004). This indicates that the reward prediction is generally important for the performance of the agent, and that methods which perform badly on the tasks are unable to predict future rewards accurately. The PVRs do not appear to provide enough information to predict the reward better than agents without PVRs do. But as can be seen in the specific case of 2.5D Segmentation (TKY) it becomes clear that low reward prediction errors do not imply good performance necessarily. We assume that the influence of other components is still important.
Reward Information in the Latent State Space. PVRs trained compress information since the representations are trained as information bottlenecks. As such the training might not capture reward-relevant data from the pre- pre-training images since reward information is usually not a component in the training objective of the PVRs. On the other hand, MBRL methods like DreamerV3 and TD- MPC2 heavily rely on reward information in their objectives. Furthermore, previous benchmarks on PVRs often examine imitation learning settings where reward information is usually irrelevant. Based on our previous results therefore examine the extractable reward information content the representations. In Figure we show UMAP [75] projections of the latent state space of model-based agents for different representations the investigated Pendulum Swingup task. For representations learned from scratch, the reward information in the latent state space projection is highly structured and states with similar rewards are more closely embedded in latent space of representations that perform better in DMC. We hypothesize that this is contributing factor to why many PVRs

---

loss or From Scratch [CNN]
VC-1 [ViT-L]
CLIP [ViT-L]
VIP [RN-50]
&
0.0
2.0
1.5
1.0
0.5
0.0
Figure 8: UMAP projections of DreamerV3 (top row) and TD-MPC2 (bottom row) encodings using different representations as input. The points are color coded by the real perceived reward. Each point represents a visited state in the Pendulum Swingup environment of DMC. The representations learned from scratch better disentangle low and high reward states whereas the embeddings of the PVRs more entangled.
underperform compared representations learned from scratch. In order learn good policy, the agent needs to be able to extract and predict reward information accurately. This is not possible if the reward information is not consistently embedded in the representation.
5 Conclusion
In this work, we evaluate the efficiency and generalization capabilities of different PVRs in the context of model-based RL. We provide empirical evidence that PVRs neither improve sample efficiency of model-basee RL, nor ultimately empower MBRL agents to generalize better to completely unseen -utoo--distribttion shifts. Experiments analyzing the quality of the trained world model suggest that model-bat basel agents utilizing PVRs are not able to learn good reward models for the tasks compared agents learning representations from scratch. This indicates that PVR-based approaches can learn comparable dynamics models but struggle to learn good reward models which are crucial for the performance of MBRL agents. We conclude that PVRs might lack the needed information for reward model learning, and that training visual representations for MBRL requires extra attention compared to model-free RL. Additionally, we conducted experiments to find relevant performance properties of PVRs. Here, diversity in the pre-training data as well VIT architecture seem to improve generalization capabilities to OOD settings of PVRs. Conversely, a language-condicee sequential training data seem to play minor roles.
Limitations. This paper aims not be the final word on the topic of OOD generalization of PVRs, RL and MBRL in particular. Our findings hopefully inspire more researchers to dig into the untapped potential of utilizing PVRs in MBRL, since more experiments are needed draw final conclusion. Naturally, benchmark like the one presented in this work is inevitably computationally demanding. was therefore necessary to make certain design decisions and restrict number of representations. From our point of view, we focused on the most important PVRs but other MBRL algorithms and PVRs are certainly of interest as well. Furthermore, we evaluate the PVRs domains, Experiments on other domains, especially in the real-world are needed.
10

---

Acknowledgments and Disclosure of Funding
Joschka Boedecker is part of BrainLinks-Brrat Tools which is funded by the Federal Ministry of Economics, Science and Arts of Waalen--Wrttemberr within the sustainability program for projects of the excellence initiative II.

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[2] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894-906 906. PMLR, 2022.
[3] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14829-14883 2022.
[4] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor, multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[5] Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. In Learning for Dynamics and Control, pages 761-770. PMLR, 2020.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245.255. Icee, 2009.
[7] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[8] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre. George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-44 489, 2016.
[9] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, (62(6419):1140-111 2018
[10] Julian Schritwiesse,, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
[11] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 104104, 2023.
[12] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep rein- forcement learning in a handful of trials using probabilistic dynamics models. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, edi- tors, Advances Neural Information Processing Systems, volume 31. Curran Associates, newteeeeredeeeeenneeeseee...pepp neurips. wwtpccppppppeeeeeeeeeeeeeeee ffpper_filee www.s/lllaaaeeee /illes//apperr218///11ee
Inc., 2018. URL 1d565488979991a3311477778886666
pdf
[13] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference Robot Learning, pages 1101-1112. PMLR, 2020.
[14] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images. In Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Syy- tems, volume 28. Curran Associates, Inc., 2015. URL /ttts//ttt///////reeednng//e http://pproceddnnggeeerrrpippppp newriis.......r.....e........... cc/ paper_ foner_filespaae httpae/eiieeaeeeooot//araaaaattn
.df.
12

---

[15] David Ha and Jürgen Schmidhuber. World models. arXiv preprint axiv:18033101222 2018.
[16] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, pages 2555-2565. PMLR, 2019.
[17] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In Internationnl Conference on Learning Representa- tions, 2020. URL httpt::t///// htttps://openreviees.cee. www.:ee//tfff mtt.://oomoooomm//ss111000000
[18] Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 27147-27166. Curran Associates, Inc.. 2023, URL URUAttps::/prooooiiinee neurgps..erpppp.. cct.pps.cc//apeee wwppaperppppppeppepppp sftorrffilllseeeeeeeeeee wiles/paperr222//111.. 654775322222201100000877736----3
pdf
[19] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Danijar Hafner, Harini Kannan, Chelsea Finn, Sergey Levine, and Dumitru Erhan. Models, pixels, and rewards: Evaluating design trade-offs in visual model-based reinforcement learning. arXiv preprint arXiv:2012.0466 2020.
[20] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference Machine Learning, pages 19561-19579. PMLR, 2022.
[21] Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextual- ized world models with in-the-wild videos for reinforcement learning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 39719-39743. Curran Associates, Inc., 2023, URL (cccbbolebbb00622233422bbbbbeee3 Coreeeeee
nww.ins. cc/paper.ppppppppp/p http:/papr_ffeeee___eeeeeeeeeeee fiweffileepppee httll///paper/22///122
pdf
[22] Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11810-2182) Curran Associates, Inc., 2020. URL http:::::::/tcteetttteeee neurips......pp.. ec/ .dff
[23] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 14129-14142. Curran Associates, Inc., 2020. /tU/httss://proceeeiii wew.rooeeedinggeeeeiip.p... cwterrisscccpppppa cc/paper.pppp...ce files/ / /p0p/////p/////lll// 2972252ee00F//22004777////o22260
pd.f
[24] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 08954-28967. Curran Associates, Inc., 2021 URL https//tttt////poooooooooooooeeo newriis.eerpppp.. cc/ wwcc/pape__ttte files/
/ pptpr///22e////le//// 722217774912233577744ccoooo22 Paper pdf
[25] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11884-19895 Curran Associates, Inc., 2020. URL httpps://proceedinngss inwp:/eeeeeddiinnnnnniiuupp.pppp newrips..i... cc/ paper_ https//eaaaaeeeeoeee.a.aaaaaaaaa
pdf
[26] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. htl.at/esspppppeererrrrre///ssss
13

---

[27] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved lata-augmented reinforcement learning. International Conference on Learning Representations, 2022. URL htt.hhtts:///peeeee http://////ppp...eeeeeeeeeree/er
-yyes8
[28] Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference Machine Learning, pages -639--650. PMLR, 2020.
[29] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang. YI WU, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave. ral Information Processing Systems, volume 35, pages 13022-13037. Curran Associates, Inc., 2022. URL Ut/httpss:/prrreeednnne htwp:ooeeeeiiingneeeee...e..eeee www.nnewrrpcco httpp//rw...uuiiieeepaaaee..eeee faarssffieeeppppee http://eeeeeaaaee////////////// waw.rdd--Paaperrreeeeeeeeeeeeeee pdf
Cho, and A. Oh, editors, Advances in Neu-
[30] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 148-456. PMLR, 2015.
[31] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. Conference on Robot Learning, pages 892-909 PMLR, 2023.
[32] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre- training. In The Eleventh International Conference on Learning Represemtations, 2023. URL httpes//ppearrvvieee.ete/oororre
[33] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning. pages
PMLR, 2023.
[34] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173. 2022.
[35] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent- Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are in the search for artificial visual cortex for embodied intelligence? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 655-677. Curran Associates. Inc., 2023. wttp://Rsttss////eeeeeiaeaa htwp://oooooioio..o.nieeeeeeeeee neurips. cc/ wc/pppppr_aa. files/ pttle////ppprr22////100 02221ba6667460222622266662111110 Ponferrerrr-rreecceeeeee pdf
[36] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[37] Alexander Sax, Bradley Emi, Amir Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXiv preprint 1rXiv:1812.11197/ 2018.
[38] Bryan Chen, Alexander Sax, Francis Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. In Conference on Robot Learning, pages 2328-2346, PMLR, 2021.
[39] Markus Wulfmeier, Arunkumar Byravan, Tim Hertweck, Irina Higgins, Ankush Gupta, Te- jas Kulkarni, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, et al. Representation matters: Improving perception and exploration for robotics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6512-6519. IEEE, 2021.
14

---

[40] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pages 17359-1755 17371. PMLR, 2022.
[41] Yingdong Hu, Renhao Wang, Li Erran Li, and Yang Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. International Conference on Machine Learning, pages 13628 13651. PMLR, 2023.
[42] Manan Tomar, Utkarsh Aashu Mishra, Amy Zhang, and Matthew E. Taylor. Learning represen- tations for pixel-based control: What matters and why? Transactions on Machine Learning Re- search, 2023. ISSN 2835-8856. BRKLLttps://ppenrevviee wtt.://ooorrrrdddxx.xelLLL....
[43] Frederik Träuble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. The role of pretrained representations for the OOD generalization of RL agents. In International Conference on Learning Representations, 2022. URL https:://peereviieeee www.wawwwwww...ee......e.e. net/forum? id=8eb12UQYxrG
[44]
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? In 8th Annual Conference on Robot Learning. 2024. URLlltttp///ppeeeviiee net/forum?id=
A1hpY5RNiH
[45] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline. In International Conference on Machine Learning, pages 12511-12522 PMLR, 2023.
[46] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024. URL httppttt/////oeeeriiii
[47] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for re- inforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processsing Systems, volume 36, pages 68760-68783, Curran Associates, Inc., 2023. URL httpps::/proceeddnggggeoo/// neurips cctppper_fieess//////eeeeppssssp paper 2appr//203///1lee/8022o0o 0rt.00/aa..4077111cccc2221111611
Paper- Paper-Conferente Conf pdf
[48] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning model the world with language. In International Conference on Machine Learning, pages 29992-30017. PMLR, 2024.
[49] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL
attps://openrevies. net. wwt.://eee...o..oimoeeaaaaaoooo.
[50] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelcrating visual model-based reinforcement learning with demonstrations. In The Eleventh International Conference Learning Representations, 2023. URL https
//openreview. peee/foorun?dddJJdoceeeee
[51] Patrick Lancaster, Nicklas Hansen, Aravind Rajeswaran, and Vikash Kumar. Modem-v2: Visuo- motor world models for real-world robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7530-7537. IEEE, 2024.
[52] Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. In Conference on Robot Learning, pages 425-445. PMLR, 2023.
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wer Wen Li,
15

---

Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Arnand Joulin, and Piotr Bojanowski. DINOv2: Learnnn robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. TRKlptttps//opperrvviee http://pwreriiie..eeeefrrrrrr//a net//oorruniccaa8SStte22
[54] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL htttp:://ppenrevee..o ott.://ee......oooooooooooooooar
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[56] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018.
[57] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, and others. Open- CLIP. Zenodo, 4:5, 2021. doi: 12228/0/2ooo. URL httth::/tpoo /tf/doi.ogg//// org/10...... 5281/
[58] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. In H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL http::t:/poooceeaiiiiaee neurips. cc/ paper_ files/
1etpocdddddd112tbbb----eeeeeee pdf
[59] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv: 1088801100000 2018
[60]
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference Learning Representations, 2023. URL https / wwwpeeevviieeeefffrreeie atte/fforurriddb__uuyyyrrrr
[61] Maxime Chevalier-Boisverr Bolun Dai, Mark Towers, Rodrigo Perez-Vicente Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J Terry. Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks, In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processsing Systems, volume 36, pages 73383-73394. Curran Associates, Inc., 2023. URL WtR.btttps://prrceeddiie hew.eddddddnnnnnnnneeeeeeeeeeeee neur iisuriippaaa cc/ paper_ files/ www.ffiaaeeeeeaeeeeea (haper/2023/flee// 68816199846666677722226661999199 Patpaaaeasaaaaaaee-a-aaattttssss and_ Batrls_annn__nnneneeeeee Benchmarks pddf
[62] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[63] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1361 1-13617. IEEE, 2021.
[64] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated bast- interactive environ- ment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097-11101 107, 2020.
[65] Kyle Hsu, Moo Jin Kim, Rafael Rafailov, Jiajun Wu, and Chelsea Finn. Vision-based manipula- tors need to also see from their hands. In International Conference on Learning Representations, 2022. URL http:://pppneeeeeeeeeeeefrrreees
16

---

[66] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 09304-29320. Curran Associates, 202., URL ttt.htttss:/pproceeddiie neurips. cc/ wc/paper_... files/ pttps//pppp/////////eee pdf
[67] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Picter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129 2023.
[68] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL httph: //pprreviee/eeeeeee//eeee net/ forun?id-11kKxiH)
[69] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[70] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0668
2022.
[71] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-acttio models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165 2183.. PMLR, 2023.
[72] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pages 8469- 8488. PMLR, 2023.
[73] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL https://openreevev.
[74] Nicklas A Hansen, Hao Su, and Xiaolong Wang- Temporal difference learning for model predictive control. In International Conference on Machine Learning, pages 8387-8406. PMLR. 2022.
[75] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018
[76] Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
[77] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages wccppappr_eaeeeeeeeeeeeeeeeeeeee haales//aperr/222/ft1e 2022. URI. http:ttt/p/peeeediiingge neurips cc/ 1125085f005550004455555666322333 Patarett and ann___echmarrkee..caaaaaaaaaaa_e pdf
Curran Associates, Inc.,
17

---

[78] Yurun Tian, Bin Fan, and Fuchao Wu. L2-net: Deep learning of discriminative patch descriptor in euclidean space. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 661-669, 2017.
[79] Filip Radenovic, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Revisiting oxford and paris: Large-scart scale image retrieval benchmarking. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5706-5715, 2018.
[80] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg. Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 8601-8610, 2018.
[81] Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self- supervised correspondence in model- based reinforcement learning. In Conference on Robot Learning, pages 693-710. PMLR, 2021.
18

---

A Implementation Details
We rum our experiments on compute cluster using cores and 32GB memory per experiment with either single NVIDIA V100 A100 GPU depending the PVR used. Depending on the representation/input size more memory might be needed (e.g. 96GB for the ManiSkill2 from scratch experiments).
A.1 D3eamer
For each task and encoding type (i.e. PVR or from scratch) combination we train instances of DreamerV3 for DMC and ManiSkill2 with another random seed each resulting in 756 trained instances. For Miniworld, we train instances of D3eamerV3 with another random seed each resulting in 16 additional trained instances. The PVR networks implemented environment wrappers taking image observations O, as input while returning the representation Zt. The representation is then fed into DreamerV3 and is stored for further training in the replay buffer. Thus, the original Dreamer V3 algorithm works solely on the representations x as input using an additional MLP or linear layer. As result, the algorithm decodes the encoding æ only and not the whole input image o. To keep the encoder-decoder structure of DreamerV3, we replace the original encoder partly only instead of removing fully. We use the implementation from https /tt///gtttuu . com dani ar /dremmev3 (MIT license). Hyperparameters are shown in Table but in most cases do not deviate from the implementation.
Table Overview of the hyperparameters for DreamerV3.
DMC
ManiSkill2
Miniworld
Corresponds to model size
S
XI.
XI.
in Hafner et al. 11
General
Replay Capacity
106
106
106
Batch Size
16
16
Batch Length
64
64
64
Start Learning (Prefill)
0
Action Repeat
2
1
Environment Steps
3 x 10
5 x 10
5 x 106
Train Ratio
512
32
32
Actor Critic
Imagination Horizon
15
15 0.95
15 0.95
Discount
0.95
Return Lambda
0.95
0.95
0.95
Learning Rate
x 10
3 x 10-5
3 x 10-5
Adam epsilon
10
1 10-5
1 10
Gradient Clipping
100
100
100
World Model
RSSM Size
512
4096
4096
Number of Latents
32
32
32
Classes per Latent
32
32
32
Learning Rate
10 10-4
1 X 10
1 1 10
Adam epsilon
x 10
1 x 10-8
1 x 10
Gradient Clipping
1000
1000
1000
Plaooexplore
Ensemble Size
10
10
10
B
1
Symlog.
The DreamerV3 baseline instances symlog their inputs as described the original paper. For the ManiSkill environments found that not applying symlog to the PVR-based instances
19

---

performs better. For the other environments we keep applying symlog for both baseline instances and PVR-based instances.
Exploration Reward. We use an additional exploration reward expl in all experiments which changes the per-timestee reward Tt for the agent
Ty r3 3rt expl
where rfth is the original reward from the environment and & determines the amount of exploration. We use Plan2Explore (76] calculate r; with ensemble size of 10 and / 1. This helps in the overall performance in the environments.
Encoder Trained From Seratch. The representations which trained from scratch using the unmodified code from the implementation. Due to high computational demands we train the DMC as well as the Miniworld baselines 64 x 64 and the ManiSkill2 baseline 128 x 128 pixels.
A.2 TD-MPC2
For each task and encoding type (i.e. PVR or from scratch) combination we train 4 instances of TD-MPC2 with different random seeds. We use the implementation from stttps://gh htt//ggttubuut/ thub com/ micklashansen/tdmie tdmpc2 (MIT license). The encoder trained from scratch uses the unchanged implementation.
The PVR encoding implemented as an environment wrapper which takes image observations o, as input and returns the representation zz. Similar to the original training from scratch, we stack the last embeddings 3:+ of a PVR into a single feature vector. The stack then fed into TD-MPC2 and is stored for further training in the replay buffer. The original TD-MPC2 algorithm works solely on the representations x as input using an additional MLP or linear layer. To keep the encoder structure of TD-MPC2 in the PVR-based approaches, replace the original encoder partly only instead of removing fully. The residual hyperparameters do not deviate from the original implementation for visual RL.
A.3 PVRs
For all representations we use the same preprocessing steps as described by the authors. We use the implementations and the pre-trained models from:
#
Name
URI.
License
1
CLIP
httpgs//gtthhbb..congggssaaaaL
MIT
2
DINOv2
Apache 2.0
3
OpenCLIP
CC BY-NC-SA 4.0 DEED
4
R2D2
atpps://gthhub..cc/aavve//22222
CC 3.0 DEED
5
R3M
MIT
6
Taskonomy
MIT
7
VC-1
CC BY-NC 4.0 DEED
8
VIP
CC BY-NC 4.0 DEED
An overview of the PVRs is given in Table 2
CLIP (Contrastive Language-Image Pre-training) 1 is method to learn joint representation space for images and text through a contrastive learning approach utilizing 400 million image-text pairs. One CLIP instance leverages two neural networks. A vision encoder network for image understanding and an additional language model for textual comprehension. The method uses contrastive objective, which encourages both networks to bring similar image-text pairs closer and push dissimilar pairs apart their shared embedding space. For our experiments we utilize the vision encoder only. More specifically, use the ResNet-50x64 and the ViT-L models.
R3M (Reusable Representations for Robotic Manipulation) tries learn generalizable visual representations for robotics from videos of humans and natural language. The pre-training of the
20

---

Table 2: Overview of used PVRs sorted alphabetically and according the used backbone. In each row we describe the loss, dataset, model architecture and embedding size specifie PVR (M corresponds to I million).
#
Model
Loss
Dataset
Backbone
Emb.
1 Autoencoder Mean squared error
In-domain
In-domain data of the re- spective task
ResNet-50
1536
1
CLIP
Contrastive image-
WIT
400M image-text pairs from the internet
ResNet-50x64
1024
language pre-training
objective
2
R3M
Time- Contrastive video- language alignment pre- training objective
Ego4D
images from sub- set of Ego4D
ResNet-50
2048
3 Taskonomy
Supervised task- dependent encoder- decoder objective
Taskonomy
4M images of indoor scenes from about 600 buildings with annota- tions for every task images from set of Ego4D
ResNet-50
2048
4
VIP
Goal-condi value function pre-training ob- jective
Ego4D
ResNet-50
1024
5 CLIP
Contrastive imaage- pre-training objective
WIT
400M image-texi pairs from the internet
ViT-L
768
language
DINOv2
Discriminative self-
142M images collected from various datasets for classification, seg- mentation, depth estima- tion and retrieval 2000M image-text pairs
VIT-G
1536
supervised pre-training objective
142M
7
OpenCLIP
Contrastive image-
LAION-
VIT-H
1024
language pre-training
2
VCI
Masked Autoencoding (MAE)
Ego4D-
5.6M images Ego4D, manipulation-
VIT-L
1024
MNI
centric data, indoor navigation data, and
ImageNet (i) IM distractor im- ages from Oxford and
9 R2D2
Unsupervised objective estimating reliability and repeatability of keypoints at the same time
(i) Oxford and Paris retrieval, (ii) Aachen Day-Night
FC L2-Net
1024
Paris, (ii) 14067 im-
ages with changing con-
ditions (day-night, sea- son, weather)
representation utilizes the Ego4D dataset using a combination of time-contrastive learning and video-language alignment. An additional LI penalty encourages sparse and compact embedding.
an approach study and understand the relationships between different visual tasks, e.g. object recognition, depth estimation, keypoint detection, semantic segmentation, etc., with the goal leveraging shared representations across tasks. The study trains different mid-level variety of visual tasks using dedicated pre-collected dataset. The works in Sax et al. (37 and Chen et al. (38) show that those mid-level representations can be used training pipeline for downstream visual navigation, well manipulation policies. We utilize subset of these representations for our experiments, namely Autoencoding, 3D Keypoint Detection, 2.5D Segmentation and Z-Depth Estimation. We denote those representations with additional TKY lettering in the experiments.
Taskonomy
56
visual representations
The Taskonomy representations we utilize for experiments use encoder-decoder-ste training objective. The encoder is trained to embed the input image while the decoder trained reconstruct the input image into the pixel-task space (e.g. the segmented image).
casts representation learning from egocentric human videos as an offline goal-conditioned reinforcement learning problem. The authors formulate a self- supervised goal-conditioned value function objective that does not depend on actions. This enables pre-training on unlabeled egocentric human videos. VIP can be understood value-implicit time
VIP Implleti Pre-Training) 32
21

---

contrastive objective that generates a temporally smooth embedding, enabling the value of a state to be implicitly defined via the embedding distance.
53 is method to learn robust visual features from images using combination of different techniques to scale pre-training in terms of data and model size. Furthermore, DINOv2 utilizes a dedicated, diverse, and curated image dataset of 142 million images. The authors trained a ViT-G model with billion parameters that surpasses the best available all-purpose vision model, OpenCLIP, most of the benchmarks at image and pixel levels. For our experiments we use the initial ViT-G model.
DINOv2
OpIICIIP 1 is an open-source implementation of CLIP [. It uses the same paradigm as CLIP, where separate models are used to encode text and images into a single embedding. OpenCLIP models have been trained on a variety of data sources and compute budgets, ranging from small-scale experiments to larger runs. We utilize an OpenCLIP ViT-H model trained on LAION-2B [77].
We utilize an OpenCLIP ViT-H model trained on LAION-2B [77] via the open-source implementation of ImageBind [69]. ImageBind initializes and freezes the image and text encoders using the OpenCLIP model resulting in the same encoders for these modalities.
VC-1 (VisualCortex-1) 1) aims to support diverse range of sensorimotor skills and environments by training them on 5.6 million images of the Ego4D-Mr MNI dataset consisting of the original Ego4D data 7 and additional data from manipulation tasks, navigation tasks and the ImageNet dataset [6]. The Masked Autoencoding (MAE) objective from He et al. (36) is utilized. Due to the similarities to Radosavovic et al. 33 and Xiao et evaluation.
we not include those PVRs based on MAE in our
[58. In order evaluate descriptor representations, we use a pre-trained R2D2 model. This self-supervised method proposes to learn keypoint detection and description in combination with predictor of the local descriptor discriminativeness. The model predicts a set of sparse locations an image that should be repeatably and reliably detected. R2D2 is the only PVR that neither uses a VIT nor a ResNet architecture, but instead employs a fully-convolutional L2-Net (FC L2-Net) [78]. The model is trained Oxford-Pait Paris retrieval [79] and the Aachen-Dat Day Night [80] datasets.
R2D2
For each processed image we choose the top-8 descriptor vectors regarding reliability, repeatability and spatial distance similar to the Spatial Descriptor Set of Manuelli et al. [81] If an image provides less than 8 descriptors we repeat the found descriptors until descriptors are reached. If descriptor is found we use a zero-vector as descriptor. Furthermore, instead of shifting multiple times over the image with different sizes we only shift once and calculate the descriptors on the original size.
A.4 Environments Details
ManiSkil12. For the ManiSkil12 environments we integrate an end-effector velocity controller which prohibits rotations around the x- and y-axis and stabilizes these rotations automatically, This results in 4-dimensional action space (pos,, pos,, pos., rotz). To further increase the realism we include a textured table and a textured floor (freely available under CCO license from hptp::///olyhae... com).
A.5
Return Normalization
For all our evaluations we normalize returns Go by
Go Ga.random
(2)
Go,mazisuum. Go,random
where Garrandom is the return of a random policy and Go.marsmun is the theoretical maximum achievable return (i.e. the maximum achievable timesteps of the environment for ManiSkill2 and DMC; 200 for ManiSkill2, 1000 for DMC and 5 for Miniworld). The return of the random policy is calculated by averaging the returns of 2500 episodes of the random policy.
22

---

B Properties Groups
Table 3: Categorization of PVRs according their properties.
#
Property
Pre-trained Visual Representations (PVRs)
1
ViT
CLIP [ViT], DINOv2, VC-1, OpenCLIP
2
Sequential Data
R3M, VIP VC-1
3
Language Conditioned
R3M, CLIP [VIT], CLIP [RN-50x64), OpenCLIP
4
Diverse Data
OpenCLIP, CLIP [ViT], CLIP [RN-50x64), DINOv2, VC-1
C Linear Layers Versus Multilayer Perceptrons
To ensure that the training does not depend on the choice of encoder size (linear layer MLP), trained such DreamerV3 instances on handful environments with VC-1 as well as representations from scratch. Comparing the results in Figure 9 shows that the performance is not dependent on the choice of encoder size and that differences between the MLPs and linear layers are negligible.
Cartpole Swingup
Cheetah
Walker
Combined (DMC)
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Return (OOD)
0.6
0.2
6.2
0.2
0.2
0.0
0.0
0.0
PickClutteYCC--O0
StackCube-vo
Combined Tasks (Manl5kili2)
1.0
1.0
1.0
0.8
Retur (DDD)
0.4
0.4
0.4
0.
0.2
0.2
0.0
0.8
0.0
0.8
0.0
0.8
(ID)
From Scratch [CNN]
From Scrotch [Lnear]
1VVT
1V1TV MLF]
Figure 9: Performance comparison with representations using linear layers versus multilayer perceptrons. Top row shows normalized ID and OOD performance DMC environments whereas the bottom row shows the performance on ManiSkill2 environments. Differences between MLPs and linear layers are negligible.
23

---

D Performance Curves (All Tasks)
Combined Tasks
Cartpole Swingup
Cheetah Run
Cup Catch
Finger Spin
Walker Walk
(DMC)
1.2
1.00
0.6
0.4
0.4
0.5
0.75
0.50
0.75
0.75
0.2
0.25
0.25
0.00
0.00
1.5M
3M
1.5M
3M
1.5M
3M
1.5M
3M
1.5M
3M
1.5M
3M
Combined Taskss
Assemblinggits-v0
PICKClutter\CB-v00
PlugCharger
StackCube-vo
(ManiSkilli2
Retur (ID)
0.4
0.3
0.12
0.3
0.1
0.0
-0.04
0.00
0.04
-0.06
0.00
0.06
0.15
0.30
0.0
0.2
2M
4M
2M
2M
4M
2M
PickupObjects-v0
(Miniworid)
Miniworld
(ID)
0.75
0.75
0.50
0.50
0.00
0.00
Step
Step
Scratch [CNN]
DINOV2
Autoencoding (TKY)
In Distribution Autoencoder (RN so]
R202 [L2 Netj
30 Keypoint (TRY) JRN 50]
OpenCLIP [VITH]
R3M [RN-50)
Z-Dopth (TRY) [RN-50]
[VIT.L]
(RN-50]
2.50 Seg. (TRY) JRN-50]
CLIP JRN 50x64)
vc.1 [VITL]
Figure 10: Performance and data-efficiency comparison for each task of ManiSkil12, DMC and Miniworld between the different representations. The solid/dashed line shows the mean over multiple runs for DreamerV3/TD-MPC2 The shaded area represents the standard deviation of the respective representation.
E Aggregated Performance of Different Properties (All Tasks)
Combined Tasks
Cartpole Swingup
Cheetah Run
Cup Catch
Finger Spin
Walker Walk
(DMC)
(0OD)
0.66
0.375
0.30
0.7
0.350
0.25
0.6
0.54
0.84
0.20
0.48
0.300
0.80
0.5
0.42
0.10
0.36
0.64
0.72
0.32
0.40
0.915
0.930
0.2
0.3
0.30
0.75
0.5
0.6
Combined Tasks (ManiSkill2)
AssemblingKitss-o
PickClutterCCB-vo
PlugChargervVO
StackCube-v0
(OOD)
0.12
0.0345
0.00100
0.056
0.0330
-0.00125
0.24
0.048
0.0315
-0.00150
0.20
0.06
0.04
0.0300
-0.00175
0.00200
0.16
0.032
0.024
0.08
0.12
0.028 0.032
0.03
0.06
0.2
0.3
0.04
0.08
PickupObjects-vo
(Miniworld)
Miniworld
Return (OOD)
0.6
0.6
0.4
0.4
0.2
0.2
0.8
0.4
Return (ID)
Sequential
Language Conditioned
Diverse
NoT
Non. Sequential Data
Languaee Conditioned
Diverse Data
Figure 11: IQM return of the different properties task level. Each marker represents the interquartile mean performance an individual group.

---

Transformer Block Ablations with VC1
Walker Walk (DMC)
0.75
Return (ID)
0.50
0.25
0.00
600K
1.2M
1.8M
2.4M
3M
Step
From Scratch [CNN]
VC-1 [Full VIT-L]
VC-1 (%/ VIT-L]
VC-1 (% VIT-L]
Figure 12: Ablation of transformer blocks in VC-1. Using of VC-1 results in similar performance compared to the full model. Transformer blocks near the final one seam to offer as much information the final output. With only 1/3 of VC-1 the performance drops significantly. It seems that earlier representations do not offer enough information for the MBRL agent to perform better or similarly.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Not needed.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Can be found in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
No proofs needed and included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We described our experimental setup in Section 3 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer:
[No]
Justification:
Not possible due to institutional regulations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. cttpsscccppbccccccc/eccceccccccc http:/ppllcggguisss.eeeee http:/CooeSSmmiiseeee ssiiiioisiyiiyiiiccoooococpiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All necessary details are described in the main paper. Further details about the implementation can be found in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report where possible and meaningful.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
At the beginning of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Not needed since we evaluate existing methods standard benchmarks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All used implementations and materials are cited and/or linked in the appendix. We state the license of those additionally.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets..papersstt wwwppppipphhheeeecoppppppoo thoode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "kj9u5dAcAG", "reviewer": "Reviewer_mY2R", "summary": "Pre-trained Visual Representations (PVRs) has been widely applied to many domains to improve OOD generalization and sample efficiency, including model-free reinforcement learning (RL). This paper explores the application of PVRs in model-based RL, which has not been done. Experiments on two suite of simulated control environments (Deepmind control and Maniskill2) show that current PVRs are outperformed by representations learned from scratch in terms of sample efficiency and OOD generalization. The paper further analyzes the reward error and dynamics prediction error of various PVRs and provides meaningful insights.", "questions": "- Different tasks can have very different reward functions even for the same environment, and we have no clue on what the reward function is when pre-training self-supervised visual representations. Then, why should we expect a single PVR to perform well on learning good reward models for all kinds of tasks? In other words, is the task of building PVRs that can learn good reward models a somewhat intractable one?", "limitations": "The proposed benchmark only evaluated on two suite of simulated control environments, which, as the authors also pointed out in the paper, is (1) somewhat insufficient, and (2) very challenging for the PVRs concerned because many of them are never trained on similar data distributions (especially for DMC). Therefore, saying that PVRs are \"ineffective\" or MBRL seems like an overstatement to me. Strong claims need strong evidence and the evidence provided in the paper is in my opinion insufficient. Of course I still appreciate the value of the proposed benchmark in that it brings the problem of applying PVRs to MBRL to the community's attention, but I do not agree with putting such an assertive statement as the title of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow. The research questions that the paper investigate are clearly stated. The proposed benchmark is well-motivated. \n- The proposed benchmark has great potential in advancing research in developing better PVRs for MBRL.\n- The paper provides sufficient detail for the experiments allowing easier reproduction of the results.\n- The experiments includes a large suite of popular PVRs, improving the credibility of the conclusions.", "weaknesses": "- The number of algorithms in each property category is small, which means that the performance of one algorithm can easily affect the average performance of the category, making the comparison of different categories (Figure 5) less significant. For example, the \"sequential data\" category and the \"temporal loss\" category only differ in one method (VC-1), and attributing the difference caused by this one method to the sequential data property might not be correct, since there are many other differences between VC-1 and R3M/VIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Weaknesses:__\n\n> “The number of algorithms in each property category is small, which means that the performance of one algorithm can easily affect the average performance of the category, making the comparison of different categories (Figure 5) less significant. For example, the \"sequential data\" category and the \"temporal loss\" category only differ in one method (VC-1), and attributing the difference caused by this one method to the sequential data property might not be correct, since there are many other differences between VC-1 and R3M/VIP.”\n\nWe integrated the mentioned categories “Sequential Data” and “Temporal Loss” since VC-1 is trained on sequential data but does not make use of this characteristic. While we agree that your statement is true for these two categories, the PVRs of the other categories represent a more diverse set of representations. In the other categories, especially “ViT” and “Diverse Data”, the performance differences are significant. Therefore, we will revise our paper accordingly and will remove the category “Temporal Loss” from our findings since the overlap between this category and “Sequential Data” is too large. Thank you for this tip.\n\n__Questions:__\n\n> “Then, why should we expect a single PVR to perform well on learning good reward models for all kinds of tasks?”\n\nThis expectation was established by various other papers referenced in our relevant work and by representations like VC-1, VIP, etc. We do not want to advocate the use of PVRs in MBRL per se. Our work is a benchmark of PVRs following the zero-shot idea established by previous works on MFRL and imitation learning. These works argue that PVRs can be helpful in downstream reinforcement learning tasks because the representations saw vast amounts of different data during training and/or were trained with losses capturing some form of value. The reason why we created the presented benchmark is precisely to challenge these claims. We do not make any claims regarding the capability of the investigated PVRs to capture features relevant for arbitrary reward models for all kinds of tasks. Our results highlight that there is not a single PVR that outperforms from scratch training in every environment.\n\n> “In other words, is the task of building PVRs that can learn good reward models a somewhat intractable one?”\n\nIt is most likely intractable to train a single PVR capturing reward information for all possible tasks. But our results suggest that e.g. diverse but curated datasets might help for specific tasks. VC-1 especially shows that a combination of diverse datasets taking e.g. manipulation aspects into account is an important element for a PVR that is used for manipulation tasks. However, VC-1 still cannot solve all tasks as effectively as learning a representation from scratch with MBRL (i.e. solving DMC tasks).\n\nFurthermore, both learning phases (pre-training of representations and downstream RL policy training using such PVRs) are heavily influenced by an objective mismatch since the pre-training objective and the RL training objective usually differ much. PVRs are not incentivized to capture reward related information from the images and might discard that information during learning whereas RL training objectives are driven by a reward function (see UMAPs in Figure 1 in PDF of Author Rebuttal).\n\n__Limitations:__\n\n> “...because many of them are never trained on similar data distributions”\n\nIndeed, the PVRs are often not trained on data distributions similar to our experiments. But we want to mention that many of the original papers of the PVRs used actually claim that their representations work on DMC and robotics environments like ManiSkill2 or even provide experiments supporting those claims (see VC-1, VIP, etc.). E.g. VIP was trained on Ego4D only. The related paper [1] shows that algorithms using VIP can solve robotic manipulation tasks. VC-1 was trained on diverse control data and was shown to work on DMC as well. The same paper includes experiments showcasing that VIP and R3M enable RL policies to be SOTA in DMC. The Taskonomy representations were used to solve tasks of the VizDoom environments and robotic manipulation tasks. We used our presented environments to show that those claims do not hold for MBRL. \n\n> “Of course I still appreciate the value of the proposed benchmark in that it brings the problem of applying PVRs to MBRL to the community's attention, but I do not agree with putting such an assertive statement as the title of the paper.”\n\nWe understand your concerns about the assertiveness of our claims and title. With an additional navigation experiment from the Miniworld [2] environment we will improve the paper to temper these claims, ensuring they more accurately reflect the evidence presented. You can see those results in Figure 2 in the PDF of the Author Rebuttal above. The new evidence still supports our original hypothesis. \nOur intention in choosing a somewhat provocative title is to highlight the above-mentioned potential limitation of current PVR approaches when applied to MBRL (as you mention yourself) and to challenge the common assumption that PVRs *generally* improve RL training. We do not claim that PVRs are worse in every environment, but many PVRs are presented as general foundation models, implying they should perform well on DMC as well. However, they do not, which is reflected in our title.\n\n[1] Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., & Zhang, A. (2023). VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training. In The Eleventh International Conference on Learning Representations.\n\n[2] Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R. D. L., Willems, L., Lahlou, S., … Terry, J. K. (2023). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X9cCNPROWR", "reviewer": "Reviewer_HgP2", "summary": "This paper conducts a thorough set of experiments to evaluate the performance of pretrained visual representations (PVR) in model-based reinforcement learning. Empirical results show that PVR performs worse than learning from scratch, which is possibly due to the large reward prediction error.", "questions": "- I think the performance degradation of PVR is mainly due to the fact that they are not trained on the experimental data used in this paper, as Autoencoder performs similarly compared to learning from scratch. Therefore, could you fine-tune the pre-trained visual encoder and then evaluate the performance? I guess slight fine-tuning can boost the performance a lot.\n- Why does VC-1 perform almost the best in maniskill (see figure 3 and figure 4)? Is VC-1 specifically pre-trained on these manipulation data?\n- Another possible reason is that models pre-trained on large datasets are usually hard to transfer to low-data regimes. See UniSim [1] for details.\n\n[1] Yang et al., Learning Interactive Real-World Simulators, ICLR 2024", "limitations": "Limitations have been discussed in this paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper shows how pre-trained visual representations perform in MBRL, which has been relatively unexplored in previous research.\n- A thorough set of experiments reveals that pre-trained representations perform worse than learning from scratch, which is surprising and could inspire future research to analyze this phenomenon further.", "weaknesses": "- This paper gives some possible reasons for the performance degradation of pre-trained representations. However, it lacks sufficient evidence to further support the claims. For example, Why does the reward prediction accuracy outweigh the dynamic prediction accuracy? Why do pre-trained representations lead to information loss about reward prediction? I encourage the author to dig deeper into the specific properties of pre-trained representations used in MBRL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Weaknesses:__\n\n> “Why does the reward prediction accuracy outweigh the dynamic prediction accuracy?”\n\nOur Figures 6 and 7 show that there exist differences in the dynamics as well as reward prediction accuracies between the trained models. The calculated correlations indicate that more subtle differences in reward predictions compared to dynamics predictions lead to poorer performing policies. Even though all models learn to predict the dynamics and are incentivized to do so via the losses of DreamerV3 and TD-MPC2, the best measure of task performance is the reward. Thus, its prediction quality is intuitively stronger correlated to task performance (as shown in our paper with the mentioned correlations). Since the predictions are directly used to update the policy, predictions must be accurate.\n\n> “Why do pre-trained representations lead to information loss about reward prediction?”\n\nWe argue that those representations lose information about rewards during pre-training since they are most often trained with different training objectives than later used during the MBRL phase. That such an objective mismatch influences the training was already shown for MBRL itself, as MBRL methods like Dreamer and TD-MPC need to learn the dynamics and a policy with conflicting objectives. The additional mismatch in objectives between PVRs and MBRL is now an additional factor. In general, the PVRs are trained to compress the given information content (i.e. information bottleneck theory) and are not incentivized to capture information in the images relevant for a possible reward function.\n\n__Questions:__\n\n> “Therefore, could you fine-tune the pre-trained visual encoder and then evaluate the performance?”\n\nWe actively decided to omit fine-tuning experiments since the original promise of many of the used representations is to perform (near) SOTA on downstream tasks without additional fine-tuning and we wanted to examine these claims. Fine-tuning is therefore not the focus of our paper. Furthermore, correctly fine-tuning might not be an easy task since it can destroy parts of the representation needed for good generalization performance [1]. Ultimately, we believe that (if done right) fine-tuning can boost performance for each of the representations. But we are unsure if the performance boost would outperform training from scratch. This aligns with mixed results from end-to-end fine-tuning in the original VC-1 paper [2]. Furthermore, fine-tuning before the RL phase would also result in an unfair advantage compared to training the representation from scratch. But we believe that the question of how to adequately fine-tune such representations for MBRL and RL in general is nevertheless important and should be the topic of future research.\n\n> “Why does VC-1 perform almost the best in maniskill (see figure 3 and figure 4)? Is VC-1 specifically pre-trained on these manipulation data?”\n\nVC-1 was trained on a combination of datasets with a connection to manipulation and navigation. More specifically, VC-1 uses Ego4D and similar human manipulation as well as navigation data (but never robot manipulation data). We hypothesize that in this case the domain gap is smaller for ManiSkill2 due to the connection of the pre-training data to manipulation. Therefore, a combination of manipulation related data seems to be somewhat beneficial; even if it's observations of humans.\n\n> “Another possible reason is that models pre-trained on large datasets are usually hard to transfer to low-data regimes. See UniSim [1] for details.”\n\nWe thank you for pointing out this important part of UniSim, since it is relevant for our conclusion. The paper states \"During joint training of the UniSim on diverse data, we found that naïvely combining datasets of highly varying size can result in low generation quality in low-data domains.\". As already mentioned, VC-1 uses a curated combination of datasets which supports this claim of UniSim.\n\n[1] Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., & Liang, P. (2022). Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. In International Conference on Learning Representations.\n\n[2] Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., … Meier, F. (2023). Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? In Thirty-seventh Conference on Neural Information Processing Systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdW3c2IDgA", "reviewer": "Reviewer_DQ5X", "summary": "This paper contains an extensive study on the potential benefits of pre-trained visual representations (PVRs) for model-based reinforcement learning (MBRL). Given the success of PVRs for model-free RL (MFRL), there are reasons to believe that PVRs are equally beneficial for MBRL, something that has not yet been explored to the same degree. Surprisingly, tests on two different RL architectures in combination with at least eight different PVR variants indicate that PVR-based solutions rarely perform better than representations learned from scratch when applied for MBRL. In most cases, it seems preferable not to use PVRs for MBRL, which should come as a surprise to many, given recent advances in applications based on PVRs.", "questions": "* What is believed to be the reason why the benefits of PVRs for MBRL seem so limited, in particular, given earlier success for MFRL?\n* To what extent do the tested PVRs preserve spatial information relevant for tasks such as manipulation?", "limitations": "The fact that more experiments are needed to draw final conclusions is highlighted as a limitation, which is true indeed given the diversity of domains for which MBRL can be used and the computational demands for experiments in each such domain.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Given the recent development of PVRs and the increasing number of successful applications of PVRs, it is interesting to know what the opportunities and limiting factors are for PVRs when adopted for training agents using reinforcement learning. A study with this as a focus, such as the one presented in this paper, should interest many. Besides the most important conclusion drawn in the presented study, i.e. the fact that the benefits of PVRs for MBRL can be questioned, it contains several experiments to paint a more complete picture. \n\nIt is concluded that the sample efficiency of MBRL agents rarely improves when supported by PVRs compared to those with representations learned from scratch. PVRs do not seem to make agents more robust of out-of-distribution (OOD) conditions either. It is also shown that the dynamics and reward prediction errors of RL agents are similar regardless of whether representations are learned from scratch or with PVRs. Also interesting is a study on the benefits of different properties of PVRs for generalization, where the most important property was shown to be data diversity during training, which might not be that surprising. The fact that language conditioning and the choice of transformer architecture have such a little impact might be more unexpected though.", "weaknesses": "From the experiments, it can be concluded that PVRs hardly benefit MBRL, but the paper does not really try to answer why there is a difference between MBRL and MFRL in this regard. Could it be that the way DreamerV3 and TD-MPC2 are modified in this study, the PVR modules $g_*$ are trained to preserve semantic information, while in effect also suppressing spatial information? Earlier studies on PVRs for MFRL, cited in the paper, do not seem to do this but instead keep a path that allows a policy to be trained with some spatial information left intact. \n\nThe authors seem surprised that language conditioning is not good for generalization, since it should provide semantically relevant features. Such a claim completely misses the fact that for e.g. manipulation it is often more important for a system to know where an object is located, how it is oriented and how large it is, than what class the object belongs to or any other semantic information. Most of our visual representations are explicitly trained to be invariant to transformations that might be particularly important for the task given to the agent.\n\nIt is worth noting that the experiments were conducted in simulation, not on real robots. The paper does not draw conclusions that go beyond what can be observed in the experiments, but the differences between simulated and real-world conditions are not really regarded. There are other potential benefits of MBRL when applied to real robots. With MBRL any action executed on the system can be exploited for pre-training, while MFRL is typically focused on only one particular type of task. It might be that this would not really affect the conclusions as expressed in the paper, but the experimental conditions between simulations and real-world experiments can be very large, The same is true regarding conclusions drawn from the OOD experiments. In the paper, the nature of randomizations is the same for both the ID and OOD sets, even if the two sets don’t overlap. If you were to do the experiments in the real world, the conditions would be very different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Weaknesses:__\n\n> \"From the experiments, it can be concluded that PVRs hardly benefit MBRL, but the paper does not really try to answer why there is a difference between MBRL and MFRL in this regard.\"\n\nWe believe that multiple objective mismatches in the different training phases, which are not all present in MFRL, result in a difference. We already discussed this in the first bullet point of the Author Rebuttal above and will include this in the revised paper.\n\n> “Could it be that the way DreamerV3 and TD-MPC2 are modified in this study, the PVR modules 𝑔∗ are trained to preserve semantic information, while in effect also suppressing spatial information?”\n\nPrevious studies (described in the related work) sometimes use the PVRs in combination with additional task information (like proprioceptive or spatial information) or use the output of the PVRs only. Since we wanted to test the capabilities of the PVRs only, we refrain from using additional task information as input for the MBRL agents since it is hard to measure to what extent the PVRs or the additional information are used to solve the task. We are unsure why the reviewer believes that we actively suppress spatial information since we adapt DreamerV3 and TD-MPC2 only slightly? Many of the analyzed PVRs learn spatial information (e.g. VC-1, R2D2, Taskonomy Autoencoder).\n\n> “The authors seem surprised that language conditioning is not good for generalization, since it should provide semantically relevant features.”\n\nWe would like to thank the reviewer for drawing our attention to this misunderstanding. We never wanted to convey this opinion but wanted to bring attention to the fact that many other approaches build up on language without utilizing additional spatial information (e.g. R3M and those mentioned in our paper). These papers utilize language in their approaches, asserting that it significantly enhances performance. We wanted to emphasize this perspective. We will revise this part of our paper for the final version.\n\n> “It is worth noting that the experiments were conducted in simulation, not on real robots. The paper does not draw conclusions that go beyond what can be observed in the experiments, but the differences between simulated and real-world conditions are not really regarded.”\n\nWe appreciate the suggestion to train on real robots and will include this as an additional limitation in the paper. Conducting experiments on real robots is computationally demanding and time-consuming, particularly since our approach involves reinforcement learning rather than imitation learning. On the other hand, our approach was to benchmark the touted zero-shot capabilities of pre-trained visual models, rather than to devise a high-performant (MB)RL agent which performs well in real world tasks.\n\n__Questions:__\n\n> “What is believed to be the reason why the benefits of PVRs for MBRL seem so limited, in particular, given earlier success for MFRL?”\n\nWe need to train a model of the environment as well as a reward predictor with those representations. The information preserved by the representations might not be enough to do both. (See also the related answer from the first weakness and Author Rebuttal above.)\n\n> “To what extent do the tested PVRs preserve spatial information relevant for tasks such as manipulation?”\n\nThis individually depends upon the training loss of the PVR. While R3M and all versions of CLIP use semantic information via language, the losses of the other PVRs allow learning spatial information implicitly.\n\n__Limitations:__\n\n> \"The fact that more experiments are needed to draw final conclusions is highlighted as a limitation, which is true indeed given the diversity of domains for which MBRL can be used and the computational demands for experiments in each such domain.\"\n\nWe added an additional task from another environment domain (i.e. PickupObjects-v0 from Miniworld [1]). You can see the results in Figure 2 in the PDF attached to the Author Rebuttal at the top. The new evidence supports our original claim that PVRs do not generally improve MBRL training and performance.\n\n[1] Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R. D. L., Willems, L., Lahlou, S., … Terry, J. K. (2023). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bj2T97gmTI", "reviewer": "Reviewer_32eA", "summary": "This paper presents a study on pre-trained visual representations (PVRs) to improve sample efficiency of model-based reinforcement learning methods (MBRL). It studies a range of PVRs with different architectures, pre-training objectives and data modalities on two recent model-based algorithms: DreamerV3 and TD-MPC2. The authors propose to replace image observations with frozen PVRs features, replacing the original CNN encoder/decoder with a trainable linear layer. The paper focuses on 9 diverse continuous control tasks from 2 different domains: The DeepMind Control Suite (DMC) and ManiSkill2. The results show that the use of PVRs surprisingly does not help to improve performance and has diminishing returns compared to training encoder representations from scratch.", "questions": "Did the authors experiment with fine-tuning the PVRs during training instead of using frozen representations ? It would have been interesting to experiment with optimizable PVRs (without reconstruction loss in the case of DreamerV3). Maybe the use of PVRs could exempt DreamerV3 from using a reconstruction loss for learning encoder representations. The use of frozen representations may also be the cause of the observed results, all information details required to achieve good performance may not be recoverable from the compressed pre-trained representations.\n\nDid the authors experiment with reconstructing the image observations from frozen PVRs in order to study the possibility of recovering image information from representations ? Also to assess the quality of trajectories imagined by the world model.\n\nDid the authors experiment with using representations of other layers than the final layer? The usefulness of representations of pre-trained models can vary greatly depending on the chosen layer. Are the results different when using the features of inner layers or a collection of layers features ?\n\nFor the in-distribution autoencoder PVR, what policy is used to generate the data used for pre-training ? Is it a random policy, a collection of policies to ensure diverse behaviors, or a pre-trained policy ? I ask the question because during training, the PVR should provide representations for the widest distribution of agent trajectories or positions. DreamerV3 has sometimes troubles to reconstruct agent situations that are seen often when training on DMC tasks.\n\nI would be ready to increase my score if additional experiments and analysis are provided by the authors.", "limitations": "The authors adequately addressed the limitations in the conclusion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a very interesting study on the application of PVRs for two recent model-based approaches. The use of PVRs for model-based RL is a promising area of research. The performance and learning speed of model-based agents like Dreamer is very linked to the quality of representations learned by the world model. Benefiting from already pre-trained representations to reduce training time or increase final performance would be very appreciated since the training time of model-based agents can take multiple days for some advanced tasks.\n\nThe paper evaluates a diverse range of pre-trained models, showing the normalized returns of each pre-trained model. The authors find that learning the encoder representation from scratch leads to better performance for DreamerV3 and TD-MPC2, contradicting expectations following previous works.\n\nThe paper also experiments with out-of-distribution (OOD) versions of the environments (colors, object sizes), finding that PVRs do not help either to improve performance in this setting.\n\nThe paper is clear and well written.", "weaknesses": "The study on the potential effect of using PVRs on MBRL methods is limited. The paper only studies the use of frozen PVRs, where the world model gradients are not back-propagated to the pre-trained weights during training. Solving RL tasks often requires the agent to pay attention to small details and learn very detailed representations, especially for DMC tasks. It would have been interesting to study the use of PVRs without frozen weights during training.\n\nAs mentioned in the conclusion limitations, the authors only study the use of PVRs for continuous control tasks. It would have been interesting to study the application of PVRs to another domain based on discrete actions (Atari games, DeepMind Lab 3D environments, Minecraft...) or more aligned with data seen during PVR pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Questions:__\n\n> “Did the authors experiment with fine-tuning the PVRs during training instead of using frozen representations ?”\n\nWe actively decided to omit fine-tuning experiments since the original promise of many of the used representations is to perform (near) SOTA on downstream tasks without additional fine-tuning. Fine-tuning itself might not be an easy task since it can destroy parts of the representation needed for good generalization performance [1]. Ultimately, we believe that (if done right) fine-tuning can boost performance for each of the representations. But we are unsure if the performance boost would outperform training from scratch. This aligns with mixed results from end-to-end fine-tuning from the original VC-1 paper [2].\n\n> \"Maybe the use of PVRs could exempt DreamerV3 from using a reconstruction loss for learning encoder representations.\"\n\nWe wanted to keep the MBRL algorithms (DreamerV3 and TD-MPC2) as original as possible. In DreamerV3, the reconstruction term is an integral part of the world model's loss and can neither be replaced nor omitted easily. As in the original algorithm, we utilize reconstructions so that the world model is able to include information about the state also in future predictions. More specifically, reconstruction is not only needed to learn the encoder and decoder networks, but also to learn long-term dependencies via the dynamics prediction. Besides, [3] showed that Dreamer without reconstruction results in poorer overall performance. On the other hand, similar reasons motivated us to include TD-MPC2 as an additional algorithm; it does not involve reconstructions and is more decision-aware (i.e. relying more on rewards).\n\n> “Did the authors experiment with reconstructing the image observations from frozen PVRs in order to study the possibility of recovering image information from representations ?”\n\nWe think the latter suggested experiment is an interesting approach for PVRs in combination with world models. Such experiments would take a similar line as our dynamics prediction accuracy experiments. Therefore, we did not perform such experiments and would leave them for future research. Besides, recovering image information is not necessarily an indicator for good reward or task-specific dynamics prediction performance since information needed for high rewards might not be present in the representation even if the overall reconstruction quality is good.\n\n> “Did the authors experiment with using representations of other layers than the final layer? “\n\nWe conducted additional experiments during the rebuttal time by removing the transformer blocks of VC-1 partly (more specifically we ablated ⅔ and ⅓ of the 24 transformer blocks). Results can be seen in Figure 3 of the PDF to the Author Rebuttal above. Using ⅔ of VC-1 results in similar performance to the full transformer. This suggests that transformer blocks near the final one offer as much information as the final output. With only ⅓ of VC-1 the performance drops significantly. It seems that earlier representations do not offer enough information for the MBRL agent to perform better or similarly.\n\n> “For the in-distribution autoencoder PVR, what policy is used to generate the data used for pre-training ? Is it a random policy, a collection of policies to ensure diverse behaviors, or a pre-trained policy ?”\n\nWe utilized data collected with a DreamerV3 policy during its training phase, including data from the exploration phases. We used an agent utilizing VC1 as an encoder. The motivation of using VC-1 as encoder was to enlarge the data distribution explored by the algorithm in comparison to a better performing agent training from scratch. This approach ensures that the autoencoder encounters a diverse data distribution similar to that of the other agents during their training, thereby ensuring also a fair comparison.\n\n__Weaknesses and Limitations:__\n\n> \"... the authors only study the use of PVRs for continuous control tasks. It would have been interesting to study the application of PVRs to another domain based on discrete actions (Atari games, DeepMind Lab 3D environments, Minecraft...) or more aligned with data seen during PVR pre-training.\"\n\nWe were able to generate additional results with a Miniworld [4] environment which supports discrete actions (see Figure 2 in the PDF attached to Author Rebuttal above). The new evidence supports our original claim that PVRs do not generally improve MBRL training and performance. Furthermore, we contend that the ManiSkill2 experiments are well aligned with the pre-training data of many PVRs (e.g., VC-1, R3M, VIP as promoted in the associated papers), and we were also interested in exploring their out-of-distribution performance in out-of-distribution domains like DMC.\n\n\n[1] Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., & Liang, P. (2022). Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. In International Conference on Learning Representations.\n\n[2] Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., … Meier, F. (2023). Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? In Thirty-seventh Conference on Neural Information Processing Systems.\n\n[3] Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2020). Dream to Control: Learning Behaviors by Latent Imagination. In International Conference on Learning Representations.\n\n[4] Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R. D. L., Willems, L., Lahlou, S., … Terry, J. K. (2023). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the Reviewers for their thorough feedback. We appreciate the detailed suggestions and the recognition of the value that our results bring to the community. We were keen to integrate the Reviewers valuable feedback into our paper and will make the revisions to the paper accordingly. Here is a summary of changes and answers:

- Reviewer HgP2 raised questions regarding the underlying causes of the poor performance in reward prediction. We ultimately believe that this is due to multiple objective mismatches which are present between the pre-training of the visual representations (PVRs) and the actual MBRL training phase, as well as in MBRL itself where we have a mismatch between dynamics learning and policy learning [1]. Especially, the first mentioned objective mismatch complicates the transfer of reward information, since PVRs are trained to compress information (information bottleneck theory) and as such the training might not capture reward-relevant data from the pre-training images. On the other hand, MBRL methods like DreamerV3 and TD-MPC2 heavily rely on reward information in their objective. We added UMAP embeddings (see Figure 1 of the attached PDF) showing that reward information is more closely embedded in representations which are learned by the agents completely from scratch. Regarding previous results we additionally want to mention that reward information was often irrelevant in previous benchmarks based on imitation learning.
- We got several reviewer questions regarding fine-tuning some of the PVRs on task-specific data (HgP2;32eA). Since we wanted to test the touted zero-shot capabilities of the PVRs, we consciously decided against additional fine-tuning experiments. Fine-tuning is therefore not a focus of our paper. Furthermore, since fine-tuning causes its own difficulties and challenges (e.g. it can distort features [2]), we want to leave those research questions for future research.
- Reviewer 32eA suggests ablating the PVRs as it was already done in previous benchmarks. Therefore, we conducted an experiment with DreamerV3 and VC-1 removing ⅓ and ⅔ of the 24 transformer blocks of VC-1. Results can be seen in Figure 3 of the attached PDF. Outputs of earlier transformer blocks do not exceed the performance of the full representation. 
- One way to gain further insights is to expand task domains beyond DMC and ManiSkill2 as also pointed out by reviewers mY2R and DQ5X. We already acknowledged this in our limitations. Both mentioned domains support continuous actions only. We now included an additional navigation experiment from Miniworld [3]. Due to the discrete action-space and the short rebuttal time, we were only able to perform the experiment with DreamerV3 and a selected number of PVRs. TD-MPC2 does not support discrete action spaces. The results can be seen in Figure 2 of the attached PDF. The new evidence supports our original claim that PVRs do not generally improve MBRL training and performance. Similar to the other experiments, DreamerV3 agents trained from scratch are more sample efficient and performant compared to agents using PVRs. Only VC-1 is able to perform comparably.

On the other hand, Reviewers highlighted the clarity and thoroughness of our paper (mY2R;32eA), the novel exploration of PVRs in MBRL (HgP2), and the well-motivated benchmark (including out-of-distribution evaluation) that advances research in this area (mY2R;32eA). Reviewer DQ5X highlighted that such a “study with this as a focus, such as the one presented in this paper, should interest many.”. The reviewers also valued our surprising findings that pre-trained representations often perform worse than learning from scratch (HgP2;32eA;DQ5X) and the large number of PVRs analyzed (mY2R;32eA).

We sincerely hope that the changes and answers have addressed reviewer concerns. If so, we kindly request that you consider revising your scores. Please let us know if you have any further comments!

[1] Lambert, N., Amos, B., Yadan, O., & Calandra, R. (2020). Objective Mismatch in Model-based Reinforcement Learning. In Learning for Dynamics and Control (pp. 761–770). PMLR.

[2] Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., & Liang, P. (2022). Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. In International Conference on Learning Representations. 

[3] Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R. D. L., Willems, L., Lahlou, S., … Terry, J. K. (2023). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Meta-review: The paper's central hypothesis is to provide experimental evidence that PVR is ineffective for MBRL and learning representations from scratch is as good (or even better) than using pre trained features. The reviewers generally agree that the presented experiments are barely enough to demonstrate this phenomenon, while suggesting a few improvements. Firstly, conducting more experiments is required to assess the generality of this hypothesis -- including but not limited to increasing the number of methods in each property category. Secondly, further analysis is needed to understand the underlying reasons for this observation. I agree with the reviewers that these are valid weaknesses of the paper. Particularly, lack of enough analysis on "why" PVR is not beneficial to MBRL can be explored further and with additional experiments and analysis, it may become apparent that the findings are limited to the specific environments and methods tested. However, I believe the paper's primary conclusions are significant and thought-provoking enough to warrant consideration as a valuable contribution to MBRL. Therefore, I recommend accepting the paper as a poster.
Decision: Accept (poster)