Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning
Authors: Zhecheng Yuan
Zhengrong Xue
Bo Yuan
Xueqian Wang
Yi Wu
Yang Gao
Huazhe Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Visual RL
Generealization
Pre-trained Models

Abstract: Learning generalizable policies that can adapt to unseen environments remains challenging in visual Reinforcement Learning (RL). Existing approaches try to acquire a robust representation via diversifying the appearances of in-domain observations for better generalization. Limited by the specific observations of the environment, these methods ignore the possibility of exploring diverse real-world image datasets. In this paper, we investigate how a visual RL agent would benefit from the off-the-shelf visual representations. Surprisingly, we find that the early layers in an ImageNet pre-trained ResNet model could provide rather generalizable representations for visual RL. Hence, we propose Pre-trained Image Encoder for Generalizable visual reinforcement learning (PIE-G), a simple yet effective framework that can generalize to the unseen visual scenarios in a zero-shot manner. Extensive experiments are conducted on DMControl Generalization Benchmark, DMControl Manipulation Tasks, Drawer World, and CARLA to verify the effectiveness of PIE-G. Empirical evidence suggests PIE-G improves sample efficiency and significantly outperforms previous state-of-the-art methods in terms of generalization performance. In particular, PIE-G boasts a 55% generalization performance gain on average in the challenging video background setting. Project Page: https://sites.google.com/view/pie-g/home.

Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning
Zhecheng Yuan', Zhengrong Xue?,
Bo Yuan Xueqian Wang', Yi Wu Yang Gaol.,, Huazhe Xu
Tsinghua University" Shanghai Jiao Tong University
Qianyuan Institute of Sciences' Shanghai Qi Zhi Institute yuanzc200mails. wtt.aaigssssssttttaaaaaaaaaaaaaa wwngghua.edu.cc. cn, mubuazhe122gmaill
Abstract
Learning generalizable policies that can adapt unseen environments remains challenging in visual Reinforcement Learning (RL). Existing approaches try to acquire robust representation via diversifying the appearances of in-domain observations for better generalization. Limited by the specific observations of the environment, these methods ignore the possibility of exploring diverse real- world image datasets. In this paper, we investigate how visual RL agent would benefit from the off-the-shelf visual representations. Surprisingly. we find that the early layers in ImageNet pre-trained ResNet model could provide rather generalizable representations for visual RI.. Hence, we propose Pre-trained Image Encoder for Generalizable visual reinforcement learning (PIE-G), simple yet effective framework that can generalize to the unseen visual scenarios in a zero- shot manner. Extensive experiments are conducted on DMControl Generalization Benchmark, DMControl Manipulation Tasks, Drawer World, and CARLA to verify the effectiveness of PIE-G. Empirical evidence suggests PIE-G improves sample efficiency and significantly outperforms previous tfate-of-the-an tttt-arrt methods terms of generalization performance. In particular, PIE-G boasts 55% generalization performance gain on average in the challenging video background setting. Project Page: https://sitee: google. com/ cttp///vew//ppaa w/ww//ii---/-ee g/ hoom
Introduction
Visual Reinforcement Learning (RL) has achieved significant success in learning complex behaviors
directly from image observations [88 35 37]. Despite the progress, RI. agents are often plagued by the overfitting problem [67]. especially in high-dimensional observation space. Previous studies show that is difficult for the visual agents to generalize to unseen scenarios limits their deployment in real-world applications.
g 40 which severely
In general, visual RL methods rely on their encoders to learn a visual representation to perceive the world. Recent studies have found that data augmentation [65] leads more generalizable representations so that the agents can adapt to the unseen environments with different visual appear- ances [58 74). However, most of those approaches only augment the observations of the training environments 37 39 68), which is unable to provide enough diversity for generalization over large domain gaps. Furthermore, naively applying data augmentation may damage the robustness of learned representations and decrease training sample efficiency 28
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Stop Gradient
Replay Buffer
Eavironment
Gradient
MAAGNET
C
Actor
Action
Representation
Observation
Critic
Q-value
Pre-truined Encoder
Figure 1: Overview of PIE-G. This figure shows the general framework of PIE-G where visual encoders embed high-dimensional images into low-dimensional representations for downstream decision-makingg tasks. Instead of training the encoder from scratch, PIE-G selects an ImageNet pre-trained ResNet model as the encoder and freezes its parameters during the entire training process.
To overcome these drawbacks, what require is universal representation that generalize to variety of unseen scenarios. Recent works representation learning demonstrate promising results in enabling pre- pre-trained models provide strong priors for downstream tasks [I The pre- traitre models contain representations obtained from a wide range of existing real-world image datasets. These representations despite the diversity and the inconsistency [2]. Based on the observations, we would like to ask the following question: is it possible train visual RI. agent that is augmented with pre-trained visual representations so that it can better generalize to novel tasks?
proved to be robust to noises and capable of distinguishing salient features
Towards answering the question, the main contribution of this paper is surprising discovery that the off-the-shelf features of frozen models trained with ImageNet can be used as universal representations for visual RL. Based on such findings, present Pretrained Image Encoder for Generalizable visual reinforcement learning (PIE-G), a visual RL framework that allows agents to obtain enhanced training efficiency and generalization ability via integrating the extracted representations from pre-traine rre-trained ResNet [30] encoder into RL training. Straightforward as the framework appears, PIE-G enjoys thoughtful details and nuanced design choices acquire representations that suitable for control and generalizable to novel scenarios. Specifically, we show that the choice of early layer features and the ever-updating Batch Normalization (BatchNorm)
33 crucial for the performance gain.
To validate the effectiveness our framework, we conduct a series of experiments on 4 benchmarks: DMControl Generalization Benchmark (DMC-GB) (26). DMControl Manipulation Tasks (72).
Drawer World [75] that is modified from Meta World [82. and CARLA, a realistic autonomous driving simulator. Empirical studies have demonstrated that PIE-G achieves better or competitive results in training sample efficiency, and significantly outperforms previous state-of-the methods in generalization capability without bells and whistles.
Our main contributions are summarized as follows: (i) We find that pre-trained encoders from off-the-shelf image datasets with the early layer features and ever-updating BatchNorm provide generalizable representations in visual RL. (ii) We propose PIE-G, a simple yet effective framework with pre-trained encoder that boost the sample efficiency and generalization ability in visual RL. (iii) PIE-G outperforms state-of-the-arr methods in visual generalization benchmarks by large margin, with 55% boost average on the hardest setting in DMC-GB.
Related Work
Representation learning in RI. large corpus of literature has sought to leverage representation
learning in the setting of RL 66 22 68 60 81 69] More recently, there are branch of methods

---

that perform unsupervised pre- pre-training to encourage exploration for better sample efficiency 53 29]. A typical approach use contrastive learning method to jointly incentivize 38
44
17
49
exploration and acquire useful representations [1]] Liu et al. [44] introduce a new type of pre- training techniques via entropy maximization embedding space for better exploration. In partieular, inspired by the new representation algorithm in computer vision, Yarats et al. [79] propose SwAV like architecture [4] for pre-training with an exploration scheme that maximizes the entropy of the state visitation distribution. However, all these methods require the data collected in the target environments, resulting in additional sample cost. We instead propose a cross-domain pre-training way to improve visual RL agents' performance without any in- iomainin interaction during the pre- training time.
Pre-trained visual encoders for RL. Applying the pre-trained vision model from other domains to the control tasks has gradually attracted researchers' attention 80 66
51 76 For example, Shah et al. 63 and Parisi et al. [52] suggest that with the help of experts' demonstrations, the pre-trained ResNet 330 representations can achieve competitive performance with state-based inputs. Moreover, human video datasets are introduced to pre-train visual representation for downstream policy learning [4)]. The pre-trained model has also been proposed for goal -pallsppciiiiattion via behavior cloning [9] However, few works explore the effectiveness of pre-trained models for generalization. contrast to prior approaches, PIE-G enables agents to generalize well to the unseen visual scenarios with a large distributional shift in zero-shot manner while achieving high sample efficiency in a standard RL training paradigm.
Generalization in visual RL. Researchers have investigated to improve visual RI. agents' generaliza-
tion ability from various aspects [
27
40
73
2
50 77. Data augmentation 440 74 28
83
18 47
and domain randomization 77 56 54
59
5 are effective ways for generalization in visually ll different
environments. Notably, Hansen et al. 126 employ a YOLL-like architecture to decouple the augmentation from policy learning for better generalization. In order to control the high variance when implementing data augmentation, Hansen al. 28 add regularization term of the Q function between un-agumented and augmented data an implicit variance reduction technique. Meanwhile, Yuan et al. [83 propose a task-aware data augmentation method with the Lipschitz constant [19] for maintaining training stability. Fan et al. [18] apply data augmentation in the imitation learning paradigm. Prior works rely on data augmentation to gain a robust representation for each task. In this work, we tackle this challenge from different perspective, utilizing single pre-trained encoder with the universal visual representations for all the tasks.
3 Preliminaries
Reinforcement learning. Due to partial state observability from images in visual RL 3. we consider learning in a Partially Observable Markov Decision Process (POMDP) [3] formulated by the tuple (8, O, r, P) where is the state space, O is the observation space, A the action space, S x A + R is a reward function, P (st+1] St, az) is the state transition function, and E [0, 1) is the discount factor. The goal is to find a policy x to maximize the expected cumulative
return arg max, ( Σ E ( at) starting from an initial state So E S and obtained by following the policy Te s:) which parameterized by learnable parameters 0.
Generalization. terms of generalization, we consider a set of POMDPs: M = {M,, M2, Mn that shares the same dynamics and structures. The only difference among them is the observation space O. This setting is more formally described as "Block MDPs" [14. During the training process, we only have the access to a fixed POMDP denoted M, Our purpose is to train an agent specific scenario M, to learn policy = which can maximize the expected cumulative return over the whole set of POMDPs in zero-shot generalization manner.
4 Method
In this section, we introduce PIE-G, simple yet effective framework for visual RL which benefits from the pre-trained encoders on other domains to facilitate sample efficiency and generalization ability.
3

---

4.1 Pre-trained Encoder
PIE-G explicitly leverages the pre-trained models as the representation extractor without any mod- ification. The pre-trained encoder projects high-dimensional image observations into compact, low-dimensional embeddings that are later used by RL policies. Note that PIE-G is as simple as importing pre- pre-trained ResNet model from the torchvision 44 library. This avoids the design of any auxiliary tasks to acquire useful representations.
For all the training tasks different benchmarks, the encoder's parameters frozen to obtain universal visual representations. Since the pre-trained model contains the priors from wide range of weal-world images, hypothesize that the inherited power from pre-trained model may help to capture and distinguish the main components of different tasks' observations regardless of the changes of visual appearances deformed shapes, and will further improve the sample efficiency and generalization abilities of RL agents.
To validate our hypothesis, first encode each observation independently to obtain embeddings. Then, the embeddings from the second layer of the pre-trained model are fused as input features to the policy networks [64 [5]]. Moreover, we enable BatchNorm [33] to keep updating the running mean and running standard deviation during the policy training. The key findings are: 1) early layers of neural network would provide better representations for visual RL generalization, which resonates with prior works in imitation learning [52]: 2) the always updating statistics in BatchNorm helps better adapt to the shift in observation space and thus improve the generalization ability. More detailed discussion can be found in Section 5.4
4.2 Reinforcement Learning Backbone
We implement DrQ-v2 [78] as the base visual reinforcement learning algorithm. DrQ-v2 is the state-of-the-art method for visual continuous control tasks, which adopts DDPG [41] coupling with clipped Double Q-learning (20) to alleviate the overestimation bias of target Q-value. The agents are trained with two Qa, value functions and their corresponding target network Q The critic loss function is as follows, and the mini-batch of transitions (8z, ae, ftattn-1: 8t4n) is sampled from the replay buffer D:
[(Ox) = EYND [100 ax) - -y)²
vk E {1,2},
(1)
with n-step TD target y:
R-1
y = Σ TT44 + + min Qo (eetnaatn))
k=12
The actor Te is trained with the following objective:
(D) -E.ND
min Qo (st, at) k=1,2
(2)
where s, is augmented by random shift, = (S.))) + e, is sampled from clip (N -c,c) with decaying exploration noise a.
Thanks to the efficiency of DrQ-v2, PIE-G enjoys faster wall-clock training time and fewer computa- tional footprints. We emphasize that PIE-G does need any other proprioceptive states and sensory information as the inputs besides the representations extracted from original image observations. In previous works [37] 78 [88 (28), different schemes of data augmentation are proposed improve sample efficiency and generalization performance. In practice, weak augmentation methods (e.g.. random shift) of DrQ (37 and DrQ-v2 (78) are found be beneficial for sample efficiency. In the setting of generalization, we follow the way of using strong augmentation methods (e.g.. mixup) of SVEA (28 and DrAC [58] to further boost the performance. It is worth mentioning that since the gradient is stopped before it reaches the encoder, all the data augmentation techniques discussed here do not affect the pre-trained visual representation. Meanwhile, unlike Rutav al. (X and Simone et
al. [52]. we purely train the agent in standard RL paradigm without any expert's demonstration.

---

Table 1: Generalization on color-jittered observations. Experiments are conducted on multiple tasks in the DMC-GB (Top) and Manipulation Tasks (Bottom) environments with varying color backgrounds. For certain task, the color of the setting in evaluation will be altered. The agent required to adapt to the changes in zero-shot manner. Compared with its counterparts, PIE-G gains comparable and better performance in 9 out of 10 settings.
DMControl
Setting
SAC
DrQ
DrQ-v2
SVEA
TLDA
PIE-G
Tasks
Caripole,
Swingup
248+24
586452
277=80
837428
660
749:46
Walker,
3651:9
7704:1
413=61
9424:26
947::26
960=15
Stand
Walker,
144+19
520+91
168=90
760+145
823458
884-20
Walk
Ball_in cup, Catch Cheetah, Run
151#36
365=210
469=99
961-7
932#32
964=7
133126
100127
109
273123
371:51
369
Manipulation,
Training
2.5 130420
204111
49148
124:132
199113
Modified, 0.3+0.5
68:220
2949
21+25
55:221
122=30
Modified, Platform Modified, Both
0,5+0.33
0.841.3
1,541.7
24+25
89+40
96:123
0.4+0.8
1.042.0
0.8
13+14
36+25
44+16
Experiments
In this section, we investigate the following questions: (1) Can PIE-G improve the agent's generaliza- tion ability? Specifically, how well does PIE-G deal with jittered color, moving video background, and deformed shapes of robots? (2) Can PIE-G improve training sample efficiency? (3) How do the choice of layers in the encoder and the of BatchNorm KX affect the performance? (4) Can further finetuned RL agents outperform those with frozen visual encoders?
5.1 Setup
We evaluate our method on a wide range of tasks, including DMControl Generalization Bench- mark (DMC-GB) (26), DeepMind Manipulation tasks [72]. and Drawer World [75]. PIE-G is trained for 500k interaction steps with 2 action repeat and evaluated with 100 episodes for every task on the testing benchmarks. All the generalization evaluations are in zero-shot manner. By default, the encoder uses the ResNet18 architecture B0 To be more specific, the feature maps of the second layer are flattened and passed through additional fully connected layer to serve as the representations of observations. More training details and environment descriptions are Appendix B
5.2 Evaluation on Generalization Ability
We compare the generalization ability of PIE-G with state-of-the-art methods and strong baselines: SAC [25]: a widely used off- polico RL algorithm; DrQ [37]: a SAC-based visual RL algorithm with augmented observations; DrQ-v2 (78): the prior state-of-the-an model-free visual RL algorithm in terms of sample efficiency; SVEA 28]: the prior state-of the-art method terms of generalization via reducing the Q-vrriacce through auxiliary loss; TLDA [83]: another state-of-the-art method in generalization by using aaske data augmentation.
Generalization on color-jittered observations. The agent's generalization ability is evaluated on DMC-GB with randomly jittered color. For manipulation tasks, the colors different objects (e.g., floors, arms) are modified. As shown in Table [ PIE-G obtains better or competitive performance in out of 10 instances. These results suggest that the visual representation from the pre-trained model more robust to the color-changing than the one trained by standard RL algorithms.
5

---

Table Generalization on unseen moving backgrounds. Episode return in two types of unseen dynamic video background environments, i.e., video easy (Bottom) and video hard (Top). PIE-G achieves competitive or better performance out of 12 tasks. In video hard setting, we significantly outperforms other algorithms with +55 % improvement on average.
DMControl
Setting
DrQ
DrQ-v2
SVEA
TLDA
PIE-G
Tasks
Campole,
13849
13043
393145
286147
401:21
(+2.0%)
Swingup
Walker,
289149
151
834146
602451
852.56
(+2.2%)
Stand
Walker.
104#22
34+11
377#93
271
600=28
(+59.2%)
Walk
Ball Ball_in_cou _up_
92+23
97#277
403+171
257457
786+47
(+95.0%%
Catch
Cheetah, Run
32:11
23:55
105137
901127
154
(+-46.6%)
Finger, Spin
71+45
2114
335458
2414:29
762
(+127%)
Cartpole, Swingup Walker.
485+105
267+:11
782=27
671
587+61
873+83
560+:88
961
973#6
957+12
Stand
Walker.
682489
175=117
819+71
873434
871=22
Walk
Ball_in_cup 318:1157
454.10
8711106
892.188
922 20
Catch
Cheetah,
102#30
64+22
249+20
366#57
287+20
Run
Finger,
533+119
456+15
808#33
744+18
837=107
Spin
Generalization on unseen and/or moving backgrounds. We then evaluate PIE-G on the more challenging settings: video easy and video hard in DMC-GB. The video hard setting consists of more complicated and fast-switching video backgrounds that are drastically different from the training environments. Notably, even the reference plane of the ground is removed in this setting.
The comparison results are shown in Table 2 PIE-G achieves better or comparable performance with the prior state-of-the-aa methods in 9 out of 12 instances. In particular, PIE-G gains significant improvement in the video hard setting over all the previous methods with +55 % improvement on average. For example, in the Finger Spin, Cup Catch, and Walker Walk tasks, PIE-G outperforms the best of the other methods by substantial margins 127.0%. 95.0%. and 59.2% % respectively.
Original observation
SVEA
PIE-G
Video observation
Feature map difference
Feature
subtract feature map
Feature
Feature map difference
Figure 2: Visualized feature map differences of two inputs from the same state with different backgrounds. The difference of the feature maps with PIE-G as the encoder closer to zero than that with SVEA, indicating PIE-G enjoys better generalization ability.

---

Training Mittal
100
Soovess Rate %6
Stovess Rate 16
80
50
60
Test
25
DrQ
40
DXQ-2
20
20
40
60
80
100
20
40
60
80
100
Wond
Number of Frumes (x10*)
Number of Frames (x109)
(a) Meta World
(b) Training Curve
Figure 4: Training Meta World. Left: The visualization of Meta World with different textures. Right: The training curves. PIE-G (Red line) demonstrates better sample efficiency than DrQv2 (Blue line) and SVEA (Green line).
Attempting explain the success, visualize the difference of the normalized feature maps extracted from the encoder whose inputs are two Walkers of the same pose but with different backgrounds, is shown in Figure 2 Ideally, well-generalizable encoder would map the observations of the two Walkers to exactly the same embedding, and therefore the difference should be zero. In practice, as shown Figure 2 the encoder of PIE-G produces difference much closer to zero than that of SVEA. Numerically, we calculate the average pixel intensity in the difference of normalized feature maps, and the intensity is decreased by 50.9% with PIE-G than that with SVEA.
Then, we evaluate PIE-G on the CARLA 13 autonomous driving system which contains realistic observations and complex driving scenarios. The default setting is adopted from Zhang al. [85]. PIE-G and other algorithms are benchmarked on 4 diverse weather environments. As shown in Figure 3 all the prior state-of-the-art methods cannot adapt to the new unseen weather with different lighting, humidity, road conditions etc. The behind rea- son is that compared to DMC-GB whose images merely consist of single control agent and the background, the observations of CARLA contain more distracting objects and factors; therefore, only depending on data augmenta- tion to provide diverse data cannot tackle this complicated visual driving task. The experimental results in Figure 3 exhibits that thanks to the ImageNet pre-trained encoder, PIE-G can generalize well on the complicated scenes without large performance drop.
CARLA
250
SYEA
TLDA
200
150
100
50-
Trainin
Figure PIE-G can well adapt unseen scenar- ios.
Performance on CARLA.
Furthermore, we conduct exper- iments on the Drawer World benchmark to test the agent's generalization ability in manipu- lation tasks with different back- ground textures. The visual- ized observations are shown in Figure 4a Success Rate adopted as the evaluation metric for its goal-conditioned nature. Table 3 illustrates that PIE-G can achieve better or compara- ble generalization performance in all the settings with +24 boost average while other ap- proaches may suffer from the CNN's sensitivity in the face of various textures 21
Task
Setting
SAC
DrQ-v2
SVEA
PIE-G
Training
100%
98%
70%
99%
Wood
0%
32%
49%
59%
Drawer-Close
Metal
0%
46%
69%
95%
Blanket
0%
8%
72%
71%
Task
Setting
SAC
DrQ-v2
SVEA
PIE-G
Training
98%
100%
75%
97%
Wood
18%
2%
47%
79%
Drawer-Open
Metal
35%
53%
71%
97%
Blanket
28%
5%
37%
85%
Table 3: Generalization Drawer World. Evaluation dis-
tracting textures. PIE-G robust to the texture changing.
Generalization on deformed shapes. To verify agent's robustness in terms of the deformed shapes, we modify the shapes of the jaco arm and the target objects in the manipulation tasks, as shown in Figure 5a Figure 5b demonstrates that PIE-G also improves the agents' generalization ability with
7

---

various shapes while other methods could barely generalize to these changes. We attribute this to the lack of shape changing in previous data augmentation techniques. Conversely, our pre-trained encoder is learned from multitude of real-world images with various poses and shapes, thus enhancing its generalization ability on deformed shapes.
Manipulatien (Shape Deforming)
Rt.0
Original
Change
Change Birkk
Eepoodae Roward
TLDM
8 12 16
Deform Arm
Defomm Brick
(a) Deform the shape of the object
(b) Generalization performance
Figure 5: Deforming the shape. Left: Aiming evaluating the agent's robustness of the shape, we deform the robot's arm and the target brick. Right: The results demonstrate that PIE-G well- generalizable in the face of deformed shapes.
5.3 Evaluation on Sample Efficiency
evaluate the sample efficiency PIE-G on 8 relatively challenging tasks on the DeepMind Control Suite and the Manipulation task (Reach Duplo). Figure demonstrates that PIE-G achieves better or comparable sample efficiency and asymptotic performance than DrQ-v2 in 7 out of 8 tasks. To eliminate the effect of varied network size, also include random network architecture with frozen random initialized parameters. addition. Figure shows that PIE-G is also superior over the basclines the Drawer world benchmark. The results demonstrate that the pre-trained encoder inherits the powerful feature extraction ability trained from ImageNet and acquires better training sample efficiency in control tasks.
baseline that has the same
cup Catch
514)
Eptood Roward
510)
610
600
500
410
4/0
400
200
F1B-0
DVQ+0
MB--0
IVD+3
M1B-0
DVQ
200
FIB-C
50
75
100
100
150
25
50
75
100
75
Quadrapped Rum
Reacher Hard
Reach Duplo
Walker Walk
Rewand
600
$10
200-
500
150
600
200
400
100
150
01
ECT0Py
Framnes (xC1
Figure 6: Training sample efficiency. Average episode rewards on 8 challenging DMControl tasks with means and standard deviations calculated over seeds. We compare PIE-G (Red line) with DrQ-v2 (Blue line) and random enc (Green line) with respect to sample efficiency. Our method achieves better or comparable performance in 7 of instances.
5.4 Ablation study
To verify the necessity of the design choices in PIE-G, we conduct series of ablation studies to take closer look at the proposed method. More results are shown in Appendix

---

Choice of layers. In convolu- tional neural networks, the later layers capture high-level seman- tic features, while the early lay-
Task
Layer
Layer 2
Layer 3
Layer 4
Walker Walk
840
884420
845+27
306#31
Cheetah Run
366=56
369
294+60
111+19
ers are responsible for extract- ing low-level information [45]
Walker Stand
953+8
964-7
957#7
625+116
Table 4: Different layers. We employ the feature map of different ResNet model as the visual representation. Among them, the Layer 2 exhibits the best generalization performance.
84 42
Figure and Table 4 in-
layers
vestigate how much control tasks
benefit from the features tracted from different layers. As shown in Figure 7 the early layers preserve rich details of edges and corners, while the later layers only provide very abstract information. Intuitively, for control tasks, a trade-off is required between low-level details and high-level semantics. Table and and Figure 8 in Appendix show that the Layer? 2 gains better generalization and sample efficiency performance than the other layers.
Batch normalization. Batch Normalization (Batch- Norm) 33 is a popular technique computer vision. However, it is not widely adopted in RL algorithms. In contrast to conventional wisdom, BatchNorm is found to be useful and important in PIE-G. Specifically, we find that calculating the mean and variance of the observations dur- ing evaluation rather than using the statistics from training data would boost the performance. Figure P demonstrates that, in the most challenging settings, PIE-G with the use of BatchNorm can further improve the generalization performance. This is largely because the distribution of observations is determined by the agent, violating the as- sumption of independent and identical distribution (i.i.d.).
300
SVEA PIE Oww BN]
Epiisode Reward
600
400
200
Ball in Cap Catch
Walker
Finger Spit
Figure 9: Leveraging BatchNorm. The ever-updating BatchNorm beneficial for better performance.
This use of BatchNorm also reassures the recommendation from loffe al. (33 that recomputation the statistical means and variances allows the BatchNorm layer to generalize to new data distributions.
Adopting other datasets for pre- training, Besides ImageNet [10], we also implement pre-trained visual en-
Tasks
ImageNet
CLIP
Ego4D
SVEA
Walker Walk
600 #28
615 #30
441 #15
377 +93
coders with other novel and popular datasets: CLIP (57) and Ego4D (23). CLIP (Contrastive Language-Image Pre-training) trained a large number of (image. text) pairs collected from Internet to jointly acquire visual and text representations. The Ego4D is egocentric human video dataset which contains massive daily-life activity videos in hundreds of scenarios. Table 5) shows that the agents pre-trained with CL.IP achieves comparable performance with those pre-trained with ImageNet. Since the Ego4D collects the videos with the first-person view, the view difference between the tasks and the dataset leads to decrease in performance; nevertheless, the Ego4D pre-trained agents still obtain comparable results with the prior state-of-the-art methods.
Cheetah Run
154 #17
115 #02
101 #13
105 +37
Walker Stand
852 #56
849 +23
647 #59
441 +15
Finger Spin
762 159
676 111
515 1004
335 158
Table 5: Adopting other datasets for pre-training. All agents pre-trained with different datasets gain considerable generalization performance.
Layerl
Layer2
Layer3
Layer4
Observation
Encoder
Feature Map
Figure 7: Visualization of the feature maps of different layers. The feature map of Layer 2 largely preserves the outline of the Walker that is advantageous to the control tasks, and at the same time discards redundant details.

---

Layer
Layer
Layer
Layer
Duple
1000
000
230
100
Sgliode Rovard
150
Airl by Ay M MU
100
230
200
100
20
40
643
100
203
660
30
100
20
100
Nanther of Einv Promms (x201))
Nonther of Bay Promas (x:30))
Nombbr of Promes (x
Figure 8: Choice of layers in terms of sample efficiency. This figure indicates that the early layers have better sample efficiency than the later layers.
Finetuning the pre-trained model. We also conduct research finetune the encoder's parameters instead of keeping it frozen. Previous works [32 55] have found that finetuning pre- trained models challenging. Con- sistent with these studies, Table [ sug- sug- gests that compared with the frozen representations from pre-trained mod- els, the finetuned representations suffer from the out-of-distribuu problem performance drop in terms of the generalization ability.
Task
PIE-G (Finetune)
PIE-G (Frozen)
Walker Walk
455+67
600
Cheetah Run
122+
150+19
Walker Stand
771
852+55
Table 6: Finetuning the pre-trained models. We compare the generalization performance between the frozen visual representations and the finetuned ones.
6 and lead to a
dopting other pre-trained models. Additionally, we investigate the effi- cacy of other visual representations. MoCo-v2 7 is a pre-trained model optimized via contrastive learning to learn representations. We find that PIE-G with the pre-trained represen- tations of MoCo-v2 can also obtain comparable performance in terms of both the sample efficiency and generalization ability. More results are shown in Appendix C
Task
PIE-G
PIE-G (w MoCo-v2)
Walker Walk
6001128
585.130
Cheetah Run
154.17
150.119
Walker Stand
852.556
8561511
Table 7: Adopting other pre-trained models. PIE-G
with MoCo-v2 can obtain comparable generalization per- formance.
6 Conclusion
In this work, we propose PIE-G, a simple yet effective framework that leverages off-the-shelf features of ImageNet pre-trained ResNet models for better generalization in visual RL. Extensive experiments on a variety of tasks in four RL environments confirm the merits of universal visual representations, which endow the agents with improved sample efficiency and better generalization performance. In addition, we show that the choice of layers and the use of BatchNorm are crucial for the performance gain. Our exploration may inspire more researchers dig into the great potential of utilizing pre-trained representations in visual RL.
Limitations. We study generalization in the simulated environments. However, there might be new challenges the real-world applications. In the future, we would like to establish and test on benchmarks of real-world scenarios.
Acknowledgements: We would like to thank Boyuan Chen, Kaizhe Hu and Guozheng Ma for giving helpful suggestions. We are sincerely grateful the anonymous reviewers for taking the time to review this work and giving insightful advice. Yang Gao is supported by the Ministry of Science and Technology of the People' 's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0100000) Yang Gao is also supported by grant from the Guoqiang Institute, Tsinghua University.
10

---

References
[1] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.
[2] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv::612.03880 2016.
[3] Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:99 33:9912-9999 12-9924, 2020.
[5] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973-8979. IEEE, 2019.
[6] Bryan Chen, Alexander Sax, Gene Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study manipulation and navigation. arXiv preprint urXiv:2011.06692 2020.
[7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0423 2020.
[8] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference Machine Learning. pages 1282-1289. PMLR, 2019.
[9] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? arXiv preprint vrXiv:2204.11333 2022.
[10] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[11] Jacob Devlin, Ming- Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.1/000000 10.04805, 2018.
[12] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmann, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pages 647-655. PMLR, 2014.
[13] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[14] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.
[15] Yilun Du Chuang Gan, and Phillip Isola. Curious representation learning for embodied intelligence. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10408-10417, 2021.
[16] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109 13396, 2021.
11

---

[17] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06077 2018.
[18] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3088-3099 PMLR, 18-24 Jul 2021.
[19] Chris Finlay, Jeff Calder, Bilal Abbasi, and Adam Oberman. Lipschitz regularized deep neural networks generalize and are adversarially robust. arXiv preprint arXiv: I 1808. 09540, 2018.
[20] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[21] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 181 12231, 2018.
[22] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 2170-2179. PMLR, 2019.
[23] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19995- 2022.
[24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-21284 2020.
[25] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[26] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13611-13611 IEEE, 2021.
[27] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. arXiv preprint arXiv:2007.04309, 2020.
[28] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q- q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
(29) Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farles and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 1906. 2019.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738. 2020.
[32] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Interna- tional Conference on Machine Learning, pages 4182-4192. PMLR, 2020.
12

---

[33] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMIR, 2015.
[34] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):9--134 1998.
[35] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651 PMLR, 2018.
[36] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. arXiv preprint arXiv:2111.09888 2021.
[37] Iya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
[38] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
[39] Misha Laskin. Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:198844 198888 2020.
[40] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning. arXiv preprint arXiv: 1910.05396, 2019.
[41] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (1590099977 2015.
[42] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[43] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference Machine Learning, pages 6736-6747. PMLR, 2021.
[44] Hao Liu and Pieter Abbcel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34, 2021.
[45] Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang. Hierarchical convolutional features for visual tracking. In Proceedings of the IEEE international conference on computer vision, pages 3074-3082, 2015.
[46] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vistee package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages 1485-1488, 2010.
[47] Berenson Dmitry Mitrano Peter. Data augmentation for manipulation. arXiv:2205.02886, 2022.
arXiv preprint
[48] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53 5333, 2015.
[49] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12661, 2022.
[50] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 2018.
1rXiv:18/00 1810. 12822
13

---

[51] Jyothish Pari, Nur Muhammad, Sridhar Pandian Arunachalam, Lerrel Pinto, et al. The surprising effectiveness of representation learning for visual imitation. arXiv preprint 1r2.02211..01511, 2021.
[52] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurpris- ing effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203. 03550, 2022.
[53] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
[54] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[55] Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv: 190300587, 2019.
[56] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint 2017.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[58] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[59] Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain random- ization via probabilistic inference for robotics simulators. arXiv preprint arXiv: urXiv:1906.0 1906.01728. 2019.
[60] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
[61] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self superviee world models. In Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8583-8592. PMLR, 13-18 Jul 2020.
[62] Younggyo Seo, Kimin Lee, Stephen James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. arXiv preprint arXiv:2203. 2022.
[63] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. arXiv preprint arXiv:2107.033388 03380, 2021.
[64] Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Picter Abbeel, and Misha Laskin. Reinforcement learning with latent flow. Advances in Neural Information Processing Systems, 34, 2021.
[65] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[66] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages
PMLR, 2022.
[67] Xingyou Song. Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.0299 2019.
[68] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning, arXiv preprint arXiv:2004.041.66 2020.
14

---

[69] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[70] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki. Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 181100800000099 2018.
[71] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. JEEE, 2017.
[72] Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. com_ottrol:: Software and tasks for continuous control. Software Impucts, 6: 100022, 2020.
[73] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 161.00/6 2016.
[74] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems, 33:7968-7978, 2020.
[75] Xudong Wang. Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6677-6687, 2021.
[76] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.061 2022.
[77] Huazhe Xu, Boyuan Chen, Yang Gao, and Trevor Darrell. Zero-shot policy learning with spatial temporal reward decomposition on contingency-aware observation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10786-10792. IEEE, 2021.
[78] Denis Yarats, Rob Fergus. Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved Sata-augmented reinforcement learning. arXiv preprint a177v:2107.09645 2021.
[79] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference Machine Learning, pages 11920-11931 PMLR, 2021.
[80] Lin Yen-Chen, Andy Zeng, Shuran Song, Phillip Isola, and Tsung- Yi Lin. Learning to before learning to act: Visual pre-training for manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7286-7293. IEEE, 2020.
[81] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvir- tual: Augmenting cycle-consistent
consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[82] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100, PMLR, 2020.
[83] Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning. arXiv preprint
2022.
[84] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In European conference on computer vision, pages 318-833. Springer, 2014.
[85] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint 100.44000.10742, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work focus on designing an algorithm for boosting the generalization ability, rather than real-world application.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] In the supplementary material
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] In the suppicmentary material
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] In the experiments
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] In the supplementary material
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 'rere using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "ySHR9FjkdO", "reviewer": "Reviewer_UwhJ", "summary": "This paper uses pretrained visual encoders to improve the sample efficiency and generalization performance of visual RL. The main methodological contributions are in the details -- which pretrained layer to transfer, updating of batchnorm parameters, freezing of the encoder, etc. Experiments demonstrate superior performance compared to RL methods that do not use out-of-domain pretraining.", "questions": "I think it would be very interesting to delve more into the question of \"what kind of pretraining helps for what kind of tasks\"? The MoCo comparison at the end of the paper was a step in that direction. More pretrained models could be tried. Are there any lessons to glean? Is it always best just to use the SOTA general-purpose vision system? How then does CLIP do, or its various follow-ups? To me these kinds of experiments would strengthen the originality of the paper, and take it beyond what are, to me, expected results.\n\nAlong similar lines, the paper could be strengthened by comparing against a variety of methods that _all use the same pretraining dataset_. It's not terribly surprising that PIE-G can outperform methods that have access to less data than it. A more apples-to-apples comparison would be to compare between methods that all are given access to ImageNet, and then see, among those, which is best, and why. The ablations in Section 5.4 are a step in this direction.", "limitations": "I think the discussion of limitations was adequate. There was not much discussion, but I don't think much is needed for this paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper are the results -- it works well! The main weakness, in my opinion, is the originality: many other papers have proposed and demonstrated essentially the same high level point. I will elaborte on these points below:\n\n**Originality**\nMaking use of powerful pretrained representations is ubiquitous in our field right now (c.f. foundation model hype). This paper characterizes the results as \"suprising\" but I think the opposite is true: these results will be seen as thoroughly expected by a large portion of the field. This is in part due to the many other papers that have shown similar results in the specific domain of visual pretraining for embodied control (e.g., [76, 72, 48] and many more cited by the paper), and partly because the same trend has been so dominant in other areas of ML. I think [48] does a better job of characterizing this finding, in its title, as \"unsurprising\". So, the main message of this paper -- that visual pretraining can help, and a lot -- is one that has already, I think, been absorbed by the field (some of the cited related work may be concurrent, but there are also many papers that were published well before the neurips deadline on this topic, including [6], [76], [62], etc). That said, I do think the details in the present paper have some originality, especially the use of batchnorm for adaptation. I personally haven't seen that used before in this specific context (although batchnorm for adaptation in general is a well known method, e.g., https://openreview.net/pdf?id=BJuysoFeg).\n\n**Quality**\nThe quality of results is impressive: the paper delivers on showing that pretraining helps, both in terms of robustness/generalization and in terms of sample efficiency.\n\n**Clarity**\nThe writing and figures were all clear to me, except one aspect. I would have liked more detail on how the ResNet was pretrained. This is barely mentioned but seems essential to the success of the method. Indeed I would be interested to see how different pretraining methods and datasets affect the results.\n\n**Significance**\nDespite that I don't think readers will be surprised by the results, getting the details right, and achieving high performance with a simple system is a significant accomplishment, and could be impactful assuming the code is released. At the same time, I don't think the specific pretrained model in this paper will be used for long, and subsequent work will have to redo the analysis to pick which layers of future models to transfer, and how to tune the other methodological details.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lHDksT-MJMo", "reviewer": "Reviewer_ni5r", "summary": "The authors investigate using a pre-trained visual backbone in a RL setting. Specifically, they use off-the-shelf ResNet encoders trained in a supervised or self-supervised manner, and use the activations in intermediate layers as input to the policy network, as opposed to the raw pixel values. They find the resulting policy to be significantly more robust to changes in appearance than policies trained on raw pixel values, with fairly dramatic improvements under strongly distracting conditions. ", "questions": "One question we are left with however, is whether higher-level features (e.g. from levels 3 and 4 of the ResNet) simply underperform because of their lower resolution, or whether the more \"semantic\" information they encode is actually detrimental for the task. Would it be possible to use these features with higher-resolution images, to control for this difference? ", "limitations": "No ethical or societal risks here. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a simple and effective idea. Although most of the components are standard (pre-trained visual encoders, the policy stacked on top), the authors carefully choose and calibrate these components, and the results are very compelling. \n\nIn particular, it is interesting that intermediate feature activations perform better than high-level ones, and that allowing the batch-norm statistics to adapt to the new environment is helpful. These insights make the difference between the method working or not, and will likely apply in a variety of other settings. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a6JdIjQ0dhh", "reviewer": "Reviewer_9V7N", "summary": "Summary: This paper suggests the use of an off the shelf, frozen, pre-trained ResNet model as an encoder for Reinforcement Learning, as opposed to learning an encoder from scratch. The paper suggests that this enables better generalization as the pre-trained ResNet model serves as a universal representation, which is robust and generalizes well to unseen scenarios. They also show that BatchNorm is critical for better generalizing RL agents.", "questions": "- Is the fundamental goal to prove that better data distributions are better than better architectures for encoding images in visual RL? That would be a promising direction, but would require significant more experiments than the current manuscript.\n\n- The current experiments show that: (1) Pre-training is better than training from scratch for the same architecture, (2) Simpler architecture with better data performs better than complex architecture with simpler data. Unfortunately, these two are both well known ideas, and building on top of them is necessary to arrive at the point mentioned above.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Generalizing well to unseen environments is a fundamentally important problem, and advances in this direction are both extremely important, and useful for the community.\n\n2. The writing is clear and easy to follow. Figures do a good job of explaining the main idea visually.\n\nWeaknesses:\n\n1. Unfair baseline: The main difference in PIE-G is that the encoder is trained on different data, while the baselines are testing different training methodologies. When evaluating generalization to unseen data, control over the training distribution is absolutely essential. For instance, was the generalization better because the testing environment was maybe overlapping with ImageNet? For the unseen backgrounds case, it appears so as the backgrounds are real-world natural images. The typical game in generalization is to improve without access to new data. Here, new and uncontrolled data is being added to PIE-G, but not to the other baselines. This makes the comparison unfair, and also hard to qualitatively gauge.\n\n2. Unsupported claims: The paper suggests that embeddings from a pre-trained ResNet are a universal representations which are both robust and generalize well. However, it is well documented in the computer vision literature that this is not true [1,2,3]. For instance, works have shown that ImageNet trained models do not generalize to new ImageNet images either [1]. Learning generalizable features is an active area of research, and there is significant evidence suggesting that pretrained ResNet features cannot be called universal, robust or generalizable.\n\nIn summary, it appears that the performance gains may be due to the additional dataset which baselines do not have access to, as opposed to learning a universal representation.\n\n\nReferences:\n\n1. Recht, B., Roelofs, R., Schmidt, L. and Shankar, V., 2019, May. Do imagenet classifiers generalize to imagenet?. In International Conference on Machine Learning (pp. 5389-5400). PMLR.\n\n2. Hendrycks, D. and Dietterich, T., 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.\n\n3. He, K., Girshick, R. and Dollár, P., 2019. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4918-4927).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept