Self-Supervised Policy Adaptation during Deployment
Authors: Nicklas Hansen
Rishabh Jangir
Yu Sun
Guillem Alenyà
Pieter Abbeel
Alexei A Efros
Lerrel Pinto
Xiaolong Wang
Venue: ICLR 2021 Spotlight (2021)
Keywords: reinforcement learning
robotics
self-supervised learning
generalization
sim2real

Abstract: In most real world scenarios, a policy trained by reinforcement learning in one environment needs to be deployed in another, potentially quite different environment. However, generalization across different environments is known to be hard. A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal. Our work explores the use of self-supervision to allow the policy to continue training after deployment without using any rewards. While previous methods explicitly anticipate changes in the new environment, we assume no prior knowledge of those changes yet still obtain significant improvements. Empirical evaluations are performed on diverse simulation environments from DeepMind Control suite and ViZDoom, as well as real robotic manipulation tasks in  continuously changing environments, taking observations from an uncalibrated camera. Our method improves generalization in 31 out of 36 environments across various tasks and outperforms domain randomization on a majority of environments. Webpage and implementation: https://nicklashansen.github.io/PAD/.

Published as a conference paper at ICLR 2021
SELF-SUPERVISED POLICY ADAPTATION DURING DEPLOYMENT
Nicklas Hansen'? 12 Rishabh Jangir Yu Sun', Guillem Alenya,, Pieter Abbeel', Alexei A Efros", Lerrel Pinto Xiaolong Wang' UC San Diego IRI, CSIC-UPC
Technical University of Denmark 4UC Berkeley 'NYU
ABSTRACT
In most real world scenarios, policy trained by reinforcement learning in one vironment needs to be deployed another, potentially quite different environment. However, generalization across different environments is known to be hard. A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal. Our work explores the use of selffsupervision to allow the policy to continue training after de- ployment without using any rewards. While previous methods explicitly anticipate changes in the new environment, we assume no prior knowledge of those changes yet still obtain significant improvements. Empirical evaluations are performed on diverse simulation environments from Deep.Mind Control suite and ViZDoom, well as real robotic manipulation tasks continuously changing environments, taking observations from an uncalibrated camera. Our method improves general- ization in 31 out of 36 environments across various tasks and outperforms domain randomization on majority of environments.
INTRODUCTION
Deep reinforcement learning (RL) has achieved considerable success when combined with convolu- tional neural networks for deriving actions from image pixels (Mnih et al., 2013; Levine et al., 2016; Nair et al., 2018; Yan et al., 2020; Andrychowicz al., 2020). However, one significant challenge for real-world deployment of vision-based RL remains: policy trained in one environment might not generalize to other new environments not seen during training. Already hard for RL alone, the challenge is exacerbated when a policy faces high-dimensional visual inputs.
well explored class of solutions to learn robust policies that are simply invariant to changes in the environment (Rajeswaran et al., 2016; Tobin et al., 2017; Sadeghi & Levine, 2016; Pinto et al., 2017b; Lee et al., 2019). For example, domain randomization (Tobin et al., 2017: Peng et al., 2018; Pinto et al., 2017a; Yang al., 2019) applies data augmentation in a simulated environment train single robust policy, with the hope that the augmented environment covers enough factors of variation in the test environment. However, this hope may be difficult to realize when the test environment is truly unknown. With too much randomization, training a policy that can simultaneously fit numerous augmented environments requires much larger model and sample complexity. With too little randomization, the actual changes in the test environment might not be covered, and domain randomization may do more harm than good since the randomized factors are now irrelevant. Both phenomena have been observed in our experiments. In all cases, this class of solutions requires human experts to anticipate the changes before the test environment is seen. This cannot scale as more test environments are added with more diverse changes.
Instead of learning a robust policy invariant to all possible environmental changes, we argue that better for a policy to keep learning during deployment and adapt to its actual new environment. A naive way to implement this in RI. to fine-tune the policy in the new environment using rewards supervision (Rusu et al., 2016; Kalashnikov et al., 2018; Julian et al., 2020). However, while it is relatively easy to craft a dense reward function during training (Gu et al., 2017; Pinto & Gupta, 2016), during deployment it is often impractical and may require substantial engineering efforts.
Wehpage and implementation: https: //t.//icckk// assansen .g thub. P

---

Published as a conference paper at ICLR 2021
In this paper, tackle an alternative problem setting in vision-based RL: adapting pre-trained policy to an unknown environment without any reward. We do this by introducing sulf--uueovision to obtain "free" training signal during deployment. Standard self-supervised learning employs auxiliary tasks designed to automatically create training labels using only the input data (see Section 2 for details). Inspired by this, our policy is jointly trained with two objectives: standard RI. objective and, additionally, a self-supervised objective applied on an intermediate representation of the policy network. During training, both objectíves are active, maximizing expected reward and simultaneously constraining the intermediate representation through self- self-supervision. During testing / deployment, only the self- self-supervised objective (on the raw observational data) remains active, forcing the intermediate representation to adapt to the new environment.
We perform experiments both in simulation and with real robot. In simulation, we evaluate on two sets of environments: DeepMind Control suite (Tassa et al., 2018) and the CRLMaae ViZDoom (Lomonaco et al., 2019; Wydmuch al., 2018) navigation task. We evaluate generalization by testing in new environments with visual changes unknown during training. Our method improves generalization in 19 out of 22 test environments across various tasks in DeepMind Control suite, and in all considered test environments on CRLMaze. Besides simulations, transfer on both reaching and pushing tasks with a Kinova Gen3 robot. After training in simulation, we successfully transfer and adapt policies to 6 different environments, including continuously changing disco lights, on a real robot operating solely from an uncalibrated camera. In both simulation and real experiments, our approach outperforms domain randomization in most environments.
also perform Sim2Real
RELATED WORK
Self-supervised learning is a powerful way to learn visual representations from unlabeled data (Vin- cent et al., 2008; Doersch et al., 2015; Wang & Gupta, 2015; Zhang et al., 2016; Pathak et al., 2016; Noroozi & Favaro, 2016; Zhang et al., 2017; Gidaris et al., 2018). Researchers have proposed to use auxiliary data prediction tasks, such as undoing rotation (Gidaris et al., 2018), solving a jigsaw puzzle (Noroozi & Favaro, 2016), tracking (Wang et al., 2019), etc. to provide supervision in lieu of labels. In RL, the idea of learning visual representations and action at the same time has been investi- gated (Lange & Riedmiller, 2010; Jaderberg et al., 2016; Pathak et al., 2017; Ha & Schmidhuber, 2018; Yarats et al., 2019; Srinivas et al., 2020; Laskin et al., 2020; Yan et al., 2020). For example, Srinivas et al. (2020) use self-supervised contrastive learning techniques (Chen al., 2020; Hénaff et al., 2019; Wu et al., 2018; He et al., 2020) to improve sample efficiency in RL by jointly training the supe-super objective and RL objective. However, this has not been shown generalize to unseen environments. Other works have applied self- seff-supervision for better generalization across envi- ronments (Pathak al., 2017; Ebert et al., 2018; Sekar et al., 2020). For example, Pathak al. (2017) use a self-supervised prediction task to provide dense rewards for exploration in novel environments. While results on environment exploration from scratch are encouraging, how to transfer a trained policy (with extrinsic reward) to a novel environment remains unclear. Hence, these methods are not directly applicable to the proposed problem in our paper.
Generalization across different distributions is central challenge in machine learning. In domain adaptation, target domain data is assumed to be accessible (Geirhos et al., 2018; Tzeng ct al., 2017; Ganin al., 2016; Gong al., 2012; Long al., 201 6; Sun et al., 2019; Julian al., 2020). For example, Tzeng et al. (2017) use adversarial learning to align the feature representations in both the source and target domain during training. Similarly, the setting of domain generalization (Ghifary et al., 2015; Li et al., 2018; Matsuura & Harada, 2019) assumes that all domains are sampled from the same meta distribution, but the same challenge remains and now becomes generalization across meta- detta-dittiiutions.. Our work focuses instead on the setting of generalizing to truly unseen changes in the environment which cannot be anticipated at training time.
There have been several recent benchmarks in our setting for image recognition (Hendrycks & Dietterich, 2018; Recht et al., 2018; 2019; Shankar et al., 2019). For example, in Hendrycks & Dietterich (2018), a classifier trained on regular images is tested on corrupted images, with corruption types unknown during training: the method of Hendrycks et al. (2019) is proposed to improve robustness on this benchmark. Following similar spirit, in the context of RL, domain randomization (Tobin et al., 2017: Pinto et al., 2017a; Peng et al., 2018; Ramos et al., 2019; Yang et al., 2019; James et al., 2019) helps a policy trained in simulation to generalize to real robots. For example, Tobin et al. (2017); Sadeghi & Levine (2016) propose to render the simulation environment with random textures and train the policy top. The learned policy is shown to generalize real
2

---

Published as a conference paper at ICLR 2021
Training
Policy Adaptation during Deployment
Replay Buffer
Environment
Self- Seff-Superiise Learning
Self-Superwised Learning
Reinforcement Learning
TT
Reinforcement Learning
Observation
Obsarvation
Figure 1. Left: Training before deployment. Observations are sampled from a replay buffer for off-policy methods and are collected during roll-outs for on-policy methods. We optimize the RL and self-supervised objectives jointly. Right: Policy adaptation during deployment. Observations are collected from the test environment online, and we optimize only the self-supervised objective.
robot manipulation tasks. Instead of deploying fixed policy, we train and adapt the policy to the new environment with observational data that is naturally revealed during deployment.
Test-time adaptation for deep learning is starting be used in computer vision (Shocher al., 2017; 2018; Bau et al., 2019; Mullapudi et al., 2019; Sun et al., 2020; Wortsman et al., 2018). For example, Shocher al. (2018) shows that image super-resolution can be learned at time (from scratch) simply by trying to upsample a downsampled version of the input image. Bau et al. (2019) show that adapting the prior of generative adversarial network to the statistics of the test image improves photo manipulation tasks. Our work is closely related to the test-time training method of Sun (2020), which performs joint optimization of image recognition and supersseee learning with rotation prediction (Gidaris et al., 2)1 then uses the self-supervised objective to adapt the representation of individual images during testing. Instead of image recognition, we perform test-time adaptation for RL with visual inputs in an online fashion. As the agent interacts with an environment, we keep obtaining new observational data in a stream for training the visual representations.
METHOD
In this section, we describe our proposed Policy Adaptation during Deployment (PAD) approach. can be implemented on top of any policy network and standard RL algorithm (both on-policy and off-policy) that can be described by minimizing some RL objective (()) w.r.t. the collection of parameters 0 using stochastic gradient descent.
3.1 NETWORK ARCHITECTURE
We design the network architecture to allow the policy and the self-supervised prediction to share features. For the collection of parameters 0 of given policy network æ. split sequentially into 0 (0.,, 0a), where Og collects the parameters of the feature extractor, and 0,, is the head that outputs distribution over actions. We define networks Te with parameters 0, and na with parameters 9 such that ((;; 0) T (T. where represents an image observation. Intuitively, one can think of Te as feature extractor, and TT as controller based on these features. The goal our method is to update Te at test-time using gradients from self-supervised task, such that Te (and consequently nо) can generalize. Let n, with parameters 02 be the self-supervised prediction head and its collection of parameters, and the input to rT be the output of (as illustrated in Figure 1). In this work, the self-supervised task inverse dynamics prediction for control, and rotation prediction for navigation.
3.2 INVERSE DYNAMICS PREDICTION AND ROTATION PREDICTION
At each time step, we always observe transition sequence in the form of (st, a, St+1), during both training and testing. Naturally, self-supervision can be derived from taking parts of the sequence and predicting the rest. An inverse dynamics model takes the states before and after transition, and predicts the action in between. In this work, the inverse dynamics model म. operates on the feature space extracted by Te. We can write the inverse dynamics prediction objective formally as
(1) For continuous actions, is the mean squared error between the ground truth and the model output. For discrete actions, the output is soft-max distribution over the action space, and & is the cross-
L(O,, 0.) = É(a,, (s:),, Te (5(+1)))
3

---

Published as a conference paper at ICLR 2021
entropy loss. Empirically, we find this self-supervise task to be most effective with continuous actions, possibly because inverse dynamics prediction in a small space of discrete actions is not as challenging. Note that we predict the inverse dynamics instead of the forward dynamics, because when operating in feature space, the latter can produce trivial solutions such as the constant zero feature for every state". If we instead performed prediction with forward dynamics in pixel space, the task would be extremely challenging given the large uncertainty in pixel prediction.
As an alternative self-supervised task, we use rotation prediction (Gidaris et al., 2018). We rotate an image by one of 0, 90, 180 and 270 degrees as input to the network, and cast this as a four-way classification problem to determine which one of these four ways the image has been rotated. This task is shown to be effective for learning representations for object configuration and scene structure, which is beneficial for visual recognition (Hendrycks et al., 2019; Doersch & Zisserman, 2017).
3.3 TRAINING AND TESTING
Before deployment of the policy, because we have signals from both the reward and self-supervised auxiliary task, we can train with both in the fashion of multi-task learning. This corresponds the following optimization problem during training e a trade-off hyperparameter. During deployment, we cannot optimize J anymore since the reward is unavailable, but we can still optimize L to update both 0g and 6.. Empirically, we find only negligible difference with keeping 0₂ fixed at test-time, so we update both since the gradients have to be computed regardless; we ablate this decision in appendix C. As we obtain new images from the stream of visual inputs in the environment, 0 keeps being updated until the episode ends. This corresponds to, for each iteration t = 1...T:
0. J [O.. 0e) + aL ((0,00,) 0e), where > 0 is
St ~ p(S/ 1, 1c-1
(t) = (1 1) - L(ri 0, 1), 0, 1))
(3)
(t) = 0. - 1) - Lsr 1), (t 1)) 1))
(4)
a, = x (s;; (()) with O()) - (l)())
where 0.(0) = 0., (0) = 0.. is the initial condition given by the environment, ao (so)), the unknown environment transition, and L is the self-supervised objective as previously introduced.
4 EXPERIMENTS
In this work, investigate how well an agent trained in one environment (denoted the training environment) generalizes to unseen and diverse test environments. During evaluation, agents have no access to reward signals and are expected to generalize without trials nor prior knowledge about the test environments. In simulation, we evaluate our method (PAD) and baselines extensively on continuous control tasks from DeepMind Control (DMControl) suite (Tassa et al., 2018) as well as the CRLMaze (Lomonaco et al., 2019) navigation task, and experiment with both stationary (colors, objects, textures, lighting) and stationary (videos) environment changes. We further show that PAD transfers from simulation to a real robot and successfully adapts to environmental differences during deployment in two robotic manipulation tasks. Samples from DMControl and CRLMaze environments are shown in Figure 2, and samples from the robot experiments are shown in Figure 4. Implementation is available at ht / wwcklashanseeeeee gi thub. io/PAD/.
Network details. For DMControl and the robotic manipulation tasks implement PAD top of Soft Actor-Critic (SAC) (Haarnoja et al., 2018), and adopt both network architecture and hyperparam- eters from Yarats et al. (2019), with minor modifications: the feature extractor Ne has 8 convolutional layers shared between the RL head n. and self-supervised head ng, and we split the network into architecturally identical heads following Te. Each head consists of 3 convolutional layers followed by 4 fully connected layers. For CRLMaze, we use Advantage Actor-Critic (A2C) as base algorithm (Mnih al., 2016) and apply the same architecture as for the other experiments, but implement n. with only 6 convolutional layers. Observations are stacks of k colored frames (k = 3 on DMControl and CRLMaze;
in robotic manipulation) of size 100 100 and time-consistent random crop is applied as in Srinivas et al. (2020). During deployment, we optimize the self seff-superviee objective online w.r.t. 0z, 02 for one gradient step per time iteration. See appendix F for implementation details.
A forward dynamics model operating in feature space can trivially achieve loss of 0 by learning map every state constant vector, e.g. An inverse dynamics model, however, does not have such trivial solutions.
4

---

Published as a conference paper at ICLR 2021
Figure Left: Training environments of DMControl (top) and CRLMaze (bottom). Right: Test en- viromments of DMControl (top) and CRLMaze (bottom). Changes to DMControl include randomized colors, video backgrounds, and distractors; changes to CRI Maze include textures and lighting.
Table 1.
Episodic return in test environments with randomized colors, mean and std. dev. for 10 seeds. Best method on each task is in bold and blue compares SAC+IDM with and without PAD.
10x episode length
Random colors
SAC
+DR
+IDM
+IDM (PAD)
+IDM
+IDM (PAD)
Walker, walk
414474
594=1044
4064:29
468-47
3830+547
55005
Walker, stand
719+74
715
743437
797:46
7832+200
8566:1121
Cartpole, swingup
592:30
647:448
585173
630=63
6528:1539
7593:1522
Caripole, balance
857+00
867#a7
835++400
848-29
77-16=526
7670+293
Ball in cup, catch
411=188
170#2622
171
563+50
Finger, spin Finger, turn easy
626-163
4651:114
757.622
803
7249
7496:1655
270.443
167
283:51
3041-46
Cheetah, run
154+41
145=22
121+:88
159:28
1117=530
1208:4487
Reacher, easy
163:45
105
201::32
2141-44
1788:441
2152:550
4.1 DEEPMIND CONTROL
DeepMind Control (DMControl) (Tassa et al., 2018) is a collection continuous control tasks where agents only observe raw pixels. Generalization benchmarks DMControl represent diverse real-world tasks for motor control, and contain distracting surroundings not correlated with the reward signals.
Experimental setup. We experiment with 9 tasks from DM- Control and measure generalization to four types of environ- ments: (i) randomized colors; (ii) natural videos as background;
15%
Relative improvement
(iii) distracting objects placed in the scene; and (iv) the un- modified training environment. For each test environment, we evaluate methods across 10 seeds and 100 random initializa- tions. If a given test environment is not applicable to certain tasks, e.g. if a task has no background for the video background setting, they
Walker, Cartpole, swingup Co tortpolc, balance
excluded. Tasks selected on the basis of di- versity, as well as the success of vision-based RL in prior work (Yarats et al., 2019; Srinivas et al., 2020; Laskin et al., 2020; Kostrikov et al., 2020). We implement PAD on top of SAC and use an Inverse Dynamics Model (IDM) for self-supervision, as we find that learning a model of the dynamics works well for motor control. For completeness, we ablate the choice of self-supervision. Learning curves are provided in appendix B. We compare our method to the following baselines: (i) SAC with no changes (denoted SAC); (ii) SAC trained with domain randomization
-5%
.00%
200
400
600
800
1000
Episode steps
Figure 3. Relative improvement in instantaneous reward over time for PAD on the random color env.
fixed set of 100 colors (denoted +DR); and (iii) SAC trained jointly with an IDM without PAD (denoted +IDM). Our method using an IDM with PAD denoted by +IDM (PAD). For domain randomization, colors sampled from the same distribution as in evaluation, but with lower variance, as we find that training directly on the test distribution does not converge.
Random perturbation color. Robustness to subtle changes such as color is essential to real- world deployment of RL policies. We evaluate generalization foreground, background and the agent itself, and report the results in Table (first columns). We find PAD improve generalization in all tasks considered, outperforming SAC trained with domain
fixed set of 100 colors of
5

---

Published as a conference paper at ICLR 2021
randomization in 6 out of 9 tasks. Surprisingly, despite a substantial overlap between training and test domains of domain randomization, it generalizes no better than vanilla SAC on a majority of tasks.
Long-term stability. We find the relative improvement of PAD to improve over time, shown in Figure 3. To examine the long-term stability of PAD. we further evaluate on 10x episode lengths and summarize the results in the last two columns in Table (goal-oriented tasks excluded). While do not explicitly prevent the embedding from drifting away from the RL. task, we find empirically that PAD does not degrade the performance of the policy, even over long horizons, and when PAD does not improve, we find it to hurt minimally. We conjecture this is because we are not learning a new task, but simply continue optimize the same superrsuper objective as during joint training, where both two tasks are compatible. In this setting, PAD still improves generalization in 6 out of 7 tasks, and thus naturally extends beyond episodic deployment. For completeness, we also evaluate methods in the environment in which they were trained, and report the results in appendix A. We find that, while PAD improves generalization to novel environments, performance is virtually unchanged on the training environment. We conjecture this is because the self-supervised task is already fully learned and any continued training on the same data distribution thus has little impact.
Non-stationary environments. To investigate whether PAD can adapt in non-stationary envi- ronments, we evaluate general- ization to diverse video back- grounds (refer to Figure 2). We find PAD outperform all base- lines on 7 out of 8 tasks, as shown in Table 2, by as much as 104% over domain random- ization on Finger, spin. Domain randomization generalizes com- parably worse to videos, which conjecture is not because the environments are non-stationary, but rather because the image statistics of videos are not cov- ered by its training domain of randomized colors. In fact, do- main randomization is outper- formed by the vanilla SAC in most tasks with video back- grounds, which is in line with the findings of Packer et al. (2018).
Table 2. Episodic return in test environments with video back- grounds (top) and distracting objects (bottom), mean and std. dev. for 10 seeds. Best method on each task is in bold and blue compares SAC+IDM with and without PAD.
Video backgrounds
SAC
+DR
+IDM
+IDM (PAD)
Walker, walk Walker, stand Cartpole, swingup
616=80
655
694s85
717:79
899
869
902
935::2
375
485-67
487-00
521
Cartpole, balance
693+109
766192
691+76
687=28
Ball in cup, catch
393+178
271
362
436-15
Finger, spin Finger, turn_easy Cheetah,
447++102
338
605
691:
355
223
355
362::10
194+30
150a34
164 1:42
206
Distracting objects
SAC
+DR
+IDM
+IDM (PAD)
Cartpole, swingup Cartpole. balance
815:100
809=21
776=58
771=61
969+20
938
964+22
960=29
Ball in cup, catch Finger, spin Finger, turn easy
177#111
3314189
482+128
545+:173
652+184
564.1288
836262
867
302
165+12
326+101
347:448
Scene content. We hypothesize that: (i) an agent trained with an IDM comparably less distracted by scene content since objects uncorrelated to actions yield no predictive power; and (ii) that PAD can adapt to unexpected objects in the scene. We test these hypotheses by measuring robustness to colored shapes at a variety of positions in both the foreground and background of the scene (no physical interaction). Results are summarized in Table 2. PAD outperforms all baselines in 3 out of tasks, with relative improvement of 208 % over SAC on Ball in cup, catch. In the two cartpole tasks in which PAD does not improve, all methods are already relatively unaffected by the distractors.
Choice of self-supervised task. We investigate how much the choice of self-supervised task con- tributes to the overall success of our method, and consider the following ablations: (i) replacing inverse dynamics with the rotation prediction task described in Section 3.2; and (ii) replacing it with the recently proposed CURL (Srinivas et al., 2020) contrastive learning algorithm for RI. As shown in Table 3. PAD improves generalization of CURL in a majority of tasks on the randomized color benchmark, and in 4 out of 9 tasks using rotation prediction. However, inverse dynamics as auxiliary task produces more consistent results and offers better generalization overall. We argue that learning an IDM produces better representations for motor control since it connects observations directly to actions, whereas CURL and rotation prediction operates purely on observations. In general, we find the improvement of PAD to be bigger in tasks that benefit significantly from visual information (see appendix A), and conjecture that selecting a self-supervised task that learns features useful to the RL task is crucial to the success of PAD, which we discuss further in Section 4.2.

---

Published as a conference paper at ICLR 2021
Table 3. Ablations on the randomized color domain of DMC. All methods use SAC. CURL represents RL with a contrastive learning task (Srinivas et al., 2020) and Rot represents the rotation predic-
tion (Gidaris al., 2018). Offline PAD is here denoted O-PAD for brevity, whereas the default usage
of PAD is in online setting. Best method is in bold and blue compares +IDM w/ and w/o PAD.
Random colors
CURL
CURL (PAD)
Rot
Rot (PAD)
IDM
IDM (O-PAD)
IDM (PAD)
Walker, walk
445+99
495=70
335+7
330+30
406:220
441
4682-47
Walker. stand
662
7534:49
673
653
743
727
797:446
Cartpole, swingup
454=110
413+67
493+52
477
585:73
578
630:
Cartpole, balance
782:13
763
710
734
835:140
796133
848
Ball in cup, catch
231+92
332478
291+54
314+00
471#75
490=16
563+50
Finger, spin Finger, turneeasy
691+12
588++22
695436
689+20
757=62
767
803+72
202.33
18612
283.168
230
283
321110
304
Cheetah, run Reacher, easy
202+22
211=20
127:23
135:12
121:238
112:25
159+28
325:32
378:222
99:22
120+7
201:322
241:24
214
Table 4. Episodic return of PAD and baselines in CRI.aaee environments. PAD improves general- ization in all considered environments and outperforms both A2C and domain randomization by large margin. All methods use A2C. We report mean and std. error of 10 seeds. Best method in each environment bold and blue compares rotation prediction with and without PAD.
CRLMaze
Random
A2C
+DR
+DDM
+DM (PAD)
+Rot
+Rot (PAD)
Walls
-870=30
380+145
2260+137
-302+150
-2882135
10062166
74:116
Floor
-3
- 167
-438
-47:198
-530=106
-294
-209=04
Ceiling
-872=30
- 1151+175
-400=74
166=215
-508=104
128+196
281
Lights
-900=22
-30+-13
-310+106
239=270
-460=114
-84+53
312
Offline versus online learning. Observations that arrive sequentially are highly correlated, and we thus hypothesize that our method benefits significantly from learning online. To test this hypothesis. we rum an offline variant of our method in which network updates are forgotten after each step. In this setting, our method can only adapt to single observations and does not benefit from learning over time. Results are shown in Table 3. We find that our method benefits substantially from online learning, but learning offline still improves generalization on select tasks.
4.2 CRLMAZE
CRL. (Lomonaco al., 2019) time-constrained, discrete-action navigation task for ViZDoom (Wydmuch et al., 2018), in which an agent is to navigate a maze and collect objects. There is a positive reward associated with green columns, and a negative reward for lanterns as well as for living. Readers are referred to the respective papers for details on the task and environment.
Experimental setup. We train agents on a single environment and measure generalization to environments with novel textures for walls, floor, and ceiling, well as lighting, shown in Figure 2. We implement PAD on top of A2C (Mnih et al., 2016) and use rotation prediction (see Section 3.2) as self-supervised task. Learning navigate novel scenes requires generalized scene understanding, and we find that rotation prediction facilitates that more so than an IDM. We compare to the following baselines: (i) random agent (denoted Random); (ii) A2C with no changes (denoted A2C); (iii) A2C trained with domain randomization (denoted +DR); (iv) A2C with an IDM as auxiliary task (denoted +IDM); and (v) A2C with rotation prediction as auxiliary task (denoted +Rot). We denote Rot with PAD as +oot (PAD). Domain randomization uses 56 combinations of diverse textures, partially overlapping with the test distribution, and we find necessary train domain randomization for twice as many episodes in order to converge. We closely follow the evaluation procedure of (Lomonaco et al., 2019) and evaluate methods across 20 starting positions and 10 random seeds.
Results. We report performance on the CRLMaze environments in Table 4. PAD improves gener- alization in all considered test environments, outperforming both A2C and domain randomization by large margin. Domain randomization performs consistently across all environments but is less successful overall. We further examine the importance of selecting appropriate auxiliary tasks by a simple ablation: replacing rotation prediction with an IDM for the navigation task. We conjecture that, while an auxiliary task can enforce structure in the learned representations. its features (and consequently gradients) need to be sufficiently correlated with the primary RL task for PAD to be
7

---

Published as a conference paper at ICLR 2021
(a) Simulation. Figure 4. Samples from the push robotic manipulation task. The task is to push the yellow cube to the location of the red disc. Agents are trained in setting (a) and evaluated in settings (b-d).
(b) Default transfer.
(c) Table cloth.
(d) Disco lights.
successful during deployment. While PAD with rotation prediction improves generalization across all test environments considered, IDM does not, which suggests that rotation prediction is more suitable for tasks that require scene understanding, whereas IDM is useful for tasks that require motor control. We leave it future work to automate the process of selecting appropriate auxiliary tasks.
4.3 ROBOTIC MANIPULATION TASKS
We deploy our method and baselines a real Kinova Gen3 robot and evaluate on two manipulation tasks: (i) reach, a task in which the robot reaches for goal marked by a red disc; and (ii) push, task in which the robot pushes cube to the location of the red disc. Both tasks use an XY action space, where the Z position of the actua- tor is fixed. Agents operate purely from pixel observations with no access state information. During deployment, we make no effort to calibrate camera, lighting, physical properties such as dimensions, mass, and friction, and policies are expected to generalize with no prior knowledge of the test environment. Samples from the push task are shown in Figure and samples from reach are shown in appendix E.
Table 5. Success rate of PAD and baselines on real
robotic arm. Best method in each environment is in bold and blue compares +IDM with and without PAD.
Real robot
SAC
+DR
+IDM
+IDM (PAD)
Reach (default)
100%
100%
100%
100%
Reach (cloth)
48%
80%
56%
80%
Reach (disco)
72%
76%
88%
92%
Push (default)
88%
88%
92%
100%
Push (cloth)
60%
64%
64%
88%
Push (disco) 60%
68%
72%
84%
Experimental setup. We implement PAD on top of SAC (Haarnoja et al., 2018) and apply the same experimental setup as in Section 4.1 using an Inverse Dynamics Model (IDM) for self-supervision, but with- out frame-stacking (i.e. k = 1). Agents are trained in simulation with dense rewards and randomized initial configurations of arm, goal, and box, and we measure gen- eralization 3 novel environments in the real-world: (i) default environment with pixel observations that roughly mimic the simulation; (ii) a patterned table cloth that distracts visually and greatly increases friction; and (iii) disco, an environment with non-stationary visual disco light distractions. Notably, all 3 environments also feature subtle differences in dynamics compared the training environment, such object dimensions, mass, friction, and uncalibrated actions. In each setting, we evaluate the success rate across 25 test runs spanning across 5 pre-defined goal locations throughout the table. The goal locations vary between the two tasks, and the robot reset after each run. We perform comparison against direct transfer and domain randomization baselines as in Section 4. We further evaluate generalization to changes in dynamics by considering variant of the simulated environment in which object mass, size, and friction, arm mount position, and end effector velocity is modified. We consider each setting both individually and jointly, and evaluate success rate across 50 unique configurations with the robot reset after each run.
Table Success rate of PAD and baselines for the push task on simulated robotic arm in test environments with changes to dynamics. Changes include object mass, size, and friction, mount position, and end effector velocity. Best method in each environment is in bold and blue compares +IDM with and without PAD.
Simulated robot
SAC
+DR
+IDM
+IDM (PAD)
Push (object)
66%
64%
72%
82%
Push (mount)
68%
58%
86%
84%
Push (velocity)
70%
68%
70%
78%
Push (all) 56%
50%
48%
76%
Results. We report transfer results in Table 5. While all methods transfer successfully to reach (default), observe PAD to improve generalization in all settings in which the baselines show
8

---

Published as a conference paper at ICLR 2021
sub-optimal performance. We find PAD to be especially powerful for the push task that involves dynamics, improving by as much as 24% in push (cloth). While domain randomization proves highly effective in reach (cloth), observe significant benefit in the other settings, which suggests that PAD can be more suitable in challenging tasks like push. To isolate the effect of dynamics, we further evaluate generalization to number of simulated changes in dynamies shown in Table 6, We find PAD to improve generalization to changes in the physical properties of the object and end effector, whereas both SAC+IDM and PAD are relatively unaffected by changes to the mount position. Consistent with the real robot results in Section 5, PAD is found to be most effective when changes in dynamics are non-trivial, improving by as much as 28% in the push (all) setting, where all 3 environmental changes are considered jointly. These results suggest that PAD can be a simple, yet effective method for generalization diverse, unseen environments that vary both visuals and dynamics.
the push task. Results
CONCLUSION
While previous work addresses generalization in RL by learning policies that are invariant any environment changes that can be anticipated, we formulate an altemnatve problem setting in vision- based RL: can we instead adapt a pretrained-policy to new environments without any reward. We propose Policy Adaptation during Deployment, a self-supervised framework for online adaptation test-time, and show empirically that our method improves generalization of policies to diverse simulated and real-world environmental changes across a variety of tasks. We find our approach benefits greatly from learning online, and we systematically evaluate how the choice of self supervised task impacts performance. While the current framework relies on prior knowledge on selecting self- supervised tasks for policy adaptation, we see our work as the initial step in addressing the problem of adapting vision-based policies to unknown environments. We ultimately envision embodied agents in the future be learning all the time, with the flexibility to learn both with and without rewards, before and during deployment.
Acknowledgements. This work was supported, in part, by grants from DARPA, NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, and gifts from Qualcomm and TuSimple. This work was also funded, in part, by grants from Berkeley DeepDrive, SAP and European Research Council (ERC) from the European Union Horizon 2020 Programme under grant agreement no. 741930 (CLOTHILDE). We would like to thank Fenglu Hong and Jocy Hejna for helpful discus- sions.
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in- hand manipulation. The International Journal of Robotics Research, 39(1):3-20,
2020. 1
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. ACM Trans. Graph., 38(4), 2019. ISSN 0730-0301.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. 2
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017. 4
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control, 2018. 2

---

Published as a conference paper at ICLR 2021
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. The Journal of Machine Learning Research, 17(1):2096-200 2016. 2
training of neural networks.
Robert Geirhos, Carlos R. Medina Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In NeurlPS, 2018. 2
Muhammad Ghifary. W. Bastiaan Kleijn, Mengjie Zhang. and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, pp. 2551-2559. IEEE Computer Society, 2015. ISBN 9781467383911.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, 2018. 3, 3,4, 7
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2066-2073. IEEE, 2012. 2
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pp. 2451-2463, Curran Associates, Inc., 2018.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2018. 4, 8, 17
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv: Learning. 2018. 2
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. ArXiv, abs/1906. 12340, 2019. 2,
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canoniie adaptation networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12619-16229, 2019. 2
R. Julian, B. Swanson, G. Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv: Learning, 2020. 2, 15
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, V. Vanhoucke, and Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv, abs/1806. 10293, 2018.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. 2020. 5, 17
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky, Ily Sutskever, and Geoftrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012. 17
Sascha Lange and Martin A. Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 International Joint Conference on Neural Networks (JCNN), pp. 1-8, 2010.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020. 2, 17
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. A simple randomization technique for generalization in deep reinforcement learning. ArXiv, abs/1910.05396, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep
visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1375, 2016. 1
Ya Feng Li. Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. AAAI, 2018.
Vincenzo Lomonaco, Karen Desai, Eugenio Culurciello, and Davide Maltoni. Continual reinforce- ment learning in 3d non-stationary environments. arXiv preprint arXiv: 1905. 101 2019. 2, 4, 7, 16, 17
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pp. 136-144, 2016.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Ricdmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, 2016. 4, 17
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019. doi: 10.1 10.1109/icc..211.0003.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances Neural Information Processing Systems, pp. 9191-9200 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016. 2
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning, 2018. 6
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In JCML, 2017. 2
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018. 1
11

---

Published as a conference paper at ICLR 2021
Lerrel Pinto and Abhinav Gupta. Supersizing sulf-superiision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406-3413. IEEE, 2016.
Lerrel Pinto, Marcin Andrychowiez, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-batt based robot learning, arXiv preprint 1TXiv:1710006522, 2017a. 1,
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2817-2826, JMLR. org, 2017b. 1
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016.
Fabio Ramos, Rafael Possas, and Dieter Fox. Bayessim: Adaptive domain randomization via probabiliitic inference for robotics simulators. Robotics: Science and Systems XV, Jun 2019. 2
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint 100,00000 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. 2
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606. 04671, 2016.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04200 2016. 1,2
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models, 2020. 2
Vaishaal Shankar, Achal Dave, Rebecca Roclofs, Deva Ramanan, Benjamin Recht, and Lud- wig Schmidt. systematic framework for natural perturbations from videos. arXiv preprint arXiv: 1906.02660000 2019. 2
Assaf Shocher, Nadav Cohen, and Michal Irani. Zero-shot super-resolution using deep internal learning, 2017. 3
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and remapping the "dna" of a natural image, 2018.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.041333 2020. 2, 4, 5, 6, 17
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through
self-supervision. arXiv preprint, 2019. 2
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. ICML, 2020. 3
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. 17
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. Technical report, DeepMind, January 2018. 2, 4, 5, 16, 17
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba. and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEFF/RSJ International Conference Intelligent Robots and Systems (IROS), Sep 2017. 1,2
12

---

Published as a conference paper at ICLR 2021
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017. 2
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008. 2
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 2
Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle- consistency of time. In CVPR, 2019. 2
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning learn how learn: Self-adaptive visual navigation using meta-learning, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3733-3742, 2018.
Marek Wydmuch, Michat Kempka, and Wojciech Jaskowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games, 2018. 2, 7, 16
Wilson Yan, Ashwin Vangipuram, Picter Abbeel, and Lerrel Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003. 200,:40003.04436,, 2020. 1, 2
Jiachen Yang, Brenden Petersen, Hongyuan Zha, and Daniel Faissol. Single episode policy transfer in reinforcement learning, 2019. 1,2
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pincau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2019. 2, 4, 5, 17
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016. 2
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1058-1067, 2017.
13

---

Published as a conference paper at ICLR 2021
A PERFORMANCE ON THE TRAINING ENVIRONMENT
Historically, agents have commonly been trained and evaluated in the same environment when benchmarking RL algorithms exclusively in simulation. Although such an evaluation procedure does consider generalization, it is still useful metric for comparison of sample efficiency and stability of algorithms. For completeness, we also evaluate our method and baselines in this setting on both DMControl and CRLMaze. DMControl results are reported in Table 7 and results on the CRLMaze enviromment are shown in Table 8. In this setting, we also compare to an additional baseline on DMControl: blind SAC agent that operates purely on its previous actions. The performance of blind agent indicates to which degree given task benefits from visual information. We find that, while PAD improves generalization to novel environments, performance is virtually unchanged when evaluated on the same environment in training. We conjecture that this is because the algorithm already is adapted to the training environment and any continued training on the same data distribution thus has little influence. We further emphasize that, even when evaluated PAD still outperforms baselines on most tasks. For example, we observe a 15% relative improvement over SAC on the Finger, spin task. We hypothesize that this gain in performance is because the self- supervised objective improves learning by constraining the intermediate representation of policies. blind agent is no better than random on this particular task, which would suggest that agents benefit substantially from visual information in Finger, spin. Therefore, learning a good intermediate representation of that information is highly beneficial to the RL objective, which we find PAD to facilitate through its self- supervuper learning framework. Likewise, the SAC baseline only achieves 51 % improvement over the blind agent on Cartpole, balance, which indicates that extracting visual information from observations is not as crucial on this task. Consequently, both PAD and baselines achieve similar performance on this task.
the training environment,
Table 7. Episodic return on the training environment for each of the 9 tasks considered in DMControl, mean and std. dev. for 10 seeds. Best method on each task is in bold and blue compares +IDM with and without PAD. It is shown that PAD hurts minimally when the environment unchanged.
Training env.
Blind
SAC
+DR
+IDM
+IDM (PAD)
Walker, walk
235
847
756+71
911
895
Walker, stand
388+10
959+11
928
96618
956=20
Cartpole, swingup
132:41
850=28
807436
849430
845+34
Cartpole, balance
646=181
978+22
971
982+20
979#21
Ball in cup, catch
150
725
469
919 1118
9104-129
Finger, spin
3=2
809
686#229
928
927+445
Finger, tur_casy
172+27
462+146
243=1124
462=152
455+160
Cheetah, run
264478
387=74
195:-46
384
380
Reacher, easy
107+11
264+113
92:45
390#126
365+:11
Table 8. Episodic return of PAD and baselines in the CRLMaze training environment, All methods use A2C. We report mean and std. error of 10 seeds. Best method is in bold and blue compares rotation prediction with and without PAD.
CRLMaze
Random
A2C
+DR
+IDM
+IDM (PAD)
+Rot
+Rot (PAD)
Training env.
-318
371+198
-355+93
585+246
416+135
729:148
681+99
B LEARNING CURVES ON DEEPMIND CONTROL
All methods are trained until convergence (500,000 frames) on DMControl. While we do not consider the sample efficiency of our method and baselines in this study, we report learning curves for SAC, SAC+IDM and SAC trained with domain randomization on three tasks in Figure 5 for completeness. SAC trained with and without an IDM are similar terms of sample efficiency and final performance, whereas domain randomization consistently displays worse sample efficiency, larger variation between seeds, and converges to sub-optimal performance in two out of the three tasks shown.
14

---

Published as a conference paper at ICLR 2021
Cartpale, swingup
Walker, stand
Walker, walk
800
020
400
200
0.0
Number (M)
trames (14)
Number frames (M)
SAC
(DR)
SAC+IDM
Figure 5. Learning curves for SAC, SAC trained with domain randomization (denoted SAC (DR) here), and SAC+IDM on three tasks from the DeepMind Control suite (DMControl), Episodic return averaged across 10 seeds and the 95% confidence intervals are visualized as shaded regions. SAC and SAC+IDM exhibit similar sample efficiency and final performance, whereas domain randomization consistently displays worse sample efficiency, larger variation between seeds, and converges to sub-optimal performance in two out of the three tasks shown.
C KEEPING TT FIXED DURING POLICY ADAPTATION
We now consider variant of PAD where the self-supervised task head is fixed at test-time such that the self-supervised objective optimized only wrt Te, as discussed in Section 3.3. We measure generalization test environments with randomized colors and report the results in Table 9 for three tasks from the DeepMind Control suite. We empirically find the difference between updating T. and keeping it fixed negligible, and we choose update π. by default since its gradients computed by back-propagation regardless.
Table 9. Episodic return test environments with randomized colors, mean and std. dev. for 10 seeds. All methods use SAC. IDM (PAD, fixed πτ,) considers a variant of PAD where is fixed at test-time, whereas IDM (PAD) denotes the default usage of PAD in which both тe and na are optimized at test-time using the self-supervised objective.
Random colors
IDM
IDM (PAD, fixed m.)
IDM (PAD)
Walker, walk
406=29
452+38
468
Walker, stand
743=37
802:411
797
Cartpole, swingup
585=73
623+57
630
D COMPARISON TO ADAPTATION WITH REWARDS
While our method does not require data collected prior to deployment and does not assume access reward signal, additionally compare method to naïve fine-tuning approach using transitions and rewards collected from the target environment prior to deployment. To fine-tune the pre-trained policy using rewards, collect datasets consisting of 1, 10, and 100 episodes in each target environment using the learned policy while keeping its parameters fixed, and then subsequently fine-tune both मe and Na on the collected data, following the same training procedure the training phase. This fine-tuning approach is analogous to Julian et al. (2020) but does not use data from the original environment during adaptation. Results are shown in Table 10. We find that naïvely fine-tuning the policy using data collected prior to deployment can improve generalization but requires comparably more data than PAD, well as access to reward signal the target environment. This finding suggests that PAD may be more suitable method for settings where data from the target environment is scarce and not easily accessible prior to deployment.
during
ADDITIONAL. ROBOTIC MANIPULATION SAMPLES
Figure 6 provides samples from the training and test environments for the reach robotic manipulation task. Agents are trained in simulation and deployed on a real robot. Samples from the push task are shown in Figure 4.
15

---

Published as a conference paper at ICLR 2021
Table 10. Episodic return in test environments with randomized colors, mean and std. dev. for 10 seeds. All methods use SAC trained with an inverse dynamics model (IDM) as auxiliary task. Our method is denoted IDM (PAD), and we compare access to transitions and rewards collected from 1, 10, and 100 episodes, respectively, from target environments prior to deployment.
naïve fine-tuning approach that assumes
Fine-tume tuning w/ rewards
Random colors
IDM
IDM (PAD)
episode
10 episodes
100 episodes
Walker, walk
29
468
395#78
189
561+62
Walker, stand
34
49 630+63
661
128
784+31
Cartpole, swingup
585
538
605
650
(a) Simulation.
(b) Default transfer.
(c) Table cloth.
(d) Disco lights.
Figure 6. Samples from the reach robotic manipulation task. The task is to move the robot gripper to the location of the red disc. Agents are trained in setting (a) and evaluated settings (b-d) on a real robot, taking observations from uncalibrated camera.
F IMPLEMENTATION DETAILS
In this section, we elaborate on implementation details for our experiments on DeepMind Control (DMControl) suite (Tassa al., 2018) and CRLMaze (Lomonaco et al., 2019) for ViZDoom (Wyd- much et al., 2018). Our implementation for the robotic manipulation experiments closely follows that of DMControl. Code is available at atttps://nickkeee
thub io/PAD/.
Shared Representation
Self- Self-Supervised Prediction
OCTVV
Policy
conv
Observation
fc
Value
Figure Network architecture for the DMControl, MRLM and robotic manipulation experiments. n and R uses shared feature extractor nh. Observations are stacks of 100 100 colored frames. Implementation of policy and value function depends on the learning algorithm.
Architecture. Our network architecture is illustrated Figure 7. Observations are stacked frames (k: = 3) rendered at 100 x 100 and cropped to 84 x 84, i.e. inputs to the network are of dimensions 9 x 84 x 84, where the first dimension indicates the channel numbers and the following ones represent spatial dimensions. The same crop is applied to all frames in stack. The shared feature extractor x² consists of 8 (DMControl, robotic manipulation) or (CRLMaze) convolutional layers and outputs features of size 32 21 x 21 in DMControl and robotic manipulation. and size 32 25 x 25 in CRLMaze. The output from n. is used as input to both the self-supervised head π and RL head 7 both of which consist of convolutional layers followed by fully-connected layers. All
16

---

Published as a conference paper at ICLR 2021
Table 11. Hyperparameters used for the DM- Table 12. Hyperparameters used for the CRL- Control (Tassa et al., 2018) tasks.
Maze (Lomonaco al., 2019) navigation task.
Hyperparameter
Value
Hyperparameter
Value
Frame rendering
x 100 x 100
Frame rendering
3 100 x 100
Prame after crop
84 x
Frame after crop
3 84 x
Stacked frames
Stacked frames
Action repeut
(finger) (cartpole) (atherwise)
Action repeat
Discount factor ?
0,99
Episode length
1,000
Discount factor 7
0.99
Learning algorithm
Advantage Actor Critic
Episode length
1,000
Self-supervised task
Rotation Prediction
Learning algorithm
Soft Actor-Critic
Number of training episodes
1,000 (dom. rand.)
Self' salesripeeiee task
Dynamics Model
(otherwise)
Number of training steps
500,000
Number of processes
20
Replay butter size Optimizer (Ir),
500,000
Optimizer
Adam (+) = 0.9. 62 0.909)
Adam (3) 0.9, 0.999)
Learning
le.4
Optimizer (o)
Adam (.1 = 01 0.999)
Learning rate (test (test-time)
1e-5
Learning rate (5","",, ,",)
3e-4 (chestah)
Batch size
20
1e:3 (otherwise)
Batch size (test-time)
32
Learning rate (n)
le-4
5" x loss coefficient
0.5
Batch
loss coefficient (test-time)
Batch size (test-time)
32
update freq.
nf, n" update freq.
x update freg. (test-time)
af, update freq. (test-time)
convolutional layers use 32 filters and all fully connected layers use hidden size of 1024, in Yarats et al. (2019).
Learning algorithm. We use Soft Actor-Critic (SAC) (Haarnoja et al., 2018) for DMControl and robotic manipulation. and Advantage Actor-Critic (A2C) for CRLMaze. Network outputs depend on the task and learning algorithm. As the action spaces of both DMControl and robotic manipulation are continuous, the policy learned by SAC outputs the mean and variance of Gaussian distribution over actions. CRLMaze has discrete action space and the policy learned by A2C thus learns soft-max distribution over actions. For details on the critics learned by SAC and A2C, the reader is referred to Haarnoja al. (2018) () and Mnih et al. (2016), respectively.
Hyperparameters. When applicable, we adopt our hyperparameters from Yarats et al. (2019) (DM- Control, robotic manipulation) and Lomonaco al. (2019) (CRLMaze). For the robotic manipulation experiments, implementation closely follows that of DMControl, only differing by number of frames in an observation. We use frame stack of k = 3 frames for DMControl and CRLMaze, and only k frame for robotic manipulation. For completeness, the DMControl and CRLMaze environments in Table 11 and Table 12.
detail all hyperparameters used for
Data augmentation. Random cropping is commonly used data augmentation used in computer vision systems (Krizhevsky et al., 2012; Szegedy et al., 2015) but has only recently gained interest. as a stochastic regularization technique in the RL literature (Srinivas et al., 2020; Kostrikov et al., 2020; Laskin al., 2020). We adopt the random crop proposed in Srinivas rendered observations of size 100 x 100 to 84 x 84, applying the same crop to all frames in stacked observation. This has the added benefits of regularization while still preserving spatio-temporal patterns between frames. When learning an inverse dynamics model, we apply the same crop to all frames of a given observation but apply two different crops to consecutive observations (8,,81+11 81+1) used to predict action
(2020): crop
Policy Adaptation during Deployment. We evaluate our method and baselines by episodic return of an agent trained in a single environment and tested in collection of test environments, each with distinct changes from the training environment. We assume no reward signal at test-time and agents are expected to generalize without pre- mr--trinnig or resetting in the new environment. Therefore, we make updates to the policy using self-supervise objective, and train using observations from the environment in an online manner without memory, iec we make one update per step using the most-recent observation.
17

---

Published as a conference paper at ICLR 2021
Empirically, find that: (i) the random crop data augmentation used during training helps regularize learning at test-time; and (ii) our algorithm benefits from learning from batch of randomly cropped observations rather than single observations, even when all observations in the batch augmented copies of the most-recent observation. As such, we apply both of these techniques when performing Policy Adaptation during Deployment and use batch size of 32. When using the policy to take actions, however, inputs to the policy are simply center-cropped.
18


Reviews:
[{"review_id": "b4sJ_iyu4cV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the Policy Adaptation during Deployment (PAD) method, which allows a policy trained on a particular visual input distribution to be transferred to a new visual input distribution, as long as the underlying system dynamics remain the same.\n\nPAD works by structuring the policy into two parts, a state embedding network pi_e and an action selection network pi_a.  Additionally, PAD learns an auxiliary prediction task with a network pi_s that predicts the inverse dynamics (necessary action to transition from s->s').  Once on the real system, pi_a is kept frozen, and pi_e is updated to minimize the inverse dynamics loss.  In practice it seems that also updating pi_s provides negligible difference although I find only the case when pi_s is kept frozen to make sense (question on this below).  An alternative loss based on predicting scene rotation is also used and shown to work generally less well than inverse dynamics.\n\nExperiments show that PAD consistently outperforms other methods, either naive baselines, simplified domain randomisations, or ablations of PAD.  \n\nPros: I enjoyed reading the paper and believe it provides an interesting approach to used IDM as a consistent aspect of the task in the face of inconsistent visual representations of state.\n\nCons: This approach is very specific to pixels and assumes consistent inverse dynamics, which might not always be true esp. in sim2real tasks.    I think the more general question is how to adapt in MDPs when part of the MDP changes but another part stays constant.  The 'trick' in my opinion in this paper is that the dynamics are consistent across task variations, so building a predictor for this consistent dimension works.  It would have been interesting to look more generally at other aspects that could change, for example what if dynamics change but pixel representations are consistent, could a similar approach be used?  Is there a 'bigger' insight here?  I am overall accepting the paper, but I feel a bit frustrated we're not getting more insights and I feel like there's something more here.\n\nQuestions: \nI am clearly being dense here, but I wasn't able to understand why the constant zero feature would be a trivial solution to a next-state prediction problem (c.f.\n\" Note that we predict the inverse dynamics instead of the forward dynamics, because\nwhen operating on the feature space, the latter can produce trivial solutions such as the constant zero\nfeature for every state.\"), could you clarify?\n\nAlthough you mention that updating both pi_s and pi_e or only pi_e seems to provide equivalent performance, intuitively it would seem that allowing pi_s to move removes the common semantic grounding of the latent observation space that is necessary for pi_a to still be able to interpret it.  Do you have any insights here?\n\nPlease comment on my 'con', and provide some insights on how you see this approach working in potentially other dimensions than the observation space.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJKsjtDApz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThe authors present a method for online policy adaptation during domain transfer in the case no reward is available in the target domain. They achieve this by adding an auxiliary self-supervised task, such as inverse dynamics prediction, that helps shape a set of features shared with the policy during training. At test time, gradient updates are then performed on these shared features based on only the self-supervised loss. The authors evaluate their method on a number of visual continuous control tasks and discrete navigation tasks and show a significant improvement over direct transfer, domain randomisation, and using self-supervision only during training.\n\n#### Pros\n- The authors present a novel, intuitive method for domain transfer in the case of absence of reward signal. This is very relevant for e.g. sim2real transfer, where reward computation often require privileged information that is not easily measurable in a real-world setup.\n- The authors perform a large sweep of experiments on a variety of different tasks, incl. sim2real transfer on a robot arm, and provide significant detail of the experimental setup.\n- The results are largely in favour of the proposed method in comparison to fair baselines.\n- The paper is very well and clearly written, with significant attention to related work.\n\n#### Cons\n- My main concern with this submission is the lack of any discussion about the assumptions and resulting limitations of the proposed method. It is only in Section 3 that it becomes clear that this method is aimed at _perceptual_ adaptation, but otherwise assumes the transition dynamics of the source and target environment, and hence the resulting optimal action, to be the same. It's unclear how this method fairs in the case of a change in dynamics. This should definitely be more emphasised in the introduction, abstract and perhaps even title. This also puts the proposed method closer to methods such as RCAN (https://arxiv.org/abs/1812.07252), which would potentially be a strong baseline to compare to.\n- While the authors acknowledge that drift in feature space may occur by also updating the self-supervision-only weights at test time, they default to this setting but do not show any quantitative results and only mention that empirically this indeed does not significantly impact performance. Perhaps related, but the IDM+PAD setting seems to be is significantly detrimental in the CRLMaze task (see Table 4, Table 7), it would good to add a comment about this failure mode.\n\n#### Conclusion\nWhile there could be some additional discussion added, I think this paper is strong enough to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PHccd7e_4i1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary of the Work\nThis work presents a method for test-time adaptation of pre-trained visual reinforcement learning policies, with a particular emphasis on robotics applications. The method uses both a self-supervised representation learning objective and an RL objective during training, then at test-time (when RL supervision may not be available) adapts the agent to a new environment using only the self-supervision objective. The self-supervision objective is to learn an inverse dynamics model $p(s_t|a_{t-1}, s_{t-1})$ using an IDM network which shares its convolutional layers with the policy network $p(a_t|s_t)$. The authors provide experiments using a variety of simulation environments augmented with adaptation target tasks, then demonstrate the effectiveness of the method using real robot experiments. Overall, the method appears to be a simple and effective idea for quick test-time adaptation to appearance changes in visual RL, though the lack of a fine-tuning baseline makes it unclear how it compares to the \"null\" hypothesis of doing naive RL updates with no auxiliary losses\n\n## Pros and Cons\n\n### Pros\n* Addresses an important problem in the field (fast test-time adaptation)\n* Extensive experiments supporting claims, including real robot experiments\n\n### Cons\n* Likely limited only to appearance changes in the environment (not dynamics nor reward functions)\n* Comparisons to more \"naive\" fine-tuning baseline missing -- this compares favorably to other self-supervision systems, but how does it compare to the most basic adaptation method available to an RL agent?\n\n## Evaluation\n### Quality\n3/5\nThe overall presentation quality of the work is very high, but I take issue with some of the scientific aspects. While the experiments are all high-quality and well thought-out, this work has some issues with data analysis which are easy to address, and would make the statistical significance of the results much more clear. Why go to the trouble of collecting 10 seeds for many experiments, then only calculate a mean and standard deviation? I also believe a couple obvious baselines (fine-tuning and from-scratch training) were unnecessarily omitted and would help readers better-position the performance of this work as compared to the \"naive\" alternatives.\n\n### Clarity\n5/5\nEvery part of the text and supplemental were very well-presented. The tables and figures are helpful for understanding the method and results, and I found the mathematical portions of the text increase the clarity of presentation rather than simply providing a \"window dressing\" of formalism intended to make the work seem more principled than it really is.\n\n### Originality\n4/5\nThis work's originality stems primarily from its simplicity. I commend the authors for avoiding overly-complicated methods when simpler ones will do.\n\n### Significance\n4/5\nAddressing test-time adaptation without costly pre-train methods (e.g. meta-learning) is an important challenge for RL in the real world, and this work provides a step in this direction.  I would also appreciate if the work spent more time discussing the limitations of the method, and avoided statements which border on over-stating its potential -- for instance, the introduction suggests that adapting to new reward functions at test time is important (it is!) but this text contains no evidence that the proposed method could achieve that kind of adaptation. \n\n\n### Misc Editorial Comments and Reviewer's Notes\n\n#### Claims\n* Adapts a pre-trained policy to an unknown environment without any reward\n\n#### Mechanisms\n* Introduces self-supervision supervised with an auxiliary task\n* Auxiliary tasks\n  - inverse dynamics prediction\n  - rotation prediction (classify rotation of image)\n* Auxiliary prediction networks share parameters with the policy (particularly, CNN layers)\n* Trains both RL and auxiliary task during training, at test time uses IDM or rotation prediction auxiliary updates to adapt (but not RL)\n\n\n#### 1. Introduction\n* The reward functions in these experiments don't seem to change, so I don't think the motivation about \"craft[ing] a dense reward function...during deployment is impractical\" really applies? This motivation would be more compelling if the work proposed adaptation to new reward functions rather than only visual changes to the environment. Adaptation to new reward functions and dynamics likely requires a different family of methods than those addressed by this work. Again, see [1].\n\n\n#### 2. Related Work\n* Please see [1] for a similar work with a similar experimental setting. In particular, this calls into question the conventional wisdom that fine-tuning is too naive for test-time adaptation and can be immediately discarded with a citation to Rusu, et al. Please consider including a baseline (at least in simulation) comparing the performance of PAD to \"naive\" fine-tuning or NSL-style fine-tuning (oversampling).\n\n#### 4. Experiments\n* \" as we find that learning a model of the motors works well for motor control.\" I didn't understand this comment. Please make this more clear -- are you introducing a modeling prior which attempts to explicitly model the actuators and joints of these models?\n* Table 2, 3, 4: I find it misleading to highlight elements of the table which don't exceed the performance of the other elements *including their confidence intervals*. Statistically, these entries might be equivalent, with some probability. However, the simple standard deviation from the mean is a poor confidence interval for comparisons-- please instead include a CI calculated with a bootstrap method and a known p-value (e.g. 95% interval) to enable better comparison. seaborn can do this for you. See [2] for a guide.\n* Table 5: Please include a confidence interval\n* Would prefer to see an ablation of the test-time performance which includes only the RL loss and RL loss+IDM(PAD), in addition to the existing \"no RL loss + IDM(PAD)\". Additionally, it would be helpful to show the reader the performance of a scratch-trained policy on the test environments, to establish an \"oracle\" or maximum expected performance for each environment given your base algorithms (A2C and SAC).\n\n#### Appendix\n* Figure 5: Please calculate a confidence interval for these curves with a known p-value (e.g. 95%) rather than simple standard deviation. This makes it much easier for readers to assess the statistical significance of your results.\n\n\n[1] https://arxiv.org/abs/2004.10190\n[2] https://arxiv.org/abs/1904.06979", "extra_scores": {}, "extra_text": {}}, {"review_id": "L0oxYhcBI9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an important problem in vision-based RL: how to adapt a pre-trained policy to an unseen environment in a self-supervised manner. To do this, the authors introduce an auxiliary task branch that can be used to tune the intermediate representation of the policy network on the fly in a self-supervised manner(e.g. inverse dynamic prediction). The experiments in the DeepMind Control suite, CRLMaze, and robot manipulation tasks show the generalization and effectiveness of the proposed method in various vision-based RL problems.\n\nStrength:\n- The paper is well-written and easy-to-follow. The demo video is impressive.\n- The problem they studied is important to the RL community, especially for virtual2real adaption. \n- The proposed method is simple yet effective. The author conducts experiments in various environments and provides a comprehensive analysis of the results. The implementation details are also detailed for reproduction.\n\nWeakness:\n- Lack of novelty. The proposed method can be viewed as a simple application of the self-supervised auxiliary task in the domain adaptation.  Every component used in the proposed method is already introduced in previous work. It is trivial to select the auxiliary task for different environments.\n-  The setting. It seems that the training environment and the test environment only differ in appearance. However, in real-world robot settings, the test environment would be of unseen transition dynamic, e.g. additional noise in action, delay in the control loop. How does this method perform in this situation?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a method for adapting an RL policy in a deployment environment that does not provide a reward signal.  This concern arises commonly when a task reward is available in a robot simulator but not on the physical robot where the policy is eventually deployed.  The proposed solution is to learn an inverse dynamics model as an auxiliary prediction task on an internal state embedding that is shared with the policy.  The policy is adapted during deployment by modifying the state embedding using this auxiliary task (with the assumption that the main objective remains unchanged).  The proposed method is tested with transfer between simulated domains and also on transfer from a simulator to a physical robot.  The experiments showed the method had consistently higher performance than alternatives.

The reviewers found many positive contributions in the presented paper. These include the problem's importance (R1, R2,R4), extensive experiments (R1, R2, R3), clear writing (R1,R4), simplicity and effectiveness in comparison to ablations (R3, R4).  The reviewers saw a weakness in the method's limitation to perceptual adaptation instead of dynamics adaptation (R1-4) and the lack of novelty (R4).  The author response addressed both concerns.  They stated that the method is novel for adapting to continuously changing environments in a self-supervised fashion without rewards.  The authors modified the paper to clarify how the method demonstrates robustness to changes in the system dynamics.  The reviewers found the author response addressed their major concerns.

Four reviewers indicate accept for the contributions stated above and expressed no remaining concerns.  The paper is therefore accepted. 
Decision: Accept (Spotlight)