Offline Meta-Reinforcement Learning with Online Self-Supervision
Authors: Vitchyr H. Pong
Ashvin Nair
Laura Smith
Catherine Huang
Sergey Levine
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
meta learning
meta reinforcement learning
offline reinforcement learning

Abstract: Meta-reinforcement learning (RL) methods can meta-train policies that adapt to new tasks with orders of magnitude less data than standard RL, but meta-training itself is costly and time-consuming. If we can meta-train on offline data, then we can reuse the same static dataset, labeled once with rewards for different tasks, to meta-train policies that adapt to a variety of new tasks at meta-test time. Although this capability would make meta-RL a practical tool for real-world use, offline meta-RL presents additional challenges beyond online meta-RL or standard offline RL settings. Meta-RL learns an exploration strategy that collects data for adapting, and also meta-trains a policy that quickly adapts to data from a new task. Since this policy was meta-trained on a fixed, offline dataset, it might behave unpredictably when adapting to data collected by the learned exploration strategy, which differs systematically from the offline data and thus induces distributional shift. We propose a hybrid offline meta-RL algorithm, which uses offline data with rewards to meta-train an adaptive policy, and then collects additional unsupervised online data, without any reward labels to bridge this distribution shift. By not requiring reward labels for online collection, this data can be much cheaper to collect. We compare our method to prior work on offline meta-RL on simulated robot locomotion and manipulation tasks and find that using additional unsupervised online data collection leads to a dramatic improvement in the adaptive capabilities of the meta-trained policies, matching the performance of fully online meta-RL on a range of challenging domains that require generalization to new tasks.

Under review as a conference paper at ICLR 2022
OFFLINE META-REINFORCEMENT LEARNING WITH ONLINE SELF-SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-reinforcement learning (RL) methods can meta-train policies that adapt new tasks with orders of magnitude less data than standard RL, but meta-training itself is costly and time-consuming. we can meta-trail on offline data, then we can reuse the same static dataset, labeled once with rewards for different tasks, to meta-train policies that adapt to a variety of new tasks at meta-test time. Although this capability would make meta-RI. practical tool for real-world use, offline Re presents additional challenges beyond online meta-RL or standard offline RL settings. Meta-RL learns exploration strategy that collects data for adapting, and also meta meta-trains a policy that quickly adapts to data from a new task. Since this meta-trained
meta-RL
policy
fixed, offline dataset, it might behave unpredictably when adapting to data collected by the learned exploration strategy, which differs systematically from the offline data and thus induces distributional shift. We propose a hybrid offline meta-RL algorithm, which uses offline data with rewards to meta-train an adaptive policy, and then collects additional unsupervised online data, without any reward labels to bridge this distribution shift. By not requiring reward labels for online collection, this data can be much cheaper to collect. We compare our method to prior work on offline meta-RL. on simulated robot locomotion and manipulation tasks and find that using additional unsupervised online data collection leads to dramatic improvement in the adaptive capabilities the meta-trained policies, matching the performance of fully online meta-RL on range of challenging domains that require generalization to new tasks.
INTRODUCTION
Reinforcement learning (RL) agents are often described as learning from reward and punishment analogously animals: the same way that person might train a dog by providing treats, we might train RL agents by providing rewards. However, in reality, modern deep RL agents require so many trials to learn a task that providing rewards by hand is often impractical. Meta-reinforcement learning in principle then acquiring new behaviors in just a few trials at meta-test time. Current meta-RL RL methods are efficient that meta-trained policies require only a handful of trajectories (Rakelly al., (209)), which reasonable for a human to provide by hand. However, the meta-training phase in these algorithms still requires large number of online samples, often even more than standard RL, due to the multi-task nature of the meta-learning problem.
mitigate this, by learning to learn using set of meta-trasn meta-training tasks, and
Offline reinforcement learning methods, which use only prior experience without active data col- lection, provide a potential solution to this issue, because a user must only annotate multi-task data with rewards once the offline dataset, rather than doing so in the inner loop of RL training, and the same offline multi-task data can be reused repeatedly for many training runs. While few recent works have proposed offline meta-RL. RI. algorithms Dorfman & Tamar (2020); Mitchell et al. (2021), we identify a specific problem when an agent trained with offline meta-RL. RI. is tested on a new task: the distributional shift between the behavior policy from the offline data and the meta-test time exploration policy means that adaptation procedures leamndd from offline data might not perform well on the (differently distributed) data collected by the exploration policy at meta-test time. This mismatch in training distribution occurs because offline meta-R RL never trains data generated by the meta-learned exploration policy. In practice, we find that this mismatch leads to a large degradation in performance when adapting new tasks. Moreover, we do want remove this distributional

---

Under review as a conference paper at ICLR 2022
Offine Meta-RL
Online Mota-RL melts-tak ((x (.,,x,r))
Offiiee Meta-RL with Self- Superreuee Online Training
anlti tosk ((..0.st.)))
mussa
sacta
mntto tramn
tratts
Figure 1: (left) In offline meta-RL, an agent uses offline data from multiple tasks 71, T2.................. ........ each with reward labels that must only be provided once. (middlc) In online meta-RI., new reward supervision must be provided with every environment interaction. (right) In semi-supervised peta-RLL, an agent uses an offline dataset collected once to learn to generate its own reward labels for new, online interactions. Similar to offline meta-RI,, reward labels must only be provided once for the offline training, and unlike online meta-RL, the additional environment interactions require neither external reward supervision nor additional task sampling.
shift by simply adopting a conservative exploration strategy, because learning an exploration strategy enables an agent to collect better data for faster adaptation.
We propose address this challenge by collecting additional online data without any reward super- vision, leading to a semi-supervee supervised offline meta-RL. algorithm, as illustrated in Figure [T, Online data can be relatively cheap to collect when it does not require reward labels, but it can still make possible to bridge the distributional shift issue. To make it feasible to use this data for meta-training, we can generate synthetic reward labels for it based on the labeled offline data.
Based on this principle, we propose semi- supisiueer meta actor-critic (SMAC), which uses reward- labeled offline data to bootstrap a semi-supervised metu-reinffrremment learning procedure, in which an offline meta-RL agent collects additional online experience without any reward labels. SMAC uses the reward supervision from the offline dataset to learn to generate new reward functions, which it uses to autonomously annotate rewards in these rewardless interactions and meta-train on this new data. Our method contains two novel contributions: first, it is the first method to combine the efficient PEARL (Rakelly et al. 2019) amortized inference method for meta-RL with AWAC (Nair et al. 2020), an effective approach for offline RL with online finetuning. This already yields an effective new offline meta-RL procedure. Second, it is the first approach to perform offline meta-R RL with self-supervised online finetuning, without ground truth rewards. We evaluate our method and prior offline meta-RI. methods on a number of benchmarks (Dorfman & Tamar 2020; Mitchell et al., 2021), as well as challenging robot manipulation domain that requires generalization to new tasks, with just a few reward-labeles trials at meta-test time. We find that, while standard Me methods perform well at adapting to training tasks, they suffer from data-distribution shifts when adapting to new tasks. In contrast, our method attains significantly better performance, on par with an online meta-RL method that receives fully labeled online interaction data.
2 RELATED WORKS
Many prior meta- RL algorithms assume that reward labels are provided with each episode of online
interaction (Duan et al. 2016; Finn et al.] [017]] Guppt et al.; (2018) Xu et al. [018 Hausman et al.
2018
Rakelly al. 2019 Humplik et al. 2019 Kirsch al. 2019 Zintgraf et al. 2020 Xu et al.
2020 Zhao et al. 2020] Kamienny et al. 2020). In contrast to these prior methods, our method
only requires offline prior data with rewards, and additional online interaction does not require any ground truth reward signal. Prior works have also studied other formulations that combine unlabeled and labeled trials. For example, imitation and inverse reinforcement learning methods use offline
demonstrations to either learn a reward function (Abbeel & Ng 2004 Finn et al., 2016 Ho &
Ermon 2016 Fu et al. 2017 7) or to directly learn a policy (Schaal) 1999] Ross & Bagnell 2010 Ho
Ermon 2016; Reddy et al. 2019 Peng et al. 2020). Semi-supervised and positive-unlans unlebeled reward
learning (Xu Denil 2019, Zoina al. 2020 Konyushkova et al. 2020) methods use reward labels provided for some interactions train reward function for RL. However, all of these methods have been studied in the context of a single task. In contrast, we focus on meta- veta-lcarning an RL procedure that can adapt to reward functions. In other words, do not focus on recovering a single reward function, because there is no single test time reward or task.
SMAC uses context- context-based adaptation procedure similar to that proposed by Rakelly al. (2019). which is related to contextual policies, such as goal-conditions reinforcement learning (Kaelbling 1993. Schaul et al. 2015 Andrychowicz et al. 2017 Pong al. 2018 Colas et al. 2018 Warde-
2

---

Under review as a conference paper at ICLR 2022
Farley et al. 2018: Péré et al. 2018 Nair al. 2018) or successor features (Kulkarni al. 2016 Barreto et al. 2017: 2019 Grimm et al. 2019). In contrast, our meta-learning procedure applies to any RL problem, does not assume that the reward is defined by single goal state fixed basis function, and only requires reward labels for static offline data.
Our method addresses similar problem to prior offline meta-RL. methods (Mitchell et al. 2021 I
Dorfman & Tamar 2020). In our comparisons, we find that after offline meta-training, our method is competitive these prior approaches, but that with additional self-supervised online fine-tuning, our method significantly outperforms these methods by mitigating the aforementioned distributional shift issue. Our method addresses the distribution shift problem by using online interactions without reward supervision. In our experiments. we found that SMAC greatly improves performance on both training and held-out tasks. Lastly, SMAC is also related to unsupervised meta-learning methods (Gupta et al.. 2018a Jabri et al.] 2019), which annotate data with their own rewards. In contrast to these methods, we assume that there exists an offline dataset with reward labels that we can use to learn to generate similar rewards.
PRELIMINARIES
Meta-reinforcemenn learning. In meta-RL., we assume there is a distribution of tasks pT()). A task T is a Markov decision process (MDP), defined by tuple T = (S,A, Po. Pa), where S the state space, is the action space, r a reward function, 7 is discount factor, (o the initial state distribution, and Pa(S s,, ax) is the environment dynamics distribution. A replay buffer DD is set of state, action, reward, next-states tuples, D {st, T { j==1 rewards come from the same task. We will use the letter to denote a the notation h D to denote that mini-batch h is sampled from replay buffer D. We will use the letter T to represent a trajectory T = (51. a1, .2....) without reward labels.
Nus, where all the or "history" and ri,
A meta- meta-episode consists of sampling a task T ~ pT()), collecting T trajectories with policy va with parameters 0, adapting the policy to the task between trajectories, and measuring the performance on the last trajectory. Between trajectories, the adaptation procedure transforms the history of states and actions h from the current meta-episode into a context z = A& (h))), which is then given to the policy To (a s, z) for adaptation. The exact representation of TT. Ag. and z depends meta-RL method used. For example, the context z can be weights of a neural network (Finn et al., 2017) outputted by gradient update, hidden activations ouputted by a recurrent neural network (Duan et al. 2016), or latent variables outputted by stochastic encoder (Rakelly et al., 2019). Using this notation, the objective in meta-RL. is to learn the adaptation parameters Φ and policy parameters to maximize performance on a meta-episode given a new task T sampled from p(T).
the specific
PEARL. Since we require an off-policy meta-RL procedure for offline meta-training we build on probabilistic embeddings for actor-critic RL (PEARL) (Rakelly et al. 2019), an online off-policy meta-RL algorithm. In PEARL, z is a vector and the adaptation procedure A, consists of sampling from distribution z 4o. (z h). The distribution qs. is generated by encoder with parameters
de. This encoder is a set-based neural network that processes the tuples in h = {s;, a, Tú, s in a permutation-invarian manner to produce the mean and variance of a diagonal multivariate Gaussian. The policy is a contextual policy Ty (a s, z) conditioned on z by concatenating z to the state s.
The policy parameter 0 is trained using soft-actor critic (Harmoja et al. 2018) which involves learning a Q-function, (s, a, z). with parameter w that estimates the sum of future discounted rewards conditioned on the current state, action, and context. The encoder parameters are trained by mack-propagating the critic loss into the encoder. The actor, critic, and encoder losses are minimized via gradient descent with mini-batches sampled from separate replay buffers for each task.
Offline reinforcement learning. In offline reinforcement learning. we assume that we have access to a dataset D collected by some behavior policy Ty. An RL agent must train on this fixed dataset and cannot interact with the environment. One challenge that offline RL poses is that the distribution of states and actions that an agent will see when deployed will likely be different from those seen in the offline dataset they are generated by the agent, and a number of recent methods have tackled this distribution shift issue (Fujimoto et al., 2019b a Kumar et al. 2019 Wu et al.) 2019 Nair et al.) 2020 [evine et al. 2020). Moreover, one can combine offline RI. with meta-RL RL by training meta-RI.
3

---

Under review as a conference paper at ICLR 2022
Ant Direction: Online Offline z-Posteriors
Ant Direction: Adapting from Online Offline Data
Ill-Adaptaaiion Returns
Dxa (notz ()) ll p(2))
250
data source offline buffer
200
online policy
2
150
offline buffer
online policy
10000
20000 30000 40000 50000
data source
Number of gradient updates
Figure Left: The distribution of the -i-deveree between the posterior qo. (z h) and the prior p(z) over the course of meta-training, when conditioned on data from the offline datasct (blue) or on online data from the learned policy (orange). Data from the learned policy results in posteriors that substantially farther from the prior, suggesting significant difference in distribution over Note that online data from the learned policy not available meta-training. but only used for measurement. Right: The performance of the policy after adaptation when adapted using data from the offline dataset (blue) or the learned policy (orange). Although the meta-RL RL policy adapts well when conditioned on sampled from the offline dataset, the performance does not increase when is sampled from the learned policy. Since the same policy evaluated, the change in z- 2-distribution is likely the cause for the drop in performance.
on multiple datasets DI,..., N DDorfman & Tamar 2020 Mitchell et section we describe some limitations of this combination.
2021). but in the next
THE PROBLEM WITH NAÎVE OFFLINE META- REINFORCEMENT LEARNING
Offline meta-RL is the composition of meta-RL and offline RL: the objective is to maximize the standard meta-RL objective using only a fixed set of replay buffers, D {De Ji=1. buffer corresponds to data for one task. Offline meta-RL methods can in principle utilize the same constraint-basee approaches that standard offline RL algorithms have used to mitigate distributional shift. However, they must also contend with an additional distribution shift challenge that is specific to the meta-RL scenario: distribution shift in z-space.
Must where each
Distribution shift in z-space occurs because meta-learning requires learning an exploration policy that generates data for adaptation. However, offline meta-RL. only trains the adaptation procedure (h) using offline data generated by previous behavior policy, which we denote as T3. After offline training, there will be a mismatch between this learned exploration policy πa and the behavior policy ng. leading to a difference in the history h and in turn, in the context variables example, in a robot manipulation setting, the offline dataset may contain smooth trajectories that were collected by a human teleoperator (Kofman al. 2005). In contrast, the learned exploration may contain jittering due learning artifacts or the use of stochastic policy. This jittering may not impede the robot from exploring the environment, but may result in trajectory distribution shift that degrades the adaptation process, which only learned to adapt to smooth, human- genneratd trajectories in the offline dataset. More formally, if p(z hottins) and p(z honline) denote the marginal distribution of given histories h sampled using offline and online data, respectively, the differences between To and Ty will lead to differences between p(z hommin during offline training and p(z h at meta-test time.
Ah)). For
To illustrate this difference, we compare p(z hommi) and p(z honline) on the Ant Direction task (see Section 6). We approximate these distributions by using the PEARL encoder discussed in Section 3 with p(z h) ~ qo. (z h), where sampled either from the offline data set or using the learned policy. We measure the KL-divergence observed at the end of offline training between the posterior p(z h) and a fixed prior Pz(z), for different samples of h. If the two distributions were the same, then would expect the distribution of KL divergences to also be similar. However, Figure shows that the two distributions are markedly different after the offline training phase of SMAC.
We also observe that this distribution shift negatively impacts the resulting policy. In Figure 2 we plot the performance of the learned policy when conditioned on z sampled from qo. (z homine compared go. (z) ) We see that the policy that uses offline data, hottine leads improvement, while
4

---

Under review as a conference paper at ICLR 2022
Offline Phase
Self-supervised Phase Reward labelling
Explore
Train
Q,la,, 4) Canti
Pala, 0,2)
sample (o.(a/h))
T)(a|s,*) - Lytarr
5,0,6
s.
Qu
up-ate
Ad. ra,(t,n,)) Cowoard
Adi
pia)
qs(2|RT)
Figure 3: (Left) In the offline phase, we sample history h' compute the posterior qo. (z h'). We then use a sample from this encoder and another history batch to train networks. In red, we then update the networks with h and the a sample. (Right) During the self-supervised phase, we explore by sampling z ~ p(z) and conditioning our policy on these observations. We label rewards using our Icarned reward decoder, and append the resulting data to the training data. The training procedure is equivalent to the offiine phase, except that we do not train the reward decoder or encoder since no additional ground truth rewards are observed.
the same policy that uses data from the learned policy, oonlne drops in performance. Since we evaluate the same policy тe and only change how is sampled, this degradation in performance suggests that the policy suffers from distributional shift between p(z hottine) and p(z hoaline.. In other words, the encoder produces z vectors that are too unfamiliar the policy when conditioned these exploration trajectories.
We note that this issue arises in any method that trains non- Markovian policies with offline data. For example, recurrent policies for partially observed MDPs (Jaakkola et al., 1995) depend both on the current observation and a history h. When deployed, these policies must also contend with potential distributional shifts between the training and test- cest-time history distributions, in addition to the change in observation distribution o. This additional distribution shift may explain why many memory- based recurrent policies are often trained online (Duan et al. 20166) Heess et al. 22017, Espeholl et al. (018) or have benefited from refreshing the memory states (Kapturowski et al. 2018). In this paper, we focus on addressing this issue specifically in the offline meta-RL RL setting.
Offline meta-RL with self-supervised online training. In complex environments where many behaviors are possible, the distribution shift in 2-space will likely be inevitable, since the learned policy is likely deviate from the behavior policy. To address this issue, introduce additional assumption: in addition to the offline dataset, we assume that the agent can autonomously interact with the environment without observing additional reward supervision. This problem statement useful for scenarios where autonomously interacting with the world is relatively easy, but online reward supervision is more expensive to obtain.
we assume that the agent can generate additional rollouts in an MDP without reward function, T \ r (S,A, Y, Pu. Pa). These additional interactions enable the agent to explore using the learned exploration policy. Since the resulting states and actions are from the learned exploration policy, we can use them to construct honline and enable an agent to train on the online context distribution, p(z honline), thus mitigating the distribution shift issue described above. However, meta-training requires that the history hoaline contain not just states and actions, but also rewards. In the next section, we describe a method for autonomously labeling these rollouts with synthetic reward labels to enable an agent to meta-train on this additional data.
Formally,
SEMI-SUPERVISED META ACTOR-CRITIC
In this section, we present our method, semi-supervised meta actor-critic (SMAC). SMAC consists of offline meta-training followed by self- supervie online meta-training to mitigate the distribution shift in z-space. The SMAC adaptation procedure consists of passing history through the encoder described in Section 3 resulting in posterior do, (z h). Below, we describe both phases.
5.1 OFFLINE META-TRAINTE
To learn from the user-provided offline data, we adapt the PEARL (Rakelly et al. 2019 to the offline setting. Similar to PEARL, we update the critic by minimizing the Bellman error:
(uiii((w) (w) E(s....s"") (x/h).a' (a
[( a, 2) (r 7Q a (s' a' 2] (1)
where T are target network weights updated using soft target updates (Lillicrap al. 2016)
PEARL has a policy updated based on soft actor critic (SAC) (Haarnoja et al. 2018). but when naively applied to the offline setting, these policy updates suffer from off-policy bootstrapping error
5

---

Under review as a conference paper at ICLR 2022
accumulation (Kumar et al.) 2019), which occurs when the target Q-function for bootstrapping Q(s', a') is evaluated at actions a' outside of the training data.
To avoid eπror accumulation during offline training, we implicitly constrain the policy to stay close to the actions observed in the replay buffer, following the approach in a single-task offline RL algorithm called AWAC (Nair et al.) (020). AWAC uses the following loss to approximate constrained optimization problem, where the policy is constrained to stay close to the data observed in D:
Q( a, V(s', 2)
}
Lactor (0) = -BBs,a.s' Diaane (z/h) log No exp
(2)
A
We estimate the value function (is, z) = Ba (ais..)) Q(s, a, z) with a single sample, and À is the resulting Lagrange multiplier for the optimization problem. See Nair et al. (2020) for a full derivation.
With this modified actor update, we can train the encoder, actor, and critic on the offline data without the overestimation issues that afflict conventional actor-critic algorithms (Kumar et al.) 2019). However, it does not address the z-space distributional shift issue discussed in Section 4 because the exploration policy learned via this offline procedure will still deviate significantly from the behavior policy тe. As discussed previously, we will aim to address this issue by collecting additional online data without reward labels and learning to generate reward labels if self-supervised meta- tettatraining..
Learning to generate rewards. To continue meta-training online without ground truth reward labels, we propose to use the offline dataset to learn a generative model over meta-training task reward functions that we can use to label the transitions collected online. Recall that during offline a latent context z that encodes the task. In the same way that train our policy iT (a 2) that conditionally decodes z into actions, additionally train a reward decoder Toz (s, a, z) with parameter dd that conditionally decodes z into rewards. We train the reward decoder ro. to reconstruct the observed reward in the offline dataset through a mean squared error loss.
learning, we learn an encoder Qs. that maps experience
Because we the latent space for reward-decoding, we pack-pro the reward decoder loss into qo.. As visualized in Figure 3 we also regularize the posteriors qo. (z h) against a prior (z( provide an information bottleneck in that latent space and ensure that samples from pa (z) represent meaningful latent variables. We found it beneficial to not back-propagate the critic loss into the encoder, in contrast to prior work such as PEARL. To summarize, we train the encoder and reward decoder by minimizing the following loss, in which we assume that z ~ qo. (h):
Lrewand De: h,
Σ
||- 1 (s, DKL (go. (. h) pa(())-
(3)
(s,a,))hh
5.2 SELF-SUPERVISED ONLINE META-TRAN TRAINING
We now describe the self-supervised online training procedure, during which we use the reward decoder to provide supervision. First, we collect trajectory 7 by rolling out our exploration policy To conditioned on context sampled from the prior p(z). To emulate the offline meta-training supervision, we would like to label 7 with rewards that are in the distribution of meta-training tasks. As such, we sample replay buffer D; uniformly from D to get history hottine D; from the offline data. We then sample from the posterior z ~ qo. (z homline) and label the reward 'ggeeerteed of a new state and action, (s, a), using the reward decoder
Tgenerated
Toh (s. a, z).
where z ~ To. (z h).
(4)
We then add the labeled trajectory to the buffer and perform actor and critic updates as in offline meta- training. Lastly, since we do not observe additional ground-truth rewards, during the self- supervisdd phase we do not update the reward decoder You or encoder Qo. and instead only train the policy and Q-function. We visualize this procedure in Figure
We note that the distribution shift discussed in Section is not an issue for the reward decoder. The distribution shift only occurs when we sample z from the encoder using online data, i.c. z qo. (z honline), but, we only sample from this reward decoder using z sampled from the encoder using offline
With this notation, pera-parameters the encoder and decoder parameters (i.c., (ge, (1)).

---

Under review as a conference paper at ICLR 2022
mota-training
busten
HATDOME
drawers
abject
Figure 4: We propose a new meta-learning evaluation domain based on the environment from Khazatsky et al. [202T]. in which a simulated Sawyer gripper can perform various manipulation tasks such as pushing button, opening drawers, and picking and placing objects. We show subset of meta-trate veta-training (blue) and meta-test (orange) tasks. Each task contains unique object configuration, and we test agent on held-out tasks.
data, i.c. z ~ qo. (z homine). The reward decoder does need to generalize to new states and actions, and we hypothesize that reward prediction generalization is easier than policy state, action, and generalization. If true, then we would expect that using the reward decoder to label rewards and train on those labels, as in SMAC, will outperform directly using the policy on new tasks (as in Figure 2
5.3 ALGORITIIM SUMMARY AND DETAILS
We visualize SMAC in Figure For offline training, we assume access to offline datasets D (D,),mnn Miva, where each buffer corresponds to data generated for one task. Each iteration, we sample a buffer D, ~ DD and history from this buffer h ~ DD We condition the stochastic encoder do. this history to obtain a sample z ~ ls. (z h). We then use this sample and a second history sample h' D; to update the Q-function, the policy, encoder, and decoder by minimizing Equation (I). Equation 2 and Equation (3) respectively. During the self-supervised phase, we found it beneficial to train the actor with a combination of the loss in Equation (2) and the original PEARL actor loss, weighted by hyperparameter Xpeat.. We provide pseudo-code for SMAC in Appendix
EXPERIMENTS
We presented method that uses self-supervised data to mitigate the z-space distribution shift that occurs in offline meta-RL. In this section, evaluate how well the self-supervised phase of SMAC mitigates the resulting drop in performance, and we compare SMAC to prior offline meta-RL Meta-RRLoe methods on range of meta-RL. benchmark tasks that require generalization to unseen tasks at meta-test time.
RI tasks. We first evaluate our method on multiple simulated MuloCo Todorov et al. 2012
meta-learning tasks that have been used in past online and offline meta-RL papers (Finn et al. 2017
Rakelly al. 2019 Dorlman & Tamar 2020 Mitchell et al. 202T) (see Figure 9). Although there are standardized benchmarks for offline RL (Fu et al. 2020) and meta-RL (Yu et al. 2020), there no standard benchmarks for offline meta-RL particular,
combination of tasks in prior work (Dorfman & Tamar 2020) Mitchell et al., 2021), and introduce a more complex robotic manipulation domain meant push task generalization to the limit.
We first evaluate five different domains, Cheetah Velocity, Ant Direction, Humanoid, Walker Param, and Hopper Param, which have been used in prior work (Rothfuss al., 2018 Rakelly al.) 2019 Dorfman & Tamar 2020). In the first three domains, the agent is rewarded for moving in an unobserved target velocity and meta-episode consists of sampling desired velocity. The last task, Humanoid, is particularly challenging due to the high space dimension of 376. The last two domains, Walker Param and Hopper Param, require adapting different dynamics parameters, such friction, joint mass, and inertia. The agent must adapt to each task within 7 = 3 episodes, each of length 200.
We also evaluated SMAC on significantly more diverse robot manipulation meta-learning task called Sawyer Manipulation, based on the goal-conditioned environment introduced by Khazatsky ctt al. (2021). This a simulated PyBullet environment Coumans & Bai 2016-2021) in which Sawyer robot arm can manipulate various objects. Sampling task 7 p(T involves sampling both a new configuration of the environment and the desired behavior to achieve, such as pushing button, opening drawer, or lifting an object (see Figure The sparse reward is 0 when the desired behavior achieved and -1 otherwise. The action space consists of 3 dimensions to control the end-effector in Euclidean space and one dimension control the gripper. The state is 13-dimensional vector,
7

---

Under review as a conference paper at ICLR 2022
which we detail in Appendix C of object configuration between each task sample make this environment difficult.
The sparse reward, precise manipulation requirement, and diversity
In all of the environments, test the meta-RL procedure's ' ability to generalize to new tasks by evaluating the policies on held-out tasks sampled from the same distribution as in the offline datasets. We give complete description of environments and task distribution in Appendix
Offline data collection. For the MuJoCo tasks, we use the replay buffer from single PEARL run that uses the ground-truth reward. We limit the data collection to 1200 transitions (i.e., trajectories) per task and terminate the PEARL run early, forcing the meta-RL agent to learn from highly suboptimal data. For Sawyer Manipulation, we collect data using a scripted policy that randomly performs one of many possible tasks in the environment. We used 50 training tasks and 50 trajectories of length 75 each. Note that the performed task (e.g. open drawer) may differ from the task that rewarded (e-g. push a button). As a result, in the offline dataset, the robot succeeds on the task in 46% of the transitions, indicating that this data is highly suboptimal. In contrast to prior work (Dorfman & Tamar 2020 Mitchell et al.] 202T), , we use this limited data because enables us to test how well the different methods can improve over suboptimal, offline data. See Appendix foo further discussion and details.
Comparisons and ablations. As an upper bound, we include the performance of PEARL with online training using oracle ground-truth rewards rather than self-generated rewards, which we label Online Oracle. To understand the importance of using the actor loss in Equation ( we include an ablation that replaces Equation (2) with the actor loss from PEARL, which we label SMAC (actor ablation). We also include a meta-imitation baseline, labeled meta behavior cloning, which replaces the actor update in Equation (2) with simply maximizing log xo (a 8s, 8,2). A gap between SMAC and this imitation method would help us understand whether or not our method improves over the (possibly sub-optimal) behavior policy. For comparisons to prior work, we include the two previously proposed offline meta-RL methods: meta-actor critic with advantage weighting (labelled MACAW) (Mitchell et al., 2021) and Bayesian offline RL (labelled BOReL) (Dorfman & Tamar 2020). MACAW and BOReL assume that rewards are provided during training and cannot make use of the supe-vupee online interaction, and so we cannot compare directly to these past works. Therefore, we report their performance only after offline training. For these prior work, we used the open-sourced code directly from each paper (PEARI., MACAW, and BOReL.), and and trained these methods using the same offline dataset. To ensure a fair comparison, we ran the original hyperparameters and matched SMAC hyperparameters (matching network size, learning rate, and batch size), taking the better of the two as the result for each prior method. We include a list of all hyperparameters and network architectures in Appendix
Comparison results. We plot the mean post-adaptation returns and standard deviation across 4 seeds in 5 We see that across all three environments, SMAC consistently improves during the self-supervised phase, and often achieves a similar performance to the oracle that uses ground-truth reward during the online phase of learning. SMAC also significantly improves over meta behavior cloning, which confirms that the data in the offline dataset is far from optimal.
We also observed even before the online phase, our method is competitive with BOReL and MACAW as a stand-alone offline meta-RL RL method. When sule--upe training included, we see that SMAC significantly improves over the performance of BOReL on all three tasks and significantly improves over MACAW on two of the three tasks, including the Sawyer Manipulation environment, which is by far the most challenging and exhibits the most variability between tasks. In this domain, we also see the largest gains from the AWAC actor update, in contrast to the actor ablation in blue, which corresponds to PEARL-style update, indicating that properly handling the offline phase is also important for good performance.
In conclusion, method is the only attains generalization performance close to the Online Oracle upper bound baseline on all three tasks, indicating that self-supervised online fine-tuning is highly effective for mitigating distributional shift in meta-RI, whereas without it offline meta-RI. generally does not exceed the performance of a meta-imitation learning baseline.
Visualizing the distribution shift. Lastly, we further investigate if the self-supervised training helps specifically because it mitigates distribution shift caused by the exploration policy. To investigate this, we visualize the trajectories of the learned policy both before and after the self- supervised phase for the Ant Direction task in Figure 6
For each plot, show trajectories from the
8

---

Under review as a conference paper at ICLR 2022
Cheetah Velocity
Ant Direction
Saweer Manipulation
Post Adiptctionn Retums
Port Adgptcttoon Roturss
5001
Peta-tactttatiion Heturns
-100
-00
-150
-30
-000
-250
-10
-500
0
25
50
75
1001
3001
2001
0
100
200
Number Environment Steps (x1000)
Number of Environment Steps (x1000)
Number Environment Steps (x1000)
Post-Adsplalion Retumss
500
Walker
Poot-Adapaation Returs
Happer
PostAAdatationn Returns
1000
Humanod
400
400
800
300
300
800
200
200
100
200
25
Number of Environment Steps (x1000)
Number of Emvrrnment Steps (x1000)
Numbar Enviranment Stops (x1000)
SMAC (Ours)
Online Oracle
SMAC (actor ablation)
MACAN
B0ReL
meta behavior cloning
Figure 5: We report the final return of meta-test adaptation on unseen test tasks versus the amount of self- supervised meta-training following offline meta-training. Our method SMAC, shown in red, consistently trains reasonable performance from offline meta-RL (shown step 0) and then steadily improves with online self-supervised experience. The offline meta-RL methods, MACAW Mitchell et al.] (202T) and BOReL are competitive with the offline performance of SMAC but have no mechanism to improve via self-suuervision. We also compare to SMAC (SAC ablation) which uses SAC instead of AWAC as the underlying RL algorithm. This ablation struggles to train value function offline, and struggles improve on more difficult tasks.
policy (a dataset (homine) or from the learned exploration policy (hanline). Since the same policy is evaluated, differences between the resulting trajectories represent the distribution shift caused by using history from the learned exploration policy rather than from the offline dataset.
z) when the encoder To. (z h) is conditioned on histories from either the offline
We see that before the self-supervised phase, there is large difference between the two modes that can only be attributed to the difference h. When using houline, the post-adaptation policy only explores difference explains the large performance gap observed in Figure ② and highlights that the adaptation procedure is sensitive to the history h used to adapt. In contrast, after the self-supervised phase, the policy moves in all directions regardless of where the history came from. In Appendix B we also visualize the exploration trajectories and found that the exploration trajectories are qualitatively similar both before and after the self-supervised phase. Together, these results illustrate the SMAC policy learns adapt the exploration trajectories by using the self-supervised phase mitigate the distribution shift that occurs with naïve offline meta RL.
mode, but when using bottine. the policy moves in all directions. This qualitative
7 CONCLUSION
In this paper, we studied problem specific to offline meta-RL: distribution shift in the context parameters z. This distribution shift occurs because the data collected by the meta-learned exploration policy differs from the offline dataset. To address this problem, we assumed that an agent can sample new trajectories without additional reward labels and presented SMAC, a method that uses these additional interactions together with state-of-the-as offline RL techniques to provide an effective meta-RL method. Experimentally, we found that SMAC significantly improves over the performance of prior offline me methods, after the supeviuee online phase.
limitation of SMAC that it assumes that the agent can gather additional unlabeled online samples. However, this assumption may not be satisfied due to safety concerns and engineering needed to ensure that automatic resets are feasible, and therefore enabling safe, autonomous environment interactions is important direction for future research.
Lastly, using self-super interaction to mitigate distribution shift in offline meta RL is orthogonal to the underlying meta RL algorithm. In this paper, built off of PEARL (Rakelly et al.) (2019) due to its simplicity, but this insight could easily be used to improve other existing (Mitchell et al. 202T Dortman & Tamar 2020 future offline meta RL. algorithms.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have included the source code for our method and environments in the supplementary file, which includes a README file that describes how install our code and run the experiments. We describe in detail the environments and hyperparameters used for the experiment in Appendix
C For clarity, we have also included pseudo-code of SMAC on the first page of supplementary material.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Megrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. In Advances in Neural Information Processing Systems (NIPS), 2017. URL hhttps://aara: Stg/pdf/1700 01495 pwffttt::/aaxii.......
hbsp:/bbs/10 abs 077117177979995
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning, In Advances in neural information processing systems, pp. 6055-4065 2017.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zídek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. arXiv preprini 19X1v 10964, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.00 01540, 2016.
Cédric Colas, Olivier Sigaud, and Pierre-Yve Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. International Conference on Machine Learning (ICML), 2018.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://bybullet. org 2016-2021.
Ron Dorfman and Aviv Tamar. Offline meta reinforcement learning. arXiv preprint arXiv:2008.02 025998. 2020.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 1rXiv:1611.022 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp.
1407-1416. PMLR, 2018
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pp. 49-58. PMLR, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. 1135. PMLR, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint 111.1:17,, 2017.
2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. 2020.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv: 1910.0/70000000 2019a.
10

---

Under review as a conference paper at ICLR 2022
Scott Fujimoto, David Meger, and Doina Precup. Off- Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019b.
Christopher Grimm, Irina Higgins, Andre Barreto, Denis Teplyashin, Markus Wulfmeier, Tim Her- tweck, Raia Hadsell, and Satinder Singh. Disentangled cumulants help successor representations transfer to new tasks. arXiv preprint arXi::1911.100S:: 2019.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv: 1806 04640, 2018a.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Proceedings of the 32nd Inter- national Conference on Neural Information Processing Systems, 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry 7hu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 18/2..58/ 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint 1rX7v:1707.000 707.02286, 2017.
Jonathan Ho and Stefano Ermon. arXiv:1606.03476 2016.
Generative adversarial imitation learning. arXiv preprint
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv: 100.. 905.06424, 2019.
Tommi Jaakkola, Satinder P Singh, and Michael I Jordan. Reinforcement learning algorithm for partially observable markov decision problems. Advances in neural information processing systems, pp. 345-352, 1995.
Allan Jabri, Kyle Hsu, Ben Eysenbach, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual reia-reeinoorementt learning. arXiv preprint 1911v:1912.0422. 2019.
P Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAI), volume vol.2, pp. 1094 8, 1993.
Pierre-Alexandre Kamienny, Matteo Pirotta, Alessandro Lazaric, Thibault Lavril, Nicolas Usunier, and Ludovic Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005. 02934, 2020.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay distributed reinforcement learning. In International conference on learning representations, 2018.
Alexander Khazatsky, Ashvin Nair, Daniel Jing, and Sergey Levine. What can do here? learning new skills by imagining visual affordances. In International Conference on Robotics and Automation. IEEE, 2021.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint arXiv:1910.0409: 2019.
Jonathan Kofman, Xianghai Wu, Timothy Luu, and Siddharth Verma. Teleoperation of robot ma- nipulator using a vision- based numan-robor interface. IEEE transactions on industrial electronics, 12(5):1206-1211 2005.
11

---

Under review as a conference paper at ICLR 2022
Ksenia Konyushkova, Konrad ZoIna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. supervipee reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06889, 2020.
Tejas D Kulkarmi, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.0223: 2016.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint 1rXiv:1906.0000 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 2016. ISSN 10769757. doi: 10.1613/jair.301. URL https://
arx.v.org/pdf/1500 020091.0299111
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning pp. 7780-7791. PMLR, 2021.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual Reinforcement Learning with Imagined Goals. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint (xXi::2006009359, 2020.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Roboties: Science and Systems, 2020.
Pierre-Yves Alexandre Péré, Sebastien Forestier, Olivier Sigaud, and Pierre Yves Oudeyer. Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration. In International Conference on Learning Representations (ICLR), 2018. URL https: http:sss//aarii.orrr/////////// pdf
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal Difference Models: Model- Free Deep RI. For Model-Based Control. In International Conference Learning Representations (ICLR), 2018. URL
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.1 108, 2019.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings. 2010.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In International Conference on Learning Representations, 2018.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6): 233-242, 1999.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal Value Function Approxima- tors. In International Conference Machine Learning (ICML), 2015.
12

---

Under review as a conference paper at ICLR 2022
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-Time Training with Self-Suuprvision for Generalization under Distribution Shifts. In International Conference on Machine Learning (ICML), 2020. URL sstaa://reet-ttmerrraaaaa
github.ic. .
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Interaational Conference on Intelligent Robots and Systems, pp. 5026-500000000 -0033. IEEE, 2012.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. CoRR, 0bs//8111.11351 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
Danfei Xu and Misha Denil. Positive-umlabeled reward learning, arXiv preprint arXiv: 1XXIv:991.000499 2019.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta Meta-gradient reinforcement learning.
Advances in neural information processing systems, 31:2396-2400 2018.
Zhongwen Xu, Hado Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Sil- ver. Meta-gradient reinforcement learning with an objective discovered online. arXiv preprint arXiv:2007.08433, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100. PMIR, 2020.
Tony Z Zhao, Anusha Nagabandi, Kate Rakelly, Chelsea Finn, and Sergey Levine, Meld: Meta- reinforcement learning from images via latent state models. arXiv preprint a3X1v:2010.13957, 2020.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep rl via meta-learning. Proceedings of ICLR 2020, 2020.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011 138855, 2020.
13

---

Under review as a conference paper at ICLR 2022
Supplementary Material
A METHOD PSEUDO-CODE
We present the podudo-code for SMAC in Algorithm
Algorithm Semi-Supervised Meta Actor-Critic
1: Input: datasets D {D: j policy Te.. Q-function Qw encoder lo. and decoder To
2: for iteration n = 1, ........ Notttin do
> offline phase
3: Sample buffer D; D and two histories from buffer h ~ Dj-
4: Use the first history sample to h to infer z encode it 9s. (h).
5: Update no. Qw t0. re by minimizing Lacuer ficitics Preward with samples z, hr.
6:
for iteration n = 1,2, Nonline do
& self-supervised phase
7:
Collect trajectory with To (a 8, 2). with 2, p(z). Sample buffer Dj ~ 'D and offline history hofflne D
8:
9:
Use hoffline label the rewards in т, in Equation (4 and add the resulting data D..
10: Sample buffer D; 'D and two histories from buffer h, h' Dj
Encode first history qs. (h).
12: Update no. Q by minimizing Lucor Lenti with samples z, h'.
B ADDITIONAL EXPERIMENTAL RESULTS
Exploration and offfine dataset visualization Figure 7 we visualize the post-adaption trajec- tories generated when conditioning the encoder the online exploration trajectories honline and the offline trajectories holtine- Similar Figure and also visualize the online and offline trajectories themselves. We see that the exploration trajectories honline and the offline trajectories homlin are very different (green red, respectively). but the sulf-supe phase mitigates the negative impact that this distribution shift has offline meta RL. In particular, the post-adaptation trajectories conditioned on these two data sources (blue and orange) are similar after the self-supervised training, whereas before the self superviedd training, only the trajectories conditioned the offline data (blue) move in multiple directions.
Addressing state-space distribution shift by self-supervised meta-training test tasks. An- other source of distribution shift that can negatively impact meta-policy is distribution shift in state space. While this distribution shift occurs in standard offline RL, we expect this issue to be
After Offlne Training
After Self Supervised Training
20
Y-positio
COM Y pasttion
post-adaptation policy, where z zgz hammel
post-adaptation where (ravaire))
-10
-10
-20
-20
-20
-10
0
10
20
-20
-10
0
10
20
COM passitio
COM position
Figure 6: Example XY-coordinates visited by a learned policy on the Ant Direction task. Left: Immediately after offline training, the post adaptation policy moves in many different directions when conditioned on hom (blue). However, when conditioned benline (orange), the policy only moves up and the left, suggesting that the post-adaptation policy is sensitive to data distribution used to collect h. Right: After the self-supervised phase, the post-adaptation policy moves many directions regardless of the data source, suggesting that the self-sun supervised phase mitigates the distribution shift between conditioning on offline and online data.
14

---

Under review as a conference paper at ICLR 2022
After Offline Training
After Self-Supervised Training
20
post-adaptation policy, where 2 - q(z hottiee
COM Y-position
10
Y-position
10
post-adaptation policy, where z q(2 (exttror)
exploration policy, where 2 p(2)
-10
offline dataset
-20
-20
-20
-10
0
10
20
-20
-10
10
20
COM X-positlon
COM X-position
Figure 7: We duplicate Figure include the exploration trajectories (green) and example trajectories from the offline dataset (red). We see that the exploration policy hoth before and after self-supervised training primarily moves up and to the left, whereas the offline data moves in all direction. Before the self-supervised phase, that conditioning the encoder on online data (orange) rather than offline data (blue) results in very different policies, with the online data resulting the post-adaptation policy only moving up and to left. However, self-supervised phase of SMAC mitigates the impact of this distribution shift and results in qualitatively similar post-adaptation trajectories, despite the large difference between the exploration trajectories and offline dataset trajectories.
Sawyer Manipulation
-25
Post- Adta-tapttaionn Returns
-30
-35
Supe-Suuerniiion on Test Tasks
Oracle Self-Supervision on Train Tasks
-40
-45
0
25
50
75
100
Number of Environment Steps (x1000)
Figure 8: Learning curves when performing self-supervised training the test environments (red) the meta-training environments (blue). We also compare to an oracle that trains on test environments combination with ground-trute rewards (black). We see that interacting with the test environment without rewards allows for steady improvement in post-adaptation test performance and obtains similar performance to meta- meta-training on those environments with ground-truth rewards.
more prominent in meta RL, where there is a focus generalizing to completely novel tasks. In many real-world scenarios, experiencing the state distribution of a novel task is possible, but it the supervision (ic. reward signal) that is expensive to obtain. Can we mitigate state distribution shift by allow the agent to meta- trait in the task environments, but without rewards?
In this experiment, we evaluate our method, SMAC,, when training online on the test tasks instead of on the meta-train teetnnta tasks as in the experiments in Section 6 Prior work has explored this idea of self-supervision with test tasks in supervised learning [unn et al. (2020) and goal-conditioned RI. Khazatsky et al. (202T). We use the Sawyer Manipulation environment supervised training mitigate state distribution shifts, as these environments contain significant variation between tasks. To further increase the complexity of the environment, we use a version of the environment which samples from set of eight potential desired behaviors instead of three.
study how self-
We compare supervuper training on test tasks to self- superviedd training on the set of meta- training tasks, which are also the tasks contained in the offline dataset. A large gap in performance indicates that interacting with the test tasks can mitigate the resulting distribution shift even when no reward labels provided.
15

---

Under review as a conference paper at ICLR 2022
We show the results in Figure 8 and find that there is indeed large performance gap between the two training modes, with self-supervision on test tasks improving post-adaptation returns while supervupe on meta- veta-traiming tasks does not improve post- adsttadaptttion returns. We also compare to an oracle method that performs online training with the test tasks and the ground-truth reward signal. We see that SMAC is competitive with the oracle, demonstrating that we do not need access rewards in order to improve on test tasks. Instead, the entire performance gain comes from experiencing the new state distribution of test tasks. Overall, these results suggest that SMAC is effective for mitigate distribution shifts in both z-space and state space, even when an agent can interact in the environment without reward supervision.
Reward Accuracy Dependence Our method involves training reward decoder, and so a natural question is: How good must the reward decoder be for SMAC to work? We found that the reward loss does not need to be particularly low for our method to work. Specifically, in the Sawyer Manipulation environment, the reward scale is either -1 or O, meaning that the maximum reward scale is 1. We observed that the reward decoder loss is around 0.2 on the training task and around 0.25 on the test tasks, indicating that the method does not need a relatively low reward decoder loss to perform well.
C EXPERIMENTAL DETAILS
C.I DATA COLLECTION DIFFERENCE FROM PRIOR WORK
BOReL and MACAW were both developed assuming several orders of magnitude more data than the regime that we tested. For example, in the BORel. paper (Dorfman & Tamar (020), the Cheetah Velocity was trained with an offline dataset using 400 million transitions and performs additional reward relabeling using ground-trun -ruth information about the transitions. In contrast, our offline dataset contains only 240 thousand transitions, roughly three orders of magnitude fewer transitions. Similarly, MACAW uses 100M transitions for Cheetah Velocity, over 40 times more transitions than used in our experiments. These prior methods also collect offline datasets by training task dask-specific policies, which converge to near-optimal policies within the first million time step (Haarnoja et al.) 201 8), meaning that they utilize very high-quality data. contrast, our experiments focused on the scenario with many fewer and much lower quality trajectories, which is likely the cause of the relatively worse performance of BOReL and MACAW than in the original papers.
C.2 ENVIRONMENT DETAILS
In this section, describe the state and action space of each environment. We also describe how reward functions were generated and how the offline data was generated.
Ant Direction The Ant Direction task con- sists of controlling a quadruped "ant" robot that can move in a plane. Following prior work (Rakelly et al. 2019; Dorfman & Tamar 2020), the reward function is the dot product between the agent's 'ss velocity and direction uniformly sampled from the unit circle. The state space is R.20 comprising the orientation of the ant (in quaternion) as well as the angle and angu- lar velocity of all 8 joints. The action space is [],, with each dimension corresponding to the torque applied to a respective joint. The reward function is the negative absolute differ- ence between the agent's 's x- veloociyy and target velocity uniformly sampled from (0, 3).
Figure 9: Illustrations of two evaluation domains, each of which has of meta-train tasks (examples shown blue) and held out test tasks (orange). The domains include (left) a half cheetah tasked with running at dif- ferent speeds and (right) quadruped ant locomoting to different points a cirele.
The offline data is collected by running PEARL (Rakelly et al., 2019) on this meta RL task with 100 pre-sampled target velocities. We terminate PEARL after 100 iterations, with each iteration
2To mitigate variance coming from this sampling procedure, we use the same sampled target velocities across all experiments and comparisons. We similarly use pre-sampled set of tasks for the other environments.
16

---

Under review as a conference paper at ICLR 2022
consisting of collecting trajectories until at least 1000 new transitions have been observed. As discussed in Appendix [.T] this results in highly suboptimal data, enabling us to test how well the different methods for each task, one for sampling data for training the encoder and another for training the policy and Q-function. We will call the former replay buffer the encoder replay buffer and the latter the RI. replay buffer. The encoder replay buffer contains data generated by only the exploration policy, in which z ~ Pz(z). The RL replay buffer contains all data generated, including both exploration and post- past-adaptation, in which z ~ qa. (z h). To make the offiin dataset, we load the last 1200 samples of the RL replay buffer and the last 400 transitions from the encoder replay buffer into corresponding RL and encoder replay buffers for SMAC.
improve over the offline data. In PEARL, there are two replay buffers saved
Cheetah Velocity The Cheetah Velocity task consists of controlling a two-legged "half cheetah" that can move forwards or backwards along the x-axis. Following prior work (Rakelly et al.] 2019 Dorfman & Tamar 2020), the reward function is the absolute difference product between the agent's x-velocity and a velocity uniformly sampled from [0. The state space is R comprising the z-position; the cheetah's x- and z- velocity; the angle and angular velocity of each joint and the
half-cheetah's 's y-angle; and the XYZ position of the center of mass. The action space is [-1, 1]6, with
each dimension corresponding to the torque applied to a respective joint.
The offline data is collected in the same way as in the Ant Direction task, using run from PEARL with 100 pre-sampled target velocities. For the offline dataset, we use the first 1200 samples from the RL replay buffer and last 400 samples from the encoder replay buffer after 50 PEARL iterations, with each iteration containing at least 1000 new transitions. For only this environment, we found that it was beneficial to freeze the encoder buffer during the self-supervised phase.
Sawyer Manipulation The state space, action space, and reward is described in Section Tasks are generated by sampling the initial configuration, and then the desired behavior. There are five objects: drawer opened by handle, a drawer opened by button, a button, tray, and a graspable object. The state is a 13-dimensional vector comprising of the 3D position of the end-effector and positions of objects in the scene: 3 dimensions for the graspable object, and dimension for each articulated joint in the scene including the robot grippers. If an object is not present, it takes on position in the corresponding element of the state space. First, the presence or absence of each of the five is randomized. Next, the position of the drawers (from 2 sides). initial position of the tray (from 4 positions). and the object (from 4 positions) is randomized. Finally, the desired behavior is randomly chosen from the following list, but only including the ones that possible in the scene: "move hand", "open top drawer with handle", or "open bottom drawer with button". The offline data is collected using a scripted controller that does not know the desired behavior and randomly performs potential tasks in the scene, choosing another task if it finishes one task before the trajectory ends. This data is loaded into a single replay buffer used for both the encoder and RL.
Walker Param This environment involves controlling a bi-pedal robot that can move along the Z- and Y-axis. The source code is taken from the rand_paraaa farrmm_enve repository [ which has been used in prior work (Rakelly et al.) 2019). A task involves randomly sampling the body mass, the joint damping coefficients, the body inertia, and the friction parameters from log-uniform distribution on the range [1.5-3 1.53]. The reward is the velocity plus a bonus of 1 for staying alive and a control penalty of 10-1 x ||a||2, where ||a|] is the l2 norm of the action. The state space is 17 dimensions and the action space is 6 dimensions (one for each joint).
Hopper Param This environment involves controlling lne-legged robot that can move along the Z- and Y-axis. This environment is also taken from the rand__adap is the same as in Walker Param, and tasks are sampled the same as for the Walker Param. The state space is 11 dimensions and the action space is dimensions (one for each joint).
repository. The reward
Humanoid This environment is based on the Humanoid environment from OpenAl Gym (Brock man et al. 2016). We reuse the standard reward function but replace the forward- velociiy reward with a velocity reward based on target direction. Each task consists of sampling a target direction
17

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Value
RL batch size
256
encoder batch size
64
meta batch size
Q-network hidden sizes
[300, 300, 300
policy network hidden sizes
[300, 300, 300
decoder network hidden sizes
64, 64
encoder network hidden sizes
2000, 200, 200
z dimensionality (dg)
hidden activation (all networks)
ReLU identity
Q-network, encoder, and decoder output activation
policy output activation
tanh
discount factor ?
0.99
target network soft target 7
0.005
policy, Q-network, encoder, and decoder learning rate
3 x 10
policy, Q-network, encoder, and decoder optimizer # of gradient steps per environment transition
Adam
Table 1: SMAC Hyperparameters for Self-Supervised Phase
Hyperparameter
Cheetah
Ant
Sawyer
Walker
Hopper
Humanoid
horizon (max # of transitions per trajectory)
200
200
50
200
200
200
AWR
100
100
0.3
100
100
100
reward scale
5
1
5
5
of training tasks
100
100
50
50
# of test tasks
30
20
10
5
# of transitions per training task
1600
1600
3750
1200
1200
1200
in offline dataset
Apearl
0
1
Table Environment Specifie SMAC Hyperparameters
uniformly random. The state space is 376 dimensions and the action space is 17 dimensions (one for each joint).
C.3 HYPERPARAMETERS
We list the hyperparameters for training the policy, encoder, decoder, and Q-network in Table [] If hyperparameters different across environments, they listed in Table For pretraining. we use the same hyperparameters and train for 50000 gradient steps. Below, we give details on non-standard hyperparameters and architectures.
Batch sizes. The RL batch size is the batch size per task when sampling (s, a, r,s') tuples to update the policy and Q-network. The encoder batch size is the size of the history h task used to conditioned the encoder qo. (z h). The meta batch size is how many tasks batches were sampled and concatenated for both the RL and encoder batches. In other words, for each gradient update, the policy and Q-network observe (RL batch size) x (meta batch size) transitions and the encoder observes (RI. batch size) (encoder batch size) transitions.
18

---

Under review as a conference paper at ICLR 2022
Encoder architecture. The encoder uses the same architecture as in Rakelly et al. (2019). The posterior is given as the product of independent factors
qo (z h) II 8,a.rch
Φ(z s,
where each factor is multi-variate Gaussian over Rd. with learned and diagonal variance. In other words,
L0. (z s, s,a, N (No, (s. a, r), Jo. (s, r)))
The mean and standard deviation is the output of single MLP network with output dimensionality 2 x dg. The output of the MLP network is split into two halves. The first half is the mean and the second half is passed through the softplus activation get the standard deviation.
Self-supervised actor update. The parameter Apear controls the actor loss during the self- supervised phase, which is
suelr-supervin supervised ised
(0) Lactor Apeart
PEPAR (0),
aettor
Lactor
where CPEARL is the actor loss from PEARL (Rakelly al. 2019). For reference, the PEARL actor loss is
exp Q.. a,
PEARL
(8) Bg (x/h) DKL To(a s,z)
actir
Z(s)
When the parameter Apent is zero, the actor update is equivalent to the actor update in AWAC (Nair ct al. 2020).
Comparisons
As discussed in Section 6 used the authors' code for PEARL
Rakelly al.
2019 BOReL (Dorfman & Tamar 2020 and MACAW (Mitchell et al. 2021 To ensure fair comparison, ran the original hyperparameters and matched SMAC hyperparameters (matching network size, learning rate, and batch size), taking the better of the two as the result for each prior method. For all comparisons, SMAC, i.c. collecting exploration trajectories with the learned policy and perform adaptation using this newly collected data.
evaluated the final policy using the same evaluation method as
19


Reviews:
[{"review_id": "mk8XHW4OL7j", "reviewer": "Reviewer_RpMh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Merits\n\n1.\tThe problem this paper aims to address is practically important for RL, that is, how to effectively utilize offline datasets to meta-train policies and enable them for fast adaptation to new tasks. \n2.\tThis paper identifies a specific problem to offline Meta-RL : distribution shift of the exploration policy for collecting data for adaption, which looks interesting.\n3.\tThe idea of using unsupervised online data collection seems novel and shows some effectiveness in alleviating the problem of the distribution shift during the adaptation. \n4.\tThis paper is generally well-written and easy to follow. \n\n\n ## Concerns and limitations\n1.\tThe technical contribution of the proposed approach is limited. It is basically a simple combination of AWAC and PEARL. One modification to this combination is to use unsupervised online data collection, whose data are labeled with synthetic rewards, instead of ground-true rewards. However, synthetic reward learning is not novel and looks straightforward.\n2.\tAnother concern is that the proposed approach seems to assume tasks share the same dynamic model and only differ in reward functions. This is because the proposed SMAC method learns the task encoder only through the backpropagation from the reward learning, in contrast to prior work like PEARL. This assumption may bring SMAC with some advantages over other more general methods in tasks only differing with rewards, but will significantly limit its applicability. How will SMAC perform in settings where tasks may have different dynamics?\n3.\tThe reviewer is also curious about how this approach works in sparse-reward tasks? \n4.\tFrom the experiments, the outperformance of SMAC is mainly due to the unsupervised online data collection, which may not be a fair comparison to baselines. Without this online data collection, SMAC underperforms MACAW in two out of three tasks. \n5.\tThis paper only evaluates the proposed method in three scenarios. It is highly recommended to include more settings to strengthen the empirical results.\n6.\tIn Algorithm 1, Line 8 is confusing. What is $D_i$ in this line?\n\n", "summary_of_the_paper": "This paper aims to enable effective meta-RL in the offline setting. It first identifies the problem of simply combining meta-RL with offline RL: distribution shift in z-space. This paper then proposes a hybrid method that uses offline datasets to meta-train an adaptive policy and then collects and utilizes additional unsupervised online data to bridge the identified distribution shift. Empirical results show that unsupervised online learning significantly improves the adaptive capabilities of the meta-trained policy.", "summary_of_the_review": "This paper identifies a specific problem of distribution drift in offline Meta-RL and proposes an interesting idea of using unsupervised data collection to alleviate this problem. This paper is generally well-written. However, the proposed approach has limited technical contributions and makes strong implicit assumptions. The experimental evaluation also needs to be improved by including more settings.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "miwPTHjAB-8", "reviewer": "Reviewer_tmzu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper notes a problem called \"distribution shift in z space\" in offline meta RL and illustrates this issue in Figure 2.\nThis paper is well written and easy to understand.\nThe visualization of the distribution shift at the end of the experiment part is informative. \n\nWeaknesses:\n\nThe problem of \"distribution shift in z space\" might require deeper discussion. In other context-based meta RL algorithms (e.g. RL^2, MQL, CaDM), where the task context feature is extracted from a sequence of past historical transitions, is there still the problem of  \"distribution shift in z space\". If yes, could the proposed unsupervised online training be beneficial to solve this issue of \"distribution shift in z space\"  in these algorithms?\n\nIn the online training process, is the distribution of tasks the same as tasks in the offline training dataset? Figure 1(right) seems a bit confusing. The task set T1, T2, T3, T4, T5, T6 in online training is different from the task set T1, T2, T3 in the offline set. Then when we compare with offline meta-RL (Figure 1 left) or online meta-RL (Figure 1 middle), the proposed method is exposed to more tasks during training? Is it a fair comparison?\n\nIn section 5.2, in the online training process, if the agent interacts with the environment of task T_i to get a trajectory \\tau, does the proposed method use D_i (offline dataset collected on the task T_i) to infer z and then generate reward? Or it is fine to use D_j (j is not equal to i)? How accurate is the estimated reward in comparison with the ground truth reward? Does the advantage of the proposed method heavily rely on the accuracy of the estimated reward? If yes, under the sparse-reward tasks where estimating reward is challenging (e.g. the sparse 2D navigation task in PEARL), will the proposed method fail in this case?\n\nIs it possible to demonstrate the advantage of the proposed method on standard offline meta-RL benchmarks (e.g. D4RL)? The results on standard benchmarks with commonly used offline datasets will be more convincing.\n", "summary_of_the_paper": "This work investigates the meta offline reinforcement learning problems. It highlights an issue called \"distribution shift in z space\" in the meta offline RL. In the meta-test phase, the agent needs to explore the test environments and the collected transitions are used to generate z representing the task feature. The policy is conditioned on the task feature z, so it is important that the transitions collected by the exploration policy are informative enough to extract the feature information. In the offline RL setting, the distritbution of transitions collected by the exploration policy deviates from the distribution of transitions in the static training dataset. Therefore, inferring task feature z becomes problematic.\n\nThis paper proposes an additional online training process (without the reward info) to fix the issue of \"distribution shift in z space\". It learns a reward function from the offline training dataset, and thus it is able to generate the reward for transitions collected during online training. These online training transitions with the estimated rewards are used to update the learned policy.", "summary_of_the_review": "The online training process with generated rewards needs more clarification. And it will be better if the proposed method is evaluated on standard offline RL benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RjwPWJZ6j2g", "reviewer": "Reviewer_fffq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ It is a simple addition to the meta-RL setup and is shown to alleviate the z-distribution shift.\n+ Related work is well covered.\n\nWeakness/Comments\n- Some of the exposition is a bit high-level. Grounding in running examples would be helpful to better understand the overall approach. For example, in a set of tasks how different can any two tasks, is the difference at the level of one being picking and one being opening a drawer or is the difference at the level of different pick locations in a pick task.\n- '... can autonomously interact with the environment ...' is a very strong assumption for many real world tasks. This needs automatic resets in place as well as safe policies that already allow the agent to explore (which if are already available and robust defeats the purpose of learning to some extent?).\n- The claim that '... reward prediction generalization is easier than policy state, action, and z generalization' is not sufficiently supported. If the reward decoder is not well trained or reliable then wouldn't the bad labels skew the learning in the wrong direction? It wasn't clear how this issue was being addressed or if this imposes limits to the overall approach and where it can be used.", "summary_of_the_paper": "This paper introduces a reward prediction module to produce reward labels in a meta-RL setting. Here the offline RL part requires reward labels while the online portion can be done in a self-supervised manner by generating reward labels using the offline trained module.\n", "summary_of_the_review": "The idea is simple and works on the problems tested. The larger applicability and constraints of the approach are not clear. Explanation of the approach can be improved.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2cvfq96NKZF", "reviewer": "Reviewer_w6Yf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper strengths and contributions\n**Motivation and intuition**\n- Tackling distributional shifts of the exploration policy in offline meta-RL is well motivated.\n- Introducing the online self-supervised learning phase is intuitive and convincing.\n\n**Novelty**\n- I believe the introduced online self-supervised learning phase is novel.\n- The proposed self-supervised learning method is an intuitive way to learn from the online self-supervised learning phase. \n\n**Clarity**\n- Writing is easy to understand. Approach, notations, and figures are well explained.\n- Figure 2 well supports the targeting problem and motivation.\n- Figure 6 qualitatively showed targeting problem is resolved.\n\n**Related work**\n- Authors clearly address the additional assumption (online phase) from related work and its motivation.\n\n**Experimental results**\n- The presentation of the experimental results is clear.\n- The experimental results demonstrate that the proposed approach \nFigure 5 shows that self-supervised learning gradually improves meta-agent even without reward labels.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.\n- Appendix gives essential details for implementation. I believe reproducing the results is possible. \n\n## Paper weaknesses and questions\n\n**Offline datasets**\nThis work (and offline meta-RL prior work) assume the availability of offline/static datasets labeled with rewards for different tasks, which could be very expensive to obtain in some cases, limiting the applicability of this line of work. This work uses scripted policies to obtain the offline datasets, which means that all the tasks are solvable already. I would like to hear the authors' opinions on how this can scale up to more complex domains.\n\n**Suboptimal data**\nThe robot succeeds on the task in 46% of the transitions in the offline dataset, which is a pretty low success rate. I wonder if the proposed framework works better partially because of the suboptimality of this dataset.\n\n**Online interactions**\nThe proposed method needs additional online interactions compared to offline meta-RL prior works. While it could be cheaper to collect, sometimes it can still be impossible to leverage online interactions.\n\n**Reward prediction performance**\nIt would be informative to see how well it can predict the rewards of online interactions and how this affects learning performance.\n\n## Other metrics\n\n### Relevance and significance\nSolid contributions to a relevant problem\n\n### Novelty\nWorthy contributions, but not surprising\n\n### Technical quality\nTechnically adequate for its area, solid results\n\n### Experimental evaluation\nSolid, informative evaluation w.r.t all 5 criteria\n\n### Clarity\nVery clear, only minor flaws.", "summary_of_the_paper": "Meta-RL methods aim to learn to learn using a set of meta-training tasks. Yet, it requires a large number of online interactions for training. Offline meta-RL methods aim to learn from static datasets labeled with rewards for different tasks. These methods usually suffer from distributional shifts between the behavior policy from the offline data and the meta-test time exploration policy. This paper aims to tackle this distributional shift issue of offline meta-RL methods. To this end, the paper proposes a framework that is robust to the distributional shift by introducing a self-supervised learning phase. In this phase, the proposed framework leverages online data without reward labels and generates synthetic reward labels for it based on the labeled offline data, allowing for smoother adaptation. The experiments show the proposed approach alleviates the distribution shift issue and thus marginally improves post-adaptation results compared to offline meta-RL prior works. I believe this work tackles an interesting and promising research direction (i.e. alleviate the distributional shift issue in offline meta-RL) and proposes a convincing framework to address it. The experimental results verify the effectiveness of the proposed framework. Therefore, I am leaning toward accepting this paper with some concerns on the assumptions made by this work and its applicability.", "summary_of_the_review": "I believe this work tackles an interesting and promising research direction (i.e. alleviate the distributional shift issue in offline meta-RL) and proposes a convincing framework to address it. The experimental results verify the effectiveness of the proposed framework. Therefore, I am leaning toward accepting this paper with some concerns on the assumptions made by this work and its applicability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a new offline meta RL technique that addresses the distributional shift problem with a self-supervised online exploration phase where reward labels are not available.  The framework is novel and interesting.  The authors addressed many concerns of the reviewers.  However, the additional experiments raised additional questions.  For instance, why does meta-BC perform so well, even better than the proposed method without online data, and other baselines seem not to work at all?  In the discussion, the reviewers expressed concerns about the experimental results in the case of changing dynamics.  Those experiments are questionable since the proposed method only considers the reward information to deal with different dynamics.  Finally, an important question regarding SMAC remains unanswered: how much does the proposed method depend on the quality of the offline dataset and the quality of the reward decoder? Overall, the work is promising and the authors are encouraged to continue their work by addressing the reviewers concerns.
Decision: Reject