Hindsight Foresight Relabeling for Meta-Reinforcement Learning
Authors: Michael Wan
Jian Peng
Tanmay Gangwani
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Meta-Learning

Abstract: Meta-reinforcement learning (meta-RL) algorithms allow for agents to learn new behaviors from small amounts of experience, mitigating the sample inefficiency problem in RL. However, while meta-RL agents can adapt quickly to new tasks at test time after experiencing only a few trajectories, the meta-training process is still sample-inefficient. Prior works have found that in the multi-task RL setting, relabeling past transitions and thus sharing experience among tasks can improve sample efficiency and asymptotic performance. We apply this idea to the meta-RL setting and devise a new relabeling method called Hindsight Foresight Relabeling (HFR). We construct a relabeling distribution using the combination of "hindsight", which is used to relabel trajectories using reward functions from the training task distribution, and "foresight", which takes the relabeled trajectories and computes the utility of each trajectory for each task. HFR is easy to implement and readily compatible with existing meta-RL algorithms. We find that HFR improves performance when compared to other relabeling methods on a variety of meta-RL tasks.

HINDSIGHT FORESIGHT RELABELING FOR REINFOR M LEARNING
Michael Wan, Jian Peng & Tanmay Gangwani
University of Illinois at Urbana- Chaamaaiin {mw3, jianpeng, gangwan2 edu
ABSTRACT
Meta-reinforcemenn learning (meta-RL) algorithms allow for agents learn new behaviors from small amounts of experience, mitigating the sample inefficiency problem in RL. However, while meta-RL. agents can adapt quickly new tasks at test time after experiencing only few trajectories, the meta meta-training process is still sample-inefficient. Prior works have found that in the multi-task RL setting, relabeling past transitions and thus sharing experience among tasks can improve sample efficiency and asymptotic performance. We apply this idea to the meta-RL. setting and devise new relabeling method called Hindsight Foresight Relabeling (HFR). We construct relabeling distribution using the combination of hindsight, which is used to relabel trajectories using reward functions from the training task distribution, and foresight, which takes the relabeled trajectories and computes the utility of each trajectory for each task. HFR is easy to implement and readily com- patible with existing meta-RI. algorithms. We find that HFR improves performance when compared other relabeling methods a variety of meta-RL. tasks.
INTRODUCTION
Deep Reinforcement Learning (RL) has achieved success on a wide variety of tasks, ranging from computer games robotics. However, RL agents are typically trained on single task and are extremely sample-inefficient,, often requiring millions of samples to learn a good policy for just that one task. Ideally, RL agents should be able utilize their prior knowledge and adapt tasks quickly, just as humans do. Meta-learning, or learning to learn, has achieved promising results in this regard, allowing agents to exploit the shared structure between tasks in order to adapt to new tasks quickly during meta-test time.
Although meta-learned policies can adapt quickly during meta-test time, training these meta-learned policies could still require large amount of data. Several popular meta-RL methods (Duan et al., 2016; Finn et al., 2017; Mishra et al., 2017; Rothfuss et al., 2018; Wang et al., 2016) utilize on-policy data during meta-training to better align with the setup meta-test time, where the agent must generate on- on-policy data for an unseen task and use it for adapting to the task. Recent works (Fakoor et al., 2019; Rakelly et al., 2019) have sought to incorporate off- off-policy RL (Fujimoto et al., 2018; Haarnoja et al., 2018) 8) into meta-RI. to improve sample efficiency.
The combination of off-policy RL and relabeling, in which experience is shared across tasks, has been utilized in the multi-task RL. setting, in which an agent learns to achieve multiple different yet related tasks, for both goal-reaching tasks (Andrychowicz et al., 2017) and more general multi-task settings (Eysenbacc et al., 2020). Experience collected for one task may be completely useless for training a policy learn that task, but could be extremely informative training a policy learn different task. For example, an agent trying to shoot a hockey puck into a net might miss to the right. This experience could easily be used to train an agent to shoot a puck into a net positioned further to the right (Andrychowicz et al., 2017).
Both meta-RL. and multi-task RL. involve training on a distribution of tasks, so it follows that we can also combine relabeling techniques with meta-RL algorithms in order to boost both sample efficiency and asymptotic performance. meta-RL, an agent learns to explore sufficiently to identify
Code: https: //www. i thub. com/ /mi chael wan1 1/hfr

---

the task it is supposed to be solving, and then uses that knowledge achieve high task returns. The agent collects exploratory pre-adaptation data, then undergoes some adaptation process using that pre-adaptation data. Finally, after adaptation, the agent attempts to solve the task. Meta-RL algorithms typically have a meta-training phase followed by a meta-test phase. The goal during meta-training is train the meta-parameterrs such that they could be quickly adapted to solve any task from the meta-train task distribution, given a small amount of data from that task. At meta-test time, given a new unseen task, the goal to rapidly adapt the learned meta- parameters for this task, using a small amount of task-specific data. The focus in this paper is to improve the sample efficiency of the veta-training phase via data sharing.
Using concepts from maximum entropy RL (MaxEnt RL), we introduce a relabeling scheme for the meta-RI. setting. Prior relabeling methods for multi-task RI. have used the total reward of the trajectory under different tasks to guide the relabeling (Eysenbach et al., 2020; Li et al., 2020). Direct application of this type of relabeling to the meta-RL. setting is potentially sub-optimal since the multi-task RL and meta-RL objectives are distinct (learning to perform many tasks vs. learning to learn a new task). Towards developing an approach more suited to meta-RL, we define the notion of the utility of a trajectory under the different tasks, where the utility captures the usefulness of the trajectory for efficient adaptation under those tasks. We call our method Hindsight Foresight use hindsight in replaying the experience using reward functions from different tasks, and we use foresight in computing the utility of trajectories under different tasks and constructing a relabeling distribution over tasks using these utilities. We demonstrate the efficacy of our method on a variety of robotic manipulation and locomotion tasks. Notably, we show that our method, as the first meta-RL relabeling technique (applied during meta-training that we are aware of, leads to improved performance compared to prior relabeling schemes designed for multi-task RL.
Relabeling (HFR)
2 RELATED WORK
Meta-learning, or learning to learn (Baxter, 1998; Naik et al., 1992; Schmidhuber, 1987; Thrun & Pratt, 1998), has been topic of interest since the 1980s. Various approaches have been developed in recent years. Prior works have attempted to represent the RL process using a recurrent neural network (RNN) (Duan et al., 2016; Miconi al., 2018: Wang et al., 2016)- the hidden state is maintained across episode boundaries and informs the policy as to what task it is currently solving. Similarly. Mishra et al. (2017) also maintain the internal state across episode boundaries while incorporating temporal convolution and attention into recursive architecture. Gradient- based meta- learning methods have also been explored (Finn al., 2017: Nichol & Schulman, 2018; Xu et al., 2018; Zheng et al., 2020). MAML (Finn et al., 2017) seek to learn a good policy initialization so that only a few gradient steps are needed to achieve good performance on unseen meta-test tasks. Stadie et al. (2018) build on this gradient-based approach but explicitly consider the effect of the original sampling distribution on final performance. Similar to these works, our relabeling method also considers the impact of pre-adaptation data on the post-adaptation performance. Another body of work focuses on designing strategics for structured exploration in meta-RL RL such that telk-relevant information could be efficiently recovered (Liu al., 2020; Rakelly et al., 2019; Zintgraf al., 2019). Rakelly et al. (2019) devise an off- off-po meta-RL. method called PEARL that trains an encoder to generate a latent context vector on which the meta-RL agent conditioned. Although we use PEARL as our base algorithm in this work, our relabeling scheme is general enough to be integrated into any off-policy meta-RL algorithm.
Experience Relabeling in Meta-RL. Recent work has studied the scope of sharing experience among tasks in the meta-RL paradigm. Mendonea et al. (2020) propose to tackle meta-RL via a model identification process, where context-dependent neural networks parameterize the transition dynamics and the rewards function. Their method performs experience relabeling only at the teta-test time, with the purpose of consistent adaptation to the out-of out-of-distribution tasks. Crucially, there is no relabeling or sharing of data amongst tasks during the meta-train time. In contrast, the goal of the relabeling in HFR is to improve the sample efficiency of the meta-training phase. Dorfman et al. (2020) study the offline meta-RL problem. They propose reward relabeling as a mechanism to mitigate the "MDP ambiguity" issue, which the authors note is specific to the offline meta-RL RL setting. Their relabeling is based on random task selection. HFR, on the other hand, operates in the online meta-RI. setting and provides principled approach to compute a relabeling distribution that suggests
2

---

tasks for relabeling. We compare with the random relabeling method used in Dorfman et al. (2020) in our experiments. A more detailed comparison to these two prior works is included in Appendix A.11.
In the context of multi-task RL (Caruana, 1997; Kaelbling, 1993; Schaul et al., 2015), recent methods have proposed relabeling to improve the sample-efficiency (Andrychowicz et al., 2017; Eysenbach et al., 2020; Li et al., 2020). HER (Andrychowicz al., 2017) relabels transitions using goals that the agent actually achieves. Doing so allows for learning even with a sparse binary reward signal. However, HER is only applicable goal-reaching tasks and cannot be incorporated into meta-RL algorithms because the meta-RL agent is trained on a batch of tasks sampled from a fixed task distribution. Similar to our work, Eysenbach al. (2020) use MaxEnt RL construct an optimal relabeling distribution for multi-task RL and apply this to both goal-reaching tasks and tasks with arbitrary reward functions.
BACKGROUND
3.1 REINFORCEMENT LEARNING
In reinforcement learning (RL), the environment modeled as a Markov Decision Process (MDP) M = (S, P. PIP). where S is the state-space, A is the action-space, r is the reward function, is the transition dynamics, 7 E [0, 1) is the discount factor, and P1 is the initial state distribution. At timestep f, the agent MT. parameterized by parameters 0, observes the state 8t E S, takes an action a₂ ~ To (o and observes the next state St-1 ~ p( (s,, a,) and the reward r (s,, az). The goal
is to maximize the expected cumulative discounted rewards: maxg Esp [E 7 Ir (8t, a:)]
3.2 MUITA-REINFRRCEMEN LEARNING
In the general meta-reinforcemenn learning (meta-RL. setting, there is a family of tasks that is characterized by a distribution p (1)), where each task is represented by an MDP My (S,A, Tu Pe., Y PI). The tasks share the components (8, (S,AA,7P)) but can differ in the reward function To (e.g. navigating to different goal locations) and/or the transition dynamics pv (e.g. locomotion on different terrains). In this work, consider the setting where the tasks share the same transition dynamics (i.e., Pv = p). but differ in the reward function. The goal in meta-learning is to learn set of meta-pane parameters such that given data for the new task, the meta pata-parameters can be efficiently adapted to solve the new task. In the context of meta-RL, given new task s/, the agent collects some initial trajectories {Tpee, }, each being sequence {$1, , ...... }. and then undergoes some adaptation procedure fo (Te, Tpre: Tw:) (e.g.. gradient update (Finn et al., 2017) or a forward pass through an RNN (Duan al., 2016)). The adaptation procedure returns a new policy T'. Using this post-adaptation policy, the agent should seek to maximize the cumulative discounted rewards it achieves. Overall, the meta-RL objective is:
task from p (1) and small amount of
oo
max (eppp(()))eedde))) (e) 8.6
Σ
Pr (St, at)
7' 7'(0, Φ) = So (TO, Tpre, rw))
(1)
where 0, e are the meta-parameters that are learned in the meta-training phase. A meta-RL agent must learn a good adaptation procedure fo that is proficient in extracting salient information about the task at hand, using few pre-adaptation trajectories Tpre. At the same time, it should learn the policy meta meta--arameters 0 such that can achieve high returns after the adaptation process, f.e., while following the policy n' J (TO, Tpee: T.)).
3.3 PEARL
In this work, we use PEARL (Rakelly et al., 2019) as our base meta-RL algorithm since it uses off- policy RL and provides structured exploration via posterior sampling. PEARL is built on top of Soft Actor-Critic (Haarnoja et al., 2018) and trains encoder network qø (2|c) that takes in the "context" c, which consists of a batch of (st, ae, Tt, $t+1) transitions, and produces the latent embedding 2. The intent is to learn the encoder such that embedding encodes some salient information about the task. The adaptation step fa in PEARL. corresponds to generating this latent z and then conditioning the policy and the value function networks on The policy अe (als, z) is trained using loss Lactor
DKL. To (als, 2) exp (Qp (s, a.a) 2))
Tegttor ESn aarot (x
(2)
Zo (s)
3

---

where B is the replay buffer. The critic Qo (s, a, 2) and the encoder qs (z|c) are trained with temporal difference learning:
Tugiitic (. [ (s, (r 2)))
(3)
where V is the target state value and v denotes that the gradient does not flow back through the latent.
4 HINDSIGHT FORESIGHT RELABELING
The objective in this section is to derive a formalism for data-sharing amongst the tasks during the meta-training phase. This is achieved via trajectory-relaheling wherein a trajectory collected for training task &e is reused or re-purposed for training a different task v Reward-basee trajectory- relabeling has received lot of attention in recent works on multi-task RL and goal-conditioned RL (Andrychowicz et al., 2017: Eysenbach et al., 2020; Li et al., 2020). The intuition is that if a trajectory collected while solving for the task vp achieves high returns under the reward definition for another task vi (i.e.) >, Tyo (st, (t) is large), then T can be readily used for policy-optimization
for the task v/ as well. The meta-RI. setting presents the following subtlety - for any given task, the meta-RL agent generates trajectories with the aim of utilizing them in the adaptation procedure and subsequently seeks to maximize the post-adaptation returns (cf. $3.2). To improve the efficiency of the meta-training stage, we would like to share these pre-adaptation trajectories amongst the different tasks, accounting for the fact that the metric of interest with these trajectories is their usefulness for task kosk-identifiiation, rather than the returns (as multi-task RL). This difference is illustrated in Figure 3. Hence, when deciding if a trajectory collected for task v/d is appropriate to be reused for task vh, it is sub-optimal to consider the return value of this trajectory under v/d. Instead, we argue that this reuse compatibility should be determined based on the performance the task v after the agent has undergone adaptation using т. Concretely, we define a function to measure the utility of the trajectory r for task v/
Uni (T) = Eeraa
Σ 1=1
rass (st,a))
(4)
where ='=Jo (TO; T, Tes)) denotes the policy after using T for adaptation. The trajectory-relaheling mechanism during meta- veta-training now incorporates this function UgA, which we refer to as the utility function, rather than the return Rai. Broadly, a trajectory collected for task vf can be relabeled for use in another task vi if Una (r) is high. Subsection $4.1 1 makes this more precise by deriving a relabeling distribution g(vllr) that informs us of the tasks for which 7 should be reused. Figure 4, and the caption therein, describe a high-level overview of our approach, HFR.
Comparison to HIPI (Eysenbach et al., 2020) with a didactic example. We consider a toy environ- ment to further motivate that return-value based data sharing and trajectory relabeling (as proposed by HIPI) is potentially sub-optimal for meLa-RL. The placed at the center of a square where each corner of the square represents a goal location, as shown in Figure 1. For each goal (task), there is section of the space in the corresponding quadrant in which the robot receives a large negative reward. Consider trajectory that hovers over the blue square in top-right quadrant. Note that T could have been generated by the agent while collecting data for any of the four tasks. We examine if can be reused for the blue task. Since Rolue (T) is highly negative, the relabeling strategy in HIPI does not reuse T for meta-training on the blue task. It is clearly evident, however, that 7 carries significant amount of signal pertaining to mess-ideniification on the blue task, making it useful pre- -daaptttt trajectory. HER reuses 7 for the blue task since the utility Vsuee (T) high.
environment consists of a point robot
Figure I: The Four- Corners environment
Four Gomannm
To quantify this effect, we include the numerical data on the returns and the utility values for a sampled trajectory that hovers over the blue square in top-right quad- rant. The values for the returns ( (Rpurpte(T) (T), Risue(T) Rorange (T), (osree(()) are {-20, -58 20, -00). while the utility values (Upurte(T) Usiue (T), (orange
Figure 2: Success-
rate on Four-Corners
Ugree((T) are 1015, - -756,- 993,, 9311 These (unnormalized) numbers show that the probability of relabeling this trajectory with the blue task is low under HIPI, but high under HFR. Further analysis is included in Appendix A.6. Figure compares HFR and HIPI in terms of the success-rate in the Cour-Corners environment, and shows the performance benefit of using the utility function for trajectory relabeling.
4

---

Goal: Meximize Feeuus
Goal: Task Idenifiication
Gcel: Maximize Peturn
Multi-takk
Meta-RL
Figure 3: An illustration of the differences between multi-task RL and meta-RL. In multi-task RL (blue) the agent simply maximizes its returns given task v5, while in meta-RI. (orange) the agent must first quickly identify the task with limited number of exploratory trajectories (first two orange stacks the figure), before adapting to the task and maximizing returns. Because of these differences, existing multi-task relabeling methods may be sub-optimal for meta-RL.
4.1 DERIVING A META-RL RELABELING DISTRIBUTION
Our derivation in this subsection largely follows HIPI (Fysenbach et al., 2020), but differs in that we adapt it to the meta-RL. setting to promote sharing of pre-adaptation trajectories amongst tasks, using the concept of trajectory utilityy Assume dataset D of trajectories gathered by the meta-RL agent when solving the different tasks in the meta-train task distribution. We wish to learn a trajectory relabeling distribution (1//7) such that, given any trajectory with high density under this posterior distribution. To that end, we start by defining a variational distribution q( (b) to designate the trajectories used for the adaptation process far for given task sb. Using the definition of the utility function (Eq. 4), the meta-RI. objective from Eq. 1 could be written maxo,6 (p)) ((r()))) [U, (r)]- For fixed meta-parameters (0, natural approach to optimize the variational distribution ((1lv)) is use this same objective since it facilitates alignment with the goals of the meta-learner. Thus, the combined objective for the variational distributions for all the tasks, augmented with entropy regularization, is:
D, we could reuse 7 for tasks
max ((p (Eragtris)( Hacrles 4
where Hatrlwy denotes the causal entropy of the policy associated with q(T (t). Now, we note that the above optimization is equivalent to a reverse-KL divergence minimization objective: milg(r.p)) DRI. [g (7, vb) ||p (T, } (ef. Appendix 10) Here, the joint distributions over the tasks and the trajectories are defined as q (T, e) = ((())
and p v) = p(T))) where
p(T()) A
Z(v) 2(1)
1
pz (((s))c p(s
Our goal is to formulate the trajectory relabeling distribution ((1b77) To make this explicit in our objective, we use the trick proposed in HIPT (Fysenbach et al., 2020) and factor q (T, 15) as q (1b(T) (r), thereby rewriting the reverse-KL divergence minimization objective as:
min E rog(r) [log g (+l7)+ ((7) - ((t)) (w)) ((..)) a
Σ log (osp((s41sss) a:)
(7)
Ignoring the terms independent of v/, we can analytically solve (by differentiating and setting to zero) for the optimal trajectory relabeling distribution for the meta-RL setting:
(vbTT) (v) e (r)))) 2(+)
(8)
Given a trajectory 7 ~ DD the HFR algorithm uses this relabeling distribution to sample tasks for which J should be reused. Concretely, we compute the utility function under for all the tasks, construct the distribution q(bb)iT) using these utilities (Eq. 8), and sample tasks from it. Please see Figure for details. We assume a uniform prior p (1) over the tasks in our experiments.
4.2 ALGORITHM AND IMPLEMENTATION DETAILS
Our relabeling algorithm is summarized in Algorithm I and fits seamlessly into the meta-training process of any of the base meta-RL algorithms. Once the meta-RL. RL agent generates a trajectory 7 for training task, is fed as input to HFR, and returns another task that reuse this experience
5

---

Sp
HFR Trajectory Relabeling
Add to B
Gollect Sie the cuck
City
Algreihen (uummary)
Tapet:: Trigstay odklided (r)
B3r
Ny. (ols, 9)
Tislegit Doaiaae traeatioss la uade (attors
Task Distribution p(s)
Compone (Kgrl)).....
Compuns
clole) sxcple
Rotren odocted
Date collection
Our contribution
PEARL Meta-training
Figure 4: During meta meta-training, after trajectory T is collected for task vi', HFR uses hindsight to relabel this trajectory using reward functions for different tasks, and then uses foresight to compute the utility of the relabeled trajectory for the different tasks. A distribution over tasks is constructed using the utilities, and task & is sampled from the distribution, with tasks for which the trajectory has higher (normalized) utility having higher probability mass. The trajectory then relabeled using the reward function Tu and added to the task-specific replay buffer Bus. Finally, the meta-training update rules are applied. This process repeats throughout the entirety of meta veeaatraiinnn, HFR uses PEARL as the base meta-RL algorithm and does not alter its data collection meta-ge veta-gradient computation rules. Please the Algorithm box for details.
Algorithm 1:
Hindsight Foresight Relabeling (HFR)
Input Trajectory he relabeled (r)
Output: Task to relabel the trajectory with (4)
Function ComputeUt (T, th)
for each training task v² do
for each (st, er, re) E do Replace r. with (8t, as) end Sample embedding using encoder z 46 (2/T) Sample a batch of initial states (si No
Ue (T) + ComputeUtilityyrr v
log Z (v²) GetlLogarttitionn""
end
Return v ~ softmax (U (T) log [v)))} (Eq. 8)
1m1 By
Function Get Get.ooPartiiiii (1)
Sample actions for these states using the post-adaptation policy T=(-18,2); {a= To (a biilsi, 2) Ny
Sample batch of trajectories {r' Jizl By for each majectory 7¹ do
Uv (r") + Cowputeuuilittteeeeee
end
Return Up I ENI Qo (si,ai,2) (Eq. 9) Ner
Return log Z (++) ~ log ( EN e
We compute the utility of the input trajectory for every training task, along with an empirical estimate of the log-pan por-parttiion function of the tasks. The task to relabel the trajectory with then sampled from a categorical distribution. For our experiments, we build on top of the PEARL algorithm (Rakelly al., 2019), which data-efficient off- policy meta-RL method. PEARL maintains task-specific replay buffers Bg. If HFR returns the task v/, then is relabeled using the reward function Tir and added to Be for meta-training on the task v/.
The adaptation procedure x' [a(To, r, rw) for a task de corresponds to a sequence of steps: 1.) augment T by marking each transition with a reward value computed using Te at), 2.) condition to obtain
the encoder to sample embedding, the post-adaptation policy, = T [s, z). The calculation of the utility function (Eq. 4) requires generation of post-adaptation trajectories, which could be computationally inefficient, especially if the number of tasks is large. To avoid this cost, for each task, we sample a batch of initial states 81 pi (s1) ) and the corresponding actions from the post-adaptation policy, and compute the utility based on an estimate of the state-action value function Q (81, a1) as:
go (2/7); and 3.) condition the policy
Up (7) (e @ (s).01)
Since we use PEARL, we can avoid training separate task-specific value functions Q and instead get the required estimates from the task-conditioned critic Q a, 2) already used by PEARL (Fqq. 3). We

---

Sawyer Reach
(b) Sawyer Push
(c) Visual Reacher
(d) Cheetah
(e) Ant
Figure 5: MuJoCo environments we evaluate on sparse reward manipulation tasks (a, b, c), as well as sparse and dense reward locomotion tasks (c, d).
Figure 6: Performance of our relabeling algorithm HFR (shown in blue) on sparse reward tasks. HFR consistently outperforms baselines on both sparse reward robotic manipulation and locomotion tasks. Visual- Rlsuhl-Reahher uses image observations, while the other environments use proprioceptive states.
highlight that HFR facilitates efficient data-sharing among the training tasks via trajectory-relablle without altering the meta-train and test-time adaptation rules of the base meta-RL. RL algorithm.
EXPERIMENTS
The goal in this section is to quantitatively evaluate the benefit of sharing experience among tasks using HFR, during the meta-train stage. We evaluate on a set of both sparse and dense reward MuJoCo environments (Todorov al., 2012) modeled in OpenAI Gym (Brockman al., 2016). Please refer to the Appendix for environment details. We compare HFR with two relabeling methods: Random, in which each trajectory is relabeled with randomly chosen task, and HIPI (Eysenbach et al., 2020), which utilizes MaxEnt RI. to devise an algorithm that relabels each transition using a distribution over tasks involving the soft Q values for that transition. In contrast, HFR proposes the concept the utility of a trajectory for given task. Using the utility aligns the relabeling methodology with the objective the meta-RL RL agent (ef. section 4). All methods are built on top of PEARL. Finally, we compare to PEARL with no relabeling at all, which we refer to as None.
5.1
RESULTS
Sparse Reward Environments. We first evaluate HFR on of sparse reward robotic manipulation and locomotion tasks. We use five environments: a goal-reaching task involving a quadruped Ant robot, pushing task the Sawyer robot, a reaching task on the Sawyer robot, velocity-matchin task involving bipedal Cheetah robot, and a reaching task involving the MuJoCo Reacher where the agent learns directly from images. The environments are described in detail Appendix A. and shown pictorially in Figure 5. Figure 6 plots the performance (average returns or success-rate) on the held-out meta-test tasks on the y-axis, with the total timesteps of environment interaction for meta-traine tota-training and achieves a higher asymptotic score. Meta-RL in sparse reward tasks hard due to the challenges of task- lak-identiffcation and efficient exploration. HFR is especially useful for these tasks the data- sharing afforded by the trajectory relabeling algorithm mitigates the need for an elaborate exploration strategy during meta-traine vetta-raiining This leads to the sample-efficiency gains exhibited in Figure 6.
the r-axis. We note that HFR tends to be more sample-efficient than the baselines
Dense Reward Environments. We next evaluate HFR on dense reward environments (Figure 7). We experiment with the Cheetah-Higgdim environment, in which the bipedal robot is required to
7

---

Cheetah-Highdim
Cheetah-Vel
Ant-Vel
4000
[ours
HR (ou))
[oers)
5000
100
Jvvorgg Roburn
100
Juece 150
Nave
2000
20000
0.35
0.09
0.55
0.35
3.3M
Timestare
Timestaps
Timesteos
Figure 7: Performance of our relabeling algorithm HFR (shown in blue) on dense reward tasks. With the exception of Cheetah Highhdi,,, relabeling in general offers no benefit in dense reward meta- RL tasks, likely due to the highly informative nature of dense reward function.
best match its state vector to a set of predefined vectors, as well as the Cheetah- Vel and quadruped Ant-Vel environments, in which the robots are required to run at various velocities.
Note that the impact of HFR is much less pronounced for these environments, with the exception of Cheetah-Highhitg We believe this is because exploration is not as critical for these environments as it was for the sparse reward tasks. This hypothesis is supported by the fact that, in these environments, PEARL with no relabeling is competitive with the various relabeling methods, which all share similar performance, whereas the sparse reward environments HFR is the relabeling method that performs best, with the two other relabeling methods also vastly outperforming vanilla PEARL. In the case of the Cheetah- Vel and Ant- environments, agents are provided with an informative dense reward that immediately informs them to which task they're supposed be solving. Although the agent in Cheetah- Cheetah-Highdin is also provided with an informative dense reward, the reward function in this task is linear combination of 18-dimensional state vector. Thus, reasonably good exploration is needed to determine optimal values for each of these 18 dimensions. HFR relabeling provides improvement over the baselines for this task.
5.2 ABLATION STUDIES
Batch Size. We investigate the impact of the batch size No used in Algorithm I for computing an empirical estimate of the state-action value: No ΣN" Qo (st, 2). We compare the effect of batch size across batch sizes (16, 32, 64, 128, 256}. We expect lower values of No to lead to a higher variance estimate of the post-adaptation cumulative discounted rewards and thus potentially worse approximation of the optimal meta-RI. relabeling distribution. In Figure 8a, we see some evidence of this in the comparatively worse performance when using N2 = 16 and Ng = 32. However, we note that even with these small batch sizes, our method still performs well, and in general achieves high returns across all choices of Nu.
Partition Function. We investigate the impact of the log-partition function log (w)) in the optimal meta-RL relabeling distribution (Eq. 8). Prior work has noted the importance of the partition function in the multi-task RL setting when tasks may have different reward scales (Eysenbach et al., 2020). We believe that in the meta-RI. setting, the partition function may be crucial even if the tasks share the reward scale, since some tasks in the meta-training distribution may be easier to solve than others. We speculate that with the omission of the partition function from our relabeling distribution, trajectories would find high utility and thus be disproportionately labeled with the easily-solved tasks, causing a degradation overall performance. In our experiments, we find that the partition function serves as an essential normalization factor; Figure 8b shows an example.
Reward Function. One assumption our method assumes access the true reward function Ty (s, a). which we can query to get the reward for an individual transition under any training task sb. This availability has also been utilized in existing works on relabeling for multi-task RL (Eysenbach et al., 2020; Li et al., 2020). In many real-world applications of meta-RL, e.g. a distribution over robotic tasks, it is reasonable to assume that the task-designer outlines a rough template for the rewards corresponding to the different tasks from the distribution. Furthermore, several of the rewards used in our experiments are success/failure indicators, which are simple to specify. Nevertheless, we consider the scenario where cannot query the true reward function for individual transitions. Figures 8c and 8d show good performance even when we use a learned reward function rather than the true reward function relabel trajectories.
8

---

11000
(a) Batch Size (Nu)
(b) Partition Function
(c) Reward Function
(d) Reward Function
Nr' Gooa
Serven Fush
Stervor Headh
Cheatah-- dir
110
(c) Soft-max vs Hard- max
(f) Soft-max vs Hard- max
(g) Soft-max vs Hard- max
(h) Soft-max vs Hard- max
Figure 8: Ablation analyses. (a) HER with different values for the batch size No used in approximating Eq. 9. HFR is relatively robust to choice of batch size, with some smaller choices of Ny leading to slightly worse performance. (b) HFR with and without the log- partition function (log Z (t=)). The partition function serves as a necessary normalization factor and prevents against simply relabeling every trajectory using easiest task. (c), (d) HPR with true reward function (orange) and HFR with learned reward function (blue). (c), (f), (g), (h)
HFR with soft-max (orange) vs hard-max (bluc) relabeling distribution
Soft-max vs Hard-max Relabeling Distribution. Given a trajectory т, the relabeling distribution derived in Section $4.1 samples tasks for which should be reused. Specifically, tasks are sampled crucial to have stochasticity in the relabeling distribution, could we deterministically select the task for which the normalized utility value is the highest, i.e., tb = argmax(Uet (T) - log (x)*)
as: 1 ~ so www.max (Unr
log }. This raises the following question:
A minor modification to the equations in Section $4.1 yields the hard-max relabeling distribution. Concretely, we can add a term to the starting objective for the variational distribution that explicitly minimizes the entropy of the relabeling distribution q(bl)):
max (p(( (rrggrive (e) (U (T) Hytries] (1 e))--(a)))()) ( [Ha(etr) Q where is a value less than Proceeding with the derivation in the exact same manner as in Section $4.1, we obtain the adjusted relabeling distribution:
9e (v/T) (g(r)) 2(0)
In the limit when e + 0, qe (vb is the hard-max relabeling distribution. Figures Se, 8f, 8g, and 8h compare the performance of HFR when sampling tasks from soft-max relabeling distribution (orange) vs a hard-mar max distribution (blue). The results indicate that stochasticity is an important factor.
CONCLUSION
In this paper, we introduced HFR, a trajectory relabeling method for meta-RL that enables data sharing between tasks during meta-train. We argue that unlike the multi-task RL setting, where the appropriateness of a trajectory for task could be measured by the returns under the task, for meta-RL, it is preferable to consider the future (expected) task-returns of an agent adapted using that trajectory. We capture this notion by defining the utility function for a trajectory-tass pair and incorporate these utilities in our relabeling mechanism. Inspired by prior work on multi-task RL, an optimal relabeling distribution is then derived that informs us of the tasks for which a generated trajectory should be reused. Hindsight is used to relabel trajectories with different reward functions, while foresight is used in computing the utility of each trajectory under different tasks and constructing a relabeling distribution. HFR is easy to implement, can be integrated into any existing meta-RL algorithm, and yields improvement on a variety of meta-Ru RL tasks, especially those with sparse rewards. To the best of our knowledge, HFR is the first relabeling method designed explicitly for the meta-RI. paradigm.
9

---

7
REPRODUCIBILITY
We provide our code at https://githut com/mi Shaalwanil/hfr Relevant hyperparam- eters can be found in Tables and well the config files in our code, while algorithm details can be found in Algorithm 1.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. 2017.
In Advances neural information processing systems, pp.
Jonathan Baxter. Theoretical models of learning to learn. In Learning to learn, pp. 71-94. Springer, 1998.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1006:010000054400 2016.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-755 1997.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008. 02598, 2020.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2, Fast reinforcement learning via slow reinforcement learning. arXiv preprint /6 2016.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint arXiv:2002.11089, 2020.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning. arXiv preprint arXiv:1910.0012 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. arXiv preprint arXiv:/703.033m /7030034000 2017.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 09477, 2018
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic, Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint /80.vv/801.01220 2018.
Leslic Pack Kaclbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Alexander C Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. arXiv preprint orXiv:2002.1 /708 2020.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning. arXiv preprint arXiv:2006. 08875, 2020.
Evan Zheran Liu, Aditi Raghumathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforrcement learning without sacrifices. arXiv preprint arXiv:2008.02799. 2020.
Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reiinoorcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint 001::20066.0177.. 2020.
10

---

Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, pp. 3559-3568.
PMLR. 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprini arXiv: 2017.
DK Naik, Richard Mammone, and A Agarwal. Meta-neural network approach to learning by learning. In Proceedings of the 1992 Artificial Neural Networks Engineering, ANNIE 1992.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint (8K300800302999, 2(2):1, 2018.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reinforceemen learning via probabilistic context variables. In International conference on machine learning. pp. 5331-5340, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint 1rXiv:1810.000
2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1512-1320. PMLR, 2015.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The mela-meta-... hook.) Diploma thesis, Institut Informatik, Tech. Univ. Munich, 1(2), 1987.
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv: 118, 2018
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: .05763, 2016.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. arXiv preprint ar Xiv: 1805.00800008000 2018
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. pp. 1094 1100, 2020.
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pp. 114661 PMLR, 2020.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sehastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint 181.0.1910,08348, 2019.
11

---

A APPENDIX
A.1 FAILED EXPERIMENTS (AN ALTERNATE UTILITY FUNCTION)
While the expected post-adaptation return is a direct indicator of the usefulness of a trajectory for task, we now mention an indirect measure that is pertinent to the meta-RL baseline algorithm we employ (PEARL). The idea behind PEARL is that the encoder network qa, conditioned on the given trajectory (referred as "context"), produces an embedding that captures some salient information about the task. The encoder is trained to minimize the Bellman error (Freitie- Eq. 3). A lower value of Leiii thus indicates that the trajectory is valuable for identifying the task v. Therefore, we experiment with using the negative of Leritic utility function:
UN (T) = [(s..ann" [ (r (x)r))
(10)
We can efficiently compute this utility function with single forward pass through the encoder, followed by computation of the Bellman error using a batch of (s, a, s') tuples. Algorithm includes the details. We note that this utility function explicitly gauges the viability of iast-identiiicatiin through the PEARL encoder, rather than measure the post- adaptttttnioon returns.
Algorithm 2: Computation of the utility function based on the Bellman error (Fq. 10), for PEARL-based me
Input :Trajectory to be relabeled (7)
Task to compute utility of the trajectory for (1)
Output:Utiligg of the trajectory under task sh Up (T)
for each (&t, at, rr) E r do
| Replace ry with ry (#s: as)
end
Sample embedding using the encoder qo (2/r)
Sample batch of transitions (st, az, s4) i=1 BB
Return the negative Bellman error (Eq. 10): Up
1 E - (Qu as,,,;, (r: + V (s(,
N
Our results when using this new utility function can be seen in Figure 9. Overall, relabeling based on the Bellman error (HFR-Bellman does not perform as well as relabeling based on the expected post-adaptation returns, likely because the Bellman error is simply an indirect measure of the true metric of interest posst-ddatttioon returns).
A.2 TRANSITION DYNAMICS
Although this work focuses the setting where the tasks share the same transition dynamics, but differ in the reward function, in this section we show results on environments where the tasks differ in both the transition dynamics and reward function. We consider two environments: Ant-Goal-Ran Gaaa---addd-aransss and Cheetah- San-paare--Rad--aaaaaae
which are identical to Ant-Goal and Cheetah- Vel- Sparse, respectively, except for the fact that the transition dynamics now also differ across tasks. We do not modify any of the methods (HFR, HIPI, None, Random) to take into consideration the different transition dynamics across tasks. Our results can be seen in Figure 10. We find that (a) Relabeling can surprisngly still provide some benefit in this setting and (b) HFR still outperforms basclines when transition dynamics differ.
Figure 10: Perfor- mance on tasks that differ reward function and transi- tion dynamics.
A.3 HYPERPARAMETERS
Table lists the hyperparameters that were shared across all environments. Hy- perparameters for the Sawyer environments were taken from Yu et al. (2020), while hyperparameters for the other environments were taken from the open-source PEARL implementation (Rakelly et al., 2019).
12

---

420
Goad
Sawyer-PPust
Sawyer- Reachh
400
SU00000 RODE
0.3
250
300
6.24
200h
250c
4009
(a)
(b)
Cheetah- Vel Sparse
Choetah-llighlim
Cheetsh-Vyl
-52
100
1.5
6.0
Avseaaee
-00
230
502
2.2M
0.5M
1:30
24
0.29
0
6.0M
3.50
Timesbeps
Timestaps
(e)
(f)
Ant-Vel
IF2
Recrogs
-92
0.35
Timestoss
Figure 9: Performance of our relabeling algorithm HFR (blue) compared to a variant of HFR based on the Bellman error (orange) Table 1: PEARL hypeparameters used for all experiments.
Hyperparameter
Value
Nonlinearity
ReLU
Optimizer
Adam
Policy Learning Rate
3e-4
Q-function Learning Rate
3e-4
Batch Size
256
Replay Buffer Size
1e6
A.4 ENVIRONMENTS
The environments we evaluate HFR on are as follows:
We create 2D navigation task in which point robot must navigate to one four goal locations. A reward of 0 is given when the robot is within a distance of 0.2 from the goal, with the episode ending. Otherwise, reward of 1 is given, with reward of -3 being given if the robot within a certain section space in the same quadrant as the goal (Figure I).
Four-Corners:
Ant-Goal:
We the Ant-Goal task from (Gupta al., 2018). Tasks correspond to goal locations sampled uniformly from a half-cirele radius 2. The reward function is 4 || I 2 [aat Fggoall la < 0.8 and otherwise, where large uninformative constant.
13

---

Table 2: Environment Details
Number of
Environment
Discount
Horizon
Train Tasks
Test Tasks
Exploration Steps
Ant-Goal
0,99
200
100
30
400
Ant-Vel
0.99
200
150
30
400
Cheetah-Higghiin
0.99
200
100
30
400
Cheetah- Vel
0.99
200
100
30
400
Cheetah-
Cheetab-Vel-Sparye Vel-Sparse
0,99
200
100
30
400
Four-Corncrs
0,90
20
4
4
380
Sawyer-Pushe 0.99
150
50
10
450
Sawyer-Reach
0.99
150
50
10
450
Visual-Reache
0.99
100
50
10
200
We use the Ant environment from OpenAl gym. Tasks correspond to goal velocities sampled uniformly in [0. 3]. The reward the negative absolute value of the difference between the agent's s velocity and the goal velocity. We take this task from (Finn al., 2017).
Ant-Vel:
Cheetah- :igghd--i We take the Gtheetah-Highhiee task from (Lin et al., 2020). Each task corresponds to an 18-dimensional vector v sampled uniformly in []1, The reward is linear in the post- transition state s' and is given as
s') vi
Cheetah- Vel: The Cheetah-Ve Vel task taken from (Rakelly et al., 2019). Velocities are sampled uniformly in [0, 3] and the reward is the negative absolute value of the difference between the agent's velocity and the goal velocity.
Cheetah- Vel-Sparse: Velocities sampled uniformly in [0, 3], with positive reward being given if the absolute value of the difference between the agent's velocity and the goal velocity is less than 0.3.
Sawyer-Push: An agent must control simulated Sawyer robot and push block specified goal location. A reward
is given when the block is within a distance of 0.07 of the goal and a reward of -1 is given otherwise. The episode ends when either the goal is reached the agent has taken 150 steps. Both the Sawyer Reach and Sawyer Push environment were taken from (Yu et al., 2020).
Sawyer-Reach: An agent
control simulated Sawyer robot and reach specified 3D goal location. A reward of 0 is given when the end effector is within a distance of 0.05 of the goal and reward of 1 given otherwise. The episode ends when either the goal is reached the agent has taken 150 steps.
Visual-Reacher: We use the standard MuJoCo Reacher environment where an agent must reach various 2D goal locations, with positive reward being given if the end effector is within a distance of 0.03 of the goal. The agent must meta-learn directly from 64x64 grayscale images of the environment as the input observations.
Table 2 includes the details for each environment and Figure 5 shows them pictorially. The average return shown in our plots is the average return obtained after N initial exploration steps, where N is value under Number Exploration Steps, after which the agent should attempt to solve the task.
A.5 TIME/SPACE COMPLEXITY
HFR incurs an O (NC) cost every time trajectory is collected, where N is the number of tasks in the training task distribution and C' is the cost of computing the utility of relabeled trajectory. Computing the utility of relabeled trajectory involves passing this trajectory through the encoder to generate a latent embedding, sampling a batch initial states from the replay buffer, passing these initial states and embedding to the policy network generate batch of actions, and evaluating the Q function on these mate-action pairs (Algorithm 1). HFR takes O (NN space to store the relabeling distribution.
A.6 ANALYSIS OF TRAJECTORIES IN THE FOUR-CORNERS CORNERS ENVIRONMENT
In this section, we analyze sample trajectories the Cour-Corners environment, as shown in Figure 11. Each trajectory's original goal is different from the goal located in the quadrant explores. In Table
14

---

(a)
(b)
(c)
(d)
Figure 11: Example trajectories from our Four-Comett Environment. The goal for the task drawn from HFR's ss relabeling distribution is shown in green, while the red square represents an area of large negative reward for that task. Encountering the red square informs the agent of the goal location. It is clear that trajectories (a), (b), (c), and (d) are most useful for meta- seetattainnng on how to reach the top left goal, top right goal, bottom left goal, and bottom right goal, respectively. HFR correctly relabels these trajectories with the appropriate task, while HIPI fails to do so due to the large negative returns achieved under them. Table shows the trajectory returns that HIPI uses, and the unnormalized Q-values that HFR uses, to sample task to relabel trajectories (a), (b), (c), and (d) with.
we show trajectory returns under the different tasks for HIPI and the unnormalized post-adaptation Q-values under the different tasks for HFR. We note that for the first trajectory, HIPI fails relabel using the top left goal due to the highly negative return (-44) it achieves for that goal. However, HFR, by considering the post- post-adapten astaaadptnooon returns after using the trajectory for adaptation, correctly relabels the trajectory for the top left goal (-727 76 is the highest value in the column). This is beneficial, as despite the highly negative return, the trajectory extremely informative about the top left goal and should be used for meta-training on it. The same phenomenon is seen for the other 3 trajectories.
Task
HFR
HIPI
Task
HFR
HIPI
Top Left
722.46
44
Top Left
1015.28
20
(a)
Top Right
-957.61
-20
(b)
Top Right
-756.99
-58
Bottom Left
-98533
-20
Bottom Left
-935.45
20
Bottom Right
937.60
-20
Bottom Right
-931.33
-20
Task
HFR
HIPI
Task
HFR
HIPI
Top Left
9116.00
20
Top Left
-1243.47
20
(c)
Top Right
799.887
20
(d)
Top Right
-1202.90
-20
Bottom Left
-780.03
- 28
Bottom Left
-1249.10
-20
Bottom Right
-829.24
-20
Bottom Right
-760.03
-30
Table 3: Unnormalized trajectory returns (HIPI) and post-adaptation Q-values (HFR) for trajectories
(a). (b), (c), (d) (Figure 11). The task for which the trajectory is most informative in bold. Given task vi and trajectory HIPI will not relabel trajectory 7 with task vé if the return of r is low under di even though may be extremely useful for meta-training on th. By considering post-adaptation returns, HFR does not suffer from this issue.
A.7 LEARNING REWARD FUNCTION
In this section, describe how learn reward function in the setting where do not assume can query the true reward function rw (Br, oe) for individual transitions, as explored in Subsection $5.2. We learned reward function ri (Se, oe) parameterized by neural network with parameters relabel trajectories. This network takes in task i and state and action 8t, 0t input. We train our reward function using the mean-squared error:
min E Rr (r (st, Tw (st, a:))
(11)
This update rule is applied during every training update during the meta-training process.
15

---

A.8 ADAPTATION META-TEST TIME
In Algorithm 3, we include the meta-test time procedure of PEARL used in HER. In PEARL, adaptation procedure So corresponds to a forward pass through encoder Qá-
Algorithm 3: PEARL Meta-testing
Input Test (1))(
Number of exploration trajectories (K)
Initialize context Co {}
for k 1 to K do
Sample embedding
qo (z|cx)
Collect trajectory = {(s,, ae, me, 8141 Ii using policy no (-1s,2) 2) Append trajectory Tu context: Cy Co U.k
end
Sample embedding z ~ qu, (zlcy)
Compute adapted policy n. = fe (TO, Go, Ce, = T0(-18,2) Roll out adapted policy s' measure returns on
A.9 IS THE LEARNED Q FUNCTION CORRECTLY CAPTURING TASK INFORMATION?
In this section, we examine whether it makes sense to use the learned Q function as the source of relabeling signal. To do this, we train HFR to convergence, freeze the weights of the HFR context encoder qs. and take set of trajectories TN { j==1 from the training process that have not been relabeled.
We split T'y into two collections of trajectories, which we denote as Timis and T Ne.. We construct Tr by taking the trajectories from T Tiu and relabeling them using HFR (Algorithm 1). We then use our trained HFR context encoder 9s generate three collections of context embeddings: Z N
{2 ~ 4s (-IT): T E I = {2 9s (-IT) T E TN... Z = {2 ~ go (-IT): : TR).
We then train a N-way classifier (1))2) on N Given context embedding z, f should correctly identify the task y that was generated from. If 2 was generated from relabeled trajectory, then should classify as coming from the task that the trajectory was relabeled with. If was generated from trajectory that was not relabeled, then f should classify coming from the task that the trajectory was originally generated for. We evaluate the accuracy of our classifier on ZN... and ZR. HFR is correctly capturing task information during relabeling, would expect the accuracy on Za to be greater than or equal to the accuracy on ZN.... For clarity, we illustrate this entire procedure in Algorithm 4.
Algorithm 4: Context Embedding Classification
Input Trained context encoder(ga)
Trajectories(TTw
Output: Accuracy of classifier original trajectories
Accuracy of classifier relabeled trajectories
Split T' into TNmm and 1 Vuu.
Construct I'x by relabeling trajectories in 7 T'M using HFR (Algorithm 1)
Generate Z Nusi = (2 qe (-IT) : T E TN.m Generate Z = {2 Qo (-IT) :T TNu1
Generate ZA = =2 qo (T
Train classifier f (ı12) on embeddings from Z ZN Return accuracy of f on ZNea. accuracy of ff on Zn
We conduct this experiment on the Four-Corners environment, as well as the Cheetah-Vel-Sparce Vel-Sparse environment with 10 training tasks. We parameterize ff as a feed-forward neural network with two hidden layers, each with 200 units, and ReLU activations. We train f using cross-entropy loss for 500 epochs training set of non-relabeled trajectories and report the accuracy relabeled trajectories and a test set of non-relabeled trajectories at the end of training. Our results are shown in Table 4. We find that the accuracy relabeled trajectories is higher than the accuracy non-relabeled
16

---

Table 4: Accuracy classifier on relabeled and non relabeled trajectories
Environment
Cour-Coreers Accuracy Relabeled Trajectories 89.65%
Accuracy Non-Relabeled Trajectories
76.98%
Cheetah- Vel-Sparse
74.33%
57.58%
trajectories. This indicates that HFR is indeed correctly capturing task information when relabeling our relabeled trajectories are easier identify as coming from specific task than the non-relabeled trajectories and are thus more useful for adaptation.
A.10 OBJECTIVE EQUIVALENCE
In this section, show that the objective in Equation 5 is equivalent to the reverse-KL KL diver- gence minimization objective: ming(r,w) DRL. [q (T, +) llp (r, v)), where y (T, 4b) = ((Tl1) and
p (T, b) p()). Further, p(T(() ) defined in Equation
min DrL la (T. +) 11l (r, bb] min Ber,)) log g(T, v)) p(T, 1b)
min E (()a(T10)) log ((T
p(T
IT ((ae(se:))
min E (PW)a(T log
(Ui(T))
max EE p(())
E E (U (T) + Hy(t)e)
where Hy(r(e) denotes the causal entropy of the policy associated with g(rlv)), that (()
A.11 COMPARISON TO PRIOR WORK IN EXPERIENCE RELABELING
A.11.1 COMPARISON WITH MIER (MENDONCA ET AL., 2020)
Mendonca et al. (2020) propose to tackle meta-RL via a model identification process, where context- dependent neural networks parameterize the transition dynamics and the rewards function. These models are trained with supervised meta-learning For a new task (at meta-train/test time), the context adapted encapsulate the task information. Although MIER uses experience relabeling, the scope and the purpose of this relabeling is very different from HFR:
Relabeling during meta-train vs meta-test. In MIER, experience relabeling is performed only at meta-test time, with the purpose of consistent adaptation to the out-of-distribution tasks. Crucially, there is relabeling sharing of data amongst tasks during the meta-train time. In contrast, the motivation the relabeling in HFR to improve the sample-efficiency of the meta-traie meta-training phase. HFR does not modify the meta-test time behavior of the underlying meta-RL algorithm.
Problem setup. MIER considers a setup where the meta-test tasks are cout-of-distribution" w.r.l. the meta-train task distribution. The authors, therefore, introduce relabeling handle this discrepancy. HFR operates in the standard meta-RL. setting where the meta-test tasks are "in-distributionn.. MIER does not perform relabeling in this case (please see Figure 2 in their paper).
Relabeling strategy. Lastly, the relabeling strategy is very different for the two approaches. While MIER relabels all of the meta-training data for the new meta-test time task, HFR selectively relabels the trajectory for new task during meta-train, basing the selection on the idea of the utility function and a relabeling distribution q (z)()))
A.11.2 COMPARISON WITH BOREL (DOREMAN ET AL., 2020)
Dorfman et al. (2020) propose a method for offline meta-RL where the meta-learning agent is provided with fixed trajectory data collected in different environments and the goal of the agent is to
17

---

learn to maximize returns in an unseen environment from the same task distribution. The authors highlight the issue of "MDP ambiguity", which refers to the difficulty in MDP identification using the belief from a VAE encoder trained with offline data. To mitigate this challenge, the authors propose reward relabeling scheme that replaces the reward in a trajectory from an MDP-i in the offline data, with rewards from randomly chosen MDP-j. There two critical differences from work:
Motivation for relabeling. Relabeling BOReL is done alleviate the "MDP ambiguity" issue, which the authors mention is specific to the offline meta-RL RL setting. HFR operates in the online meta-RL setting and our motivation for relabeling to improve the sample- efficiency of the online meta-training phase via data sharing.
Relabeling strategy. Perhaps the more important difference is in the strategy used to select the task for which a given trajectory should be relabeled. While BOReL chooses the task at random. HFR provides principled procedure of computing relabeling distribution q(1) 7 over the tasks using the concept of the utility function, and sampling task from this distribution for the relabeling. Empirically, we observe that this approach is more performant than random task relabeling. Figure in the paper includes the learning curves for both the methods HFR outperforms Random in all those tasks.
18


Reviews:
[{"review_id": "rKGyAxevTk-", "reviewer": "Reviewer_P7KG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n\n1. Usefulness of relabelling\n\nThe problem considered is an important one, since even though hindsight relabelling is standard in multi-task RL, and has been shown to enable learning on sparse reward environments (which are otherwise very difficult to solve), this approach hasn’t been applied to the meta-RL setting yet. This is despite the fact that meta-RL also considers a multi-task distribution, and can benefit from explicitly using data for a different task and relabelling it under the corresponding reward function. The mathematical formulation of the approach closely follows HIPI[2], with the difference that post-adaptation trajectory return is considered instead of current trajectory return, to be aligned with the meta-learning objective. The authors show experimentally that current meta-RL approaches do not begin to make progress on sparse-reward tasks, showing the importance and effectiveness of relabelling.\n\n2. Extent of Evaluation and Analysis\n\nThe paper includes evaluation on 5 different sparse reward environments, 3 dense reward environments which show that the relabelling scheme offers benefits over current meta-RL approaches mainly in the sparse reward setting. The authors also include ablations/analysis of specific components, such as using a learned reward instead of the true reward, using hardmax instead of softmax for sampling the relabelling task etc. The paper is well written, the presentation is clear and well-motivated. \n\nWeaknesses \n\n1. Small performance gap with HIPI, Simplistic Environments \n\nOut of the 8 experimental domains chosen, the performance of the proposed approach (HFR) is significantly better than HIPI on only two domains (ant-goal and sawyer-push). This indicates that most of the benefit is coming from the relabelling scheme for most environments, either because the adaptation procedure doesn't actually lead to better performance, or because the environments are too simple to require a lot of adaptation to held-out tasks. Given that performance is much better than HIPI on the hardest environments (ant-goal and sawyer-push), I am inclined to think the issue is the latter instead of the former, which can be addressed by evaluating on harder environments. \n\nThese could include other single-family robotic tasks from meta-world (eg: sawyer-door-open, sawyer-box-close etc). Even better would be meta-training across task families, using MetaWorld ML-10 or ML-45. This would test adaptation to tasks that are semantically different, and would make the paper a lot more compelling. \n", "summary_of_the_paper": "The paper proposes an approach for using data relabelling for meta-RL for better sample efficiency and to enable training on sparse reward environments. Specifically the proposed method combines Pearl[1] with a modified version of HIPI[2], where the trajectories chosen for relabelling are effective for adaptation, and not necessarily high in reward themselves.\n\n[1] : Efficient Off-Policy Meta-RL vis Probabilistic Context Variables (Rakelly et al.)\n[2] : Rewriting History with Inverse RL (Esyenbach et al.) \n\n\n\n\n", "summary_of_the_review": "The approach introduces relabelling (which has already shown to be important in multi-task RL) in meta-RL, and shows superior performance on sparse reward environments. The paper would be more compelling if it included evaluations on more challenging environments, to establish the importance of the adaptation component. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "r6-ZwsHymxJ", "reviewer": "Reviewer_YEZ4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper overall. The motivation is sound: meta-RL is almost by definition slow, by using a slower timescale for meta-learning than the fast learning or adaptation, and so data-efficient methods are key. Task relabelling, like in multi-task or goal-conditioned RL, makes a lot of sense in this context.\n\nThe particular proposed method seems reasonable, although I have some concerns about the detail of the exposition – I found section 4.1 fairly difficult to follow.\nFirst, it’s not 100% clear to me how to map eq(1) to the objective written in terms of utilities because eq(1) does not define where tau_pre come from. Presumably this is just following pi_theta, and the conditioning of q(tau | psi) on psi only results in differing rewards in tau, not differing state-action sequences?\nThen, most importantly for understanding this section, I don’t follow why the objective for (theta, phi) should be maximised by adjusting this q for fixed (theta, phi). The paper says this “facilitates alignment with the goals of the meta-learner” but I’m not sure what this means.\nThe derivation then continues in a very brusque manner. I’m not a fan of “it is easy to show”: in general if it is easy rather write it yourself (in an appendix if need be for space) or cite appropriately. Eventually we arrive at an ‘optimal’ relabeling distribution but I don’t understand in what sense it is optimal due the previous confusion.\nIt could be I’m missing something simple or these things are all straightforward and clear to a reader with the right context.\nHowever, I encourage the authors to substantially clarify and elaborate this section to engage with a broad audience.\n\nThe issue that I have more intuitively with the method is that the optimal task inference should depend on the true distribution of tasks. By altering this distribution through relabeling, it seems it would change the optimal (theta, phi). Can the authors elaborate on whether or not this should be a consideration, perhaps by clarifying the exposition given in S4.1?\n\nThe implementation of the approach is quite neat. I like the use of PEARL’s particular type of value function to efficiently estimate the value of the post-adaptation policy without sampling any fresh transitions.\n\nI also like the empirical study. The performance gains seem substantial in several tasks, and I appreciate the credible baselines which are more naive but not just vanilla PEARL without any task relabeling.\nI appreciated the informative ablations.\n\nMinor comments or questions:\n - Paragraph 2 of the intro says meta-RL is \"inherently on-policy\": this is incorrect.\n - Why relabel with just one task sampled from q(psi|tau)? Why not several samples, or weighted samples?\n - In Algo 1, maybe use a different letter to distinguish N in GetLogPartition and in N in ComputeUtility\n - The paper would benefit from more details on the setup with a learned reward function.\n\n----------------------\n\nThe authors were able to clarify the points that had confused me in my initial reading. I am persuaded that the optimal meta-learned solution will not be biased by the proposed relabelling; and that the derivation is sound. Optimising the relabelling distribution for the immediate post-adaptation returns makes sense as a somewhat myopic heuristic to accelerate meta-learning.\nI also appreciate the additional experiment carried out for HrwG. Further, while the connection to prior work is close in many ways, I believe the adaptation of the method for this context is sufficiently novel and effective to warrant acceptance.", "summary_of_the_paper": "This paper studies task relabelling in hindsight to increase the efficiency of meta-reinforcement-learning.\nThe authors propose a strategy for calculating a distribution of tasks for which a particular batch of data would be useful for adaptation, and sample from this distribution to construct a relabelled batch which augments the training data.\nThe authors show empirically that this improves sample efficiency over more naive relabelling schemes, particularly for sparse reward tasks.\nA series of ablations further justifies several design decisions or investigates robustness to hyperparameters.", "summary_of_the_review": "The work is well-motivated intuitively, but the mathematical justification for the specific method is difficult to follow (so I cannot quickly verify its soundness).\nThe empirical study is well done overall, so I lean to accept the paper but would likely increase my score and confidence if the authors can clarify the theoretical motivation for their relabeling strategy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PB1XjctYOmI", "reviewer": "Reviewer_vofF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(a). I am a little concerned with the novelty of the paper.  The authors made an interesting point that compared to multi-task RL, the objective in meta-RL is to learn to learn a new task, so the metric of interest for relabeling trajectories in meta-RL is their usefulness for task-identification, rather than the returns like in multi-task RL (Section 4, page 4). The paper’s main contribution is a trajectory-relabeling algorithm in meta-RL setting based on this intuition. However, the proposed algorithm still seems to try to relabel trajectories based on their returns on other tasks. Specifically, the resulting learning objective Equation (8) is quite similar to that of HIPI (Eysenbach et al., 2020), as the utility function similarly aims to maximize the expected return of the trajectory on the new task, which I think would be exactly the total reward of trajectory on the new task in the multi-task setting. This seems to be contradictory to what the papers says about the difference of relabeling trajectories of previous work in multi-task rl and relabeling trajectories in meta-RL proposed in this work. Plus, the actual implementation looks like a straightforward combination of HIPI and PEARL (Rakelly et al., 2019) to me.\n\n\n(b). The proposed algorithm makes an important assumption that the actual reward function is known for each task in this meta-RL setting. This makes the meta-RL problem setting confusing as the paper also says the tasks share the same dynamics and only differ in the reward function. But like the paper mentions, there may exist some scenarios where this assumption is reasonable. The experimental results show that the proposed algorithm improves performance compared with other relabeling methods (HIPI and random) in such settings. First, I think there might exist methods that, under the same assumptions, are able to do better than meta-RL algorithms. For instance, one can relabel all the collected data with the new task reward and run some kind of offline RL algorithms on it (without meta-learning). In my opinion, that would be another good baseline to strengthen the author’s claim under the same assumptions. Secondly, in the experiment section the paper mentions a variant of the proposed method that also considers the scenario where the true reward function cannot be queried for individual transitions. This is a more interesting setting and I think the authors should elaborate on this part (e.g. how do you learn the reward functions? Is there anything specifically designed for meta-RL settings?) And it would be better to show more experimental results under these settings. For instance, the author can compare to the state-of-the-art meta-RL algorithms under such more common settings, especially in sparse-reward environments where the proposed algorithm is more competent. Potential baseline: 1. MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration, Zhang et al., ICML 2021;  3.  Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning, Fu et al., AAAI 2021. \n\n\n(c). I have some minor comments listed below:\n\n1. Equation (4), On the left hand side, $j$ should be superscript instead of subscript.\n\n2. Figure 4, the actor $\\pi$ and critic $Q$ should be parameterized using different denotations, instead of jointly using $\\theta$.\n\n3. Figure 6, could the authors explain why randomly relabeling the trajectories can achieve competitive performance or even better performance than HIPI and PEARL?\n\n4. The adaptation procedures listed on page 6 before equation (9) confuse me. Could the authors provide an algorithm bar for the meta-test phase (maybe in appendix)?\n", "summary_of_the_paper": "The paper proposes a trajectory relabeling method for meta Reinforcement Learning (meta-RL), aiming to share some of the collected trajectories to improve sample efficiency during meta-training. The relabeling method is built on HIPI (Eysenbach et al., 2020). Instead of relabeling the trajectory based on the total reward as in HIPI, the paper argues that in meta-RL, the metric of interest for trajectories from different tasks is their usefulness for task-identification rather than returns. The paper further proposes a meta-RL algorithm based on PEARL (Rakelly et al. 2019). The experimental results on several sparse-reward tasks show that the method outperforms other relabeling methods as well as PEARL.", "summary_of_the_review": "The idea in the paper is well presented and carefully investigated. The proposed method is simple and effective. However, I am not quite convinced about the novelty of the proposed idea and I think the experimental settings can be improved to strengthen the paper’s claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E9qqIUbh-G", "reviewer": "Reviewer_HrwG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this paper presents an interesting idea for sharing data between tasks of a meta-RL problem. The paper is well written and the ideas are presented clearly.\n\nPros:\n\n1\\. I find the main insight of the paper simple and intuitive. The idea that we need to relabel not according to how much return we achieve but according to how much information we can gather for task identification gives us a clear distinction between multi-task RL and meta-RL. The derivation of relabeling according to the exponentiated post adaptation return follows naturally.\n\n2\\. The ablation study in the paper is very informative. The ablation study gives us clear comparisons, showing us which component is more important. From the ablation study, it seems that using the partition function and softmax relabeling distribution are the most important components.\n\n3\\. The paper is well written. The insights, ideas, algorithms and experiments are easy to follow. \n\n\nCons:\n\n1\\. I am somewhat skeptical about the approach of using the learned Q function to estimate return after adaptation. In the base meta-RL algorithm PEARL, the context encoder is trained to identify the task from a distribution of tasks, producing a posterior distribution of context z corresponding to that task. This means that given the relabeled trajectory, even if the context encoder predicts the context corresponding to a wrong task, as long as the produced context is within the distribution of tasks, the expected return will still be high because the policy is trained to do well also on that wrong task. Therefore it is not clear to me why using the learned Q function is a good way to estimate return on that specific task. \n\nIn order to verify this, I’d like to ask the authors to include the following experiments. First train the proposed algorithm to convergence and freeze the weights of the context encoder. Then train a N-way classifier on top of the context encoder to classify the context into one of N training tasks without using any relabeled trajectories, and then report the accuracy of the classifier. Finally relabel the trajectories according to the proposed method and report the classifier’s accuracy on top of the relabeled trajectories. If the relabeling mechanism using the learned Q function is correctly capturing the task information, we would see that the classifier’s accuracy on the relabeled trajectories is comparable to that on the true trajectories. In fact, this experiment could also lead to an even simpler relabeling strategy: directly use the true task’s probability under the classifier’s prediction as the source of relabeling signal. \n\n\n2\\. The empirical performance of the proposed method does not seem very strong. Only in 3 of 5 sparse reward tasks the proposed method significantly outperforms the baselines, and the proposed method does not show much improvement on dense reward tasks.\n\n\nGiven these limitations, I’m leaning slightly towards not accepting the paper. I’d highly encourage the authors to conduct the experiment I suggested in order to verify that the proposed method is indeed capturing the task information correctly.\n\n\n## Update After Author Response\nThe authors conducted additional classifier experiments I requested and the results suggest that using learned Q function to estimate returns is highly informative about the task. Therefore my main concern about the proposed method has been addressed, and I'm now leaning towards accepting the paper.\n\n", "summary_of_the_paper": "This paper proposes a way to share data across different tasks in meta-reinforcement learning (meta-RL), where the data from one task is reused in another task by relabeling the rewards. Based on the HIPI method ([1]), the authors construct a relabeling distribution to relabel the pre-adaptation trajectories from one task to be used for another task. The relabeling probability of a trajectory is chosen to be proportional to the exponentiated utility function, which is defined as the expected return after the agent uses that trajectory to adapt. In practice, the post-adaptation return is approximated using the learned Q function. The authors apply this relabeling distribution to PEARL, an existing off-policy actor-critic style meta-learning algorithm.\n\nThe authors conduct experiments on simulated robotics experiments. The results suggest that the proposed method outperforms prior methods on sparse reward tasks, while performing roughly the same on dense reward tasks. \n\n\nReferences\n\n[1] Eysenbach, Benjamin, et al. \"Rewriting history with inverse rl: Hindsight inference for policy improvement.\" arXiv preprint arXiv:2002.11089 (2020).\n", "summary_of_the_review": "The paper presents an interesting idea about reusing data across tasks in meta-RL. The idea is very intuitive and the paper is well written. However, I’m not sure about whether the approach used to implement the idea of the paper really does what the authors claim it does. Therefore I’d like to see more evidence before I can recommend accepting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Hindsight Foresight Relabeling (HFR), an approach for reward relabeling for meta RL. The main contribution is a measure of how useful a given trajectory is for the purpose of meta-task identification as well as the derivation of a task relabeling distribution based on this measure.

Reviewers agreed that the paper tackles an interesting problem and found the main insight to be simple and intuitive. While the initial reviews raised some concerns regarding novelty, the performance gap, and using the learned Q-function to estimate post-adaptation returns the rebuttal did a good job of addressing these concerns. Overall, the paper proposes a non-trivial extension of hindsight relabeling to meta RL and while the results could be stronger I think the paper provides useful ideas and insights so I recommend acceptance as a poster.
Decision: Accept (Poster)