Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL
Authors: Charles Packer
Pieter Abbeel
Joseph E. Gonzalez
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta Reinforcement Learning
Meta-RL
Deep RL
Hindsight
Sparse Reward

Abstract: Meta-reinforcement learning (meta-RL) has proven to be a successful framework for leveraging experience from prior tasks to rapidly learn new related tasks, however, current meta-RL approaches struggle to learn in sparse reward environments. Although existing meta-RL algorithms can learn strategies for adapting to new sparse reward tasks, the actual adaptation strategies are learned using hand-shaped reward functions, or require simple environments where random exploration is sufficient to encounter sparse reward. In this paper we present a formulation of hindsight relabelling for meta-RL, which relabels experience during meta-training to enable learning to learn entirely using sparse reward. We demonstrate the effectiveness of our approach on a suite of challenging sparse reward environments that previously required dense reward during meta-training to solve. Our approach solves these environments using the true sparse reward function, with performance comparable to training with a proxy dense reward function.

Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL
Charles Packer UC Berkeley
Pieter Abbeel UC Berkeley
Joseph E. Gonzalez UC Berkeley
Abstract
Meta-reinforreement learning (meta-RL) has proven be successful framework for leveraging experience from prior tasks to rapidly learn new related tasks, how- ever, current meta-RL. approaches struggle learn in sparse reward environments. Although existing meta-RL algorithms can lear strategies for adapting to new sparse reward tasks, the actual adaptation strategies are learned using hand-shaped reward functions, require simple environments where random exploration is sufficient to encounter sparse reward. this paper, we present a formulation of hindsight relabeling for meta-RI, which relabels experience during meta-training to enable learning to learn entirely using sparse reward. We demonstrate the ef- fectiveness of our approach environments that previously required dense reward during meta- veta-training to solve. Our approach solves these environments using the true sparse reward function, with performance comparable to training with a proxy dense reward function.
suite of challenging sparse reward goal-reaching
Introduction
Reinforcement learning (RL) has seen tremendous success applied to challenging games (Mnih et al. Silver Silver al. 2017) and robotic control (Lillicrap et al. 2015 Levine et al. 2016), driven by advances in compute and the use of deep neural networks as powerful function approximators in RI. algorihmss.. However, agents trained using deep RL often struggle meaningfully utilize past experience to learn new tasks, even if the new tasks differ only slightly from tasks seen during training (Zhang al. 2018 Packer al. 2018 Cobbe et al. 2019). In contrast, humans adept at utilizing prior experience to rapidly acquire new skills and adapt to unseen environments.
Meta-reinforcement learning (meta-RL) aims to address this limitation by extending the RL frame- work to explicitly consider structured distributions of tasks (Schmidhuber) 1987 Bengio et al., 1990
Thrun & Pratt 1998). Whereas conventional RL is concerned with learning single task, meta- RL is concerned with learning to learn, that is, learning how to quickly learn a new task by leveraging prior experience related tasks. Meta-RL methods generally utilize limited amount of experience in new environment to estimate latent task embedding which conditions the policy, or to compute a policy gradient which is used to directly update the parameters of the policy.
major challenge in both RL and meta-RI. learning with sparse rewards. When rewards are sparse delayed, the adaptation stage in meta-RL becomes extremely difficult: inferring the task at hand requires receiving reward signal from the environment, which the sparse reward setting only happens after successfully completing the task. Due to this inherent incompatibility between meta-RL and sparse rewards, existing meta-RL algorithms that consider the sparse reward setting either only work in simple environments that do not require camporally-extendee adaptation (Duan et al. [2066] Stadic et al. 2018). ) or train exclusively using dense reward functions (Gupta et al. 2018b Rakelly al. 2019), which are designed to encourage an agent to learn adaptation strategies that can be directly applied to the original sparse reward setting.
exploration strategies for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Goal-conditioned RL [ 88,
Metn-RL 3. (x 7, =())82))
T
g'
(a) Hindsight relabeling in goal-conditioned RL.
(b) Hindsight Task Relabeling (HTR) meta-RL.
Figure I:
In goal-conditioned RL (a), an agent must navigate to a provided goal location g (filled circle, revealed to the agent). An unsuccessful attempt for goal g provides no sparse reward signal, but can be relabelled as successful attempt for goal g'. creating sparse reward that can be used to train the agent. In meta-RL (b), the task T (i.c., goal, hollow circle) is never revealed to the agent, and instead must be inferred using experience on prior tasks and limited experience (T1.t-1) on the new task. In (b), there is shared optimal task 7' to relabel all attempts with. HTR relabels each attempt 7 under its own hindsight task T', and modifies the underlying meta-RL training loop to learn adaptation strategies
the relabelled tasks. Note that include multiple trajectories T in (b) vs a single trajectory in (a) to highlight the adaptation stage in meta-RL. which does not exist in goal-conditioned RL and requires significantly different sampling and relabeling procedures.
In this paper, we show that the concept of hindsight relabeling (Andrychowicz al. 2017) from conventional RL can be applied in meta-RL to enable learning to learn in the sparse reward setting, Our key insight is that data collected on the true training tasks can be relabelled as pseudo-expert data for easier hindsight tasks, bootstrapping meta- trata-trining with the reward signal needed to train the agent. We introduce new algorithm for off off-policy meta-RL, which we call Hindsight Task Relabeling (HTR), and demonstrate its effectiveness by achieving state-of offthof--heeatt performance collection of challenging sparse reward environments that previously required shaped reward solve. Not only is our approach able to learn adaptation strategies only using sparse reward, but it also learns strategies with comparable performance to existing approaches that use shaped reward functions.
2 Related Work
In meta-RL, an agent learns an adaptation strategy by repeatedly adapting to training tasks in the inner loop of meta-training, with the hope that the learned adaptation procedure will generalize to new, unseen tasks during mela-testing. Context-based methods use recent experience (i.e., context, in the form of trajectories or transitions) mainly in how this embedding is computed: Duan et al. (2016); Wang et al. (2016); MMshrr et al. (2018) propose aggregating context into the hidden state of the policy, whereas Rakelly et al. (2019) propose explicitly feeding the task embedding as input to the policy. Gradient-
new task estimate a latent task embedding, and differ
methods use
recent context to update differentiable hyperparameters (Xu et al. 2018b), loss functions (Sung et
2017 Houthooft et al. 2018), or to directly update policy parameters (Finn et al.
2017: Stadie et al. Xu al. 2018a Zintgraf et al. 2018 Gupta et al.) 2018b Rothluss et al.
2018
2019).
Many of the aforementioned approaches struggle to learn effective exploration strategies for tasks with sparse or delayed rewards. Methods that directly update the policy either implicitly (e.g.. with hidden state) or explicitly (e.g.. with gradients) are generally unable to learn policy that meaningfully explores, since the primary source of randomness action-space, and thus time-invariant, [Guptal ett al. (2018b) and |Rakelly et al. (2019) propose using probabilistic task variable that is sampled once per episode, which makes the primary source of variability task-dependent and enables temporally- extended exploration. While this approach enables effective adaptation in the sparse reward setting, both Gupta al.) (2018b) and Rakelly et al. (2019) learn the actual adaptation strategies using shaped (dense) reward functions during meta- veta-training. In the environments they consider, the adaptation strategies learned using dense rewards generalize to sparse rewards despite the difference reward structure: however, engineering a suitable reward function can be costly (Ng et al.) 1999) and error prone (Clark & Amodei| 2016). and thus is highly desirable devise a mechanism for learning to leamn in the sparse reward setting that does not require auxiliary dense reward function.

---

Our proposed method is closely related to prior work in unsupervised meta-RL and goal generation. Unsupervised meta-RL (Jabri et al. 2019 Gupta et al. 2018a) considers the meta-RL setting without access to training task distribution; training tasks are instead gelf-generated with the aim of learning skills useful for the test task distribution. Our proposed method also generates a curriculum of training tasks, but with the intent of learning adaptation strategies in the sparse reward setting, as opposed to learning transferable skills in a setting with no rewards at all. Several methods exist for curriculum generation in the context of goal-conditioned policies, including adversarial goal generation (Florensa)
2018) and goal relabeling (Kaelbling 1993] Andrychowicz et al. 2017 Levy et al.) 2017 Nair ctt al. et al. 2018). Recent work (Li et al. 2220;; Eysenbach et al. 2020) has proposed the use of inverse RL to generalize goal relabeling to broader families of tasks. Unlike prior work on goal and task relabeling that use goal- or task- conditiood policies, we focus on the meta-RL setting where the task
is unknown to the policy and must be inferred through deliberate and coherent exploration.
3 Background
In this section we formalize the meta-RL problem setting and briefly deseribe two algorithms which our approach, Hindsight Task Relabeling, builds on: Probabilistic Embeddings for Actor-critic RL (PEARL) and Hindsight Experience Replay (HER).
3.1 Meta-Reinforcement Learning (Meta-RL)
Conventional RL assumes environments are modeled by a Markov Decision Processes (MDP) M, defined by the tuple (S, A. Y, Pu. 7') where S is the set of possible states, A is the set of actions, p: S A R20 is the transition probability density, r S A - the reward function, y is the discount factor, Pa : S + Rgo is the initial state distribution at the beginning of each episode, and T is the time horizon of an episode (Sutton Barto 2017).
Let St and a be the state and action taken at time t. At the beginning of each episode, So ~ po(-). Under stochastic policy mapping sequence of states actions, a, T(ar st. ,8) and 1 ~ p(S St, az), generating a trajectory T = {S,, a r(s,, a,)], where t= 0, ..... Conventional RL algorithms, which assume the environment is fixed, learn to maximize the
where r, r(s:, ar). They often utilize the concepts of a value function V (s), the expected reward conditional on So = and state-action value function (T, a). the expected reward conditional on So = and au a.
expected reward per episode (T) ET
Meta-RL. algorithms assume that there is a distribution of tasks P(T), and usually maximize the expected reward over the distribution, EE (pT) [J]( The distribution of tasks corresponds to
distribution of MDPs M (T), where T can paramaterize an MDP's dynamics or reward. In this work MDP with different reward function under the
we focus on the latter case, where each T implies same dynamics, The agent is trained on a set of training tasks Treain ~ p(T) during meta-training, and evaluated
held-out set of testing tasks Trest p(T) during meta-testing.
An important distinction between meta-RL and multi-tasè or goal-conditioned RL is that in meta-RL, the task is never explicitly revealed to the agent or policy through observations s. Instead, π must explore the environment to infer the task to maximize expected reward. This crucial difference is one reason why Hindsight Experience Replay cannot be directly applied to the meta-RI. setting.
3.2 Off-Policy Meta-Reinforcemmet Learning
We demonstrate the effectiveness of our approach by building top of Probabilistic Embeddings for Actor-critic RL (PEARL) by Rakelly et al. (2019), an off-policy meta-RL algorithm which itself is built on top of soft actor-critic (SAC) by Haarnoja al. (2018). SAC actor-critic algorithm based on the maximum entropy RL objective that optimizes a stochastic policy π with off-policy data. In addition to the policy (actor) n, SAC concurrently learns twin state-action value functions (critics) Q. Transitions collected using the policy π are added to the replay buffer B, and Q and x are optimized using loss functions regularized by the entropy of ा.
Rakelly et al. (2019) extend SAC to the meta-RI. setting by adding a context encoder that uses recent history in an environment to estimate a task embedding, which is then utilized by the policy.
3

---

(a) Point Robot
(b) Wheeled Locomotion
(c) Ant Locomotion
Figure 2: Sparse reward environments for meta-RL. that require temporally-extender exploration. In each environment, the task (the top-left circle in (a), the green sphere in (b) and (c)) is not revealed to the agent via the observation. The agent must instead infer the task through camporall--xxtendee exploration (illustrated by the dotted lines in (a)), since reward signal is provided until the task is successfully completed. Prior meta-RL. methods such as PEARL (Rakelly et al. (219)) and MAESN (Gupta et al., 2018b) are only able to (meta-)learn meaningful adaptation strategies using dense reward functions. Our approach, Hindsight Task Relabeling (HTR), can (meta-)train with the original sparse reward function and does not require additional dense reward functions.
Specifically, PEARL uses network qo (z c) that takes as input recently collected data (i.e., context c) to infer probabilistic latent context variable Z. Samples from the latent context variable condition the actor, (a s, z). and critic, Qo a, z). Instead of single replay buffer (as in SAC), there are T separate buffers B......T. corresponding to the number of tasks sampled for Tirain. The separate per-task replay buffers enable sampling off-policy (but on-task) context during the inner loop of meta-training, which significantly improves data efficiency.
During meta-training, transitions are collected for each B, by sampling
qo(z c) and acting according to .e [a S, z, ) z is periodically sampled during data collection such that the context contains transitions collected using the policy conditioned on the prior, as well the same policy conditioned on the posterior. Gradients used to optimize qá, Ty and Qo are computed in task-specific batches across a random subset of training tasks Te (T),, each with an associated replay buffer B, During meta-testing, for each test task T; ~ p(T). an empty buffer B, is initialized, is sampled from the prior, and the policy conditioned on z is rolled out. As additional context is added to the replay buffer, the belief over narrows, enabling the policy to adapt to the task at hand.
3.3 Hindsight Experience Replay
Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) is method for off poolico goal- conditioned RL in environments with sparse reward. The key insight behind HER is that unsuccessful attempts (i.e., attempts that received no reward signal) generated via condiiinit policy can be relabelled as successful attempts for a 'hindsight' goal that was actually achieved, e.g., the final state (see Figure []. HER can be applied to any off-policy RL algorithm that uses goal-conditioned policies (where the actor and/or critic receive the desired goal as part of the state, i.e., π s, g) and Q(s, a, g)). When a transition {s,, a, r(,,, a,, g) g) is sampled from the replay buffer during training, with some probability HER rewrites the desired goal g with an achieved goal g' and recomputes the reward under the new goal. The modified transition {st, a,, rSr, ar, g') g') is used to optimize π s, g') and Q(s, a, g'). Hindsight relabeling generates an implicit curriculum: initially, sampled transitions rewritten with relatively 'easy" goals achievable with random policies, and as training progresses, relabelled goals are increasingly likely to be near the true desired goals.
4 Leveraging Hindsight in Meta-Reinforcemen Learning
Our algorithm, Hindsight Task Relabeling, applies ideas from hindsight relabeling to the meta-RL setting to enable learning adaptation strategies for tasks with sparse or delayed rewards. Similar to how policies in HER are conditioned on a goal g, policies in context-based on latent task embedding z. A crucial difference in meta-RL is that the task is hidden from the agent: the agent must infer relevant features of the task at hand using experience on prior tasks and a limited amount of experience gathered in the environment. Our key insight is that an unsuccessful experience collected in an unknown task can be relabelled as a successful experience for known hindsight task, i.e., a transition {st, ar, r(s,, a,, T)} generated under an unknown T can be rewritten under known T' as {st, a r(St, a, T')}. regardless of whether T' E Ttrasn-
me are conditioned

---

Data Collection
ti (s)
maa |s,
xxa 1,,
B
7-p(7)
Agent Training
Task Relabelling
( s)
æ7aa is
@ [sals,2
T, SIT,
By
b'= R(b, TW
b- By
Figure 3: Illustration of Hindsight Task Relabeling (HTR) in meta-RL. training loop. HTR is agnostic the underlying (off- (off-policy) meta-RL algorithm; the agent architecture and/or training specifics (e.g., the encoder é, actor и and Q-function neural networks shown in blue) can be modified independently the relabeling scheme. HTR can also be performed an 'eager' fashion at the data collection stage (as opposed to 'lazy" relabeling in the data sampling stage), see Section 3 for details.
Relabeling transitions under hindsight tasks serves as an additional source of supervision in learning the latent task embedding z < [(c) by enabling gradient-based optimization of f in the absence of meaningful reward signal (in PEARL., this embedding parameterized by the context encoder qa). Similar to how HER generates an implicit curriculum of goals, HTR generates an implicit curriculum of tasks that gradually shifts from easier hindsight tasks towards the true training distribution Trrain p(T): initially, the agent is unable recover reward signal on the true training tasks, but it can bootstrap the learning of an adaptation strategy by learning on easier hindsight tasks.
We outline our method for meta-training with Hindsight Task Relabeling in Algorithm | (task relabeling is only used during veta-training; meta-testing unchanged). We evaluate our approach using PEARL as the off oof-co meta-RL algorithm A, though the approach we describe is general to any off-policy meta-RL. algorithm, and could potentially be integrated with on-poliey context-based meta-RI. algorithms (such as RI.²) via hindsight policy gradients (Rauber et al. 2019)
4.1 Algorithm Design
Two important considerations in adapting hindsight relabeling meta-RL are (i) how choose the hindsight task, and (ii) how to sample the transitions to be relabelled. Together these choices form the task relabeling strategy S in Algorithm T HER, transitions are relabelled with goals that are optimal at the trajectory-leve (see Figure I The key difference between our method and HER is that our meta-RL relabeling scheme needs to construct batches of data that share the same pseudo-task (or real task), which is fundamentally different from standard HER. HER does not consider the setting where batches trajectories are collected in different MDPs; HER, transitions are relabeled independently with locally-optimal pseudo-goals, and batches of training data for the agent contain many different pseudo-godd -ooul (and real goals). the meta-RL setting, the agent must estimate the task from recent context of transitions with the same underlying task (unknown to the agent), and therefore relabeling should take into consideration more than a single trajectory; the agent explores an environment which has a fixed task T ~ p(T), and it accumulates context uses the context to identify the singular task and update its policy accordingly.
A naive application of hindsight relabeling to meta-RL would assign each transition in the batch its own hindsight task (e.g., give each r in Figure Pp its own T'). This a simple and straightforward way to apply HER to MI that may seem correct at the surface-level, however, this is bound to fail the RL. training process is left unchanged, since the context encoder is trained to estimate the task from collection of transitions generated in that task (not batch of transitions generated
5

---

Algorithm 1: Hindsight Task Relabelling for Off-Policy Meta-Reiifforeement Learning
Require: Off-policy meta-RI. algorithm A, training tasks T1:T ~ p(T), context encoder Φ,
actor critic Q, relabelling function Rt task relabelling strategy S, and task relabelling probability K
Initialize replay buffer B; for each training task Ti while meta-training do for each T. do
Collect data on T. according to A (e.g., by rolling out π conditioned on ()
Add data to task replay buffer B;
for each Ti do
Ph ~ [1 1) if ph K then
Select minibatch of transitions b; and hindsight task T, using strategy S sample transitions from trajectory L, and select state reached in for Tr.)
10
11
Relabel transitions in by according to Th i.e., R(b,, Th)
12
else
Sample minibatch of transitions b; B, Compute gradients and update Φ, n, and Q using b, according to
14
across several distinct tasks). A similarly flawed approach is to generate single hindsight task from random transition in the batch, and then to relabel the entire batch of transitions with same hindsight task (c.g., choose T' Figure [ using random 7 to relabel all TS with). The issue with this approach is that the hindsight task is only guaranteed to be optimal for a single trajectory, and may in fact be highly sub-optimal for other trajectories in the batch. We present two relabeling strategies, Single Episode Relabeling (SER) and Episode Clustering (EC), that are inspired by the relabeling strategy used in AAdrychhwicc et al. (2017) but are specifically designed to work in the meta-RL. setting.
4.2
Single Episode Relabeling (SER) strategy
In SER, single episode (i.e., trajectory) is randomly selected to generate the hindsight task and sample transitions from (i.e., sample N transitions from trajectory, each relahelled under new task). Despite the fact that this reduces the pool of context from the entire task buffer to single episode, this setting closely resembles meta-testing (where context drawn purely online), and we found this resampling strategy work well in practice since it results in more relabelled transitions with non-zero reward. HTR using SER outlined in Algorithm
4.3 Episode Clustering (EC) strategy
An alternative strategy to Single Episode Resampling is to cluster trajectories that satisfied similar hindsight tasks into same task buffers BR, essentially creating additional task buffers for hindsight tasks that can be used for training in-place of the real task buffers B; (which are also sampled with some probability K). For example, in goal-reaching tasks, trajectories can easily be mapped to hindsight buffers with trajectories from separate trials by discretizing the state space and creating buffer for each partition. In more complex tasks, alternative clustering approaches (e.g., learning- based) can be used to group together trajectories that can be relabeled with high reward conditioned on the same tasks. Episode clustering relabels a similar number transitions with non-zero reward to the resampling strategy, but with less duplicate transitions bat-hatch. In practice. we found the SER strategy to be far less complex and similarly effective to the EC approach. See Section 55 for an empirical comparison between SER and EC, and the supplement for more detailed comparison.
4.4 Comparison of HTR and HER
Both SER and EC are similar to the relabeling strategy used HER some respects, but differ in others: HER samples N transitions from the full replay buffer, and relabels each transition independently (N different goals). HTR with SER samples trajectory, and samples N transitions from that trajectory all relabelled with new task. HTR with EC samples N transitions from I

---

hindsight replay buffer, where each transition in that buffer has been relabeled with 1 new task. In HER, the hindsight goal for given transition is chosen by selecting random transition that occurs after the sampled transition in the same trajectory. In HTR, hindsight tasks are assigned in the same way as HER, but single hindsight task applied to an entire batch of transitions.
An important distinction between HER and HTR is that in HER, bootstrapping training with hindsight goals does not change the optimal policy, since a goal-conditioned policy with sufficient capacity should (in theory) be able to represent an optimal policy for every goal (including both original goals and hindsight goals). In contrast, the optimal exploration strategy for a particular meta- RLL environment depends on its true task distribution, and bootstrapping training with hindsight tasks may expand the training task distribution in manner that changes the optimal exploration strategy. One way to benefit from HTR's 's bootstrapping while ensuring the final learned strategy is optimal for the original task distribution is to only relabel data from tasks on which the agent has never received (non-zero) reward. Similarly, another solution is to anneal the relabelling probability K to zero during meta-training. Neither method was required to achieve the results shown in this paper, however, we expect they may be useful if applying HTR to other meta-RL RL environments.
4.5 Limitations
The key limitation of our method is that it assumes trajectories can be relabeled under new task (i.e., a mapping s -> T or -> 'I exists), which is reasonable assumption for goal-reaching environments (as studied in [Rakelly et al. (2019); Gupta et al. (2018b); Andrychowicz et al. (2017)) but may be significantly more challenging in other environments. Imagine complex robotic manipulation environment (e.g., retrieving an object from a drawer) where reward is sparse and only received upon successful completion of the entire task; in this scenario, relabeling the rero-reward transitions generated by an initial random policy into useful signal for training an optimal policy is less straightforward than the goal-ne setting, and may require explicit sub-task specification using domain expertise.
Similar to how HER is not necessarily only for goal-reaching goul-conditioned RI., HTR is not necessarily only for goal-reaching meta-RL., and a good relabeling function exists then either approach can be successfully applied in their respective setting (HER for goal-conditioned RL, HTR for me)a-RL). However, as in the original HER work, we focus our experiments on goal-reaching tasks, where the reward function is simple function of the state and can be easily repurposed for relabeling. It may be possible to relax this assumption and extend HTR to more families of tasks by employing more complex task relabeling schemes, e.g., by extending work inverse RL for hindsight relabeling (Li al., 2020 Eysenbach et al. (2020) to the meta-RL setting, or by carefully engineering reward functions specific to each environment, however we leave this to future work.
Additionally, our method adds a new hyperparameter to the underlying off-policy meta-RI. algorithm, which may need to be tuned for optimal results. The specification and tuning of a relabeling probability is also required in standard HER Ssee/Andrychooicz et al. (2017).
Experiments
In evaluating our proposed method, we aim to answer the following questions. (i) Can Hindsight Task Relabeling enable (meta-)learning of adaptation strategies in challenging sparse reward environments, where existing meta-RL RL methods fail without shaped reward? (ii) How do the adaptation strategies learned using Hindsight Task Relabeling compare to those learned using shaped reward functions? (iii) How do key implementation choices (such as relabeling probability K) affect its performance?
5.1 Environments
We evaluate our method on a suite of sparse reward environments based those proposed by Gupta jtt al. (2018b) and Raakelly et al. (2019) (see Figure In prior work, each environment exposes two reward functions: a dense reward function used during meta- vettatraiinng, and sparse reward function used during meta-testing. The key difference in our experimental setup is that we consider the setting where sparse reward is used both during meta-testing and mela-training. In each environment, set of 100 tasks sampled for meta-training, and set of 100 tasks is sampled from the same task distribution for meta-testing. The environments were each modified to increase their difficulty, such
7

---

Point Robot
Wheeled Locomotion
Ant Locomotion
Average return
3.4
stageei lucadlo
Average roturn
slaaped (urudko) spanse + BTB spons
shaped focudds)
syerse HJR
sgario e HIK sparss
sperse
-230
Vion
1950 3000 ma
2010
200 400
100
1980
200
Mota- traintg iterations
Meta-training iterations
Mcta- Miota-trainig iterations
Figure 4: Evaluating adaptation to train tasks progressively during meta-training, Y-axis measures average sparse return during adaptation throughout meta-training (shaded std dev), though the oracle still trained using dense reward. Conventional me methods struggle learn using sparse reward. Hindsight Task Relabeling (HTR) is comparable to dense reward meta- veta-training performance.
Point Robot
Wheeled Locomotion
Ant Locomotion
Average roturn
slapped laradel
Sperraa rettrmm
slaped (aruck))
Average retur
shaped (acade) HIX
sparse MIS
spurse BTS sporse
sparso
-200
-200
220 Episodes of adaptation
Episodes of adaptation
Episodes of adaptation
Figure 5: Evaluating adaptation to test tasks after meta-training. Y-axis measures average (sparse) return during adaptation using context collected online, using sparse reward only. Adaptation strategies learned with Hindsight Task Relabeling (HTR) generalize to held-out tasks as well as the oracle which is learned using shaped reward functions. Without HTR or access a shaped reward during meta-training, the agent is unable to learn a reasonable strategy.
that random exploration is far less likely to encounter sparse reward. Refer to the supplement for further details on the experimental setup.
Point Robot. A point robot must navigate 2D plane to different goals located along the perimeter of a semicireie...... The state includes the robot's 's coordinates but does not include the goal, therefore the agent must explore the environment to discover the goal. In the dense reward variant of the environment, the reward is the negative L2 distance to the goal. In the sparse reward variant, reward is given only when the agent is within a short distance the goal. An example optimal exploration strategy in the sparse reward variant is to efficiently traverse the perimeter of the semicircle until the goal is found.
Wheeled Locomotion. To test if approach generalizes more complex state and action spaces, we use several continuous control environments. In the wheeled locomotion environment, the agent must navigate to the goal distribution by controlling two wheels independent to turn. Similar to the point robot, the dense reward function is the negative distance the goal, while the sparse reward function only provides reward signal when the agent is nearby the goal.
Ant (Quadruped) Locomotion. This environment requires controlling quadruped ant with a gigeesimeniional state and action space. The task distribution and reward functions are the same as the point robot and wheeled locomotion environment, however, exploration requires coordinating movement with four legs navigate specific locations.
5.2 HTR enables meta-training using only sparse reward
To answer question (i), we compare our hindsight task relabeling approach (using PEARL as the base meta-RL algorithm) to standard PEARL, as well as an oracle PEARL which uses the dense reward function during training. Our oracle baseline is equivalent to the standard meta-RL. setup in the 'sparse reward' experiments in Gupta et al. (2018b) and Rakelly et al. (2019), despite it never training on the sparse reward function. Note that the term 'oracle" is somewhat of a misnomer, since training proxy dense reward function designed to aid the learning of an RL agent (to be evaluated
8

---

Point Robot
Wheeled Locomotion
Ant Locomotion
Shaped reward
Sparse reward
Sperse roward HTR
Figure 6: Visualizing exploration behavior learned during meta- meta-training training using 300 pre-adaptation trajectories (i.e., sampled from the latent task prior). In the sparse reward setting, without HTR (middle row) the agent is unable to learn a meaningful exploration strategy and appears to explore randomly near the origin. With HTR (bottom row), the agent learns explore near the true task distribution (grey circles), similar to an agent trained with shaped dense reward function (top row).
on the sparse reward function) is inherently optimizing a different objective. Optimizing an RL agent for proxy objective has been shown lead to inadvertently bad performance the true objective (Clark & Amodei 2016), and therefore it often better to directly optimize for the true objective (e.g., the original sparse reward function) when possible.
As discussed in prior work, meta-learning on sparse reward environments proves extremely challeng- ing: in Figures and we see that PEARL is unable to make learn reasonable policy just using sparse rewards in same amount time it takes the oracle (PEARL trained on dense reward) and HTR (PEARL trained on sparse reward using Hindsight Task Relabelling) to converge. In all of our tested environments, HTR is not only comparable to using shaped reward during meta-training, but it also learn adaptation strategies that perform as well as the oracle during meta-testing the sparse reward environment. In Figure 5 we see that the adaptation strategies learned with Hindsight Task Relabeling similarly effective to those learned by the oracle, despite the oracle having access to dense reward function during meta-training.
Given an indefinite amount of meta-training time, shorter goal distance), PEARL should be able learn similarly optimal strategy to the oracle and HTR only using sparse reward. If PEARL is able successfully learn only using sparse rewards, HTR can be used to improve sample efficiency during meta- meta-training. In the case that the amount of time computational resources needed to train purely sparse rewards is intractable, HTR extremely effective at learning adaptation strategies comparable to using hand aand-designed dense reward function.
easier environment configuration (e.g.,
Additionally, as mentioned in Section 2] in many sparse reward settings designing dense reward function to aid training is infeasible or undesirable, in which case HTR is strong alternative to reward engineering. Because HTR does not train on a proxy reward, there is no opportunity for mismatch between reward functions at meta-training and meta-testing: HTR optimizes for the true reward function on superset of tasks Thindsight Trrain ~ p(T), whereas using a shaped reward optimizes for proxy reward function the true of training tasks Terain p P(T).

---

Averago return
abaged (ocudle) spane HTR SER sparne ITTTA pC:
Averago return
Relabellod task distance 1
sparne
Meta-training iterations
vettattaiinng iterations
Meta-training iterations
Figure 7: Comparing HTR with Figure 8: Average return when
Figure 9: Average task distance when varying K on Point Robot.
SER vs EC on Point Robot.
varying K on Point Robot.
5.3 Varying key hyperparameters
In our experiments we found a relatively low relabeling probability (e.g., K = 0.1 and 0.3) was often most effective for HTR (see Figure B). K = 0.1 means that relabelled data from a hindsight task is roughly as likely to be used for training as data from any of the 100 tasks in Terain:
Num, tronsitions with reward
2.07
Genead truth Relabeleed
= 0.1 corresponds to using far less relabelled data for training than in conventional HER: in contrast, Andrychowicz et al. (2017) relabel over 80% of sampled transi- tions. We hypothesize that low K works best for HTR since it provides enough reward signal to bootstrap learning, without bias- ing the task distribution too far from the ground truth distribution; in Figure 9 we can see that both K = 0.5 and 0.8 converge on mean relabelled task distance of 0.5, whereas K 0.1 and 0.3 are closer to the ground truth task distance of 2.0.
Meta-training iterations
Figure 10: Relative reward sig- nal from hindsight vs ground truth tasks using Point Robot.
In both HER and HTR, the relabeling probability is an important hyperparameter that should be tuned for optimal performance. In the original HER paper, Andrychowicz et al. (0077) reported that a relabelling probability of 80% 90% worked best on their three goal-conditioned RL environments. For both algorithms, there exists a range of relabelling probabilities (alternatively a range of ratios of higginal-to-relbellle data) where the relabeling algorithm works well, and range of values where the algorithm performs poorly (the HER paper did not include results for under 50% relabelled data, which may include more negative results).
In Figure 9 we see that the average distance reached by the policy during mmtaa-rraining gradually in- creases over time, increasing the average relabelled task distance. Hindsight tasks used for relabelling initially correspond to easily achievable tasks, and shift towards the ground truth task distribution. Initially, the only reward signal available for training comes from the relabelled transitions, however, as training progresses the agent is able to recover true reward signal from the environment (see Figure 10 and the hindsight tasks more likely to resemble the ground truth tasks.
As described Section 4 we found that EC generally performed similarly to SER (see Figure 7 despite being significantly more complex to implement and requiring additional tuned hyperparam- eters. We expect that more significant performance difference between the two approaches may become apparent different sparse reward meta-RL tasks, through experimenting with more general episode clustering approaches than the state space discretization used in this work.
6 Conclusion
In this paper, we propose a novel approach for mentoreeiforremment learning in sparse reward environ- ments that can be incorporated into any off-policy meta-RL algorithm. The sparse reward meta-RL setting is extremely challenging, and existing meta-RL algorithms that learn adaptation strategies for sparse reward environments often require dense or shaped reward functions during meta-training.
Our approach, Hindsight Task Relabeling (HTR), enables learning adaptation strategies in challenging sparse reward environments without engineering proxy reward functions. HTR relabels data collected on ground truth meta-training tasks as data for achievable hindsight tasks, which enables meta-training on the original sparse reward objective. Not only does HTR allow for learning adaptation strategies in challenging sparse reward environments without reward engineering, but it also generates adaptation strategies comparable to prior approaches that use shaped reward functions.
10

---

Acknowledgements
We thank Katelyn Gao for providing valuahle feedhack and insights work. This project is in part hased upon work supported by the National Science Foundation under NSF CISE Expeditions Award CCF-1730628, Berkeley Deep Drive, and gifts from Amazon Web Services, Ant Group, Ericsson, Facebook, Futurewei, Google, Intel, Microsoft, Scotiabank, and VMware. Any opinions, findings, and conclusions or recommendations expressed this paper are those of authors and not necessarily reflect the views of the National Science Foundation, Berkeley Deep Drive, and other sponsors.
References
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abheel, P., and Zaremba, W. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), pp.
5048-5058 2017.
Bengio, Y., Bengio, and Cloutier, Learning synaptic learning rule. Citeseer, 1990.
Clark, J. and Amodei, D. Faulty reward functions in the wild. http::/fwo..rreea...ooottyyyyyrr 2016.
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Duan, Y.. Schulman, J., Chen, X., Bardett, P. L., Sutskever, L,, and Abbeel, P. RL2: Fast reinforcement learning via slow reinforvement learning. arXiv:1611.02779, 2016.
Eysenbach, B., Geng, X., Levine, S., and Salakhutdinov, R. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint arXiv:2002. 1089, 2020.
Finn, C., Abbeel, P., and Levine, Modcl-agnostic meta-lcaming for fast adaptation deep networks. In International Conference on Machine Learning (ICML), 2017.
Florensa, C., Held, D., Geng, X., and Abheel, P. Automatic goal generation for reinforcement learning agents.
In International Conference on Machine Learning (ICML). 2018.
Gupta, A., Eysenbach, B., Finn, C., and Levine, S. Unsupervised meta-lcarning for reinforcement learning. arXiv preprint arXiv: 188806004411 2018a.
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S. Mete-reinforcement learning of structured exploration strategies. In Neural Information Processing Systems (NeurIPS), pp. 5302-531 5311, 2018b.
Haaroja,, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning (ICML), 2018.
Houthooft, R., Chen, Y., Isola, P., Stadie, B., Wolski, Ho, and Abbeel, Evolved policy gradients. In Neural Information Processing Systems (NeurIPS), pp. 5400-5409, 2018.
Jabri, A., Hsu, K., Gupta, A., Eysenbach, B., Levine, S., and Finn, C. Unsupervised curricula visual meta-reine reinforeentt learning. In Neural Information Processing Systems (NeurIPS), pp. 10519- 10530, 2019.
Kaelbling, L. Learning achieve goals. International Joint Conferences on Artificial Intelligence (IJCAI), pp. 1094-1099, Citeseer, 1993.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research (JMLR), 17(1):1334 1373, 2016.
Lovy, A., Platt, R., and Sacnko, K. Hicrarchical actor-critic. arXiv preprint
2017.
Li A. C., Pinto, L., and Abbeel, P. Generalized hindsight for reinforcement learning. arXiv preprint arXiv:2002.11708 2020.
Lillicrap, T. P., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Mishra, N., Rohaninejad, M. Chen. X. P.. and Abbeel, P. A simple neural attentive meta-learner, In International Conference on Learning Representations (ICLR), 2018

---

Mnih. V. Kavukcuogiu, K., Silver, D., Rusu, A. A., Veness, J.. Bellemare, M. G., Graves, A.. Riedmiller, M.. Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., et al. Human-level control through deep reinforcement learning. Nature, 518(7540). 2015.
Nair. A. V., Pong. V., Dalal, M., Bahl, S., Lin. S., and Levine, S. Visual reinforcement learning with imagined goals. Neural Information Processing Systems (NeurIPS), pp. 9191-9200. 2018.
Ng. A. Y., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference Machine Learning (ICML), volume 99, pp. 278 287, 1999.
Packer. C., Gao, K., Kos, J., Krähenbühhl. P.. Koltun, V., and Song, D. Assessing generalization in deep reinforcement learning. arXiv: 12282, 2018.
Rakelly, K., Zhou, A., Quillen, D., Finn, C., and Levine, S. Efficient off-policy meta-reinforcement learning via probabilistic context variables. International Conference on Machine Learning (ICML), 2019.
Rauber, P., Ummadisingu, A., Mutz, F., and Schmidhuber. J. Hindsight policy gradients. In International
Conference on Learning Representations (ICLR), 2019.
Rothfuss, J. Lee, D., Clavera, I,, Asfour, T., and Abheel, Promp: Proximal meta-policy search. International Conference on Learning Representations (ICLR), 2019.
Schmidhuber, J. Evolationary principles in self-referential learning, or learning how to learn: the meta-mela- hook. PhD thesis, Technische Universität München, 1987.
Silver, D., Hubert, T., Schritwweser, J., Antonoglou, I., Lai, M., Guez, A., Lanetot, M., Sifre, L., Kumaran, D., Graepel. T., et Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint wrXiv:1712.01815, 2017.
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y. Wu, Y., Abbeel, P., Sutskever, Some considcrations on learning to explore via meta-reinforcemmnt learning. arXiv preprint arXiv:1803. 18, 2018.
Sung, F. Zhang, L., Xiang, T., Hospedales, T., and Yang, Y. Learning to learn: Meta-critic networks for sample efficient learning. 17X55 2017.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2017.
Thrun, S. and Pratt, L. Learning to learn: Introduction and overview. Learning to learn, pp. 3-17, Springer, 1998.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning to reinforcement learn. (tfi::/61100766. 2016.
Xu, T., Liu, Q., Zhao, L.,, and Peng. Learning to explore via meta-policy gradient. In International Conference on Machine Learning (ICML., pp- 5463-5472, 201 8a.
Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. Neural Information Processing Systems (NeurIPS), 2396-22660022 2006224072222 2018b.
Zhang, A., Ballas, N., and Pineau, J. dissection overfitting and generalization in continuous reinforcement learning. arXiv: 0100.800077773 2018.
Zintgraf, L. M., Shiarlis, K., Kurin, V., Hofmann, K., and Whiteson, S. Fast context adaptation via meta- meta-learning arXiv preprint -xxi:/810000644 2018.
12


Reviews:
[{"review_id": "mQhspfku-Z", "reviewer": "Reviewer_hvDW", "summary": "This paper extends the idea of hindsight experience replay (HER) to a meta reinforcement learning setup. The primary focus is on sparse reward settings, where it is difficult for HER to generalize and even harder to conceptualize and optimize in the meta RL setting. \n\nThis paper provides a simple learning algorithm to relabel experiences in an off policy learning settings and provides experimental validation on continuous control tasks.\n\nKey idea: the task is unknown in the meta RL setting. An arbitrary experience trajectory collected during policy evaluation (unknown task) can be reused for an artifically induced task (therefore known). This lets the algorithm leverage past experiences under unknown task distributions to meta learn strategies for an unknown task with sparse rewards. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper studies an important and interesting problem in meta RL. Given a goal conditioned policy and the ability to explore somewhat efficiently, this algorithm can get the learning signal necessary for the purpose. However, the biggest  limitation is that it won't work for tasks where it is difficult to explore in the first place in order to achieve a positive reward value. So this formulation is deeply tied to exploration feasibfility. \n\nIn similar ways, what is the relationship and limitation of this formulation to reachability of goals and reversibility? If certain goal states are irreversible and the episodes are infinite horizon or long range (e.g. in robotics) then what are the failure models?\n\nDoes the relabeling probablity K affect results more than in the HER setting? \n\nHow does HTR compare to intrinsic motivation based approaches on the same environments (e.g. count or density based)? Is there any inherent advantage for learning to learn over intrinsic motivation? I do think that HTR is an interesting setting by itself to study and explore but it would be helpful to understand and map out how it could go beyond other efficient exploration strategies. Especially because this paper is not claiming to solve tasks that were unsolved before (except under certain assumotion on the reward structure -- which is also entirely reasonable for hypothesis driven research).\n\nI also believe that this setup and algorithm is different HER despite close similarities. This setting is fundamentally studying multiple MDPs which is quite different from HER. I think section 3.1 is quite helpful to understand and appreciate the key differences. \n\nFinally I think the experiments are reasonable and well motivated. The paper could have gotten stronger if there was at least one large scale/real world tasks in the mix, either from robotics, games or navigation. Currently there are simpler experiments in each of these domains.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "seems adequate", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "eLlfmPc9cmL", "reviewer": "Reviewer_iJzp", "summary": "This work extends HER to meta-RL setting where the task parameters are not provided to the agent. The authors propose two different ways to relabel transitions and demonstrate their effectiveness through experiments. On goal reaching tasks, the performance of HTR on sparse reward setting is comaparable to baseline under dense reward setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Low. HTR is a direct extension of HER. The authors make minor modifications to HER to adapt to meta-RL setting.\n\nQuality: Good. While empirical and straightforward, the proposed methods (SER and EC) are sounding and achieved good performance on widely-used environments. However, it would be more convincing if the authors could provide a more formal description of HTR.\n\nClarity: Medium. The story is well-written but some details of the algorithm is unclear. E.g. in l.199, how to determine the new hindsight task is not answered until l.221.\n\nSignificance: Medium. While sparse reward+HTR achieved near-dense reward setting performance in experiments, the algorithm is limited to goal reaching task and it remains questionable whether it will work for for complex meta-RL tasks.\n\nMajor concerns:\n 1. The authors claim that \"HTR generates an implicit curriculum of tasks that gradually shifts from easier hindsight tasks towards the true training distribution\". I believe this is overclaimed and not supported by the experiments in the supplementary. The distribution of the generated hindsight tasks simply follows the current policy.\n2. Compared with SER which is a modified version of HER, EC is a more sounding approach to me. I believe that the potential of EC is underclaimed in the paper. If I understood correctly, the EC method allows mixture of transitions coming from different episodes and could therefore train the context encoder with much more samples. In simple task such as goal reaching, it could be hard to demonstrate the effectiveness of EC since the context encoder is simply an exclusion of visited coordinates. I suggest that the authors add more discussion on this or even experiments, if possible. If the authors could use SER as a baseline and demonstrate how EC works better than SER in more complex settings, this paper would definitely be a valuable one.\n\nMinor issues:\n 1. Algorithms are too shortened that it is hard to tell the difference between Alg. 1 in the main text and Alg. 1 in the supplementary. According to l.216~219, in SER all transitions in a batch comes from the *same* trajectory and share the same hindsight task. However, this is not outlined in the algorithm.\n 2. The authors claimed several times that HTR creates a \"dense\" reward. This may not be true since the agent is still rewarded at goal states only.\n 3. Introduction section missing.\n\nQuestions:\n 1. In supp. Figure 2, why is the num. of rewarded relabelled transitions decreasing?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addresed the limitations adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "PNntEH_eiBq", "reviewer": "Reviewer_brT1", "summary": "This paper describes an extension of Hindsight Experience Replay (HER) to a meta-RL setting with task inference.\nLike HER, trajectories are relabelled with rewards as if they had achieved a goal. Unlike HER, the goal cannot be explicitly relabelled. Instead, a context is provided from which the agent attempts to learn a task embedding: this task inference is also trained using the reward-relabelled data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a neat extension of the nice HER idea to a task-inference setting. The main idea makes sense, the paper is fairly well written, and the results show a substantial qualitative change in performance compared to a more standard use of a sparse reward. Since it’s a (relatively) incremental extension of HER to a new setting, I can’t rate the originality or significance of the work as extremely high, but I think it’s a nice paper overall.\n\nI have one major concern about the method that deserves some further discussion, which is about the task distribution induced by the hindsight relabelling.\nIn HER, a biased goal distribution is not asymptotically problematic if the model has sufficient capacity, eventually it can represent the optimal solution for every goal (although certainly the relabelling bias will impact generalisation in practice).\nIn HTR however, the optimal meta-learned exploration strategy depends on the true task distribution, as does the optimal task inference strategy.\nThe authors seem to be somewhat aware of this issue from the brief comment in line 309 in discussion of the K hyperparameter, but it seems like a more fundamental qualitative difference between the HER and HTR settings that deserves further discussion, unless I’ve misunderstood something.\n\nMy other comments are more minor on exposition and experiments:\nI found the paragraph starting line 183 quite confusing; the ‘naive’ alternatives felt a bit like strawmen, or maybe they rest on assumptions that are more implementation-level, e.g. about what is/isn’t being done in a single batch. For example, I don’t see a problem with relabelling each transition in a batch, as long as they also each get their own associated context (this might be inefficient or undesirable in practice?).\nI’d maybe recommend presenting the two approaches that ‘make sense’ first, and then contrasting with alternatives, or trying to make a bit more clear why one might consider the 'naive' approaches first.\n\nFor the experiments; while I understand that the point of the method is to show a step change from the inability to learn with a sparse reward, it’s a little unsettling to see a baseline method that completely flatlines on all tasks. It would be encouraging to see a simple experiment with e.g. a point-mass that needn’t travel as far to achieve a sparse reward (with the expectation that HTR increases sample efficiency, but the baseline clearly works).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I discuss one potentially important limitation in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0-3SlhvAxZM", "reviewer": "Reviewer_6Fvi", "summary": "This paper proposes an algorithm. Hindsight Task Relabeling, for meta-reinforcement learning in sparse reward environments. It can be viewed as applying the hindsight relabeling in a meta-RL setting, instead of in the goal-conditioned setting of the original HER. With the Meta-RL setting, the task is not revealed to the agent during test time, and thus needs to be inferred as well (compared to the goal-conditioned setting). Specifically, they proposed two relabeling strategies, Single Episode Relabeling (SER) and Episode Clustering (EC). The key difference from naive HER style relabeling is that SER samples an entire episode, subsamples transitions, and then relabels them to the same hindsight task. With EC, similar trajectories are clustered into the same hindsight buffers (e.g. by discretizing the state space into bins) with stratified sampling during model training. The major hyperparameter introduced is K, which determines the probability of sampling original vs hindsight experience when training the model. \n\nThe experiments were applied on goal-reaching tasks (2D navigation, wheeled locomotion, and ant locomotion). They used PEARL as the underlying off-policy meta-RL algorithm. The results indicate that HTR is crucial for sparse reward tasks, and can learn as well (sometimes better) than with shaped dense reward. Unlike HER where higher hindsight ratio helped more, in HTR only a K=0.1 (10% hindsight) is best in their environments. Finally, despite EC being more complicated to implement, it performed similarly to the much simpler SER strategy. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The paper can be seen as a generalization (or application) of hindsight relabeling in the goal-conditioned setting to the meta-RL setting. The relabeling strategy needed to be at the episode level in order to mimic the meta-training set up, where they share the same pseudo/real task for the context. The paper discusses the similarity and differences to HER, as well as other sparse reward and unsupervised meta-RL. \n\n**Quality**: The proposed method was evaluated empirically, as there is no theoretical analysis for the proposed framework. However as pointed out in Appendix C4, neither did the original HER paper. Several limitations/assumptions for the method were discussed in section 3.1, including that trajectories can be relabeled under a new task, and that this method could be applied beyond the goal-reaching tasks as proof of concept in this work. \n\nOne experiment that would make the need for sparse reward in meta-RL setting convincing is to have an environment where the shaped dense reward (i.e. L2 distance) does not work well. For example, we can have an environment with obstacles, to demonstrate that a hand-engineered dense reward would perform worse than using meta-training on sparse reward with HTR. \n\n**Clarity**: Overall the paper is fairly well written and well organized. One clarification would be the detail about how the hindsight sampling connects with the context vs RL sampler when using PEARL. For example, with SER, how does this interact with the context and RL batches, i.e. is it sampling only 1 episode for the context, and then for the RL batches are sampled independently with the same hindsight task? \n\n**Significance**: The problem of sparse reward in meta-RL settings is a natural extension to existing meta-RL environments. The proposed hindsight task relabeling schemes are relatively straightforward to implement in simple settings where inferring the task is easy/given. It remains to be seen how well this method would scale to more complicated sparse reward tasks beyond goal-reaching type of environments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impacts were discussed in Appendix D. The discussions were fairly general, touching upon the common concerns in deep learning models, as they also apply to the deep meta-RL. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues with this paper", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for rebuttal and the reviewers for engaging in discussions. We reached a consensus for weak acceptance of the paper. However, the method is straightforward given PEARL and HER, and the reviewers and I agree that the current tasks are not challenging enough and do not convincingly demonstrate the value of this approach, and highly encourage the authors to add additional validations.  
Decision: Accept (Poster)