Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an algorithm. Hindsight Task Relabeling, for meta-reinforcement learning in sparse reward environments. It can be viewed as applying the hindsight relabeling in a meta-RL setting, instead of in the goal-conditioned setting of the original HER. With the Meta-RL setting, the task is not revealed to the agent during test time, and thus needs to be inferred as well (compared to the goal-conditioned setting). Specifically, they proposed two relabeling strategies, Single Episode Relabeling (SER) and Episode Clustering (EC). The key difference from naive HER style relabeling is that SER samples an entire episode, subsamples transitions, and then relabels them to the same hindsight task. With EC, similar trajectories are clustered into the same hindsight buffers (e.g. by discretizing the state space into bins) with stratified sampling during model training. The major hyperparameter introduced is K, which determines the probability of sampling original vs hindsight experience when training the model. 

The experiments were applied on goal-reaching tasks (2D navigation, wheeled locomotion, and ant locomotion). They used PEARL as the underlying off-policy meta-RL algorithm. The results indicate that HTR is crucial for sparse reward tasks, and can learn as well (sometimes better) than with shaped dense reward. Unlike HER where higher hindsight ratio helped more, in HTR only a K=0.1 (10% hindsight) is best in their environments. Finally, despite EC being more complicated to implement, it performed similarly to the much simpler SER strategy. 

main_review: **Originality**: The paper can be seen as a generalization (or application) of hindsight relabeling in the goal-conditioned setting to the meta-RL setting. The relabeling strategy needed to be at the episode level in order to mimic the meta-training set up, where they share the same pseudo/real task for the context. The paper discusses the similarity and differences to HER, as well as other sparse reward and unsupervised meta-RL. 

**Quality**: The proposed method was evaluated empirically, as there is no theoretical analysis for the proposed framework. However as pointed out in Appendix C4, neither did the original HER paper. Several limitations/assumptions for the method were discussed in section 3.1, including that trajectories can be relabeled under a new task, and that this method could be applied beyond the goal-reaching tasks as proof of concept in this work. 

One experiment that would make the need for sparse reward in meta-RL setting convincing is to have an environment where the shaped dense reward (i.e. L2 distance) does not work well. For example, we can have an environment with obstacles, to demonstrate that a hand-engineered dense reward would perform worse than using meta-training on sparse reward with HTR. 

**Clarity**: Overall the paper is fairly well written and well organized. One clarification would be the detail about how the hindsight sampling connects with the context vs RL sampler when using PEARL. For example, with SER, how does this interact with the context and RL batches, i.e. is it sampling only 1 episode for the context, and then for the RL batches are sampled independently with the same hindsight task? 

**Significance**: The problem of sparse reward in meta-RL settings is a natural extension to existing meta-RL environments. The proposed hindsight task relabeling schemes are relatively straightforward to implement in simple settings where inferring the task is easy/given. It remains to be seen how well this method would scale to more complicated sparse reward tasks beyond goal-reaching type of environments. 
limitations_and_societal_impact: The broader impacts were discussed in Appendix D. The discussions were fairly general, touching upon the common concerns in deep learning models, as they also apply to the deep meta-RL.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends the idea of hindsight experience replay (HER) to a meta reinforcement learning setup. The primary focus is on sparse reward settings, where it is difficult for HER to generalize and even harder to conceptualize and optimize in the meta RL setting. 

This paper provides a simple learning algorithm to relabel experiences in an off policy learning settings and provides experimental validation on continuous control tasks.

Key idea: the task is unknown in the meta RL setting. An arbitrary experience trajectory collected during policy evaluation (unknown task) can be reused for an artifically induced task (therefore known). This lets the algorithm leverage past experiences under unknown task distributions to meta learn strategies for an unknown task with sparse rewards. 

main_review: 
This paper studies an important and interesting problem in meta RL. Given a goal conditioned policy and the ability to explore somewhat efficiently, this algorithm can get the learning signal necessary for the purpose. However, the biggest  limitation is that it won't work for tasks where it is difficult to explore in the first place in order to achieve a positive reward value. So this formulation is deeply tied to exploration feasibfility. 

In similar ways, what is the relationship and limitation of this formulation to reachability of goals and reversibility? If certain goal states are irreversible and the episodes are infinite horizon or long range (e.g. in robotics) then what are the failure models?

Does the relabeling probablity K affect results more than in the HER setting? 

How does HTR compare to intrinsic motivation based approaches on the same environments (e.g. count or density based)? Is there any inherent advantage for learning to learn over intrinsic motivation? I do think that HTR is an interesting setting by itself to study and explore but it would be helpful to understand and map out how it could go beyond other efficient exploration strategies. Especially because this paper is not claiming to solve tasks that were unsolved before (except under certain assumotion on the reward structure -- which is also entirely reasonable for hypothesis driven research).

I also believe that this setup and algorithm is different HER despite close similarities. This setting is fundamentally studying multiple MDPs which is quite different from HER. I think section 3.1 is quite helpful to understand and appreciate the key differences. 

Finally I think the experiments are reasonable and well motivated. The paper could have gotten stronger if there was at least one large scale/real world tasks in the mix, either from robotics, games or navigation. Currently there are simpler experiments in each of these domains.
limitations_and_societal_impact: seems adequate
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes an extension of Hindsight Experience Replay (HER) to a meta-RL setting with task inference.
Like HER, trajectories are relabelled with rewards as if they had achieved a goal. Unlike HER, the goal cannot be explicitly relabelled. Instead, a context is provided from which the agent attempts to learn a task embedding: this task inference is also trained using the reward-relabelled data.
main_review: This paper is a neat extension of the nice HER idea to a task-inference setting. The main idea makes sense, the paper is fairly well written, and the results show a substantial qualitative change in performance compared to a more standard use of a sparse reward. Since it’s a (relatively) incremental extension of HER to a new setting, I can’t rate the originality or significance of the work as extremely high, but I think it’s a nice paper overall.

I have one major concern about the method that deserves some further discussion, which is about the task distribution induced by the hindsight relabelling.
In HER, a biased goal distribution is not asymptotically problematic if the model has sufficient capacity, eventually it can represent the optimal solution for every goal (although certainly the relabelling bias will impact generalisation in practice).
In HTR however, the optimal meta-learned exploration strategy depends on the true task distribution, as does the optimal task inference strategy.
The authors seem to be somewhat aware of this issue from the brief comment in line 309 in discussion of the K hyperparameter, but it seems like a more fundamental qualitative difference between the HER and HTR settings that deserves further discussion, unless I’ve misunderstood something.

My other comments are more minor on exposition and experiments:
I found the paragraph starting line 183 quite confusing; the ‘naive’ alternatives felt a bit like strawmen, or maybe they rest on assumptions that are more implementation-level, e.g. about what is/isn’t being done in a single batch. For example, I don’t see a problem with relabelling each transition in a batch, as long as they also each get their own associated context (this might be inefficient or undesirable in practice?).
I’d maybe recommend presenting the two approaches that ‘make sense’ first, and then contrasting with alternatives, or trying to make a bit more clear why one might consider the 'naive' approaches first.

For the experiments; while I understand that the point of the method is to show a step change from the inability to learn with a sparse reward, it’s a little unsettling to see a baseline method that completely flatlines on all tasks. It would be encouraging to see a simple experiment with e.g. a point-mass that needn’t travel as far to achieve a sparse reward (with the expectation that HTR increases sample efficiency, but the baseline clearly works).

limitations_and_societal_impact: I discuss one potentially important limitation in the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work extends HER to meta-RL setting where the task parameters are not provided to the agent. The authors propose two different ways to relabel transitions and demonstrate their effectiveness through experiments. On goal reaching tasks, the performance of HTR on sparse reward setting is comaparable to baseline under dense reward setting.
main_review: Originality: Low. HTR is a direct extension of HER. The authors make minor modifications to HER to adapt to meta-RL setting.

Quality: Good. While empirical and straightforward, the proposed methods (SER and EC) are sounding and achieved good performance on widely-used environments. However, it would be more convincing if the authors could provide a more formal description of HTR.

Clarity: Medium. The story is well-written but some details of the algorithm is unclear. E.g. in l.199, how to determine the new hindsight task is not answered until l.221.

Significance: Medium. While sparse reward+HTR achieved near-dense reward setting performance in experiments, the algorithm is limited to goal reaching task and it remains questionable whether it will work for for complex meta-RL tasks.

Major concerns:
 1. The authors claim that "HTR generates an implicit curriculum of tasks that gradually shifts from easier hindsight tasks towards the true training distribution". I believe this is overclaimed and not supported by the experiments in the supplementary. The distribution of the generated hindsight tasks simply follows the current policy.
2. Compared with SER which is a modified version of HER, EC is a more sounding approach to me. I believe that the potential of EC is underclaimed in the paper. If I understood correctly, the EC method allows mixture of transitions coming from different episodes and could therefore train the context encoder with much more samples. In simple task such as goal reaching, it could be hard to demonstrate the effectiveness of EC since the context encoder is simply an exclusion of visited coordinates. I suggest that the authors add more discussion on this or even experiments, if possible. If the authors could use SER as a baseline and demonstrate how EC works better than SER in more complex settings, this paper would definitely be a valuable one.

Minor issues:
 1. Algorithms are too shortened that it is hard to tell the difference between Alg. 1 in the main text and Alg. 1 in the supplementary. According to l.216~219, in SER all transitions in a batch comes from the *same* trajectory and share the same hindsight task. However, this is not outlined in the algorithm.
 2. The authors claimed several times that HTR creates a "dense" reward. This may not be true since the agent is still rewarded at goal states only.
 3. Introduction section missing.

Questions:
 1. In supp. Figure 2, why is the num. of rewarded relabelled transitions decreasing?
limitations_and_societal_impact: The authors have addresed the limitations adequately.
review_title: null

Meta-review: We thank the authors for rebuttal and the reviewers for engaging in discussions. We reached a consensus for weak acceptance of the paper. However, the method is straightforward given PEARL and HER, and the reviewers and I agree that the current tasks are not challenging enough and do not convincingly demonstrate the value of this approach, and highly encourage the authors to add additional validations.  
Decision: Accept (Poster)