Efficient meta reinforcement learning via meta goal generation
Authors: Haotian Fu
Hongyao Tang
Jianye Hao
Venue:  (2020)

Abstract: Meta reinforcement learning (meta-RL) is able to accelerate the acquisition of new tasks by learning from past experience. Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions. However, for complex tasks which requires sophisticated control strategies, it would be quite inefficient to to directly learn such a meta-policy. Moreover, this problem can become more severe and even fail in spare reward settings, which is quite common in practice. To this end, we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL (MGHRL) by leveraging hierarchical actor-critic framework. Instead of directly generate policies over primitive actions for new tasks, MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks. Our empirical results on several challenging simulated robotics environments show that our method enables more efficient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings.

Under review as a conference paper at ICLR 2020
EFFICIENT META REINFORCEMENT LEARNING VIA META GOAL GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Meta reinforeement learning (meta-RL) is able to accelerate the acquisition of new tasks by learning from past experience. Current meta-RL. methods usually learn adapt to new tasks by directly optimizing the parameters of policies over primitive actions. However, for complex tasks which requires sophisticated control strate- gies, it would be quite inefficient to directly learn such a meta-policy. Moreover, this problem can become more severe and even fail in spare reward settings, which is quite common in practice. To this end, we propose a new meta-RL algorithm called meta goal- geleeeteettoonn for hierarchical RL (MGHRL) by leveraging hierar- chical actor-critic framework. Instead of directly generate policies over primitive actions for new tasks, MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals independent RL subtasks. Our empirical results on several challenging simulated robotics environments show that method enables meta-learning from past experience and outperforms state-of-the art meta- -eta-RL, and Hierarchical RL methods in sparse reward settings.
efficient and effective
INTRODUCTION
Deep Reinforcement Learning (DRL) has recently shown great success on a wide range of tasks, ranging from games (Mnih et al., 2015) to robotics control (Levine et al., 2016 Bengio & LeCun 2016). However, for more complex problems with larger state and action spaces or sparse reward set- tings, traditional DRI. methods hardly works. Hierarchical reinforcement learning (HRL. in which multiple layers of policies are trained to learn to operate on different levels of temporal abstraction, has long held the promise to learn such difficult tasks (Dayan & Hinton| 1992 Parr & Russell 1997] Barto & Mahadevan 2003). By decomposing complex problem into subproblems, HRL signifi- cantly reduces the difficulty of solving specific task. Learning multiple levels of policies in parallel is challenging due to ntatitaai state transition functions. Recent HRL approaches (Nachum ctt al. 2018, Levy et al. 2019) use states as goals directly, allowing simple and fast training of the lower layer.
Human intelligence is remarkable for their fast adaptation to many new situations using the knowl- edge learned from past experience. However, agents trained by conventional DRL methods men- tioned above can only learn one separate policy per task, failing to generalize to new tasks without additional large amount of training data. Meta reinforcement learning (meta-RL) addresses such problems by learning how to learn. Given a number of tasks with similar structures, meta-RL RL methods enable agents learn such structure from previous experience on many tasks. Thus when encountering new task, agents can quickly adapt to it with only small amount of experience.
Most current meta-RL methods leverage experience from previous tasks adapt to new tasks by directly learn the policy parameters over primitive action space. (Finn et al. 2017 Rakelly et al.) 2019). Such approaches suffer from two problems: (i) For complex tasks which requires sophisti- cated control strategies, it would be quite inefficient to directly learn such policy with one nonlinear function approximator and the adaptation to new tasks is prone to be inaccurate. This problem can become more severe in spare reward settings. (ii) When the task distribution is much wider (riding bicycle as meta-train task and riding motorcycle as meta-test task), these methods can hardly be effective since primitive action execution mechanism is entirely different although they may share

---

Under review as a conference paper at ICLR 2020
similar high-level strategy. Moreover, existing current meta-RL methods perform badly in sparse reward settings, which are quite common in real world.
In this paper, aim at tackling the problems mentioned above by proposing an efficient hicrarchical meta-RL method that realizes meta learning high-level goal generation and leaves the learning of low-level policy for independent RL. Intuitively, this is quite similar how human being behaves: we usually transfer the overall understanding of similar tasks rather than remember specific actions. Our meta goal-generation framework is built on top of the architecture of PEARL (Rakelly al.) 2019) and a two level hierarchy inspired by HAC (Levy et al.) 2019). Our evaluation on several simulated robotics environments (Plappert al., 2018) shows the superiority of MGHRL to state- of-the-art art meta-RL and hierarchical RL methods in sparse reward settings.
Generally, our contributions are as follows:
We propose an algorithm that achieves efficient meta reinforcement learning on challenging robotics environments with sparse reward settings and outperforms other leading methods.
Similar to the way humans leverage past experience to learn new complex tasks, our al- gorithm focuses on meta learning the overall strategy for different tasks, which provides a much simpler and better way for meta RL comparing with directly learning the detailed solution.
Since we focus on meta goal-generation and leave the low level policy for independent learning, we believe our algorithm can still accelerate the acquisition of new tasks sampled from much wider task distributions. For example, to learn tasks such as riding bicycles and riding motorcycle, the two primitive action execution mechanism are entirely different but the two learning process still share similar high-level structures. Through meta generrenerttion learning, expect our method can still accelerate the acquisition of such tasks. We leave these for future work to explore.
RELATED WORK
Our algorithm is based on meta learning framework (Thrun & Pratt 1998; Schmidhuber, 1987; Ben- gio al. 1991), which aims learn models that can adapt quickly new tasks. Meta learning al- gorithms for few-shot supervised learning problems have explored a wide variety of approaches and architectures (Santoro et al.) 2016 Vinyals al. 2016: Ravi Larochelle| 2017). In the context of reinforcement learning, recurrent Duunn et al. 2016 Wang et al. (016) and recursive (Mishra et al., 2018) meta-RL. methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned. Another set of methods is gradient-based meta reinforcement learning (Finn et al., 2017; [Stadie et al., 2018 Rothfuss et al., 2019 Xu et al.) 2018). Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task. These methods focus on on-policy meta learning which are usually sample inefficient. Our algorithm is closely related to probabilistic embeddings for actor-critic RL (PEARL) |Rakelly et al. (201)). which is an off-policy meta RL algorithm. PEARL leverages poste- rior sampling to decouple the problems of inferring the task and solving it, which greatly enhances meta-learning efficiency. However, when facing complex tasks that require sophisticated control strategies, PEARL cannot effectively learn a proper meta-policy as we will show Section
Discovering meaningful and effective hierarchical policies is longstanding research problem in RL (Dayan & Hinton 1992. Parr & Russell 1997, (Sutton et al., 1999. [acon et al. 2017 Dietterich 2000). Schmidhuber 1987 proposed a HRL approach that can support multiple levels. Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into set of short-horizon subproblems. Nachum et al. 2018 proposed HIRO, a 2-level HRI. approach that can learn off-policy and outperforms two other popular HRL techniques used in continuous domains: Option-Critic (Bacon et al. (2017) and FeUdal Networks (Vezhnevets et al.) 2017]. Our algorithm is built on Hierarchical actor-critic (Levy et al. 2019), which is a framework that can learn multiple levels of policies in parallel. Most current HRL works focus on the learning problem in a single task and few of them considers to take advantage of HRL for multi-task or meta- learning tasks. MLSH (Frans et al.) is such work which also combines meta-RL RL with Hierarchical RL. It focuses on meta learning on the low level policy and need to retrain its high level policy when facing new tasks. In contrast, with the key insight that humans leverage abstracted prior knowledge
2

---

Under review as a conference paper at ICLR 2020
obtained from past experience, method focus meta learning high level overall strategy using past experience and leave the detailed action execution for independent RL.
3 BACKGROUND
3.1 META REINFORCEMENT LEARNING
In our meta learning scenario, we assume distribution of tasks p(r) that we want our model to adapt to. Each task correspond a different Markov decision process (MDP), M, = {S, Ti, R. with state space S, action space A, transition distribution T,, and reward function R, We assume that the transitions and reward function vary across tasks. Meta RL aims to learn policy that can adapt to maximize the expected reward for novel tasks from p(r) as efficiently possible.
Ti,
PEARL (Rakelly et al. 2019) is an off-policy meta-reinforcement learning method that drastically improves sample efficiency comparing previous meta-RL algorithms. The meta-training process of PEARL learns policy that adapts to the task at hand by conditioning the history of past transi- tions, which we refer context c. Specifically, for the ith transition in task T, of (8i, 0s, ri, r()..) PEARL leverages an inference network 4 (=(c)) and outputs probabilistic latent variable z. The parameters of (2]ec) are optimized jointly with the parameters of the actor ae(a)s, 2) and critic Qo(s, a, 2), using the reparametrization trick (Kingma & Welling 2014) to compute gradients for parameters of go through sampled 2's.
3.2
HIERARCHICAL ACTOR-CRITIC
HAC Lev et al. 2019) aims to accelerate learning by enabling hierarchical agents to jointly learn a multi-level hierarchy of policies in parallel. HAC is comprised of two components: a particular hierarchical architecture and a method for learning the multiple levels of policies in parallel given sparse rewards. The hierarchies produced by HAC have a specific architecture consisting of a set of nested, conl-condi policies that use the state space the mechanism for breaking down task into subtasks. HAC extends the idea of Hindsight Experience Replay (Andrychowicz et al.) 2017) by creating two types of hindsight transitions. Hindsight action transition simulates transition function that uses the optimal low level policy while hindsight goal transition use the final states achieved as the goal state in each step's 's transition. They enable agents to learn multiple policies in parallel using only sparse reward functions.
ALGORITHM
4.1 TWO-LEVEL IIERRRCIY
We up hierarchical two-layer RL structure similar HAC. The high level network uses policy u² to generate goals for temporally extended periods in terms of desired observations. In our task they correspond the positional features of the gripper. The low level policy je' directly controls the agent and produces actions for moving towards the desired goals.
As shown in Figure (a), the high level policy μ² observes the state and produces high level action (or goal) g². Low level policy μd has at most K attempts of primitive action to achieve g". Here, which can be viewed as the maximum horizon of subgoal action is hyperparameter given by the user. As long as the low level policy μ² run out of K attempts or g is achieved, this high level transition terminates. The high level policy uses agent's current state as the new observation and produced another goal for low level policy to achieve.
We use an intrinsic reward function in which reward of O is granted only if the goal produced by high level policy is achieved and reward of -1 otherwise. Note that the environment's return (i.e. whether the agent successfully accomplished the task) will not affect the reward received by the low level policy. In our evaluation simulated robotics environments, the observations as the representation for g¹. A goal g is judged to be achieved only if the distance between g and the gripper's current position 8,+1 less than threshold l.
the positional features of
3

---

Under review as a conference paper at ICLR 2020
5
0
n4/
Context
LRL
encoder
4 (z|ch)
1e
cj
4
z
S
an
On+1
by
(o(s.g.2)
Leritic (b
Replay buffer
Environment
m (g|s,
h (bn, 2)
(a) Two-level hierarchy
(b) High level meta-training framework
Figure 1: (a) Two level hierarchy of MGHRL. High level policy Y² takes in state and outputs goals at intervals. Low level policy μá takes in state and desired goals to generate primitive actions. (b) High level meta-training framework of MGHRL. The context encoder network uses high-level context data C₁ to infer the posterior over the latent context variable z, and optimized with gradi- ents from the critic as well as from an information bottleneck on z. The actor network (g)0))) and critic network Qo(s, 9, both treat as part of the state.
4.2 META GOAL-GENERATION FOR HIERARCHICAL REINFORCEMENT LEARNING
One primary motivation for our hierarchical meta reinforcement learning strategy is that, when peo- ple try to solve new tasks using prior experience, they usually focus on the overall strategy we used in previous tasks instead of the primitive action execution mechanism. Most state-of-the-art meta learning methods (Rakelly et al. 2019 Finn et al. 2017) leverage experiences from previous tasks to quickly adapt to the tasks and directly learn the policy parameters. However, it can be difficult to meta learn a proper policy that consider both the overall strategy and detailed action execution in some complex tasks. Using only level of non-linear function approximator may lead the agents to learning an inaccurate meta-policy when both the overall structure and primitive action execu- tion mechanism are complex. Moreover, in sparse reward settings which is a common situation in real-world problems, current meta learning algorithms do not perform well enough since their training methods are based on non-hierarchical RI. methods like TRPO (Schulman et al., (015), SAC (Haarooj et al. 2018), etc. These methods suffer from the difficulty of effective exploration and the lack of positive update signals.
To address the problem mentioned above, we take advantage of our two-level hierarchy structure and propose a new meta reinforcement learning framework called meta goal-generation for hierar- chical RI. (MGHRL.) Instead of learning to generate detailed strategy for new tasks, MGHRL learns to generate overall strategy (goals) given past experience and leaves the detailed method of how to achieve the goals for independent RL. We leverage PEARL framework (Rakelly et al. 2019) and independently train a high level meta-policy which is able to quickly adapt to new tasks and gen- erate proper goals. Note that off- off-policy RL method is indispensable in structure when training high level policy due to its excellent sample efficiency during meta-training. Good sample efficiency enables fast adaptation by accumulating experience online, and performs structured exploration by reasoning about uncertainty over tasks, which is crucial to hierarchical parallel training framework. We leave the low level policy be trained independently with non-meta RL algorithm using hind- sight experience replay mechanism. In our simulated robotics experiments, the low level policy aims to move the gripper to the desired goal position which can be reused when switching to other tasks. Thus we only need to train a single set of low-level polices which can be shared and reused across different tasks. On the other hand, in other situations where the tasks are from different domains, for example, when we use our experience of learning riding bicycle to help us learn how to ride motorcycle, the primitive action execution mechanism are entirely different. In this case, we can train low level policy independently that when dealing with entirely new tasks, the primitive action exccution mechanism can be entirely different but the general strategy of how to accomplish the new tasks and prior tasks can be similar.
tasks without using past experience. Our main insight is
4

---

Under review as a conference paper at ICLR 2020
With meta learning high level policy, our algorithm still greatly accelerate the acquisition of new tasks.
Our high level meta-RL network uses probabilistic embedding actor-critic framework similar to PEARL. The network consists of two parts. The first part is a context encoder which leverages data from variety of training tasks learn infer the value of from recent history of high-level experience in the new task, where 2 functions as a latent probabilistic context variable. The encoder network parameterized by yp takes context (experience) Ca as input and output posterior qu (z(ce) as a permutation-inee -nnvariinnn function (Rakelly et al., 2019) of prior high level experience. The context on consists of experience {s. r. }'} collected using hindsight technique as will introduce in Section 4.3 Then we can sample 2 from the posterior and compute policy output and Q value conditioned on it. Through posterior sampling to infer new tasks efficiently using past experience. The second part is built top of soft actor-critic algorithm (Haarnoja et al.) 2018). As we mentioned before, samples from the posterior belief are passed to actor P [g|s, ss, 2) and critic Qe(s, 9, to make predictions of the sampled task. Note that we treat as part of the state when we implement with SAC.
latent contexts, the high level network can learn
The actor and critic are trained to predict optimally given & with batches of transitions drawn uni- formly from the entire replay buffer. The context encoder is optimized using gradients from the critic. We summarize our aeta-training procedure in Algorithn[] and [ (b). Concretely, for each training task drawn from task distribution, we sample context and generate hindsight transitions for both levels hierarchy (line 13) by performing current policy. Then we train high level and low level networks with the collected data (line 16 22).
Algorithm 1 MGHRL Meta-training
Require: Batch of training tasks (TiJi=t...., from p(T), maximum horizon K of subgoal action 1: Initialize replay buffers Bi, for each training task
2: while not done do
3: for each task 72 do
4: Initialize high-level context cf = 0
5: for m= do
6: Sample (2(c))) 9. < (g)5,,
7:
8:
for K attempts until gi achieved do
9:
Gather data using a; f my (a|s,g)
10:
Generate hindsight action transition, hindsight goal transition and add to Bj
11:
end for
12:
Generate hindsight transitions, subgoal test transitions and add B% Sample high level context cj {sj 9i Ta, 5/ J1j= i=.,....N Bj end for
13:
14:
15:
end for
16:
for steps in training steps do
for each task 7; do
18:
Sample high level context of Bi and RL batch bf, Bi, bj ~ Bi
19:
Sample 2 ~ (2(c), and calculate I (b), 2), Liratic (b), 2), L Update low level actor and critic network with bf end for
20:
21:
22:
Update high level networks with E, Th Tuccoor Ei Tb Dooitticr E, L
23:
end for
24:
end while
4.3 PARALLEL TRAINING STRATEGY
Efficient meta reinforcement learning requires parallel training for the two levels of our networks. To achieve parallel training paradigm, there exists main issues in MGHRL framework. The first issue for meta learning hierarchies is that agents need to act randomly to reach their goals and obtain the sparse reward which proves to be quite difficult for both levels. We need other strategies to ensure
5

---

Under review as a conference paper at ICLR 2020
each level learn effectively in sparse reward settings. The second issue is the non-stationary problem when we do parallel training for the high level and low level networks. Whenever low level policy n' changes, the high level transition function is likely to change well. Old off- aff-policy experience may exhibit different transitions conditioned on the same goals, making the transition invalid for training. The same problem occurs when the low level is exploring with some random noise. Thus in algorithm, we rewrite the past experience transitions as hindsight action transitions (Andrychowicz al. 2017), and supplement both levels of hierarchy with additional sets of transitions as was done in HAC.
Hindsight action transition simulates a transition function that uses the optimal low level policy which enables our framework to train both levels in parallel. It substitutes the action component in high level transition to the next state achieved in low level. If the original high level transition (st, St, rt, st+1], I. the hindsight action transition will be [st, 8:11; 1: rt, St+1]. where s represents the component vector of next state that matches the goal vector. The new transition we get is indepen- dent of changing exploring low level policy since it's always optimal.
We utilize hindsight goal transition and subgoal test transition to further address the problems men- tioned before. Hindsight goal transition is created for both levels. After at most K attempts is executed, the final states achieved is used as the goal state in each step's transition instead of the original goal state. And the reward will be updated to reflect the goal state. Subgoal test tran- sition meant to compensate for the drawbacks brought by hindsight action transition. Hindsight action transition prefer the shortest path of goals that have been found but may ignore the range of goals that the low level policy is able to reach. Thus, subgoal test transition add penalty of - to the reward if the goal is not achieved after K attempts by low level policy and the discount rate to to avoid non-stationary issues.
5 EXPERIMENTS
We evaluated our algorithm on several challenging continuous control robotics tasks (inte- grated with OpenAl Gym) (Plappert et al.) 2018). simulated via the MuJoCo physics sim- ulator Todorov et al.) 2012). Visualizations of these environments are shown Fig- ure More details on each environment can be found ingredients- for tobotics-research
(a) fetch-reach
(b) fetch-push
(c) fetch-slide
(d) fetch fetch- pickandplace
Figure 2: The four evaluated robotics environments
Fetch-Reach Fetch has to move the gripper to the desired goal position. This task very easy to learn and is therefore suitable benchmark to ensure that idea works at all.
Fetch-Push A box placed on a table in front the robot and Fetch has to move a box by pushing it until it reaches desired goal position. The robot fingers are locked to prevent grasping. The learned behavior is usually mixture of pushing and rolling.
Fetch-Slide A puck is placed long slippery table and the target position is outside of the robots reach so Fetch has to hit the puck across a long table such that slides and comes to rest on the desired goal.
Pickandiinnd Fetch has to pick up a box from a table using its gripper and move it to a desired goal located the table.

---

Under review as a conference paper at ICLR 2020
5.1 ENVIRONMENTAL SETUP
In all our experiments, we compare our algorithm to baselines including PEARL with dense reward, PEARL with sparse reward and HAC with shared policy. The last one means we train a shared HAC policy jointly across all meta-train tasks sampled from the whole task distribution. Note that Rakelly al. (2019) has already proved that PEARL greatly outperforms other existing meta RL methods like MAML (Finn et al. 2017), ProMP (Rothfuss et al. 2019) at both sample efficiency and final performance. Thus we mainly compare our results with PEARL using its public source code. In addition, for fair comparison, considered to be much powerful than DDPG in the original implementation (Haarnoja et al. 2018). ensure the consistence PEARL and MGHRL.
modify the HAC source code with SAC algorithm which
We set the goal space to be the set of all possible positions of the gripper, in which a goal is a 3-d vector. In the environments, the low level policy of our algorithm aims move the gripper to the desired goal position. Such policy won't t change at all when switching to other tasks since the mechanism of moving gripper keeps the same between different tasks. Thus policy trained jointly across all tasks for the low level of MGHRL. all four scenarios, we set the maximum low-level horizon K be 10 and the distance threshold to be 0.05. The high level context data sampler St samples uniformly from the most recently collected batch of data, which recollected every 1000 meta-training steps. Unlike HAC, we use target networks for both levels, which updates with т = 0.005. All context encoder, actor and critic neural networks had three hidden layers, with 300 nodes in each layer. The discount factor was set to 7 0.99. We use sparse reward function in which a reward of 0 is granted only if the terminal goal given by the environment achieved and a reward of otherwise. The dense reward used in our baseline is a value corresponding to the distance between current position of the box (or gripper in fetch-reach case) and desired goal position. In all four scenarios, we do our experiments on 50 train tasks and 10 test tasks, where the difference between each task is in the terminal goal position want the box or gripper to reach.
shared
AveragesuC rate
Averagesuc_ rate
COM
Number of train steps total
BAM
GBM
0300
TM
Number of train steps total
(a) Reach
(b) Push
Averagesuc, rate
Averagesuc,,
6AM
0.5M
6.M
Number of train steps total (c) Slide
Number of train sleps total (d) PickandPlace
HAC with shared policy
MGHRL
PEARL with dense reward PEARL with sparse reward
Figure 3: Average success rates for MGHRL, HAC, PEARL agents in each task, each algorithm was trained for leó steps. The error bar shows standard deviation.

---

Under review as a conference paper at ICLR 2020
5.2 RESULTS
We evaluate the performance of approaches in the term of the average success rate. As shown in Figure [] in Fetch-reach environment which is very easy to learn as we mentioned before, the tested methods except PEARL with sparse reward all reach final performance of 100% success rate. In other three scenarios, MGHRL significantly outperforms the other three method in such sparse reward settings. Our two-level hierarchy and hindsight transitions significantly decrease the difficulty of meta learning with sparse reward. As we expected, PEARL performs badly in sparse reward settings. The original version of PEARL is based on SAC, such non-hierarchical RL method has been proved to perform badly before on challenging tasks with sparse reward settings. Thus it is reasonable that PEARL, which can be viewed as meta-version of SAC, performs badly as well in sparse reward settings. HAC with shared policy generally performs better than PEARL in Fetch-slide and Teech--icckandllcee environments. We assume that it is because in our settings since we only change the terminal goals' positions to create different tasks, thus it is possible that the policy learned from one task will work on other task whose terminal goal positions are very close to previous training ones. But such method lacks generalization ability and cannot always achieve good performance when tested on varied tasks as shown in our results.
We also compare our method to PEARL with dense reward to demonstrate that MGHRL is able to more efficiently and accurately meta leam from past experience. Shown in Figure 3 generally, our algorithm still outperforms PEARL and adapts to new tasks much more quickly. In such environ- ments with sophisticated control strategies, directly using PEARL to meta learn a policy that con- sider both overall strategy and detailed execution would decrease prediction accuracy and sample efficiency. Thus it is better to decompose the meta-RL training process and focus on meta goal- generation learning. Moreover, under dense reward settings of these challenging tasks, the critic of PEARL has to approximate a highly non-linear function that includes the Euclidean distance be- tween positions and the difference between two quaternions for rotations (Plappert et al., 2018). As our method use hierarchical structure, learning with the sparse return is much simpler since the critic only has to differentiate between successful and failed states.
CONCLUSION
In this paper, we have presented a hierarchical meta-RL algorithm, MGHRL, which realizes meta goal-generation and leave the low-level policy for independent RL. MGHRL aims to more effi- ciently and accurately meta learn from past experience by focusing on learning the overall strategy of tasks instead of learning detailed action execution. Our experimental results on a range of sim- ulated robotics environments show the superiority of MGHRL over sate--of-the-ar meta RL and hierarchical RL methods in challenging and practical sparse reward settings.
We believe our work open up many directions in training agents that can quickly adapt to new tasks sampled from much wider distribution efficiently. Currently, we have only conducted experiments on meta learning tasks with relatively narrow task distribution (e.g. different goal positions of the box). As future work, we expect our algorithm can accelerate the acquisition of entirely new tasks (i.e. using fetch-push and fetch-slide meta train tasks and using aeth--pickandplcee task) by only meta learning overall strategy and leaving the details of primitive action execution mechanism for further separate low-level policy learning. Moreover, tasks are still far from perfect. There is still much work left for future research to improve meta-RI. methods' performance on those tasks.
meta test
note our results on some
REFERENCES
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Ad- vances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, pp. 5048-5058 2017. URL htt/http////pprrs ni nips. cc/paper/ 7090-hindsighe oegp replay
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-Firs First AAAI Conference on Artificial Intelligence, pp. 1726-1734, 2017. URL http:
8

---

Under review as a conference paper at ICLR 2020
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13(1-2):41-77, 2003. doi: 10.1023/A:1022140911 URL.
httpss://oil org. /tt/://.....0.111112222211111111
Yoshua Bengio and Yann LeCun (eds.). 4th International Conference on Learning Representa- fions, ICLR 2016, 2016. URL
1clr2016:accepttr mainn.ainn.. meaain.he html
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning synaptic learning rule. IJCNN-91-
Seattle International Joint Conference on Neural Networks, ii:969 vol.2,, 1991.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in Neu- ral Information Processing Systems 5, [NIPS Conference), pp. 271 271-278, 1992. URL http: /apperss.om nips wwtpppppppppceeeececeeeeeee 714 heanee//t---ecaaaaeeeeeerreeeetr
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function de- composition. J. Artif. Intell. Res., 13:227-303, 2000. doi: 10. 10.1613/jair..33 URL https:
org/i..rrg/...o 16130..6.1.iii.. 639
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RI$"2S: Fast reinforcement learning via slow reinforcement learning. CoRR, 1.02779, 2016. URL
http://arxiv.org/
027799
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pp. 1126-1135, 2017. URL http::/prrooeedings. mlr wtt.s//o0//ii/nn/////... ottpapeeeee.aaaa...eeeeeee///o/r html
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In 6th International Conference on Learning Representations, ICLR 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pp. 1856-1865, 2018. URL
html
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International Conference Learning Representations, ICLR 20/4, 2014. URL httt::/tt///i...////// 1312.617 6114
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep vi- suomotor policies. J. Mach. Learn. Res., 17:39:1-39:44 2016. URL http::/ org/
paaprs/v17/15-522 html
Andrew Levy, George Konidaris, Robert Platt Jr., and Kate Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference Learning Representations, ICLR 2019, 2019. URL https://openreviee net. wtt.://www.....oo..meeeee.e 2tte/ommmmmmeeyeccaac
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbecl. A simple neural attentive meta- leamner. In 6th International Conference on Learning Representations, ICLR 2018, 2018. URL
atttps:/oppeneviiee http:///ppeeereeeeeeeeetotoooaee www.m/id-mmiiooooooooom Bi1B-1-UBBBBBAA
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A. Rusu, Joel Veness Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, 2015. doi: 10. 10.8////ae 14236. URL www.ddo..ogg/....0 wwt.//atture11400m

---

Under review as a conference paper at ICLR 2020
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hi- erarchical reinforcement learning. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pp. 3307-3317, 2018. URL
Ronald Parr and Stuart J. Russell. Reinforcement learning with hierarchies of ma- chines. In Advances in Neural Information Processing Systems 10, INIPS Con- URL http://paper:: nips. ocppaper/ with of
ference]. pp. 1043-1049, 1997.
1384 www.iforreeeetteeerrree
Matthias Plappert, Marcin Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1802.09964 2018.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, pp. 5331-5340, 2019. URL. http::/proceed//
mlr, www.sppess/eee v97 wwwprawtellleeeee 11 9a html
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In JCLR, 2017.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In 7th International Conference on Learning Representations, ICLR 2019,
2019. URL https:: http://ppppeeeeeeeeeeeeeeeeeeeee / wwwp::/wwweeoooooomee.mm http:/orrmm.i.ikxxkcc10p0op.0
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. Meta-learning with memory-augmente neural networks. In Proceedings of the 33nd Inter- national Conference on Machine Leurning, ICML 2016, pp. 1842-1850, 2016. URL http: wtwpiiinggg.......e.........
//proceedings mll. prw.s/ess/eee.. /trees//////// wantoooo//oooolooooooooooooooooo 6. html
Juergen Schmidhuber. Evolutionary principles in self-referential learning. 1987.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference Machine Learn- ing. ICML 2015, pp. 1889-1897, 2015. URL httt:://proceeeiins...... mwt.press..pp..
schulman15.cm hhmml
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. CoRR, 011880030118... 2018. URL http::/ wttp:v/arrvvvvv/ma orq/ abs//80s..1111/. 01118
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell, 112(1-2):181-
211. 1999.
doi: 00.0016/00000-3779990002222222 URL https:::/cc...gg/////t 17004-3702 (99) 00052-1
Sebastian Thrun and Lorien Y. Pratt. Learning to learn. In Springer US, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems. IROS 2012, pp. 5026-5033, 2012. doi: 10. 10.119//000.2220266660099 URL https:://cci org 1109/ TROS. 2012. 6386109
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pp. 3540- 3549, 2017. JRtlptttp///rroceeeie wttp:iiinsmmms..see..eeeeeeeeeee wwwps//e////sppppp httas//////eeeeeeeeeeaaaaa htmll
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wier- stra. Matching networks for one shot learning. In Advances in Neural Infor- mation Processing Systems 29: Annual Conference Neural Information Process- ing Systems 2016, pp. 5630-3638, 2016. URL hsttp://papers sips.cc/paper.. wtw.aatthhiiitattoooaooooooooooe for- one -hot-hht.- learning
10

---

Under review as a conference paper at ICLR 2020
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs//6l 1.05663, 2016. /RR:htt::/ http/arxvvvvv..... www.i..gg.wa............ wetp:/61ll... 05763
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning explore via meta-policy gradient. In Proceedings of the 35th International Conference on Machine Learning, (CML20/5) pp. 5459- 5468, 2018. URL http::/prroceeeigggs..t mlr http://000xxxx....0d0


Reviews:
[{"review_id": "rkg-Q1gy9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary ###\n\nIn this paper, the authors focus on the problem of meta-reinforcement learning (meta-RL). Specifically, the authors consider the setting of meta-RL for goal reaching tasks where each task corresponds to an unknown goal. Existing meta-RL algorithms directly train for a policy that output low level actions, which might be inefficient in this goal-reaching setting. In this paper, the authors combine the hierarchical RL framework of HAC[1] with the probabilistic task context inference method of PEARL[2], and propose the meta-goal generation for hierarchical RL (MGHRL) algorithm. In this algorithm, a two layer hierarchical policy is used where the high level policy generate goals for the low level goal-reaching policy to reach. In order to adapt to an unknown goal, the high level policy is conditioned on the output of a task inference module to generate goals for the unknown ground truth goal. The goal-reaching policy would then use the generated goal to interact with the environment.\n\nThe authors evaluated the proposed method on simulated robotic manipulation tasks and compare to PEARL as baseline. The experiment results show that the proposed method outperforms the baseline method significantly, especially under sparse reward settings.\n\n\n### Review ###\n\nOverall I think this paper presents an interesting idea in learning fast adapting goal-reaching policies. The idea is very well presented and authors include many empirical evidence to support the proposed method. However I do find a number of shortcomings that need to be addressed.\n\nPro:\n\n1. The idea for this paper is really well presented. The structure of the paper is well organized and  the authors include informative illustration to explain the architecture of the hierarchy of policies. The experiment results are easy to interpret.\n\n2.  The authors provide a detailed description of the configurations and the hyperparameters for each experiments. Such description would be very helpful if the results in this paper are to be reproduced.\n\nCon:\n\n1. The experiments presented in this paper do not include appropriate comparisons to baseline methods. While indeed the proposed method outperforms PEARL, this comparison is inherently unfair. PEARL is a general meta-RL algorithm, which can adapt to arbitrary variations of reward functions and dynamics in the distribution of tasks. The proposed method only applies in the setting of goal reaching meta-RL, where each task corresponds to an unknown goal. With this information artificially encoded into the hierarchical architecture, the proposed method should certainly perform better than any general meta-RL algorithms. Therefore, directly comparing the proposed method to any general meta-RL algorithm is unfair. Instead, the authors could compare with baseline methods with builtin goal-reaching components, such as the following one: train a goal reaching policy using HER[3], and then meta-train a goal conditioned reward function using standard supervised meta-learning methods. At test time, find the goal that maximizes the adapted reward function, and then feed that goal into the HER policy for evaluation. Note that this baseline is different from the proposed method in the way that the goal reaching and goal inference were done separately both using existing methods.\n\n2. I’m not convinced about the novelty of this paper. The proposed method seems like a straightforward combination of HAC and PEARL, and it seems to me that the two methods are combined in order to apply an existing meta-RL algorithm in a goal reaching setting rather than to create a better general meta-RL algorithm.\n\nThe idea in the paper is well presented and carefully investigated. However, I am still not convinced about the novelty of the proposed idea and the magnitude of performance improvement given the lack of proper baselines. Therefore, I would not recommend acceptance before these problems are addressed. \n\nReferences\n\n[1] Levy, Andrew, et al. \"Learning multi-level hierarchies with hindsight.\" (2018).\n\n[2] Rakelly, Kate, et al. \"Efficient off-policy meta-reinforcement learning via probabilistic context variables.\" arXiv preprint arXiv:1903.08254 (2019).\n\n[3] Andrychowicz, Marcin, et al. \"Hindsight experience replay.\" Advances in Neural Information Processing Systems. 2017.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1lVZ2-ZqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update 11/21\nI maintain my score. I like the idea and hope the authors improve the paper and submit to a future conference.\n\nSummary\nThis paper combines hierarchical RL with meta-learning. The idea is that high-level plans transfer across settings (e.g. picking up a mug), while low-level execution may differ across tasks (e.g. different robot morphologies). To this end, the approach meta-learns a two-level hierarchical policy. The higher level policy conditions on a latent task context to produce high-level actions, or goals for the lower level policy. The lower level policy is trained via HER to reach these goals (it may need to be completely re-trained at test time). \n\nConcerns and Questions\nI am very concerned about the experimental results. I do not think that these tasks require hierarchy to solve, as the exact same tasks (with the same simulated robot) were solved in Hindsight Experience Replay, Andrychowicz al. 2017. Thus HER (preferably implemented with SAC rather than DDPG for fair comparison) is a vital baseline that is missing from Figure 3. Could the authors please address this point?\nWhile the introduction and Section 4 claim that one important benefit of such a hierarchical approach is that one could transfer to more disparate tasks, there are no experiments supporting this idea. I think the addition of these experiments would greatly strengthen the paper.\nIn Figure 3, does “PEARL with sparse reward” refer to only the encoder receiving sparse rewards or also the actor-critic?\n\nWriting suggestions\nA suggestion about the title: consider including the word “hierarchical” \nIn some places the writing is quite informal, I suggest revising it (in the intro: “DRL barely works”.\nI disagree with the sentence in the intro “Intuitively this is quite similar to how a human behaves”, which is said in support of the idea of transferring high-level goals instead of low-level execution. Human behavior also supports the opposite view - that we reuse primitive motions over and over in support of new goals. So I think it’s best to avoid the appeal to human behavior here (as well as in related work).\nThe first paragraph of section 4.2 is redundant and can be removed, or at least moved to the beginning of Section 4.\n\nIn conclusion, my current impression is that while the idea is interesting, the results achieve the same performance as a non-hierarchical method, which is not included as a baseline.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyllxgTtFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of leveraging past experience to quickly solve new control tasks. The starting point (and perhaps the main contribution) is the observation that some tasks have similar high-level goals, while differing in how those goals are achieved. To that end, the paper introduces an meta-RL algorithm that, given a new task, attempts to solve it by adapting a high-level, goal-setting module, and learn a new, low-level policy to reach each commanded goal. The proposed method might be viewed as a combination of PEARL [Rakelly 19] and HAC [Levy 19]. The proposed method is compared against state-of-the-art hierarchical RL and meta-RL methods on four robotic manipulation tasks. The proposed method outperforms the baselines on each task.\n\nWhile the proposed method is quite strong empirically, I am leaning towards rejecting this paper because many of the claims made in the paper are not empirically validated. While much emphasis is put on the hierarchical aspect of the algorithm, I don't think that the tasks used in the experiments require hierarchy to solve (see [Plappert 18]). In the introduction, the second claim is that the proposed method \"focus[es] on meta learning the overall strategy … [and] provides a simpler and better way for meta RL.\" While the experiments show that the proposed method learns better than baselines, I don't think the paper show that the proposed method learns some sort of \"overall strategy.\" I don't think that the proposed method is simpler than the baselines.\n\nA second concern is that I'm confused about the experimental protocol. If the high-level policy outputs a desired XYZ position for the gripper (Section 5.1), how can the high-level policy indicate when the gripper should be closed to pick up the block? How is the reward function for the low-level policy defined? PEARL doesn't have access to this extra information (the reward function), right?\n\nA third concern is the large number of grammatical errors in the paper.\n\nI would consider increasing my review if (1) a new plot were added to visualize the commanded subgoals (I have a hunch that the high-level policy directly outputs the true goal, obviating the need for hierarchy and contradicting the claim that the method \"learns to generate high-level meta-strategies over subgoals\"); (2) the experimental protocol were clarified; and (3) the number of grammatical errors were significantly reduced.\n\nOther comments:\n* \"inefficient to to directly learn such a meta policy\" -- Why? Also, \"to to\" is repeated.\n* \"Deep Reinforcement learning\" -- \"Reinforcement\" shouldn't be capitalized.\n* \"failing to generalize\" -- Can you add a citation?\n* \"it would be quite inefficient to directly learn such a policy…\": Doesn't [Plappert 18] do exactly this?\n* \"When the tasks distribution is much wider … these methods can hardly be effective…\" -- Where is this claim substantiated? Also, \"tasks\" should be singular.\n* \"sparse reward settings which is\" -> \"sparse reward settings, which are\"\n* \"the above mentioned problems\" -> \"the problems mentioned above\"\n* \"our algorithm focus on meta learning … which provides a much simpler …\" -- Where is it shown that the proposed method is simpler? Also, \"focus\" should be \"focuses\"\n* \"1991),which\" -- Missing space\n* \"complex tasks which requires\" -> \"complex tasks that require\"\n* \"Nachum et al … set of sub-policies\" -- Run on sentence.\n* \"... human leverage…\" -- Don't humans also transfer low-level knowledge across tasks, in addition to high-level knowledge? Also, \"human\" should be plural.\n* \"algorithms.The\" -- Missing a space, I think.\n* \"PEARL leverages … latent variable Z\" -- This sentence doesn't make sense as written.\n* \"z's\" -- This should not be a possessive.\n* \"Good sample efficiency enables fast adaptation … and performs structured exploration …\" -- Isn't the first part true by definition? Why does good sample efficiency perform structured exploration?\n* \"a goal is a 3-d vector\" -- If the goal output by the high-level policy is the XYZ coordinates of the \n* \"SAC, such non-hierarchical\" -- Grammar doesn't make sense here.\n* \"such non-hierarchical RL method has been proved to perform badly before on …\" -- Can you add a citation? Generally, \"proved\" is reserved for mathematical proofs.\n* \"In this paper, We have\" -- \"We\" should not be capitalized.\n\n------------------------ UPDATE AFTER AUTHOR RESPONSE ------------------\nI thank the authors for at least reading the reviews. My concerns with experiments and clarify remain unaddressed, and are amplified by reading the other reviews. I therefore vote to \"reject\" this paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper combines PEARL with HAC to create a hierarchical meta-RL algorithm that operates on goals at the high level and learns low-level policies to reach those goals. Reviewers remarked that it’s well-presented and well-organized, with enough details to be mostly reproducible. In the experiments conducted, it appears to show strong results.

However there was strong consensus on two major weaknesses that render this paper unpublishable in its current form: 1) the continuous control tasks used don’t seem to require hierarchy, and 2) the baselines don’t appear to be appropriate. Reviewers remarked that a vital missing baseline is HER, and that it’s unfair to compare to PEARL, which is a more general meta-RL algorithm. The authors don’t appear to have made revisions in response to these concerns.

All reviewers made useful and constructive comments, and I urge the authors to take them into consideration when revising for a future submission.
Decision: Reject