Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks
Authors: Lukas Jendele
Sammy Christen
Emre Aksan
Otmar Hilliges
Venue:  (2020)
Keywords: Hierarchical reinforcement learning
planning
navigation

Abstract: Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl

Under review as a conference paper at ICLR 2020
LEARNING FUNCTIONALLY DECOMPOSED HIERARCHIES FOR CONTINUOUS NAVIGATION TASKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity. it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on lower level with an image-based planning policy on higher level. We evaluate our method various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos ht:ps://sites.ce google com/ woom/viee /vv./viwwwww.. wwww/hide-w/ hide-rl
INTRODUCTION
Reinforcement learning (RL) has been succesfully applied to sequential-decision making tasks, such as learning how to play video games in Atari (Mnih al. (013), mastering the game of Go (Silver eet al. 2017) or continuous control in robotics (Lilicrrap et al. 2015 Levine et al. 2015 Schulman al. 2017). However, despite the success of RL agents in learning control policies for myopic tasks, such as reaching a nearby target, they lack the ability to effectively reason over extended hori- zons. In this paper, we consider continuous control tasks that require planning over long horizons in navigation environments with sparse rewards. The task becomes particularly challenging with do-
sparse and delayed rewards since an agent needs to infer which actions caused the reward main where most samples give no signal at all. Common techniques to mitigate the issue of sparse rewards include learning from demonstrations (Schaal 1999 Peng et al. 2018) or using enhanced exploration strategies (Bellemare et al., 2016 Pathak al. 2017 Andrychowicc et al.) 2017).
Hicrarchical Reinforcement Learning (HRL) has been proposed in part to solve such tasks. Typi- cally, a sequential decision making task is split into several simpler subtasks of different temporal and functional abstraction levels (Sutton et al. 1999 Andre & Russell, 2002). Although the hierar- chies would ideally be learned in parallel, most methods resort curriculum learning (Frans al.) Florensa et al. 2017 Bacon et al. [016 Vezhnevets et al. 2017). Recent goal-conditioned hi- erarchical architectures have successfully trained policies jointly via off-policy learning (Levy al. 2019 Nachum et al. 2018 2019). However, these methods often do not generalize to unseen envi- ronments as we show 1.c We argue that this is due lack of true separation of planning and low-level control across the hierarchy. In this paper, we consider two main problems, namely functional decomposition of HRI. architectures in navigation-ban domains and generalization of RL agents to unseen environments (figure I].
2017
To address these issues, we propose a novel multi-level HRL architecture that enables both func- tional decomposition and temporal abstraction. We introduce a 3-level hierarchy that decouples the major roles in a complex navigation task, namely planning and low-level control. The benefit of modular design is twofold. First, layers have access only task-relevant information for pre- defined task, which significantly improves the generalization ability of the overall policy. Hence, this enables policies learned single task solve randomly configured environments. Second,

---

Under review as a conference paper at ICLR 2020
Training
Test
b)
d)
Figure I: Navigation environments. The red sphere indicates the goal an agent needs to reach, with the starting point at the opposite end of the maze. The agent is trained environment To test generalization, we use the environments with b) reversed starting and goal positions, c) mirrored maze with reversed starting and goal positions and randomly generated mazes.
planning and control layers are modular and thus allow for composition cross-agent architec- tures. We empirically show that the planning layer of the hicrarchy can be transferred successfully to new agents. During training we provide global environment information only to the planning layer, whereas the full internal state of the agent is only accessible by the control layer. The actions of the top and middle layers are in the form displacement in space. Similarly, the goals of the middle and lowest layers relative the current position. This prevents the policies from overfitting to the global position in an environment and hence encourages generalization to new environments.
In our framework (see figure 2). the planner (i.e., the highest level policy т2) learns to find trajec- tory leading the agent to the goal. Specifically, we learn a value map of the environment by means of value propagation network (Nardelli et al., 2019). To prevent the policy from issuing too ambitious subgoals, an attention network estimates the range of the lower level policy To (i.c., the agent). This attention mask also ensures that the planning considers the agent performance. The action of 22 the position which maximizes the masked value map, which serves goal input to the policy The middle layer implements an interface between the upper planner and lower control layer, which refines the coarser subgoals into shorter and reachable targets for the agent. The middle layer is crucial in functionally decoupling the abstract task of planning (T₂) from agent specific continuous control. The lowest layer learns control policy the policies are functionally decoupled, they are trained together and must learn to cooperate.
steer the agent intermediate goals. While
In this work, focus solving long- monghhorizon tasks with sparse rewards in complex continuous navigation domains. We first show in a maze environment that generalization causes challenges for sfate-of-the-ar approaches. We then demonstrate that training with the same environment configura- tion (i.e., fixed start and goal positions) can generalize to randomly configured environments. Lastly, show the benefits of functional decomposition via transfer of individual layers between different agents. In particular, we train our method with simple 2DoF ball agent in maze environment to learn the planning layer which later used to steer a more complex agent. The results indicate that the proposed decomposition of policy layers is effective and can generalize to unseen environments. In summary our main contributions include:
A novel multi-layer HRL architecture that allows functional decomposition and temporal abstraction for navigation tasks.
This architecture enables generalization beyond training conditions and environments.
Functional decomposition that allows transfer of individual layers across different agents.
RELATED WORK
Learning hierarchical policies has seen lasting interest (Sutton et al. 1999 Schmidhuber 1991 Dietterich 1999 Parr & Russell 1998 McGovern & Barto 2001 Dayan & Hinton 2000). but many approaches are limited to discrete domains or induce priors.
More recent works (Bacon al. 2016 Vezhnevets al. 2017 Tirumala al. 2019 Nachum al. 2018 Levy et al. 2019) have demonstrated HRL architectures in continuous domains. Vezhnevets al. 2017) introduce FeUdal Networks (FUN), which was inspired by feudal reinforcement learn-
2

---

Under review as a conference paper at ICLR 2020
Say
joints
2
92
91
T
Control
Actions
Figure 2: Our 3-layer HRL architecture. The planning layer T2 receives birds eye view of the environment and the agent's position Say and sets an intermediate target position f2. The interface layer m2 splits this subgoal into reachable targets 91- A goal-conditioned control policy no learns the required motor skills to reach the target 91 given the agent's joint information $joints.
ing (Dayan & Hinton, 2000). In FUN, a hierarchic decomposition is achieved via a learned state representation in latent space. While being able to operate in continuous state space, the approach limited to discrete action spaces. Tirumala et al.] (2019) introduce hierarchical structure into KL- divergence regularized RL using latent variables and induces semantically meaningful representa- tions. The separation of concerns between high-level and low-level policy is guided by information asymmetry theory. Transfer of resulting structure can solve speed up training of new tasks or different agents, Nachum et al. (2018) present HIRO, an off-policy HRL method with two levels of hierarchy. The non-stationary signal of the upper policy mitigated via off-policy corrections, while the lower control policy benefits from densely shaped rewards. Nachum et al. (2019) propose an extension of HIRO, which we call HIRO-LR, by learning representation space from environ- ment images, replacing the state and subgoal space with neural representations. While HIRO-LR can generalize to flipped environment, it needs to retrained, as only the learned space repre- sentation generalizes. Contrarily, HiDe generalizes without retraining. Levy et al. (2019) introduce Hicrarchical Actor-Critic (HAC). an approach that can jointly learn multiple policies in parallel. The policies trained in sparse reward environments via different hindsight techniques. HAC, HIRO and HIRO-LR consist of a set of nested policies where the goal of a policy is provided by the top layer. In this setting the goal and state space of the lower policy is identical to the action space of the upper policy. This necessitates sharing of the state space across layers. Overcoming this limitation, introduce modular design decouple the functionality of individual layers. This allows us to define different state, action and goal spaces for each layer. Global information about the envi- romment is only available the planning layer, while lower levels only receive information that is specific to the respective layer. Furthermore, HAC and HIRO have state space that includes the agent's position and the goal position, while (Nachum et al. 2019) and our method both have access to global information in the form of a top-down view image.
In model-based reinforcement learning much attention has been given learning of a dynamics model of the environment and subsequent planning (Sutton) 1990 Sutton et al. 20122; Wang et al.) 2019]. Eysenbach et al. (2019) propose planning method that performs graph search over the replay buffer. However, they require to spawn the agent at different locations in the environment and let it learn a distance function in order to build the search graph. Unlike model-based RL, we do not learn state transitions explicitly. Instead, we learn spatial value map from collected rewards.
Recently, differentiable planning modules that can be trained via model -rree reinforcement learning have been proposed (Tamar et al., 2016 Oh et al. 2017]] Nardelli et al.) 2019 Srinivas al. 2018). Tamar et al. (2016) establish a connection between convolutional neural networks and Value Itera- tion (Bertsekas) 2000). They propose Value Iteration Networks (VIN), an approach where model- free RL policies are additionally conditioned on a fully differrentiable planning module. MVProp (Nardelli et al. 2019) extends this work by making it more parameter-efficient and generalizable. The planning layer in our approach is based on MVProp, however contrary to prior work we do not rely on a fixed neighborhood mask to sequentially provide actions in its vicinity in order to reach a goal. Instead we propose to learn an attention mask which is used to generate intermediate goals for the underlying layers. Gupta et al. (2017) learn a map of indoor spaces and planning on it using a multi-scale VIN. In their setting, the policy is learned from expert actions using supervised learning. Moreover, the robot operates on discrete set of high level macro actions. [Srinvas et al. (0018) pro- pose Universal Planning Networks (UPN), which learn how plan an optimal action trajectory via latent space representation. In contrast to our approach, the method relies on expert demonstrations and transfer to harder tasks can only be achieved after retraining.
3

---

Under review as a conference paper at ICLR 2020
3 BACKGROUND
3.1
ROAL-CONDTTIONED REINFORCEMENT LEARNING
We model a Markov Decision Process (MDP) augmented with of goals G. We define the MDP as a tuple M = {S, A, R. T, Po. 7}, where S and A are set of states and actions, respectively, R₁ = rst,, ats 9t) reward function, 7 discount factor [0. 1], T p(*++ |s, at) the transition dynamics of the environment and Po p({1) the initial state distribution, with & S and E A. Each episode initialized with a goal g E and initial state is sampled from Pa- We aim find policy S G + A, which maximizes the expected return.
We train our policies by using an actor-critic framework where the goal augmented action-value function is defined as:
Q(s, = B
Σ x ((()))), =. 9.
(1)
The Q-function (critic) and the policy (actor) approximated by using neural networks with parameters 0 and fx The objective for 0 minimizes the loss:
L( =EM
EM [Q(5L) 9t, ar; 08) 91)
where
(2)
yt rst, 9c. a²) + (1 I++1, a141; 82)
The policy parameters are trained to maximize the Q-value:
T(0*) F (s It, az; 02 |8e; 9t. at (8t, 91;
(8t, 91;0)))))) 0*]
To address the issue of sparse rewards, we utilize Hindsight Experience Replay (HER) (Andrychow- icz et al., 2017), technique to improve sample-efficieng in training goal-conditioned environ- ments. The insight is that the desired goals of transitions stored the replay buffer can be rela- beled states that were achieved in hindsight. Such data augmentation allows learning from failed episodes, which may generalize enough to solve the intended goal.
3.2 HINDSIGHT TECUNIQUES
In HAC, Levy et al. (2199) apply two hindsight techniques address the challenges introduced by the non non-stationary nature hierarchical policies and the environments with sparse rewards. In order train policy mi optimal behavior of the lower-level policy is simulated by hindsight action transitions. More specifically, the action a; is replaced with state 8;-1 that is actually achieved by the lower-level policy -- Identically to HER, hindsight goal transitions replace the subgoal 9:-1 with an achieved state s;-1, which consequently assigns reward to the lower-level policy T;-1 for achieving the virtual subgoal. Additionally, a third technique called subgoal testing proposed. The incentive of subgoal testing is to help a higher-level policy understand the current capability of a lower-level policy and to learn Q-values for subgoal actions that are out of reach. We find both techniques effective and apply them to our model during training.
3.3
VALUE PROPAGATION NETWORKS
Tamar et al. (2016) propose differentiable value iteration networks (VIN) for path planning and navigation problems. Nardelli et al. (2019) propose value propagation networks (MVProp) with better sample efficiency and generalization behavior. MVProp creates reward- and propagation maps covering the environment. The reward map highlights the goal location and the propagation map determines the propagation factor of values through a particular location. The reward map is an image Fr, of the same size as the environment image 7, where Ft, = 0 if the pixel (i,j) overlaps with the goal position and -1 otherwise. The value map V is calculated by unrolling max-pooling operations in neighborhood N for ls steps as follows:
Fs.j
(4)
(k-1) max Uij
(Fi. - Pr Fii)) ((N
0 [.j
= max
4

---

Under review as a conference paper at ICLR 2020
2
Attention
M
G
MVProp
arginax -xy -
T
Figure 3: Planner layer (2(S G) Given the top-view environment image I and goal G on the map. the maximum value propagation network (MVProp) calculates a value map V. By using the agent's is current position Suy. we estimate an attention mask M restricting the global value map V to a local and reachable subgoal map V. The policy π2 selects the coordinates with maximum value and assigns the lower policy #1 with a sugboal that is relative the agent's current position.
The action (i.e., the target position) selected be the pixels (i',j") maximizing the value in predefined 3r3 neighborhood N (i). of the agent's 's current position (do:
x ss (io. jo))
argmax E
(5)
Note that the window ((io: Jo) is determined by the discrete, pixel-wise actions.
4 HIERARCHICAL DECOMPOSITIONAL REINFORCEMENT LEARNING
We introduce novel hierarchical architecture, HiDe, allowing for an explicit functional decompo- sition across layers. Similar to HAC (Levy et al. 2019). our method achieves temporal abstractions via nested policies. Moreover, our architecture enables functional decomposition explicitly. This achieved by nesting i) an abstract planning layer, followed ii) by a local planer to iii) guide control component. Crucially, only the top layer receives global information and responsible for planning trajectory towards goal. The lowest layer learns control policy for agent locomotion. The mid- dle layer converts the planning layer output into subgoals for the control layer. Achieving functional decoupling across layers crucially depends reducing the state in each layer to the information that relevant to its specific task. This design significantly improves generalization (see Section 5
4.1 PLANNING LAYER
The highest layer of hierarchical architecture is expected learn high-level actions over longer horizon, which define a coarse trajectory in navigation- -based tasks. In the related work (Levy et al.) 2019 Nachum et al. 2018 2019). the planning layer, learning an implicit value function, shares the same architecture as lower layers. Since the task is learned for specific environment, limits to generalization are inherent to this design choice. In contrast, we introduce planning specific layer consisting of several components to learn the map and to find a feasible path to the goal.
The planning layer is illustrated figure We utilize value propagation network (MVProp) (Nardelli et al., 2019) to leam an explicit value map which projects the collected rewards onto the environment image. Given top-down image of the environment, a convolutional network deter- mines the pixel flow probability Pt,j- For example, the probability value of pixel corresponding to wall should be and that for free passages respectively.
Nardelli et al. (2019) use predefined x 3 neighborhood of the agent's current position and pass the location of the maximum value in this neighbourhood as goal position to the agent (equation 5). We augment& MVProp network with an attention model which learns define the neighborhood dynamically and adaptively. Given the value map V and the agent's 's current position 8zy. we estimate how far the agent can go, modeled by 2D Gaussian. More specifically, we predict a full covariance matrix Σ with the agent's global position Sxy as mean. We later build a 2D mask M of the same size the environment image / by using the likelihood function:
mij
((t,))))aa x)
5

---

Under review as a conference paper at ICLR 2020
Figure 4: A visual comparison of (left) our dynamic attention window with a (right) fixed neighborhood. The green dot corresponds to the se- lected subgoal this case. Notice how our window is shaped so that avoids the wall and induces a fur- ther subgoal.
Intuitively, the mask defines the density for the agent's success rate. Our planner policy selects an action (i.e., subgoal) that maximizes the masked value map as follows:
V
( G.I) argmax Vijj
1
92 T2 (
Szy
where Us corresponds to the value at pixel (i,j) on the masked value map V. Note that the subgoal selected by the planning layer I2 is relative to the agent's current position Suys which improves generalization performance of our model.
The benefits of having attention model are twofold. First, the planning layer considers the agent dynamics in assigning subgoals which may lead to fine- or coarse-grained subgoals depending on the underlying agent's performance. Second, the Gaussian window allows us to define dynamic set of actions for the planner policy T2, which is essential to find a trajectory of subgoals on the map. While the action space includes all pixels of the value map V, it is limited the subset of only reachable pixels by the Gaussian mask M. Qualitatively we find this leads to better obstacle avoidance behaviour such as the corners and walls shown in figure
Since our planner layer operates in discrete action space (i.e., pixels), the resolution of the projected maze image defines the minimum amount of displacement for the agent, affecting maneuverability. This could be tackled by using a soft-argmax (Chapelle & Wul 2010) to select the subgoal pixel, allowing to choose real-valued actions and providing in- in-variance to image resolution. In our exper- iments we see difference in terms of the final performance. However, since the former setting allows for the use of DQN (Mnih et al., 2013) instead DDPG (Silver et al. 2014), we prefer the discrete action space for simplicity and faster convergence.
Both the MVProp (equation 4 and Gaussian likelihood (equation 6 operations are differentiable. Hence, MVProp and the attention model parameters trained by minimizing the standard mean squared Bellman error objective as defined equation?
4.2 INTERFACE LAYER
The middle layer in our hierarchy interfaces the high-level planning with low-level control by intro- ducing an additional level of temporal abstraction. The planner's 's longer-term goals are further split into a number shorter-term targets. Such refinement policy provides the lower-level control layer with reachable targets, which in return yields easier rewards and hence accelerated learning.
The interface layer policy is the only layer that is directly interacting with the environment. More specifically, the policy Ti only receives the subgoal ga from the upper layer T2 and chooses an action (i.e. subgoal Ø1) for the lower-level locomotion layer To. Note that all the goal, state and action spaces of the policy are in 2D space. Contrary to Levy et al. (2019), we use subgoals that are relative to the agent's ' position Say. This helps to generalize and learn better.
4.3 CONTROL LAYER
The lowest layer learns goal-conditioned control policy. Due to explicit functional decomposi- tion, it is the only layer with access to the agent's 's internal state jjoints including joint positions and velocities. Whereas the higher layers only have access to the agent's position. In navigation task, the agent has to learn locomotion to reach the goal position. Similar to HAC, we use hindsight goal transition techniques that the control policy receives rewards even in failure cases.

---

Under review as a conference paper at ICLR 2020
100
MAC
Experiment 1
Forward
Backward
Flipped
EERO
Success Rate
MiDe
HАC
82 + 16
0 0 0
0 + 0
HIRO
99 =1
0 0
0
HIRO-LR
97
0
0
m
RelHAC
27
4
0+0
20
HiDe-R
89 +3
61 + 14
90 + 3
HiDe
85
55
69
40
SOOK
2000K
2500K
Total steps
Figure 5: Success in forward maze wrt the number of environment steps. All algorithms eventually learn the task. HIRO converges the fastest because it benefits from dense rewards. The results are averaged over seeds.
Table 1: Success rates of achieving goal with Ant agent in simple maze environment. All al- gorithms except for HiDe have been trained with randomly sampled goal positions. The results are averaged over seeds.
All policies in our hierarchy are jointly-trained. We use the DDPG algorithm (Lillicrap et al. 2015) with the goal-augmented actor-critic framework (equation 2 and DQN (Mnih et al. 2013) for the planning layer (see section 4.1
B for the control and interface layers,
EXPERIMENTS
We evaluate our method on a series simulated continuous control tasks in navigation based en- vironments] All environments are simulated in the MuloCo physics engine (Toorovv et al. 2012) Experiment and implementation details are provided in the Appendix B First, in section 5. we compare to various bascline methods. In section 5.2 move to new maze with more complex design in order to show our model's generalization capabilities. Section 5.3 demonstrates that our approach indeed leads to functional decomposition by composing planning layer of one agent with the locomotion layer of another. Finally, in section 5.4 we provide an ablation study for our design choices. We introduce the following task configurations:
agents via combining the
Maze Forward: the training environment in all experiments. The task is to reach a goal from a fixed pre-determined start position.
Maze Backward: the training maze layout with swapped start and goal positions.
Maze Flipped: a mirrored version of the training environment.
Maze Random: a set of randomly generated mazes with random start and goal positions.
In experiments, we always train in the Maze Forward environment. The reward signal during training is constantly -1, unless the agent reaches the given goal (except for HIRO and HIRO-LR, see section[5.1] (.1).. We test the agents on the above tasks with fixed starting and fixed goal position. For more details about the environments, we refer to Appendix We intend to answer the following two questions: 1) Can our method generalize to unseen test environments? 2) it possible transfer the planning layer policies between agents?
5.1 SIMPLE MAZE NAVIGATION
We compare our method tfate-of-the-ar approaches including HIRO (Nachum 2019 HIRO-LR (Nachum et al. 2019), HAC (Levy et al. 20199 and modified version of HAC called Rel- HAC in a simple Maze Forward environment as shown figure For fair comparison, we made number of improvements to the HAC and HIRO implementations. For HAC, we introduced target networks and used the hindsight experience replay technique with the future strategy (Andrychow- observed that oscillations around the goal kept HIRO agents
icz et al. 2017). In experiments from finishing the task, which was solved via doubling the distance-threshold of success. HIRO-LR method, as it also receives top-down view image of the environment. Note that both HIRO and HIRO-LR have access to dense negative distance reward, which is an advantage HAC and HiDe that only receive reward when finishing the task.
the closest
Videos available htttss:///sstttt www.oo/......ee..ooooooo http://oggl....cco///e/occcaeaee

---

Under review as a conference paper at ICLR 2020
Experiment
Ant
Ball
Experiment
Ant
Ball
Ball + Ant
Forward
81 = 8
96 7
Forward
100 = 0
66 = 14
Random
89 =
96 =
Random
97 =
86 =
Backward
56 8
100 0
Backward
98 +
53
Flipped 74 = 11
99 +
Flipped
100 0
59 + 27
Success rates in the complex maze. We train ant and ball For- ward maze and test on unseen Random, Backward, and Flipped environments.
Table 2:
Table 3: Success rates of achieving a goal in complex maze environment with transferred agents. We train with a different agent on For- ward maze.
We train modified HAC model, dubbed RelHAC, to asses our planning layer. ReiHAC has the same lowest and middle layers HiDe, whereas the top layer has the same structure the middle layer. therefore missing an effective planner. Preliminary experiments using fixed start and fixed goal positions during training for HAC, HIRO and HIRÓ-LR yielded 0 success rates in all cases. Therefore, the baseline models are trained by using fixed start and random goal positions. allowing to receive a reward signal without having reach the intended goal the other end of the maze. Contrarily, HiDe is trained with fixed start and fixed goal positions, whereas HiDe-R represents HiDe under the same conditions as the baseline methods.
All models learned this task successfully as shown in figure 5 and demonstrates slightly better convergence and final performance, which can be attributed to the fact that it is trained with dense rewards. RelHAC performs worse than HAC due to the pruned state space of each layer and due to the lack of an effective planner. HIRO-LR takes longer to converge because it has learn latent goal space representation.
column). HIRO
Table summarizes the models' generalization abilities to the unseen Maze Backward and Maze Flipped environments (see figure While HIRO, HIRO-LR and HAC manage solve the training environment (Maze Forward) with success rates between 99% and 82%, they suffer from overfiting the training environment, indicated by the 0% success rates in the unseen test environments. Contrarily, our method is able to achieve 54% and 69% success rates in this generalization task. As expected, training our model with random goal positions (i.e., HiDe-R) yields more robust model outperforming vanilla HiDe. In subsequent experiments, we only report the results for our method, as our experiments have shown that the baseline methods cannot solve the training task for more complex environments.
5.2 COMPLEX MAZE NAVIGATION
In this experiment, we train an ant and a ball agent (see Appendix A.I with more complex environment layout (ef. figure D. while keep both the start and goal positions intact. We then evaluate this model in 4 different tasks (see section 5)
in the Maze Forward task
Table reports success rates of both agents in this complex task. Our model successfully transfers its navigation skills to unseen environments. The performance for the Maze Backward and Maze Flipped tasks is similar the results shown in section [. the increased difficulty. Since the randomly generated mazes are typically easier, our model shows similar or better performance.
5.3 TRANSFER OF POLICIES
To demonstrate that the layers in our architecture indeed learn separate sub-tasks we transfer indi- vidual layers across different agents. We first train agent without our planning layer, i.e., with RelHAC. We then replace the top layer this agent with the planning layer from the models trained in section 5.2 Additionally, we train a humanoid agent and show proof of concept that transfer to very complex agent can be achieved.
We carry out two sets of experiments. First, we transfer the ant model's planning layer to the simpler 2 DoF ball agent. As indicated in Table the performance of the ball with the ant's planning layer matches the results in Table 2 The ball agent's success rate increases for random (from 96% to 100%) and forward (96% to 97%) maze tasks whereas decreases slightly the backward (from 100% to 90%) and flipped (from 99% to 88%) configurations.
8

---

Under review as a conference paper at ICLR 2020
Ant agent
Fwd
Back
Flip
Ant agent
Fwd
Rand
Back
Flip
HiDe-A
+ 0
0 0+0 0
0 + 0
HiDe-A
0 +
0
+ 0
+ 0
HiDe-AR
95 + 1
52 + 33
34 + 45
HiDe-AR
0
0+0
0 0
0 0+0
HiDe-NI
10 10=5 5
46 +
16
0
-
0 3
Table 4: Success rates in the simple maze. HiDe-A is our method with absolute sub- goals. HiDe-AR has absolute goals and samples random goals during training.
Table 5: Success rates of achieving goal in the complex maze environment. HiDe-A and HiDe- AR as Table 4 HiDe-NI is our method without the inferface layer.
Second, we attach the ball agent's planning layer to the more complex ant agent. Our new com- positional agent performs marginally better or worse in the Flipped, Random and Backward tasks. Please note that this experiment is an example of a case where the environment is first learned with fast and easy-to-train agent (i.e., ball) and then utilized by more complex agent. We hereby show that transfer of layers between agents is possible and therefore find our hypothesis to be valid. Moreover, an estimate indicates that the training roughly 4 times faster, since the complex agent does not have to learn the planning layer.
To demonstrate our method's 's transfer capabilities, we train humanoid agent (17 DoF) in an empty environment with shaped rewards. We then use the planning and interface layer from a ball agent and connect as is with the locomotion layer of the trained humanoid
5.4 ABLATION STUDIES
To support the claim that our architectural design choices lead to better generalization and functional decomposition, we compare empirical results for different variants of our method with an ant agent. First, we compare the performance of relative and the absolute positions for both experiment 1 and experiment 2. For this reason, we train HiDe-A and HiDe-AR, the corresponding variants of HiDe and HiDe-R that use absolute positions. Unlike for relative positions, the policy needs learn all values within the range of the environment dimensions. Second, we compare HiDe against a variant of HiDe without the interface layer called HiDe-NI.
The results for experiment I are in Table 4 HiDe-A does not manage to solve the task at all, similar to HAC and HIRO without random goal sampling. HiDe-AR succeeds solving the Forward task. However, it generalizes worse than both Hide and HiDe-R in the Backward and Flipped task. Both HiDe-A and HiDe-AR fail to solve the complex maze for experiment 2 as shown the Table 5 These results indicate that 1) relative positions improve performance and are an important aspect of our method to achieve generalization to other environments and 2) random goal position sampling can help agents, but may not be available depending the environment. As seen in Table [5] the variant of HiDe without interface layer (HiDe-NI) performs worse than HiDe (cf. Table 2 in all experiments. Thus, the interface layer is important part of architecture.
We also run an ablation study for HiDe with a fixed window size. More specifically, we train and evaluate an ant agent on window sizes 33 5,5, and 9x9. The results are included in Tables 12 13 and 14 The learned attention window (HiDe) achieves better or comparable performance. In all cases, HiDe generalizes better in the Backward complex maze. Moreover, the learned attention eliminates the need for tuning the window size hyperparameter per agent and environment.
6 CONCLUSION
In this paper, we introduce a novel HRL architecture that can solve complex navigation tasks in 3D-based maze environments. The architecture consists a planning layer which learns an explicit value map and is connected with a subgoal refinement layer and low-level control layer. The frame- work can be trained end-to-end. While training with a fixed starting and goal position, our method is able to generalize to previously unseen settings and environments. Furthermore, we demonstrate that transfer of planners between different agents can be achieved, enabling us to transfer a planner trained with simplistic agent such ball to more complex agent such as In future work, we want to consider integration of a more general planner that is not restricted to navigation-based environments.
humanoid.
Videos available httpt:tt///// / tes http:/sssssssssslsoooeo//sssssst googl wttp://www/.eeeeeeeeteeeeeeooeee

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Andre and Stuart Russell. State abstraction for programmable reinforcement learning agents. 08 2002.
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Re- play. In Advances in Neural Information Processing Systems, pp.
2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. abs/1609.00 htss1609.05140, 2016.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. abs/1606.01 0668. 2016.
Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 2nd edition, 2000. ISBN 1886529094.
Olivier Chapelle and Mingrui Wu. Gradient descent optimization of smoothed information retrieval metrics. Information retrieval, 13(3):216-25 235, 2010.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in Neural Information Processing Systems, 5, 09 2000. doi: 10. (0000///07021442.paa/o//0
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decom-
position. cs.9G99905014, 1999.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Rein- forcement Learning for Continuous Control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. arXiv preprint arXiv:1906.05253 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. abs/1704.03011. 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. abs/1710.09767, 2017.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2616-2625, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. abs/1504.00702, 2015.
Andrew Levy, Robert Platt, and Kate Saenko. Learning Multi-Level Hierarchies with Hindsight. In International Conference on Learning Representations, 2019.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning, arXiv preprint 1009::500000250 2015.
Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pp. pp.36p..111. 361-368, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-778-1.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. abs/131,.5602, 2013.
10

---

Under review as a conference paper at ICLR 2020
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient Hierarchical Reinforcement Learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-Optimal Representation Learn- ing for Hierarchical Reinforcement Learning. International Conference on Learning Repre- sentations, 2019.
Nantas Nardelli, Gabriel Synnaeve, Zeming Lin, Pushmeet Kohli, Philip H. S. Torr, and Nicolas Usunier. Value Propagation Networks. In International Conference on Learning Representations, 2019.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. abs/1 /a/1777.03399, 2017.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Proceed- ings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS '97, pp. 1043-1049, Cambridge, MA, USA, 1998. MIT Press. ISBN 0-262-10076-2.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. abs/1705.05363. 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. N0V/1804.0277., 2018.
Stefan Schaal. Is imitation learning the route humanoid robots? Trends in Cognitive Sciences, 3: 233-242, 1999.
Jürgen Schmidhuber. Learning to generate subgoals for action sequences. IICNN-91-Seatle Inter- national Joint Conference Neural Networks, ii:453 vol.2-, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic Policy Gradient Algorithms. In International Conference Machine Learning, pp. 387-395, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-, October 2017.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. abs/1 10b/00844 2018
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In MI., 1990.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artif. Intell., 112(1-2):181221 August 1999. ISSN 0004-3702. doi: 10. (0.006650044-3702099000005222
Richard S. Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael Bowling. Dyna-style plan- ning with linear function approximation and prioritized sweeping. abs/1206.3285, 2012.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Picter Abbeel. Value Iteration Networks. In Advances in Neural Information Processing Systems, pp. 2154-2161 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Ried- miller. DeepMind Control Suite. Technical report, January 2018.
11

---

Under review as a conference paper at ICLR 2020
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Tch, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized RL. CoRR, abs/1903.07438, 2019. httt::/aaaxxi
abs/1 1903. (0t0/1400307.88
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. abs/1703.01161,
2017. 2017.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. abs/1907.02057, 2019.
12

---

Under review as a conference paper at ICLR 2020
A ENVIRONMENT DETAILS
We build on the Mujoco Todorov et al. environments use dt 0.02. Each episode in experiment is terminated after 500 steps and after 800 steps in the rest of the experiments or after the goal in reached. All rewards are sparse as Levy et al. 2019), i.e., 0 for reaching the goal and -1 otherwise. We consider goal reached 8 max 1. Since HIRO sets the goals in the far distance to encourage the lower layer to move to the goal faster, can't 't stay exactly at the target position. Moreover, they do not terminate the episode after the goal is reached. Thus for HIRO, consider goal reached if gl2 < 2.5.
2012) environments used in Nachum et al. (2018) All
A.1 AGENTS
Our Ant agent is equivalent to the one in Levy al. (2119) In other words, the Ant from Rllab (Duan et al. (016) with gear power of 16 instead 150 and 10 frame skip instead of 5. Our Ball agent the PointMass agent from DM Control Suite (Tassa al. 2018). We made the change the joints so that the ball rolls instead of sliding. Furthermore, we resize the motor gear and the ball itself to match the maze size.
A.2 MAZES
All mazes are modelled by immovable blocks of size 4 of 8 x x 8. The environment shapes are clearly depicted in figure ! For the randomly generated maze, we sample each block with probability being empty sampled randomly at uniform. Mazes where start and goal positions are adjacent or where goal is not reachable are discarded. For the evaluation, we generated 500 such environments and reused them (one per episode) for all experiments.
Nachum et al. (2018) uses blocks
0.8,start and goal positions are also
Maze Forward
Maze Backward
Maze Flipped
T
Figure 6: Environments used for testing in Section 5.1
The red sphere indicates the goal of the task.
B IMPLEMENTATION DETAILS
Our PyTorch (Paszke et al. 2017) implementation will be available at the project website.
B. BASELINE EXPERIMENTS
For both HIRO and HAC we used the original authors implementatiorfl In HIRO, we set the goal success radius for evaluation described above. We ran the hiro tiro_xy variant, which uses only position coordinates for subgoal instead of all joint positions, to have a fair comparison with our method. To improve the performance of HAC in experiment one, we modified their Hindsight Expe- rience Replay (Andrychowicz et al.) 2017) implementation so that they use FUTURE strategy. More importantly, we also added target networks to both the actor and critic improve the performance.
afficient-hr* "HAChttps: hMt.tt/tess////aooooo//aaaraeeee
13

---

Under review as a conference paper at ICLR 2020
B.2 EVALUATION DETAILS
For evaluation, we trained seeds each for 2.5M steps on the Forward environment with continuous evaluation (every 100 episodes for 100 episodes). After training, we selected the best checkpoint based on the continuous evaluation of each seed. Then, we tested the learned policies for 500 episodes and reported the average success rate. Although the agent and goal positions are fixed, the initial joint positions and velocities are sampled from uniform distribution as standard in OpenAl Gym environments. Therefore, the tables in the paper contain means and standard deviation across 5 seeds.
B.3 NETWORK STRUCTURE
B.3.1 PLANNING LAYER
Input images for the planning layer binnarized in the following way: each pixel corresponds to one block (0 if was a wall or it was corridor). In our planning layer, we process the input image size 32x32 (20x20 for experiment I) via two convolutional layers with 3 kernels. Both layers have only input and output channel and are padded so that the output size is the same as the imput size. We propagate the value through the value map as in Nardelli al. (2019) K 35 times using a 3 x 3 max pooling layer. Finally, the value map and agent position image (a black image with a dot at agent position) processed by convolutions with 32 output channels and x filter window interleaved by 2 x max pool with ReLU activation functions and zero padding. The final result flatten and processed by two fully connected layers with 64 neurons each producing three outputs: d1, 22 P with softplus, softplus and tanh activation functions respectively. The final covariance matrix given by
of
p0102 02
= 60
po102
so that the matrix is always symmetric and positive definite. For numerical reasons, we multiply by binnarized kernel mask instead of the actual Gaussian densitics. We set values higher than mean to and others to zeros. In practice, we use this line:
kernel there (kernel >= kernel mean [1, keepdin=True), t.ones_ (kernel), t.zeros_ like (kernel)
B.3.2 MIDDLE AND LOCOMOTION LAYER
We use the same network architecture for the middle and lower layer proposed by Levy et al. (2019), i.c. we use 3 times fully connected layer with ReLU activation function. The locomation layer is activated with tanh, which is then scaled the action range.
B.3.3 TRAINING PARAMETERS
Discount 7 0.98 for all agents.
Adam optimizer. Learning rate 0.001 for all actors and critics.
Soft updates using moving average; 0.05 for all controllers.
Replay buffer size was designed store 500 episodes, similarly in Levy al. (2019)
We performed 40 actor and critic learning updates after each epoch on each layer, after the replay buffer contained at least 256 transitions.
Batch size 1024.
No gradient clipping
Rewards 0 and -1 without any normalization.
Subgoal testing (Levy et al. 2019) only for the middle layer.
Maximum subgoal horizon H = 10 for all layers algorithms and H = 25 for ablations
without the inferace layer. See psuedocode [
Observations also were not normalized.
14

---

Under review as a conference paper at ICLR 2020
2 HER transitions per transition using the FUTURE strategy (Andrychowicz et al. 2017
Exploration noise: 0.05, 0.01 and 0.1 for the planning, middle and locomotion layer re- spectively.
C ADDITIONAL RESULTS
In this section, we present results collected for this paper including individual runs.
Experiment
Forward
Backward
Flipped
HAC
96.4
00.0
00.0
HAC
82.0
00.0
00.0
HAC
85.6
00.4
00.0
HAC
92.8
00.0
00.0
HAC 5
55.6
00.0
00.0
HIRO 1
100
00.0
00.0
HIRO 2
99.8
00.0
00.0
HIRO 3
99.0
00.0
00.0
HIRO 4
99.6
00.0
00.0
HIRO 5
97.2
00.0
00.0
HIRO-LR
88
00.0
00.0
HIRO-LR 2
100
00.0
00.0
HIRO-LR
97.6
00.0
00.0
HIRO-LR 4
98.8
00.0
00.0
HIRO-LR
100
00.0
00.0
ReIHAC
77.6
18.8
00.0
ReIHAC
86.4
00.0
00.0
ReIHAC 3
77.8
00.0
00.0
ReIHAC
19.6
00.0
00.0
ReIHAC 5
71.0
00.0
00.0
HiDe-R
87.6
72.2
90.6
HiDe-R 2
87.8
70.4
89.6
HiDe-R 3
86.4
38.2
89.8
HiDe-R 4
90.6
69.4
94.0
5
94.4
56.2
87.0
HiDe
80.6
21.2
00.0
HiDe 2
94.6
71.8
96.8
HiDe
81.6
53.4
90.2
HiDe 4
79.4
61.4
91.2
HiDe
87.0
66.0
66.6
Table 6: Results for experiment for individual seeds.
Ant
Ant
Ant
Ant
Ant
Ball
Ball 2
Ball
Ball 4
Ball 5
Forward
82.0
80
92.2
70.6
82.0
99.6
99.0
100
83.2
100
Random
85,8
88.2
89.2
91.4
92.0
97.0
94.2
97.6
97.2
96.0
Backward
61.8
61.4
56.0
42.0
60.4
100
100
100
100
100
Flipped
87.4
68.2
64.8
64.0
85.4
100
96.4
100
100
100
Table 7: Results for experiment 2 for individual seeds.
15

---

Under review as a conference paper at ICLR 2020
A-+B
A-+ 2
A-
A- 4
A-+B 5
Forward
100
100
100
100
100
Random
96.2
96.6
96.8
97.8
96.6
Backward
90.2
99.8
99.8
99.2
100
Flipped
100
100
100
100
100
Table 8: Results for Ant to Ball transfer for individual seeds.
B-+A1
B-+A 2
B-+A 3
B- 4
B-+A 5
Forward 45.8
81.2
75.8
64.8
64.4
Random
81.6
85
94.4
83.4
88.0
Backward
49.4
45.4
68.6
48.0
54.6
Flipped
32.0
83.8
79.6
26.8
70.4
Table Results for Ball to Ant transfer for individual seeds.
Ant
Ant
Ant
Ant 4
Ant 5
Averaged
Forward
94.0
96.0
96.2
95.0
93.4
95 +
Backward
84.2
40.6
1.2
59.4
76.4
52 + 33
Flipped
2.0
1.8
75.4
90.4
0.0
34 + 45
Table 10: Results for experiment on HiDe-AR.
Ant
Ant
Ant
Ant
Ant 5
Averaged
Forward
7.0
12.6
6.2
16.8
6.2
10 +
Random
29.0
40.2
37.6
67.2
57.8
46 + 16
Backward
0.0
0.2
0.0
0.8
0.0
0 +
Flipped
0.0
3.6
0.0
9.4
0.0
3 4
Table 11: Results for experiment 2 on HiDe without interface layer.
Ant
Ant
Ant
Ant 4
Ant 5
Averaged
Forward
38.6
49.8
42.2
75.6
33.8
48 17
Random
69.4
83.8
70.8
86.4
64.2
75 + 10
Backward
7.2
55.4
25.4
72.6
0.0
32 31
Flipped
0.0
69.8
0.0
0.0
0.0
14 + 31
Table 12: Results for experiment with fixed 3x3 attention window.
Ant
Ant
Ant 3
Ant 4
Ant 5
Averaged
Forward
89.0
88.0
78.8
96.4
86.6
88 + 6
Random
87.8
93.0
89.2
92.0
87.0
90 +
Backward
58.2
73.6
45.2
0.0
3.2
36 + 33
Flipped
59.0
84.0
46.4
0.0
81.0
54 + 34
Table 13: Results for experiment with fixed 5x5 attention window.
Ant
Ant
Ant
Ant
Ant 5
Averaged
Forward
92.0
75.4
80.2
91.0
94.6
87 + 8
Random
84.2
83.4
85.0
91.2
89.2
87 +
Backward
6.4
48.2
55.2
85.0
29.8
45 + 29
Flipped
85.2
64.2
81.6
93.6
71.8
79 + 12
Table 14: Results for experiment with fixed 9x9 attention window.
16

---

Under review as a conference paper at ICLR 2020
Algorithm/ Hicrarchical Decompositional Reinforcement Learning (HiDe)
Input:
Agent position Say goal position Jzy. and projection from environment coordinates image coordinates and its inverse Proj.Proj-1.
Parameters:
maximum subgoal horizon H/ 10, subgoal testing frequency 0.3
Output:
k = 3 trained actor and critic functions To,-.., #k-1, Qu. Qk-1
for M episodes do
Train for M episodes Get initial state and task goal b Begin training
< Sinit, 9 <- Gk-1 trainntop_leve((s. g)
Update all actor and critic networks
end for
function (2 :: state, :: goal)
Umap MV Prop(I,
Run MVProp on top-down view image and goal position D Predict mask parameters
01, 22,02, <- CNN (Umaps Proj(s (8ry))
= = 0102P, 0102P, ob
Mask the value map return (2 Proj (arg max Boy D Output relative subgoal corresponding to the max value pixel
Umpp (-1ssyy Σ)
end function
function
(T state, rellaeeee.sseeeeeee subgoal)
return 01 < MLP(g)
b. Output fine-grained relative subgoal
end function
function
io(s jointsssstteee, g relative..udgoall return < MLP(s.g)
b Output actions for actuators
end function
function TRAIN level, :: state, g :: goal)
8; 5, 9i 9
Set current state and goal for level i Sample (noisy) action from policy
for H attempts or until ga. S n achieved do
a; [T( 9;) noise (if not subgoal testing)
if > 0 then
Determine whether to test subgoal 0;
8, train_levelli 1, (4.)
Train level 1 using subgoal o;
else
Execute primitive action ag and observe next state Su
end if
Create replay transitions
if > 0 and not reached then if ag was subgoal tested then
Penalize subgoal a;
Buf + [s = Si, a = 0; Penalty, s 9.. = 9t, 7 = 0]
end if
a; + s end if
Replace original action with action executed in hindsight
Evaluate executed action on current goal and hindsight goals
Replay.Buf t- [s= 8;, = a;, E ()1,0,, 8 = 9i, 7 E {7, 0}]
HER.Storage; + [s= S;,
TBD, 8;,9 TBD, = T TBD]
8 end for Replay. Buf fer; Perform HER using HER Httooogggee, transitions return 8 end function
Output current state
17


Reviews:
[{"review_id": "HJg9A-usKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses hierarchical deep reinforcement learning (RL), an important problem in control learning and RL. Based on my understanding of this paper and recent prior work, the most important difference between the proposed approach (HiDe) and other recent approaches, such as HIRO and HAC, is that the top-level goal proposal policy uses a learned planner based on VIN and a learned attention mask to decide on a subgoal. There are also other differences, e.g., this policy outputs a goal position that is relative to the agent's position, rather than an absolute position. HiDe seems to demonstrate impressive transferability to both unseen mazes and new agent embodiments, which are important problems to address for hierarchical RL.\n\nIntroducing learned planning and attention into the top-level policy seems to also introduce additional assumptions into the method. For example, it is my understanding that HIRO and HAC use the same state representation, e.g., joint positions and velocities of the agent, as input to their top-level policies. In contrast, HiDe uses a top down view of the maze and the x y position of the agent, which certainly is more privileged information. If this is correct, first, this should be discussed more thoroughly and directly in the paper. Second, the experimental setup should be elaborated on: is HIRO or HAC modified to include the same information for the top-level policy? Or can HiDe somehow be extended to not require this information? I do not think that requiring this information is egregious, but currently the experimental comparison is not clear in this regard.\n\nThe experiments are arguably the strongest part of the paper, and the transfer results and videos are quite nice. But there is still room for improvement. Table 1 and Figure 5 seem disconnected. In particular, the numbers reported in Table 1 are clearly not achieved in Figure 5. Is the figure cut off early? Furthermore, an additional experiment on a more complicated domain would greatly strengthen the paper. A humanoid agent, for example, seems easy to test for the current method. Another option would be the movable blocks tested in HIRO, though it is unclear if this readily fits into the current method's assumptions. In my opinion, as the paper currently rests heavily on the results, this section should be further improved. Doing so would also improve my rating of the paper.\n\n------\nEdit after author response: I appreciate the authors' efforts in providing extensive responses to all of the reviewers' concerns as well as a significant general response detailing what seems to be a large amount of additional experimental work. I think that all of this warrants a change to my score, and seeing that the authors have more or less addressed my concerns, I am bumping up to a weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bye3eya2KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a novel method for explicit decomposition of hierarchical policies for long-horizon navigation tasks. The approach proposes to separate a policy into 3 modules, high-level planner, intermediate planner and low-level control. The evaluation shows that explicit decomposition is well suited for generalisation across a limited set of RL domains.\n\nThe proposed method integrates aspects from a variety of recent work including planning layers from value propagation networks, hindsight training paradigms from hierarchical actor critic and hindsight experience replay and related techniques. The variety of different techniques combined instead of a single main contribution renders it challenging to follow all aspects and in particular to trace relevant contributions to performance - which is rendered harder by a limited evaluation section.\n\nWhile the approach shows good performance against a couple of start of the art methods, it is necessary to provide sufficient ablations to enable long-term insights for the community. The submission high level goal (explicit decomposition and information asymmetry) is clear, the execution involves the combination of many existing techniques plus variations such that it is hard to make solid statements about the relevance of any part.\n\nIt is commendable that the authors have introduced adaptations and improvements to their baselines for a stronger and fairer comparison but the evaluation remains very limited. \nI suggest to run different domains as given by other domains from OpenAI gym or DeepMind control suite. But more importantly I suggest to run further ablations without the intermediate planning layer & with absolute goal positions. Furthermore, since HAC seems to perform worse when combined with the proposed low and mid level policies (RelHAC), it would make sense to compare to the proposed high-level policy using low and mid level from HAC instead. \n\nThe submission provides an overall interesting perspective but makes it hard to narrow down on contribution and important insights by being unclear in formulation and providing only very limited ablations. \n\nMinor issues include:\n- Missing description of the mid level policy - what encourages the proposal of closer short-term goals.\n- Missing literature on information asymmetry in RL (e.g. see Tirumala et al 2019 ‘Exploiting Hierarchy for Learning and Transfer in KL-regularized RL’)\n- Unclear description of how models that get attached to existing planning layers have been trained (Sec 5.3).\n- Additional unclear description in the description of the experiments and method sections (4.2, 4.3)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxSfTH8KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neat framework for creating HRL framework that will be able to generalize its application to slightly different environment layout. This is done via an image-based top-down from as input to the high level. An intermediate layer is used to help create more fine-grained goal specification for a final goal-based control layer. These layers are trained together using HAC. Overall, the method shows promise but there needs to be more analysis to understand which parts of this combination of ideas are the most important. The results are also only shown for a single environment. Last, the generalization analysis in the paper does not appear to be overly thorough. It would be good to perform this on more than one type of environment also the random environment is not very random.\n\nMore detailed comments:\n- The results seem very similar to some of the work in \"Universal Planning Networks\" that did not need a more complex HRL design to achieve subgoal specification via images. This should be discussed more in the paper.\n- The authors point out that the use of relative goal positions \"ensures generalization to new environments\", this is a rather strong statement. The use of relative goal specification my help improve generalization but that can only be shown empirically.\n- The demonstration to show that the method generalizes to other configuration after being trained on a fixed environment should be evaluated over many randomly generated environments so that we have a non-biased estimate of the true generalization performance. In Table 2, it is rather surprising that the HiDe trained model does better on the \"Random\" environment vs the \"FOrward\" environment it is trained on. Can more details be provided on how the \"Random\" environment is created? Are the locations of the walls randomized? Is the initial position of the agent and goal randomize?\n- The video seems to contradict the ordering of operations for training the planning network. The video suggests that first it is learned with the Ant then transferred to the ball which is less complex to control.\n- At the beginning of the prior work section, it is noted that many other methods require prior knowledge of the environment I would say this method also requires certain kinds of prior knowledge about the task. For example, a top-down view of the environment is needed which is not often feasible.\n- HIRO and HAC use a more proprioceptive state space but I don't think the sharing of global states is intentional. I am not convinced that this choice, in particular, is what makes the approaches prone to overfitting.\n- You show a comparison to the \"windows\" created from your method vs a fixed neighbourhood. Do you perform any empirical evidence that your introduced methods provide an improvement over this fixed window?\n- It is mentioned at the end of section 4.1 that MVProp is differential so it can be trained with the Bellman error objective. Because many policies are being trained concurrently does the MVProp attention model need to be recomputed after every sub-policy update? Does the frequency of updates have a large effect on performance?\n- Section 4.2 introduces an interface layer that is not a very common practice. It would be good to include an ablation study of the effects of this introduced layer.\n- In section 4.3 it says that the control layer is the only layer with access to the agent's proprioceptive state. Would it not be good to at least include the agent facing direction or current average velocity to higher layers to improve the attention mask estimation?\n- In figure 5 it says HIRO converges the fastest because it has dense rewards. Can you be more specific? Also, If different agents are using different reward signals I am not sure this evaluation is a fair comparison.\n- Are tables 2 and 3 just for the HiDe algorithm? Is it possible to include data for the other algorithms? \n- You perform an experiment to train HiDe with random initial and goal locations for comparison. I think running this comparison for HIRO and HAC would be a good additional point of comparison. This would help the reader know if the generalization is not biased to the particular initial environment configuration for Maze Forward.\n- In the generalization analysis for the paper, how is the analysis performed? There are percentages for the success of the policy, where does the randomness come from is the agent state and goal are always fixed? Are these averaged because the agent has a stochastic policy during evaluation? If this is the case how many random trajectories are collected to compute these statistics?\n- It would be very helpful to have a description of the algorithm in the paper. How the algorithm works is not very clear and some details about how the goal and states are passed to the different policies would be very helpful if anyone wanted to reimplement this work.\n\nUpdated comments:\n\n- The addition of more results and added analysis helps show the improvement of this method over the most related baselines.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a complex, hierarchical architecture for continuous control RL that combines Hindsight Experience Replay, vision-based planning with privileged information, and low-level control policy learning. The authors demonstrate that the approach can achieve transfer of the different control levels between different bodies in a single environment.

The reviewers were initially all negative, but 2 were persuaded towards weak acceptance by the improvements to the paper and the authors' rebuttal. The discussion focused on remaining limitations: the use of a single maze environment for evaluation, as well as whether the baselines were fair (HAC in particular). After reading the paper, I believe that these limitations are substantial. In particular, this is not a general approach and its relevance is severely limited unless the authors demonstrate that it will work as well in a more general control setting, which is in their future work already. 

Thus I recommend rejection at this time.
Decision: Reject