$\epsilon$-Invariant Hierarchical Reinforcement Learning for Building Generalizable Policy
Authors: Yihan Li
Tianren Zhang
Jinsheng Ren
Feng Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: hierarchical reinforcement learning
generalizable policy
zero-shot generalization

Abstract: Goal-conditioned Hierarchical Reinforcement Learning (HRL) has shown remarkable potential for solving complex control tasks. However, existing methods struggle in tasks that require generalization since the learned subgoals are highly task-specific and therefore hardly reusable. In this paper, we propose a novel HRL framework called \textit{$\epsilon$-Invariant HRL} that uses abstract, task-agnostic subgoals reusable across tasks, resulting in a more generalizable policy. Although such subgoals are reusable, a transition mismatch problem caused by the inevitable incorrect value evaluation of subgoals can lead to non-stationary learning and even collapse. We mitigate this mismatch problem by training the high-level policy to be adaptable to the stochasticity manually injected into the low-level policy. As a result, our framework can leverage reusable subgoals to constitute a hierarchical policy that can effectively generalize to unseen new tasks. Theoretical analysis and experimental results in continuous control navigation tasks and challenging zero-shot generalization tasks show that our approach significantly outperforms state-of-the-art methods.

Under review as a conference paper at ICLR 2023
E- I-IVAARINTT HIERARCHICAL REINFORCEMENT LEARNING FOR BUILDING GENERALIZABLE POLICY
Anonymous authors Paper under double-blind review
ABSTRACT
Goal-conditioned Hierarchical Reinforcement Learning (HRL) has shown re- markable potential for solving complex control tasks. However, existing methods struggle in tasks that require generalization since the learned subgoals are highly task-specific and therefore hardly reusable. In this paper, we propose a novel HRI. framework called e-Invariant HRI that uses abstract, task-agnostic subgoals reusable across tasks, resulting in a more generalizable policy. Although such subgoals are reusable, a transition mismatch problem caused by the inevitable in- correct value evaluation of subgoals can lead to non-stationary learning and even collapse. We mitigate this mismatch problem by training the high-level policy be adaptable to the stochasticity manually injected into the low-level policy. As a result, our framework can leverage reusable subgoals constitute a hierarchical policy that can effectively generalize to unseen new tasks. Theoretical analysis and experimental results in continuous control navigation tasks and challenging zero-shot generalization tasks show that our approach significantly outperforms tfate-ofthhe-e methods.
INTRODUCTION
Goal-conditioned Hierarchical Reinforcement Learning (HRL) methods have shown remarkable po-
tential to solve complex Nachumm et al. (0188) et al. 2021 (Csttales et al., [202I). such as continuous control tasks that require long- long-horizon navigation Goal-conditioned HRI. uses subgoals to decompose the original task into several sub-tasks, training a high-level policy to output subgoals and low-level policy that executes raw actions conditioned on
al.. [202] [ Gürtler et al. [021).
the subgoals. Therefore, the performance of coal-conditioned HRL mainly relies on a desl-des subgoal space. In particular, in many complex realistic scenarios such as robotics, reusable subgoals necessary for building generalizable policy that is widely applicable to different tasks.
Prevailing strategies usually utilize task-specific subgoals such as representations extracted from essential states trajectories (Nachum et al. 2018a (Li et al. [202 202ay[iaag et al. (219b)), or select subgoal space based on prior knowledge such as (a subspace of) the raw state space (Nachum et al.) 2018b) (Zhang et al. 2020). Such subgoals depend on the specific task and thus often cannot be reused in different tasks. For instance, in a maze navigation task, a subgoal representing coordinates in maze may not be reachable in another maze, although the two mazes can be similar. As a result, policies based on these subgoals cannot be reused in different tasks.
To construct reusable subgoals, a readily applicable choice is use invariable abstract physical quantities as subgoals such as directions in navigation task. These abstract subgoals are usually aask-agnostic and thereby naturally reusable. However, how build a generalizable policy based on these abstract subgoals remains a challenge due to the transition mismatch problem. which is common in HRL (Zhang et al. 2022) (Levy et al., 2018) and can lead to the non-stationary learning process. In general, the transition mismatch problem emerges when the high-level policy evaluates the subgoals with incorrect transition and rewards due to an inadequately trained low-level policy. To introduce the problem clearly, we will show it in prevailing challenging task considered by prior HRL methods, i.e., a long-horizon maze navigation problem based on a legged robot, despite our idea is not limited to this particular task. Consider that the high-level produces a subgoal to go to position with coordinates (x, y), but the immature low-level policy reached (a', v²).. Then, the high-level will evaluate the subgoal by the incorrect reward from (x', v'), which will lead to

---

Under review as a conference paper at ICLR 2023
inaccurate value estimation and even the collapse of the hierarchical policy learning process [g et all 2020b) Wang et al., 2021). Previous work proposes to alleviate the mismatch problem in single-task settings by refining the wrong transition, such as relabeling the wrong transition by the correct transition obtained from the replay buffer (Levy et al. 2018) (Nachum et al. 2018b) (Kim) jet al., (02T). However, these methods fail to solve the transition mismatch problem in the multi-task setting, where this problem can be exacerbated by the confusion of changeable tasks with the same abstract subgoals. As successful trajectories are not identical across different tasks, relabeling by sampled trajectories may aggravate the mismatch problem.
In this paper, we propose novel HRL framework called e-Invariant HRL to leverage task -agoosicc abstract subgoals as well as alleviate the influence of the transition mismatch problem. We propose a method called e-Invariant Randomization to inject controllable stochasticity into the low-level policy during the training of the high-level policy. The key idea is to see the mismatch as kind of randomness and train the high-level policy to be adaptable to this randomness. We term the subgoals introduced with randomness e-Invariant subgoals (the definition is in 3.1 Based on this framework, we propose a parallel synchronous algorithm from A2C Mnih et al. (0016) to evaluate the online policy and update the network by an expected gradient from multiple trajectories sampled with the same parameter instead of updating by a single gradient, which can alleviate the mismatch problem with the expected transition, reducing of influence of incorrect transitions and rewards. The parallel algorithm is proven to possess the potential to solve changeable scenarios and randomness in the environments (Hou et al., [022 (Espcholt et al. 2018). After solving the mismatch problem, our HRL policy can leverage general task-agnostic subgoals and generalize even to unseen tasks.
To demonstrate the superiority of our HRL framework, we extend the widely-used benchmark based on (Duan et al.] 2016)'s 's work using the MuJoCo (Todorov et al. 2012) simulator. These experi- ments are designed to control high-dimensional robots to solve long-horizon maze navigation tasks in different mazes with sparse rewards (see details in section 4.1p. Some tasks are in a stochastic environment with changeable structures to check the robustness of policy. In these difficult tasks, our method achieves state-of-the-ar results compared with the most advanced RI. and HRI meth- ods. Our method also shows novel abilitics in generalization tasks, in which the structures of the mazes are unseen, even with unseen pre-trained robots. To the best of our knowledge, such complex generalization tasks can hardly be solved by previous methods, and we are the first to build an HRL. policy that exhibits generalization capability across different mazes and different robots.
In summary, our contributions are three-fold:
1. We devise HRL framework for generalizing in high high-dimensional controlling maze nav- igation tasks with a theoretical guarantee.
2. We propose a randomization method for the transition mismatch problem in generalization tasks along with an algorithm that enables stable learning.
3. We provide a new benchmark for evaluating the approaches for high-dimensional maze navigation tasks, and our method outperforms SOTA algorithms. To the best of our knowl- edge, we are the first to build policies that can generalize to such zero-shot navigation tasks with unseen mazes and unseen robots.
2 PRELIMINARIES
We formulate the task in this paper as goal-conditioned Markov decision process (MDP) (Sutton & Barto 2018), defined as tuple < 9, P. R. S the state space, .A is the action space, and g is the goal space, which is set of consistent invariant actions. In this paper we focus on maze- navigating tasks, so that we choose the relative displacement of directions "up, down, left, right" (or in the coordinate system of the environment) the goals (see details in Section 3.1. P is the transition probability matrix and P(s'|s, a) is the one-step transition probability. R(s.a) a) is the reward function, 7 [0, 1) is the discount factor.
"zt, 2-+,,-,30 B.c..yt.y-
Goal-conditional HRL. We consider the framework that consists of two hierarchies: a high level policy x" [g] s; 0%) and low-level policy n/ n r[ass, g: 0t), where 0n, 0, is the parameter of the two policies parameterized by neural networks respectively. At a high- timestep t, the high-level policy generates a high-level action, i.e. subgoal ge E 9 by ge a (g]se; The
2

---

Under review as a conference paper at ICLR 2023
Learning Task of Random Maze
Generalization Task of Unseen Maze
Zero-shot Generalization
Figure I: Illustration of Zero-shot Generalization Maze Task. From left to right is Maze-g2. Maze- g3, Maze-gl. Details are in section 4.1
high-level policy gives goal every k steps and the low-level policy executes the subgoal 9r in k Then the high-level policy receives the accumulative external rewards from the environment 1=th R(s;, a;) in & steps. The goal of the high-level is to maximize the expected return tk++-1
steps.
r
E
/ by driving the low-level policy.
The low-level policy receives additional intrinsic reward rl for efficient learning follows: r' (Stts St. aek, $tk+1) = a cos < gt: (2.:((11111) (p(()))))))))))))))))())))) > 111+(514+11 - (((x))]
(I)
where 9r gt is invariable direction vector of "( , セー}" the y is coordinate extracting function form state and a is a constant coefficient. The reward means that the further the agent goes towards the goal direction, the more reward methods as done in (Kim et al.) [202T) (Nachum et al. 2018b), we do not use the relative distance of specific coordinates as the intrinsic reward function but use the fixed distance towards invariable and orthogonal goal directions. By this reward function, the agent will learn to walk towards an abstracted direction according to the subgoal instead of towards specific location, which preserves the generalization potential for different tasks.
obtain. Different from current HRL
3 APPROACH WITH ANALYSIS
Motivation. We hope to build an HRL policy to solve nave-navigation and generalization tasks of high-dimensional continuous control- ling robots. The difficulty of these tasks is usually caused by large exploration space and sparse rewards. Thus aim build a sub- goal setting, which can not only reduce the difficulty exploration but also are reusable in different tasks. Consider that many previ- ous RI works research on tabular mazes, such as the BabyAl platform (Chevalier-Boisee) jct al.) 2018). the maze with continuous state space can be discretized into several blocks, the learning process will more ef- ficient than the original one. So that the subgoal space by abstracted task-agnostic subgoals like directions of "front, back, left, right". These subgoals mean the agent should move in the direction setting, the maze is divided into blocks ac- cording the distance of the movement in the high-level perspective. The high-level policy can make decision on finite discrete space, which will significantly improve the exploration efficiency. Meanwhile, such subgoals are physically invariable so that can reused in any maze. Policy based on these subgoals can also be general among tasks.
Enviroument
High-level policy
Miskala
Granulis diffirat mer
e-mmanast
Postere
Low-level policy
Gesantize
Figure 2: HRL framework illustration. By the e- invariant subgoals, the high-level policy can adapt to different maze tasks and different pre-trained low-level robots.
fixed step. In such
Challenges. Although the motivation concise, it needs overcome three challenges for imple- mentation. (1) Will the hierarchical policy in our setting seriously break the performance of the original optimal policy? How to evaluate the error? the error controllable? (2) HRL the fixed oracle subgoals setting with respective learning process often lead collapse due to the mismatch of the two hierarchies. How to let the agent learn stably and safely and overcome the mismatch?
In the following paragraphs, will answer the questions respectively by theoretical analysis for discussing the effectiveness our method, and framework building with algorithm design for stable policy learning.

---

Under review as a conference paper at ICLR 2023
3.1 THEORETICAL ANALYSIS
Limited by the length of the paper, in this section we just show the main idea of proof. The details be seen in Appendix A
Our main idea to prove the effectiveness our method is to prove the error between optimal policy and ours in a metric is bounded and can be controlled so that our methods can be used as a sub- optimal policy to approximate the optimal one. The results in stochastic MDPs show that the error between our hierarchical policy and the optimal policy can be bounded.
Firstly, we give the mathematical definition of the e-invariant subgoals (as the high-level abstracted actions).
Definition 3.1. (e-invariant subgoals) For every environment & and an e-invariant subgoal ga E Ga transitions of the subgoal from 's' to 's"" and "8" to "8" with optimal low-level policy, for any salisfying:
Ex(p(() - pp(ss) = Eg [E (p((s)) -=((s))()
(2)
where 0 (ps')) O Se
Here 7 (g]s) is the high-level policy. s' and 8" the achieved states by the e-invariant subgoal Is with optimal low-level policy. is the coordinate extracting function and yo(s) is the coordinate vector of observation s. "x(s"),xx (')(") are the variance of achieved state in æ, y directions. We consider our high-level policy learning in stochastic MDP that the s' is random variable de- pending on and ga. and we limit the variance of s' by a little constant e. The equation means such subgoals will give an excepted invariable direction the coordinate system with little randomness.
Error Bound in Stochastic MDP. As shown in Fig 3 the trajectories of RL method, traditional HRL method, and our method are different, and for that, we build our high-level policy in stochastic MDP with stochastic subgoals.
We consider a goal-come conditonniii HRL where the high-level policy makes decision every steps. To compare with the HRL method, the optimal value function of original RL methods in deterministic- MDP can be rewritten with & step as:
(3) where т trajectory in k steps with optimal policy :" (Tk(st), R((T)) is the accumulative dis- counted reward in i steps. And equation can be rewritten as: (st) Σn, Pa (R(T):)(R(TT)) + >*++(S++))
(s,) E., [R(Tk) + +VV((S(+))
With equation
3 we can define the error between traditional RL method and our method: [V (st)) To analyze and control the error, we introduce the value Vij (8t) in optimal stochastic HRL with coordinates as subgoals:
(v:(s)) (s)) S (V. VV)))-((ss)))
+|
Vi
HRL error
subgoal error
Here we only give the conclusion of our analysis, that is:
1 VÅ 1000000 < UakB
2(1 7b)) + 2/ Rmaz
(5)
2(1 - xbp
and
Tnp (20mar + +3v26) Rmax (k 1-
7
|
V
(6)
7k)
where 8mar
max
[lgell [lol]::1 [///|]]. Rmar is the bound of reward function, v% is the transition mis- 9.84
match rate between optimal RL method and HRL method with stochastic coordinate subgoals. L is the Lipschitz constant. The proof is shown in Appendix A
The result shows that the error between method and optimal policy indeed can be bounded and controlled. That means our policy can be used as a sub-optimal policy to approximate the original optimal one.

---

Under review as a conference paper at ICLR 2023
3.2 FRAMEWORK
In this section, we will show how to build hierarchical framework to handle the abstracted e- invariant subgoals. Generally, the high-level policy decides the direction as subgoals by pixel ob- servation, and the low-level policy receives the expected direction and walks towards the direction for a distance. Thanks to the abstractness of the subgoals defined in 3.1 although the high-level and the low-level policy can learn together, we train them respectively to accelerate the learning process. So that our method decomposes the learning process into two stages with discrete general subgoals, where the high level learns by external rewards and the low level learns by intrinsic rewards. Es- sentially, the discrete general subgoals are invariable displacement of fixed directions, which will reduce the difficulty of exploration and improve learning efficiency by discretizing the state space into little blocks. Meanwhile, they are reusable and general in any maze-navigation tasks, so as to preserve the generalization abilities of the high-level policy.
Modular Pixel Observation. As we utilize the di- rection as abstracted subgoals for the high-level pol- icy, the influence of the subgoals can hardly be per- ceived by the agent with the posture of the robot. We add pixel observation at the top view to look down at the agent. (See in figure 2 It will observe a re- gion of fixed size by a camera following the robot. From this perspective, the influence of the subgoal of directions is invariable and consistent among maze environments. The agent can observe the change in the environment close to the robot. Meanwhile, pixel observation is more adaptable than raw data of pos- ture, which can improve generalization abilities. We provide the pixel observation for the high-level pol- icy decide which direction to go.
Task1
method with stochastic subguals Task2
HRI methed with e-nvvariant subgnals
Optimal
method
e-Invariant Randomization. As we train the two hi- erarchies respectively with abstracted subgoals, we train the high-level by the movement directly from the simulator instead of the real walking of the robot for faster learning. However, abstracted subgoals will bring the mismatch problem between the two levels. Because to control robot moving by legs strictly to- wards a direction is very hard for RL methods. So that even the well-learned walk policy will contain a slight deviation in the vertical direction of the goal direction. Such inevitable deviation (a per- formance of mismatch) will cause incorrect evaluation of the subgoals, which will lead to non- stationarity and even collapse in a changeable environment.
Trijsttry
State
Coordiaate subgoal
Invalid subgoul
--iovarianl subgus!
Possibic rouching region
Figure 3: Comparison of different frame- works. e-invariant subgoals (invariable di- rections with little randomness) can be reused in different navigating tasks.
To alleviate the problem, we introduce the e-Invariant Randomization method build the high-level learning process as a stochastic MDP. When training the high-level policy, the simulator moves the robot with random postures and random positions. The postures are sampled from the walking postures of well-learned low-level policy with random subgoals and recorded as offline data. The random position means that a random deviation will be added to the original movement. For in- stance, the high-level policy gives a "x+" direction as subgoals, then the simulator will move the robot to "x+" " with a little offset of A,, A, ~ ( o), where a S E much less than the moving distance. As a result, if the high-level policy can overcome the randomness, the wrong execution of the lewelevel policy will be seen as a sample of the distribution of randomness, which will improve the affordance of the high-level policy.
Network structure and more details can be seen in Appendix C
Algorithm. As we introduce randomness into the learning process to alleviate the mismatching problem, the high level should overcome the stochasticity in the environment to obtain a well- performed policy. To solve the randomness, we propose parallel algorithm called the parallel expected gradient advantage actor-critic (PEG-A2C) algorithm because the parallel algorithm is proven possess the potential to adapt to a changeable environment (Hou al. 2022 (Espeholt
5

---

Under review as a conference paper at ICLR 2023
Figure 4: illustration of the shape of maze environments of our benchmark. They are Random Square, D Maze, S-shaped Maze, and Spiral Maze respectively. The blue arrow is the successful trajectory.
et al. 2018). The main idea of our algorithm is sample multiple trajectories with the same pa- rameter by several agents, then concentrate the trajectories and calculate the expected gradients of different trajectories and different steps and update the parameters synchronously. That is, different from previous works which update parameters by 0 + - aV, we utilize < 0 + aE, (Ve()) to train the high-level policy. Such gradients can hopefully reduce the impact of randomness, as taking the expectation can be seen as a special kind of automatic gradient clipping by tasks (Zhang et al., (Bjorck et al., 2021). That is our algorithm evaluates subgoals by expected return and high-level transition. It will alleviate the non-stationarity of the learning process.
The pseudo code is shown in Appendix B
4 EXPERIMENTS
In our experiments we aim to answer the following questions: (1) Can our method learn stably in different complex maze navigation by a robot with high-dimensional action space? (2) How does our method perform compared with advanced RL and HRL algorithms? (3) Can our method generalize to different unseen maze tasks even unseen robots without retraining?
4.1 ENVIRONMENTS SETTING
We evaluate suite of MuloCo Todorov et al. 2012) maze-navigating tasks modified from the benchmark from (Duan et all 2016). Different from the setting of (Kim et al.] (20T) et al. 2018b), we utilize sparse external rewards to make them more challenging. In these tasks, the agent should control an ant robot to achieve a specified area in the maze which is far away from the initial position. To compare completely, we also add experiments on dense reward, where the reward is 1/(1 + d) of Euclidean distance d between the agent and the current goal in the coordinate system for every step. The mazes are Ant D-Maze, Ant Random Square Maze, Ant S-shaped Maze, Ant Spiral Maze, Ant Spiral Maze, and Generalization Maze (See in Fig 4) They are all difficult mazes with the long navigating horizon. In these mazes, the goal is to pass the door or go to a specific region. Especially Ant Random Square Maze is task in stochastic environment with random initial positions of both the agent and the door. Generalization Maze is task with three unseen mazes of different structures. Ant S-shaped Maze and Ant Spiral Maze extremely long-hou long-horizon mazes that require at least thousands of steps by the optimal policy. More details can be seen in Appendix D
Implementation. We build our HRL agent by two policies. The high-level policy is learned by PAG-A2C algorithm, and the low-level policy is goal-conditioned policy modified from DroQ (Hiraoka et al. (202I) algorithm, of which the subgoals are directions of 'x+',y++ "y+''xx",''" yy-"". As the simulator MoJoCo allows direct movement of coordinates, we train the two policies respectively in two stages. The high-level policy learns with direct movement with randomness, and the low-level learns by intrinsic reward defined in equation TT As the subgoals and the low-level policy can be reused in different tasks, we mainly show the high-level training curves and success rates in different tasks and use the same well-learned low-level policy for the ant robot.
4.2 BASELINES
We compare method to several state-of-the-art (SOTA) model-free RL and HRL algorithms in the high-dimensional continuous control tasks said above.
DroQ. It is the SOTA RL algorithm for dimensionn continuous control tasks (Hiraoka et al. 2021), which is the most efficient RL method. effective for both simple robots like Hopper and complex robots with large numbers of degrees of freedom like Humanoid.

---

Under review as a conference paper at ICLR 2023
Readom-sqquare
D-shape
S-shape
2.0
Avengg Ilveell
1.6
0.4
12
2007
5333
9300
2300
1300
2000
1330
2007
0002
687
1303
7002
Eploodes
Higl
our
Dra
Hess
our onscle
RAND-F
Figure 5: Comparative experiment results with strong baselines in sparse-reward tasks. The mean and variance are calculated by 3 runs.
HIGL. is the SOTA HRL algorithm for high-dimensional maze-navigating tasks with both sparse rewards and dense rewards for MuJoCo suite (Kim et al. (021).
HESS. is of the most advanced subgoal learning HRL methods for continuous control tasks LL al. 2021).
RAND-H. It variety of our HRL method, of which the high-level policy is a random policy and the low-level policy is well-learned. The baseline is used as an ablation study of our method to show the capabilities of our subgoal setting.
our-oracle. It is a variety of our HRL method, of which the high-level policy is trained without the low-level policy. The robot will walk by the oracle movement of coordinates executed directly by the simulator instead of the low-level policy. The baseline is also used as an ablation study of our method to show the efficiency of the high-level policy.
4.3 RESULTS OF COMPARATIVE EXPERIMENTS
For fair comparison, we utilize the evaluated reward curves with both the high-level and the low- level, although our hierarchical policies can be trained respectively. The curves are shown in figure 5] We can see that our method outperforms both the SOTA RL and HRL methods. The curves are cut by the same episode to align the results different methods. All the curves are smoothed by sliding window. More details can be seen in Appendix D
Robustness to Stochasticity and Mismatch. In these tasks, the "Ant Random Square' square maze with random initial position of robots and goals. In this task, the mismatch problem will be- come critical as the successful trajectory is episodically changeable. So that correction by historical samples is invalid. As shown in the curve in figure 5 the HIGL method obtain reward first, but gradually reduced with downward trend. Our method can learn with gradually increasing return. The curve shows that our method can adapt to such a stochastic environment and learn stably, proving that our algorithm indeed can overcome randomness. But other methods can hardly adapt such stochastic environment.
Comparative Result. The non- non-hierarchical method DroQ performs poorly in all the tasks with sparse rewards, demonstrating the strength of the hierarchical structures solving long-horizon tasks with sparse rewards. HESS method also performs not so well with little accumulate return, due to the requirement large exploration episode steps for sparse reward, which is consistent with the results reported in their paper (Li et al.) (2021). The SOTA HIGL method outperforms other baselines while underperforming our method, which learns with gradually rising average reward but slower than ours, showing the superiority of our methods.
In tasks with dense rewards (figure()) we utilize average step reward and goal-reaching success rate to evaluate the performance of these methods. The performance of previous methods is improved lot but the curves of success rate are still under our method.
Ablation Study. The RAND-H and our-oracle show the ablation result of our method. The high- level policy with oracle movement (our-oracle) can learn stably with our algorithm. We can see that the low-level policy (RAND-H) can obtain rewards in a certain frequency with our subgoal setting even with random high-level policy, so as to preserve stable feedback for the learning process. The two ablation experiment curves show the reason why method learns faster.

---

Under review as a conference paper at ICLR 2023
D-shape-dense
S-shape-dent
-- shape-dense
S D-shape-deee
Avirag Step Rewaard
0.3
Ca
Avorage Strrass
C3
12000
3324
Higl
241
ou0 voadee
Big
V01
OUP ureure
(a)
(b)
Figure 6: Comparative experiment results in dense-reward tasks. (a) is curve of average reward of every step. (b) is curve of average success rate = achieved goals/ goals
Table I: Generalization Task Result for Different Mazes (Zero-shot Success Rate %)). 'Maze-ori' is the trained maze, and the other three are zero-shot generalization maze.
Method/Maze Maze-ori
Maze-gl
Maze-g2
Maze-g3
Ours
51.8 4.4
45.6 + 5.2
27.8 6.7
73.6 + 4.6 84.8 + 6.6
Ours-Orack
66.4 = 5.4
72.0 6.3
58.4 = 8.3
HIGL
25.4
S
4.4 RESULTS OF ZERO-SHOT GENERALIZATION EXPERIMENTS
Zero-shot Maze. these experiments, we utilize the policy trained 'Ant Square Maze' and test in new unseen mazes of fixed shapes without retraining. We compare our method with the HIGL method by the well-performed policy of training process in the Ant Random Square environment with sparse reward. Result is shown in table [ The test mazes fixed shapes without randomness. so that sometimes the test result can be better than the training result.
Zero-shot Robot. In this experiment, and the whole HRL policy respectively. In table we show zero-shot result of the mere high-level policy in 'Maze C shape'. It is show the adaptive capabilities of high-level policy with our subgoal setting.
the single high-level policy with the oracle movement
From the results, we can see that such generalization tasks can hardly be solved by previous RL and HRL. methods due to the specific subgoal goal setting. They can neither adapt to different maze tasks nor change the low-level policy without retraining the high-level policy. Our HRL method uses invariable abstracted subgoals and can adapt to different scenarios. Our method is also flexible and can adapt to different unseen robots with the low-level policy receiving the same subgoals with certain success rate.
Visualization Results. To show the generalization ability of our HRL policy, we show the heatmap of trajectories of our method and HIGL (figure 7 We can see that the HIGL agent mainly moves near the initial position and is blocked by unseen structures, but our agent can achieve the door more frequently.
Table 2: Generalization Task Result for Different Robot (Zero-shot Success Rate %). Except Ant', the other robot are unseen. Average success rate achieved goals/ goals/total total goals
Method/Robot
Ant
Point
Swimmer
Humanoid
Ours-Oracle
99.6 + 0.5
53.2 1.4 47.0 + 2.4 48.2 1.3
HIGL
67.0 + 5.2
RELATED WORK
Coal-conditioned HRL methods. In the recent few years, hierarchical reinforcement learning methods have been widely studied for complex high nigh-dimensional control and long-horizon tasks. There are many goal-conditioned HRL methods for subgoals learning, handling and discovery for these difficult tasks (Nachum et al. (0188) et al.) 201]]222)p) (Kim et al., (Q2T)) et al., [ et al. 2020 /Zhng et al. 2020). These methods usually leverage task-specific subgoals such as spe-
8

---

Under review as a conference paper at ICLR 2023
Our method
Maze-gl
Position
Maze-g2
Position
Maze-g3
Position
histogram
histogram
histogram
HIGL
Visualization result of zero-shot maze generalization tasks. The red and blue points rep- resent the achieved positions in the maze. The more lightly position is, the more frequently the agent achieving. The histogram is to straightly show the frequency of the achievement of every position.
Figure 7:
cific coordinate (Nachum et al. 2018b). sampled abstracted original states (Nachum et al. 201 Sa) or abstracted trajectory descriptions (Jiang et al. 2019a) to complete complex tasks. Different from previous methods, our framework utilizes the e-invariant subgoals (direction with randomness) as abstracted high-level actions, which not stand for task-specific descriptions the invariable relative effects of action sequences. Such subgoals can be reused ignoring the change of states in different tasks, which are more general.
Generalizable Policy Learning. Constructing policies can be reuse (or generalize) in new (even unseen) tasks a long-standing challenge, which researched by many works different perspec- tives (Igl al. 2020a Xu al. 2022 (022)(War) (Wang et al. 2019) Wang al. 2020). Recent works
mainly three lines for generalizable policy learning.
(1) Learning reusable or transferable skills, op-
tion policies for different tasks(Shah et 2021) (Nam al. 2021 ( Klissarov Precup 2021
(2) Learning policies adapting to a distribution of environments or tasks with different dynamics
by risk-sensitive objective functions(Lyle et al. 2022) (Lei & Ying (irrch et al. 2019). (3)
Learning to extract reusable abstracted representations like language, logical symbols or graph from
visual observation or state-action trajectories (Jain al. (Agarwal et al. 2020 Vaezipoor
et al. 2021.
Our method is related to the third line, i.e., utilize the abstracted representations for improving the generalization abilities of the policy. But the difference is that we do not leverage representations expressing tasks a distribution invariable abstracted trajectories. Instead, we use the abstracted representation to represent the physically invariable relative locomotion. Such representations are usually general in the kind of navigaaavi tasks, thus leading to strong generalization abilities.
6 CONCLUSION
In this paper, propose novel HRL framework to build generalizable hierarchical policy with abstract task-agnostic subgoals. We give theoretical analysis to prove the effectiveness of our method and design algorithm overcome the transition mismatch problem in generalization tasks struct a stable policy learning process. Strong results in challenging experiments show the superior- ity of our method. Meanwhile, our method can achieve zero-shot generalization different unseen tasks, which cannot be dealt with by previous methods. We believe that idea of our method could be used beyond our task setting. For future work, we will focus on more general policy learning to solve more complex tasks, not limiting to HRL.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learnin. In International

---

Under review as a conference paper at ICLR 2023
Conference Learning Representations, 2020.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is high variance unavoidable in rl? a case
study in continuous control. In International Conference on Learning Representations, 2021.
Maxime Chevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2018.
Robby Costales, Shariq lqbal, and Fei Sha. Possibility before utility: Learning and using hierarchical affordances. In International Conference on Learning Representations, 2021.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp. 3734-3744, PMLR, 2021.
Nico Gürtler, Dieter Büchler, and Georg Martius. Hierarchical reinforcement learning with timed subgoals. Advances in Neural Information Processing Systems, 34:21732-21743, 2021.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations, 2021.
Xiaohan Hou, Zhenyang Guo, Xuan Wang, Tao Qian, Jiajia Zhang, Shuhan Qi, and Jing Xiao. Parallel learner: A practical deep reinforcement learning framework for multi melti-scenario games. Knowledge-Bard -Baeed Systems, 236: 107753, 2022.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2020a.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2020b.
Ayush Jain, Andrew Szot, and Joseph Lim. Generalization to new actions in reinforcement learning. In International Conference on Machine Learning, pp. 4661-4672. PMLR, 2020.
Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019a.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019b.
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guideer subgoal generation in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 34:28336-288 2021.
Louis Kirsch, Sjoerd van Steenkiste, and Juergen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Repre- sentations, 2019.
10

---

Under review as a conference paper at ICLR 2023
Martin Klissarov and Doina Precup. Flexible option learning. Advances in Neural Information Processing Systems, 34:4632-4644 2021.
Yunwen Lei and Yiming Ying. Sharper generalization bounds for learning with gradient-dominated objective functions. In International Conference on Learning Representations, 2020.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2018.
Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference on Learning Representations, 2020.
Siyuan Li, Jin Zhang, Jianhao Wang. Yang Yu, and Chongjie Zhang. Active hierarchical exploration with stable subgoal representation learning. In International Conference on Learning Represen- fations, 2021.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynam- ics and generalization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaha Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14560-14581, PMLR, 17-23 Jul 2022. URL htttss://proceediin .ml.. mmrr.
press/v162
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1929999 1937. PMLR, 2016.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representa- tions, 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances neural information processing systems, 31, 2018b.
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J Lim. Skill-based meta- reinforcement learning. In International Conference on Learning Representations, 2021.
Devavrat Shah, Dogyoon Song, Zhi Xu, and Yuzhe Yang. Sample efficient reinforcement learning via low-rank matrix estimation. Advances in Neural Information Processing Systems, 33:12092- 12103, 2020.
Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander T Toshev, Sergey Levine, et al. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. In International Conference on Learning Representations, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. Lilacction: Generalizing instructions for multi-task rl. In International Conference on Machine Learning, pp. 10497-10508. PMLR, 2021.
Huan Wang, Stephan Zheng, Caiming Xiong, and Richard Socher. On the generalization gap in reparameterizable reinforcement learning. In International Conference Machine Learning, pp- 6648-6655 6658. PMLR, 2019.
Rundong Wang, Runsheng Yu, Bo An, and Zinovi Rabinovich. 12hrl: Interactive influence-batt hierarchical reinforcement learning. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence, pp. 3131-3138, 2021.

---

Under review as a conference paper at ICLR 2023
Yining Wang, Ruosong Wang, Simon Shaolei Du, and Akshay Krishnamurthy. Optimism in rein- forcement learning with generalized linear function approximation. In International Conference Learning Representations, 2020.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Kamalika Chaud- huri, Stefanie Jegelka. Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pp. 24631 -2445.. PMIR, 17-23 Jul 2022. URL. https:
1/proceedi ngs. m.e.s..rrr.prr.......mmm........ v1 62 xu22g html
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations, 2019.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning, Advances in Neural Information Processing Systems, 33:21579-21590 2020.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Adjacency constraint for efficient hierarchical reinfrrcment learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
12

---

Under review as a conference paper at ICLR 2023
A THEORETICAL ANALYSIS
With equation 3
we can define the error between traditional RL method and our method: (s)
V (se)] To analyze and control the error, we introduce the value VH (8x) in optimal stochastic HRL with coordinates as subgoals:
VV (st) - (s)) & (s)) (i) (it
(s))
HRL
subgoal
so that control the error by two parts: (1) the error from RL policy stochastic HRL policy and (2) the error from ordinates subgoals to c-invariant subgoals.
HRL Error. For the former (denote as eμ [V] V (sx D. the enror bound be controlled by the corollary of conclusion of Zhang's work (Zhang et al.] (022). We firstly formalize a critical factor termed transit tion mi sma tch rate between RL policy and stochastic HRL policy. (The difference between our work and Zhang's 's is that the factor in our work represents the difference between RL and HRL policy, but the difference between two HRL methods Zhang's 's work.)
Definition A.1. (k-step transition mismatch)
For goul-conditooned MDP M with (8 9c) as its k-step transition probability under an optimal condiiondi policy x (ge(ss)) the k-step transition mismatch rate of M is defined as:
Uk:
max Br.kk.
(T=(TT)))) n (galss) (s gc)l
The mismatch rate P& captures the difference between RL policy and HRL policy by the part of trajectories in k steps. The less uncertainty the high-level policy performs with, the less Vi is. The deterministic-MDD of high-level policy will lead to v 0. Then for error CH, we have the conclusion follows:
Lemma A.2. (Corollary of Theorem 2 in (hang et al. (022 With discounted factor y, bounded reward function Rmaz= maxx,a R(s,0 a) and us defined in equation & there is
2(1 7)) + 7 1/ Rmaz
Vak PykBmar
V VH S
(9)
2(1 - 74)2
Proof. See Appendix A.I
Lemma A.2 means the error between RL method and stochastic HRL method will be controlled by Pk. If the distribution of the randomness of subgoals is in a little region with little variance, the HRL policy approximate the optimal RL. policy with controllable error.
Subgoal Error. To analyses the error caused by our subgoals (the latter error is denoted as es = (s:) V/ (s))) ), we should introduce a new metric. We first consider the actually reached region caused by traditional coordinate subgoals and our e-invariant subgoals. If the region and trajectories of the two HRL policies are close to each other in the coordinate system, to some extent, that means the policy and transition are similar. On the other hand, measure the influence of the traditional subgoals and the c-invariant subgoals, the distance and trajectories in the coordinate system can directly reflect the difference between them. Thus, we assume that:
Assumption A.3. (Lipschitz condition) In the k-step reachable region under optimal goal- conditioned policy, for a traditional noal-conditionel MDP M with subgoal gr and pk (* St, gi) as its k-step transition probability, as well as an M. with e-invariant subgoals ga and transitions pk (Se*k St, 92), there is:
(s 1t, 9t) pk (5 g2)l S ( +(((++))l1
(10)
where Lug is the Lipschitz constant, &p the coordinate extructing function defined above.
The Lipschitz condition is common and widely used in many other RL or HRL works (Wang al. (001)(Shan) et al. (020) (oggianu et al. 202T). Assumption A.3 means that, if the low-level policy learns well and can execute the subgoals given by the high-level policy, the difference between the
13

---

Under review as a conference paper at ICLR 2023
two HRL policies be measured by the Euclidean distance of displacement in the coordinate system. The distance is in the reachable region with optimal low-level policy in & steps, from the same start state s,, If the reached states closed, the policies are considered similar.
By the setting above, we have the following theorem which provides a suboptimality upper bound for enrrr eg:
Theorem A.4. With discounted factor bounded reward function Ranax = max,,0 R(s, 0), Lips- chitz constant Lug, variance bound defined above, with a high probability there
Lup (20mm + +3322) Rmaz (k
IIV Vo I
(1I)
2(1 -+)
1-
where 8mar = max (Igg]) ||
IIall and || are the distance of the relative displacement from any 81 to 81+K by subgoal and ga respectively.
9:93
Proof. See in Appendix. A.2
The theorem A means that the error between the traditional HRL method with coordinates as subgoals and our method can be constrained by the similarity metric of the two policies. By the conclusion above, our method has theoretical guarantee to approximat to the original optimal policy with controllable error bound. Meanwhile, in our methods, the subgoals can be reused in different maze environments, which leads to superiority in generalization tasks.
A.1 PROOF OF LEMMA 1
Proof. Consider state 8, E S, for the value function of k step:
(s:) = Σ (T)(ss)))(((TT)l + +
(12)
TE
where PM (Tx(81) = pM (8*+* (8e) is the k-step transition probability with policy n, R(Tx(s)) = R($14A,88) = E Li=ttl 7 +R(ss) is the bounded accumulative return in The The is the possible k-step trajectory by optimal policy n² executing & times. The optimal policy should maximize the external reward from the environment:
T
π arg max E
(a.a) wm
r R(st,a)))
(13)
equation 12 can be rewritten as vector form for any state 9t S, ie:.
(st) = Σ Pd (St). R(st) + +
(14)
where ( the inner product of vectors, Pa (st) E [0, 10.),, Vs E S.|[P*(s<)ll) denote the probablistic distribution vector of the k-step trajectory starting from under the policy n².
R(st) [0, kRmoz|lS is the accumulative rewards vector of k-step of the k-step trajectory. V is the optimal value vector function of all the states.
Similarly, value of HRL method with stochastic subgoals can be written as follow:
Σ H1 (grls))
Σ
( (R 8e) + 7 " VV (S.4k)
95g
S1+LES
(15)
Σ
" (9nls)) (1),
st), R(xe) + +
9E0
where 7 (g(gg (s) is the high-level policy to choose subgoal 9r. Pk (8t+* (sc 9t) is the k-step transi- tion probability of the level policy with subgoal gt- R(S 8t) is also the k-step accumulative rewards. pł (St, ge) is the probablistic distribution vector of the k-step trajectory starting from s with subgoal gt.
14

---

Under review as a conference paper at ICLR 2023
Denote that (s. 92) - E IgEg x (gilss) 8t), that definition A.1 can be rewritten
Pk max
|| (sc) Pa (st. li)ll
(16)
Then for every 8t E S, there is:
v (st) - VH
Σ pd (s:), R(st) +2'
Σ n" (gels)) (M 8t), R(se) + +>
Th
9E9
Pd (st) Σ
π (gi ( ( (gt, st); R(s))
(17)
DEG
tg
Σ
PT (SL), v*a)
Σ x (9)l)) (gc, (g), VH gEG
Th
note that (En, 1) = and (Eeeg x ( (ge. st), 1) = 1, where 1 is an all-one
vector of [S| (S--dmmensoo,, for the first term of equation 17 we have
(=((s)
Σ DEg
n (9rls)) (g), R(se)
(18)
k R.maa
P (st)
Σ gEg
π (*,,,)) st), R(s()
1
2
by Holder inequality, there is:
k RRRaz
1 <
Σ PT Σ TH (eclss) (k((
R(s:)
1
2
9EG
kRmaz
(19)
PT PH 9:) (nn(ssg)))l R((ss)
max
Nt.gt
2
Vi k Rmar
The second term can be similarly bounded by:
Σ Pd (st),
Σ "H (o.ls)))*(gg. st), Vij
gEG
PTT(st)
Σ p (st). Vn
(20)
b /
Σ (st), Vă
Σ π (gel))) (ge, st), Vi
74
gEg
where
15

---

Under review as a conference paper at ICLR 2023
y/
{I Pr ((s)) vn
)
Σ
(8t) V/ ) (st),
s S/ 1 -V lc
(21)
and
8
Σ pd (st), Vii
Σ π (gelss) (ge, st). VH
19E
-
k.Rmax
Σ pd (8r) (5))
Σn TH (gn) ph
pa (ges (g))
2(1 - gK)
(22)
9E9
k R
5> max 1M (s 9:)1l teift kRlmax
V2
1
1
2(1 -1))
5- Vk 2(1 7*)
So that with and 22 there is
1 PikkRaar
(s:)s - (+t) S
+ gh) + +* * Viill
(23)
2
Since 24 holds for all S, so there is:
I Vil - V2
+ T
(24)
2(1 xk)
2(1 - k2
A.2 PROOF OF THEOREM 1
Proof. Consider assumption A.3 there is:
[|((()+))) - -p(((++))
(25)
[|(((((*)) ((()) + p(s)) ((((4+)))
restate that the (p(s) is the coordinate of state 8t, so that (s - (p(s) is the direction vector of the movement from step to L k.
By definition 3.1 we denote the relative displacement A, = |ل = E (p(st) - ( for any state St with e-invariant subgoal Ja- So that the displacement vector be written as Ts = A + Se, where A, is fixed vector of expected direction and Σ is random vector. De.r, Ae.y ~ (O, o). where a I is much less than the moving distance A,. Thus, the equation 25 can be rewritten as:
((((+))) - (p())) (á, Σ 25
(26)
S ((8)4)) (p(s)) A + A
Consider that in normal distribution, the random variable has a high probability fall into the region
of [-3o, 3d]. So that with high probability, there is ||Å.|| < /2 x (3o)² < 3/2e. Thus, the assumption A.3 can be rewritten as follows:
(*(ssel(s - ph (5/tx) St,
S Lig (25maz + +3v2c)
(27)
where 8maz max(|lel) Ae). 9.92
16

---

Under review as a conference paper at ICLR 2023
The equation 27 can be seen the HRL transition mismatch rate between method and optimal HRL method with stochastic coordinate subgoals. is bounded by the maximal k-step movement of the agent with the variance of the random variable. With lemma A.2 the similar conclusion
be obtained by changing the k-step transition mismatch to equation 27 Then we can get the result
of A.4
B ALGORITHM
Our algorithm for the high-level policy learning is as follows (algorithm
Algorithm PEG-A2C Algorithm
1: Initialize multi-process actor parameters Of for E [1,
2: Initialize multi-process value parameters 0g for i [1,n]
3: for episodes in 1,M do
4: for E [1.n] do
5: Reset gradients: d0A and d0t
6: Synchronize thread-specific parameters
7:
repeat
8:
Perform a according to policy (())))))
9:
Receive reward and new state $++1
10:
t t ++++1 until terminal ST or tmax
11:
12:
Set R=
13:
for j (1- 1,.., 0} do
14:
R R R +
Accumulate gradients w.r.t. 6%
1 Veg log (a)))ls;00)) v(s:0()) d0, d0, + n
Ve,
Accumulate gradients w.r.t. 0%
11 0
dd dd. +
(R, - V(ss;:))))
n a0,
17:
end for
18:
end for
19:
Synchronize and update parameters
20:
end for
17

---

Under review as a conference paper at ICLR 2023
High-dimensional Action
Goal-condilieeed DroQ
Environment
Subgoal Vector
Discrete Subgoal Distribution
Pixel Input
300 + PRelL
2C0 + PReLU
FC 2000 + PReLU
FC 512 Embedding
Three-Layer Conv Net 64x128x256
FC 300 - RelU
FC 200 + ReLU
FC0
Value
V
LSTM Net
Reward Signal
Actor and Critic
Figure Structure of the network.
C DETAILS OF FRAMEWORK
C.I NETWORKS
The structure of our high-level network is shown as follows (figure g
D DETAILS OF EXPERIMENTS
Ant D-Maze. this task, the agent should navigate from the bottom to the top. Different from setting in previous works, the maze is larger, and the agent will only attain reward 'I' once when it passes the corner and reach the final region.
Ant Random Square Maze. It is empty will start at a random initial position every episode, to walk towards the yellow door, which also has random position on the wall. Only when the agent passes the door will it obtain reward.
with door and robot. In this task, the agent
Ant S-shaped Maze. this task, the agent will start at the left region and should pass three doors. The trajectory is circuitous and hong-horizon. especially the final door is more difficult to achieve than the region in Ant Mazee Every first-time transiting of the doors will give the agent a reward.
Ant Spiral Maze. This task is in large maze with spiral routes and five doors and the agent will start at the middle region. Such a long-horizon task requires at least thousands of steps to move to the final region. Also, every first time transiting the doors will give the agent reward.
Generalization Maze. This task includes three fixed mazes of 'Maze-gl', 'Maze-g2', and 'Maze- g3' (figure D which variants of Ant Random Square Maze' with different unseen structures. Only when the agent passes the door will it obtain reward.
D.1 STEPS OF EVERY EPISODE OF DIFFERENT TASKS
Table 3 shows the maximal steps of every episode of every task.
18

---

Under review as a conference paper at ICLR 2023
Table 3: Steps of every episode of different tasks
Maze
Maze-random
Maze-U
Maze-S
Maze-spiral
Maze-gl
Maze-g2
Maze-g3
Steps per ep
600
600
1000
5000
600
600
600
D.2 DETAILS REWARD SETTING
For the sparse reward, it will be obtained by the agent when the agent goes across the door, i.c., the coordinates the agent fall into region the door. For the dense reward, is 1/(1 + d) of Euclidean distance d between the agent and the current goal in the coordinate system for every step. Once the agent goes to the current goal and gets the reward, the goal will update, and the reward will be calculated by the new goal. As a result, the curves of average reward in these tasks may decline sometimes.
D.3 HYPER-PARE PARAMETERS
Table 4: Steps of every episode of different tasks
Hyper-parameters
Value
Details
Subgoal dimension
8
two-hot vector of four subgoals abstract subgoals Represented by vector grids of 'random' "'',""''' ''pirrl" size in coordinates system of MuJoCo
Subgoals
t, 1, t-
Room size
1747, 7*7, 14*7, 21
Grid size
3#3 I
Sparse reward
Dense reward
[0.03, 1.0] 25
range of dense reward steps of the low-level to execute
High-level frequency
Intrinsic reward coefficient
3
19


Reviews:
[{"review_id": "f-uk6Oiz2C", "reviewer": "Reviewer_LpyT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to address the transition mismatch problem in GCHRL using (1) more readily reusable abstract subgoals, and (2) manually injecting stochasticity into the low-level policy as a form of regularization. Both of these contributions are ultimately proposed to improve generalization on unseen tasks where the same subpolicies and high-level policy are applicable. In addition to these contributions, they also propose multiple additional modifications, including using expected gradient updates, as well as a new form of intrinsic reward to train their abstract subgoals. They provide theoretical analysis showing that the error between their method’s hierarchical policy and the optimal policy can be bounded. Lastly, they compare their method to SOTA hierarchical and non-hierarchical algorithms, and demonstrate that their method outperforms these prior approaches, and achieves significantly better zero-shot generalization performance. ", "summary_of_the_review": "* This work is generally well-written and the method is well-motivated, but there are a number of contributions the authors make in this paper, and without the proper ablations, it is difficult to understand the relative importance of each. For instance, without these ablation results, the performance benefits may be merely due to the form of the abstract subgoals provided / corresponding intrinsic rewards, rather than the injected stochasticity, and this would make for a comparatively weaker contribution. These concerns will need to be addressed in order for me to recommend acceptance of this paper.", "strength_and_weaknesses": "Strengths:\n\n* Both major contributions (i.e. using more general subgoals for maze navigation tasks and injecting stochasticity into low-level policies to regularize/improve robustness of the high-level policy) are well-motivated and appear to be novel for this problem setting.\n\n* All important details are described clearly. \n\n* The results (especially the generalization results) are impressive and indicate a clear benefit of the proposed approach over baselines in continuous control maze navigation tasks. \n\n* The discussion of related work appears to be complete.\n\nWeaknesses: \n\n* The main weakness is that there are a number of contributions presented in this work, and there is no comprehensive ablation study demonstrating to what degree each contributes to the improved performance. It would be very helpful to see an ablation study where the effects of (1) abstract (and discretized) subgoals, (2) injected stochasticity, and (3) expected gradients are controlled. Without this, it is difficult to make sense of the results, and to understand the relative importance of the contributions. It would also be useful to see how well the baselines perform when these tricks are applied.\n\n* There are a number of grammatical errors throughout the document that make some points unclear. I suggest another careful pass through to make sure each sentence is well-structured for clarity. There are a number of examples throughout the document, but here is just one example (with some rough ideas on how to improve the grammar/clarity in brackets); in the last sentence of ‘motivation’ paragraph: “[Thus, a] policy based on these subgoals [may also generalize better across related] tasks.”\n\n* There are some minor formatting issues. E.g. backticks are not used as opening quotes in some instances (e.g. defn 3.1, motivation paragraph), and when multiple works are cited they are not included in the same set of parentheses (they appear to each be different calls to \\cite). \n\n* It is unclear how useful the theoretical analysis is. Instead of bounding the the error between an optimal RL method and the proposed method, it could have been nice to see some theoretical motivation for using stochasticity in the first place. But it is possible I am misinterpreting the results—-I did not carefully read the proofs.\n\n* There does not appear to be any results demonstrating the effects of different levels of injected stochasticity. This would be helpful to see how robust the method is to this hyperparameter. It should also point to an important trade-off.\n\nQuestions:\n\n* It is not clear to me why the transition mismatch problem is not mitigated by previous relabeling approaches in the multi-task setting. It is claimed that because “successful trajectories are not identical across different tasks, relabeling sampled trajectories may aggravate the mismatch problem”. Is there evidence of this (I see no citation), or some clearer intuition for why this is the case? An example could also be helpful.\n\n* Which of the implementation tricks are applied to the baseline methods? I.e. between (1) abstract subgoals, (2) expected gradients, and (3) using discretized high-level action space. This is unclear to me from the paper, but I could have missed it somewhere.\n\n* What value of $\\sigma$ is used for the results in the paper? It is possible I missed this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qn2gkkbpb0", "reviewer": "Reviewer_9r3Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript describes the construction of an Hierarchical Reinforcement Learning (HRL) policy for solving the navigation in a maze task. \nTechnically, the paper proposes the epsilon-Invariant HRL to leverage task-agnostic abstract subgoals. This approach tries to mitigate the transition mismatch problem ((Zhang et al., 2022) (Levy et al., 2018)) in HRL. Basically, this problem emerges when we try to learn reusable (or task agnostic) subgoals (using some invariable abstract physical quantities as subgolas, e.g. navigation directions), to later build a generalizable high-level policy. The latter evaluates the subgoals with incorrect rewards caused by poorly trained low-level policies. \nThe approach in the paper is simple: consider the transition mismatch as a sort of randomness process, this way the high-level policy is trained to adapt to it.\nA simple parallel algorithm (the parallel expected gradient advantage actor-critic (PEG-A2C)) is introduced to calculate the expected gradients of different trajectories and update the parameters synchronously, sampling multiple trajectories.\nMaze navigation experiments are performed in the MuJoCo suit for maze-navigation tasks. Multiple baselines are used in the experiments (DroQ - SOTA RL model, HIGL - SOTA HRL model, etc.)\n", "summary_of_the_review": "Overall, the paper could be of interest to the rest of the community. The ideas are somewhat incremental. The main problem is in the experimental evaluation, and in the clarity with which the model has been laid out (especially in its comparison with the work of Zhang et al. 2022). For the moment, I am inclined to consider the article below the acceptance threshold.", "strength_and_weaknesses": "Strengths:\n+ The paper described an HRL policy learning process that offers generalization capability across different mazes and different robots. This is an interesting contribution of the work. The zero-shot robot generalization experiment (Section 4.4, table 2) opens an interesting research direction.\n\n\nWeaknesses:\n\n-The experimental evaluation has some important limitations:\n\na) Not enough previous HRL methods which have been specifically designed for the same problem are considered in the experimental evaluation. The manuscript just includes the use of HIGL model (Kim et al., 2021)., but others should have been considered: e.g. (Li et al., 2021), HRAC (Zhang et al. 2022) .\n\nb) The evaluation metric proposed is not appropriate. Average reward can be useful for some problems, but in maze navigation tasks, the success rate is probably the most informative metric. See how all previous works use it previous papers (HIGL for instance), which will allow a direct comparison with them. So, I encourage the authors to change figure 5 to include this metric (and remove figure 6). The reward that the proposed approach accumulated is considerably higher that the one gathered by the rest of models, so I do not expect any surprise when the metric changes, however, the reader will be able to see how \"fast\" the new model is in achieving a SR of 100% with respect to the rest of the methods in the literature.\n\nc) I encourage the authors to also include previously used maze shapes in the experimental evaluation ((Li et al., 2021)(Kim et al., 2021))\n\nd) For the ablation study, the inclusion of RAND-H and our-oracle is an appropriate choice. I am intrigued, however, by the performance of Ours-Oracle in the generalization experiment in Table 1. How can the model trained using the oracle for a particular type of maze work well in any other type of maze? This means that the high-level policy learned by the proposed model (without the oracle), is far from being effective in generalization tasks.\n\n-See my comments below about the novelty, which a one of the weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NRFYcHvx25", "reviewer": "Reviewer_BQSm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework to mitigate a transition mismatch problem in hierarchical reinforcement learning, which utilizes abstract, task-agnostic subgoal generation. Under a certain condition on invariance in subgoals, this work proves thae the error between the hierachical RL and the original full RL can be bounded. The efficacy of the proposed method has been demonstrated on Ant Maze problem in Mujoco, compared against several baselines. ", "summary_of_the_review": "This paper is handling an important problem in hierarchical reinforcement learning by presenting a reasonable framework. The generalizability of the proposed scheme needs to be better justified. ", "strength_and_weaknesses": "Strengths: \n- This paper is clearly written and easy to follow. The motivation is clearly indicated and the methodlogy is described clearly. \n- The issue of handling mismatch in hierarchical decision settings is a timely and important. \n- The proposed framework looks like making sense in general. \n- The numerical evaluation is done with reasonable details. \n- It is good to provide a theoretical proof on some properties of the proposed scheme.\n\nWeaknesses:\n- The theoretical proof relies on the assumption on the epsilon-invariance. It is not clearly explained whether or not this assumption makes practical senses and/or is a simplying one.\n- The generalizability of the proposed scheme is not very clearly explained/defended. The notion of subgoals is claimed to be abstract and generic, but it is not very clear to come up with subgoals other than in a geographical domain. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1BE4MDNf9h", "reviewer": "Reviewer_pFzR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to use four directions (top, down, left, right) as the general subgoals to learn the high-level policy in a hierarchical reinforcement learning (HRL) setting. Since the directions are task-agnostic and agent-agnostic for the typical navigation jobs, the proposed framework could be more generalizable. Further, to overcome the mismatch problem while training an HRL policy, the authors (1) trained the high-level and low-level policies separately and (2) introduced a slight noise controlled by \\epsilon into the perfect low-level controller in the environment. The experimental results in MuJoCo suggest that the proposed framework outperforms various baselines.", "summary_of_the_review": "The work proposed a simple but intuitive way to discretize subgoal space to improve the generalization ability of an HRL policy. The experimental results show the effectiveness of the proposed framework. However, since the paper is not well polished, it is hard to read and understand the keys under the hood. In addition, because there are many missing details in the experiment section, I have no confidence in the reproducibility. Finally, the usefulness of one of the claimed contributions about “expected policy gradients by A2C” is not validated, which makes the technical contributions weak.", "strength_and_weaknesses": "+) The studied problem is essential, and the proposed idea is reasonable. Using four directions as subgoals for navigation tasks makes more sense than using absolute coordinates in prior works. The results also suggest that the learnable high-level policy is reusable for different low-level controllers (different agents).\n\n\n-) The manuscript has not been well written. Several typos, grammar errors, duplicated words, or redundant spaces exist. In addition, some notations are misleading. Taking “x+, x-, y+, y-“ as an example, are the x and y variables? Or do they only indicate direction? All of them make the paper hard to follow.\n\n\n-) Some algorithms proposed in prior works use collected on-policy rollouts to compute empirically expected policy gradients to perform the policy optimization, such as SAC, PPO, and TRPO. Is there a specific reason for using A2C in the experiments? Since the authors claimed the expected A2C algorithm is one of the contributions, a comparison to the mentioned policy gradient algorithm is necessary.\n\n\n-) There are many missing details in the experiment section. For example, what is the action space for different agents (e.g., the low-level controllers for different agents?)? What are the learning rates to train the high-level policy and the low-level controller? What optimization algorithm (e.g., SGD?) is used in the training stage?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject