SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now extend this result to hierarchical MDPs. Before doing so, it is essential to notice that even the best hierarchical policy (as constructed in Eq. (1)) might be sub-optimal. This is a direct consequence of the goal-conditioned architecture. If, while executing a sub-episode, it appears that another sub-goal becomes more valuable, the architecture proposed does not allow interruptions. The agent will first have to complete the current sub-episode before being able to adapt to the new circumstances. Let  $V_{o}^{\pi_{l}^{*},\pi_{h}^{*}}$  denote the value of the optimal hierarchical policy value function in the original MDP. Then, the sub-optimality gap is larger than the gap between the current policy pair and the optimal hierarchical policy  $\Delta_{k} = V_{o}^{*} - V_{o}^{\pi_{l}^{k},\pi_{h}^{k}} \geq V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{k},\pi_{h}^{k}}$ . Therefore, if for some  $N$ ,  $V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{k},\pi_{h}^{k}} \geq \epsilon$  for at least  $N$  episodes, it must also be the case that  $\Delta_{k} \geq \epsilon$  for at least  $N$  episodes. Hence,  $N$  is a lower bound on the number of episodes where the algorithm must follow a sub-optimal policy.
In the following theorem, we lower bound the number of episodes required to learn a pair of policies  $(\pi_l,\pi_h)$  which are  $\epsilon$ -accurate with respect to the optimal hierarchical policy  $(\pi_l^*,\pi_h^*)$ . By the above argument, this will also be a lower bound on the number of episodes necessary to learn an  $\epsilon$ -accurate policy with respect to the optimal policy  $\pi^*$ .
Theorem 3.1. There exist positive constants  $c_{l}$ ,  $c_{h}$  and  $\delta_0$  such that for every  $\delta \in (0, \delta_0)$  and for every algorithm  $A$  that satisfies a PAC guarantee for  $(\epsilon, \delta)$  and outputs a deterministic policy, there is a fixed horizon MDP such that  $A$  must interact for
$$
\mathbb {E} [ N ] = \Omega \left(\max  \left(\frac {\left| S _ {l} \right| \left| \mathcal {A} _ {h} \right| \left| \mathcal {A} \right| H _ {l} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {l}}\right), \frac {\left| S _ {h} \right| \left| \mathcal {A} _ {h} \right| H _ {h} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {h}}\right)\right)\right) \tag {3}
$$
episodes, in the original MDP, until the policy is  $(\epsilon, \delta)$ -accurate.
The complete proof is in Appendix A.1. In the following, we highlight the main steps.
Sketch of the proof: An  $\epsilon$ -accurate pair of policies must satisfy the following inequality,  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l},\pi_{h}}|\leq \epsilon$ . To find a lower bound on the number of episodes  $N$  before we obtain an  $\epsilon$ -accurate pair of policies  $(\pi_l,\pi_h)$  we used the following steps:
(i) We decompose the objective using the triangle inequality,  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| + |V_{o}^{\pi_{l}^{*},\pi_{h}} -$ $V_{o}^{\pi_{l},\pi_{h}}|\leq \epsilon .$
(ii) We show that the number of samples required to guarantee  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| \leq \epsilon /2$  is bounded by  $\Omega \left(\frac{|\mathcal{S}_h||\mathcal{A}_h|H_h^2}{\epsilon^2}\ln \left(\frac{1}{\delta + c_h}\right)\right)$
(iii) We show that the number of samples required to guarantee  $|V_{o}^{\pi_{l}^{*},\pi_{h}} - V_{o}^{\pi_{l},\pi_{h}}| \leq \epsilon /2$  is bounded by  $\Omega \left(\frac{|S_l||A_H||A|H_l^2}{\epsilon^2}\ln \left(\frac{1}{\delta + c_l}\right)\right)$
Combining these three steps gives us the result in Theorem 3.1; see A.1 for more details.
3.1 Interpretation of the Sample Complexity Bound:
By comparing this lower bound to that in the original MDP, we can identify the problem characteristics that might lead to improved sample efficiency. In general, only one of the two MDP characteristics will dominate the overall sample complexity because of the max operator in the bound in Eq. 3. To maintain the max as small as possible, the complexity should be distributed between the low- and high-level MDP as evenly as possible. We discuss some of these key insights below:
State abstraction: Only one of the two-state space cardinalities will dominate the bound in Eq. 3. This suggests that an efficient decomposition tends to separate the original state space as evenly as possible between the two levels of the hierarchy. Another phenomenon at stake is the low-level re-usability. Due to the state abstraction, the low-level agent can re-use its learned policy in different states (i.e. different states  $s_1, s_2 \in S$  whose low-level component  $s_l$  are the same). We rewrite the lower bound 3 in terms of the re-usability index  $\kappa = \frac{|S|}{|S_l|}$ .
$$
\mathbb {E} [ N ] = \Omega \Bigg (\max \left(\frac {\frac {| \mathcal {S} \times \mathcal {A} _ {h} |}{\kappa} | \mathcal {A} | H _ {l} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {l}}\right), \frac {| \mathcal {S} _ {h} | | \mathcal {A} _ {h} | H _ {h} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {h}}\right)\right) \Bigg).
$$
Eq. 4 highlights that a large re-usability index improves the sample efficiency.
Temporal abstraction: Similarly, only one of the two-time horizons will dominate the bound, again suggesting a fair repartition of the load. The temporal abstraction (reducing  $H$  to  $H_{h}$  and  $H_{l}$ ) simplifies the credit assignment problem for the high-level and the low-level policies by giving denser feedback. The low-level agent is rewarded for completing sub-tasks that are significantly shorter than the original task, and the high-level trajectory consists of significantly fewer (high-level) steps than a trajectory in the original MDP.
High-level action space: This is the only term that appears on both sides of the  $\max (\cdot ,\cdot)$  in Eq. 3. This highlights that both the high-level and the low-level benefit from a compact sub-goal representation.
It is interesting to note the contrast between the state space decomposition and the design of the high-level action space. To find efficient state decomposition, the amount of information available at each level must be distributed among each level of the hierarchy. In the case of the sub-goal space, it appears that both levels benefit from a compact representation.
The above discussion highlights properties of the hierarchical decomposition that could improve sample complexity. Note, however, that our bound also shows that a hierarchical decomposition does not always improve the sample efficiency. Indeed, there will be some settings where using a "bad" hierarchical decomposition does not improve the sample complexity. Our bound can, therefore, provide a sanity check to determine whether a hierarchical decomposition could lead to an improved sample complexity. However, finding an algorithm that achieves this improved sample complexity can still be challenging. Nevertheless, the proposed Q-learning-based hierarchical algorithm empirically demonstrates the potential benefits of leveraging the considered decomposition. In Section 5, we consider several MDP decompositions and empirically validate that when our bound suggests the hierarchical decomposition is beneficial, our algorithm (see Sec. 4) leverages this to achieve lower sample complexity.
4 Stationary Hierarchical Q-Learning
Once we know that we are in an MDP where the hierarchical decomposition could lead to improved sample complexity, the next challenge is to design an algorithm to exploit this. This section proposes the Stationary Hierarchical  $Q$ -learning algorithm (SHQL) for this purpose. One of the most challenging aspects of jointly learning a pair of policies is the non-stationarity of the high-level transition dynamics,  $P_{h}$ . It was briefly mentioned (in Sec. 2.2) that the high-level transition function,  $P_{h}$ , is non-stationary since it depends on the low-level policy,  $\pi_{l}$  with the next high-level state depending on whether  $\pi_{l}$  managed to reach the sub-goal. To address this issue, we leverage the fact that the algorithm knows what a successful sub-episode is, i.e. it knows if the low-level agent managed to
<sup>1</sup>Note that this is a lower bound - we still do not know if there exist algorithms which achieve this lower bound.
Figure 2: The grid of plots depicts, on the top row, the mazes whose size ranges from 4 rooms to 1024 rooms. The bottom row shows the steps required for SHQL (in blue) and Q-learning (in red) to complete the maze. The standard deviation is obtained by running ten different seeds.
arrive at the desired sub-goal. In the case of off-policy methods, this is typically leveraged with hindsight correction [17]. However, in the considered on-policy setting, we propose that the algorithm only makes an update if the low-level agent behaves reasonably well (i.e. solving the sub-goal). In this way, the algorithm filters all bad examples from the training set, and the behaviour of  $P_{h}$  is more stable. Note, however, that the reward function of the high-level agent remains non-stationary. At first, sub-goals won't be solved optimally, incurring a small reward to the high-level agent. However, the associated reward will increase as the low-level agent learns to solve sub-goals more efficiently. As detailed in the function LowLevelUpdate in Algorithm 1, the low-level agent performs Q-learning updates on the observed low-level transitions and rewards. The high-level agent also performs Q-learning updates, but only on successful transitions, as specified at line 10 of Algorithm 1.
Algorithm 1: Stationary Hierarchical Q-learning (SHQL)
Input:  $Q_{\vdots \vdots}^{l} = 0$ $Q_{\vdots \vdots}^{h} = 0$  done  $h =$  False,  $t = k = 0$    
1 while not done and  $k <   K$  do   
2 Observe  $\mathfrak{sl}$    
3  $g_{sub} = \pi_h(s_h)$    
4 done  $l =$  False /True if sub-goal solved \*/   
5   
6 while not done and  $t <   T$  do   
7  $\begin{array}{rl} & {\mathrm{a}_l = \pi_l(s_l)}\\ & {\mathrm{Observe~s'_l,~r_l,~done_l}}\\ & {\mathrm{LowLevelUpdate((s_l,a_l,r_l,s'_l,g_{sub}))}}\\ & {\mathrm{s}_l = s'_l}\\ & {t = t + 1} \end{array}$    
8   
9 LowLevelUpdate((sl, al, r, sl, gsub))   
10  $\begin{array}{rl} & {\mathrm{s}_l = \mathrm{s}_l^{\prime}}\\ & {t = t + 1} \end{array}$    
11   
12 Observe  $\mathbf{s}_h^\prime ,\mathbf{r}_h$  done h   
13 if done then   
14  $\begin{array}{rl} & {\mathrm{Q}_{nxt}^{h} = \max_{g}\mathrm{Q}_{s_{h}^{\prime},g}^{h}}\\ & {\mathrm{Q}_{s_{h},g_{sub}}^{h} = \mathrm{Q}_{s_{h},g_{sub}}^{h} + \alpha *(r_{h} + Q_{nxt}^{h})} \end{array}$    
15   
16  $\begin{array}{rl} & {\mathrm{s}_h = \mathrm{s}_h^{\prime}}\\ & {k = k + 1} \end{array}$    
17   
18 Function LowLevelUpdate(s, a, r, s' gsub):   
19  $\begin{array}{rl} & {\mathrm{Q}_{nxt}^{l} = \max_{a^{\prime}}\mathrm{Q}_{g_{sub},s^{\prime},a^{\prime}}^{l}}\\ & {\mathrm{Q}_{g_{sub},s,a}^{l} = \mathrm{Q}_{g_{sub},s,a}^{l} + \alpha *(r + Q_{nxt}^{l})} \\ & {\mathrm{return}\mathrm{Q}^{l}} \end{array}$
We now empirically evaluate<sup>2</sup> the impact of the decomposition on various MDPs to validate the lower bound found in Section 3 and evaluate the performance of our proposed SHQL algorithm. To satisfy the assumption of hierarchical structure, the environments considered are a generalization of the four-room problem with an arbitrary number of rooms, called the  $n$ -rooms problem. The entire maze is built by arranging an arbitrary number of rooms on a grid. The high-level task would involve
$^{2}$ Experiments were run on a  $12^{th}$  Gen Intel Core i7 with 16GB of RAM, to train the agents on the largest maze considered takes  $\sim 7$  minutes.
learning the shortest sequence of rooms that lead the agent from the starting position (the top left room) to the goal room (the bottom right room). The low-level task is to learn how to navigate within each room and to reach the instructed hallway. To further modulate the task's difficulty (in addition to the maze size), we vary the room profiles used, as depicted in the rightmost plot of Fig. 4.
The set of MDPs generated by these environments are the following:
The original MDP: This is a standard grid-world MDP, where the state space indicates the cell where the agent is located, and the action space allows the agent to move one cell in any cardinal direction (North, South, East, West). To obtain stochastic environments, each action has a success probability of  $p_{success} = 4 / 5$ . In case of failure, the action will be chosen at random.
The high-level MDP: The high-level state space is restricted to the room where the agent is currently located, and the exact position of the agent within that room is abstracted away. The high-level actions instruct the low-level to reach one of the available hallways. Note that not all rooms have access to the four hallways.
The low-level MDP: The low-level agent only observes the agent's current location within a room and the goal instructed by the high-level agent (one of the reachable hallways). It then uses the primitive action space (the four cardinal directions) to reach the desired hallway. All required code to reproduce the experiments is made available online [1].
5.1 Identical Rooms
We first introduce the experimental setting in its simplest form. The environments considered in this subsection are mazes built by assembling identical rooms without obstacles (i.e. the top room profile in Fig. 4). Fig. 2 illustrates the empirical performance of our SHQL algorithm against Q-learning in the original MDP. As expected for simple mazes (e.g. with 4 or 16 rooms), the hierarchical decomposition does not provide much improvement. Still, as the problems grow more complex, the empirical evaluation suggests a significant improvement in sample efficiency. This is also confirmed by our bound (yellow curve on the rightmost plot of Fig. 4), which highlights that the efficiency gain of HRL is mostly achievable in complex MDPs (i.e. MDPs with large state and action spaces). It is essential to notice that in this experiment, the low-level decomposition remains constant for a given set of room profiles. This is why the benefit of HRL increases with the number of rooms (i.e. the high-level state space) until a plateau is reached. Once the bound is dominated by the high-level MDP, the unchanging complexity of the low-level MDP causes the ratio between the RL bound (Eq. 2) and the high-level part of the HRL bound (Eq. 3),  $\frac{|S||A|H}{|S_h||A_h|H_h}$ , to remain constant (even though number of rooms might still grow).
5.2 Different  $n$ -rooms & Gymansium Taxi Task
To make the task more challenging, we next increase the number of room profiles used to construct the mazes. As depicted in the rightmost plot of Fig. 4 we considered four different room profiles, each one with a different obstacle in the room. The low-level agent must now learn to navigate multiple
Figure 3: Those plots are similar to the ones shown in Fig. 2, showing the performance obtained on mazes built from four different room layouts.
Figure 4: The left-hand plot shows the evolution of the ratio between the RL bound Eq. (2) and the HRL bound Eq. (3) for various mazes and different room profiles. The plateau is obtained when the high-level MDP dominates the bound, leading to the following ratio:  $\frac{|S||A|H}{|S_h||A_h|H_h}$ . The curves are colour-coded such that a darker curve indicates more room profiles were considered. The right-hand side of the plot shows the different room profiles available to build the mazes.
types of rooms to reach the sub-goal sent by the high-level agent. The performance of the algorithms with four different rooms is shown in Fig. 3. The introduction of different room profiles allows us to modulate the complexity of the low-level MDP, in contrast to varying the number of rooms, which only affects the complexity of the high-level MDP. This additional complexity results in a larger state space  $S_{l}$  but may also result in a longer horizon  $H_{l}$  as the optimal trajectory might require more time to navigate around obstacles to reach the desired hallway successfully. It also becomes evident that, as the number of rooms increases, the hierarchy's benefits become more significant. Nevertheless, comparing Fig. 2 and Fig. 3 we can observe that the introduction of various room layouts has little effect on the Q-learning curve (in red). At the same time, it makes the task slightly more challenging for the HSQL learning curve (in blue), especially when the number of rooms is small since it suffers from the increased complexity of the low-level MDP. But, when the number of rooms is sufficiently large for the high-level MDP complexity to dominate the bound, the benefit of hierarchical decomposition becomes evident. The evolution of the bound ratio (HRL/RL) for the various MDPs considered is shown in the leftmost plot of Fig. 4. It shows that the low-level MDP dominates the bound when the maze consists of a small number of rooms. However, the curves clearly indicate that the expected sample efficiency improves as the high-level MDP becomes more complex (i.e., balancing the complexity between the two levels of the hierarchy). This result is also supported by empirical evidence as illustrated in Figs. 2, 3, 5, and 6. The Gymnasium Taxi environment [9] experiments presented in Appendix A.2.2 further validate our approach and conclusion on an entirely different task.
In this work, we analysed the sample complexity of goal-conditioned HRL. To the best of our knowledge, we provide the first result that analyses the decomposition induced by goal-conditioned HRL. In particular, our lower bound offers a valuable tool for practitioners that could help them decide whether they should consider a hierarchical decomposition for their problem. We also designed a novel algorithm that can leverage the hierarchy to improve its sample efficiency and implemented this on a set of hierarchical tasks. These experimental results further emphasizes the usefulness of the proposed bound since our theoretical findings support empirical efficiency gains.
Although this paper has taken a significant first step in bettering our understanding of the benefits of hierarchical decomposition, there is still scope for further work in this area. Three immediate open questions are: (i) whether our lower bound could be refined by explicitly accounting for the interactions between the low-level and the high-level agent, (ii) is it possible to design an algorithm that can theoretically match the proposed lower bound, (iii) the current results only consider the cardinality of the sub-goal space because we assumed that the sub-goal-space was given and that all sub-goals where solvable. Methods that design efficient sub-goal spaces remain largely unexplored and are a critical aspect of the design of HRL algorithms. Moreover, the insights we proposed are framed in a tabular setting and do not yet extend to a continuous setting where function approximation could be leveraged to allow the low-level agent to generalise over sub-goals and to consider setting beyond the tabular case described in this article. Overcoming those limitations is an interesting direction for future work.
AR was supported by an EPSRC CASE studentship supported by Shell, and AAF was supported by a UKRI Turing AI Fellowship (EP/V025449/1).
[1] A. A. Faisal A. Robert, C. Pike-Burke. Code and resources for running Hierarchical Stationary Q-learning: doi.org/10.6084/m9.figshare.24291172.
[2] Ekaterina Abramova, Luke Dickens, Daniel Kuhn, and Aldo Faisal. Hierarchical, heterogeneous control of non-linear dynamical systems using reinforcement learning. In European Workshop On Reinforcement Learning, at ICML, volume 2012, 2012.
[3] Peter Auer and Ronald Ortner. Online regret bounds for a new reinforcement learning algorithm. In 1st Austrian Cognitive Vision Workshop, pages 35-42. Österr. Computer-Ges., 2005.
[4] Benjamin Beyret, Ali Shafti, and A Aldo Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5014-5019. IEEE, 2019.
[5] Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International Conference on Machine Learning, pages 316-324. PMLR, 2014.
[6] Christoph Dann. Strategic Exploration in Reinforcement Learning-New Algorithms and Learning Guarantees. PhD thesis, Google, 2019.
[7] Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning. Advances in Neural Information Processing Systems, 28, 2015.
[8] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[9] Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303, 2000.
[10] Thomas G Dietterich et al. The maxq method for hierarchical reinforcement learning. In International Conference on Machine Learning, volume 98, pages 118-126, 1998.
[11] Ronan Fruit and Alessandro Lazaric. Exploration-exploitation in mdps with options. In Artificial intelligence and statistics, pages 576-584. PMLR, 2017.
[12] Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Emma Brunskill. Regret minimization in mdps with options without prior knowledge. Advances in Neural Information Processing Systems, 30, 2017.
[13] Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 1851-1860. PMLR, 2018.
[14] Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172-221, 2022.
[15] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in Neural Information Processing Systems, 31, 2018.
[16] Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In AAMAS (1), pages 299-306, 2008.
[17] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948, 2017.