SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arnaud Robert
Brain & Behaviour Lab
Dept. of Computing
a. robert20@imperial.ac.uk
Dept. of Mathematics
A. Aldo Faisal
Depts. of Computing & Bioengineering
Chair in Digital Health & Data Science
University of Bayreuth, Germany
a.faisal@imperial.ac.uk
Hierarchical Reinforcement Learning (HRL) algorithms can perform planning at multiple levels of abstraction. Empirical results have shown that state or temporal abstractions might significantly improve the sample efficiency of algorithms. Yet, we still do not have a complete understanding of the basis of those efficiency gains, nor any theoretically-grounded design rules. In this paper, we derive a lower bound on the sample complexity for the considered class of goal-conditioned HRL algorithms. The proposed lower bound empowers us to quantify the benefits of hierarchical decomposition and leads to the design of a simple Q-learning-type algorithm that leverages hierarchical decompositions. We empirically validate our theoretical findings by investigating the sample complexity of the proposed hierarchical algorithm on a spectrum of tasks (hierarchical  $n$ -rooms, Gymnasium's Taxi). The hierarchical  $n$ -rooms tasks were designed to allow us to dial their complexity over multiple orders of magnitude. Our theory and algorithmic findings provide a step towards answering the foundational question of quantifying the improvement hierarchical decomposition offers over monolithic solutions in reinforcement learning.
Hierarchical Reinforcement Learning (HRL) [27, 8, 9, 4] leverages the hierarchical decomposition of a problem to build algorithms that are more sample efficient. While there is significant empirical evidence that hierarchical implementations can drastically improve the sample efficiency of Reinforcement Learning (RL) algorithms [20, 21, 29, 8], there are also cases where temporal abstraction worsens the empirical sample complexity [16]. Therefore, a natural question to ask is: when does HRL lead to improved sample complexity, and how much of an improvement can it provide?
Theoretical work on sample-complexity bound in Machine Learning has been integral to the development of the field. Moreover, theoretical results (e.g. [7, 18, 3, 15, 26]) often uncover interesting principles useful for improving algorithm design. For example, the Q-learning algorithm analysed in [15] improved our understanding of exploration strategies in model-free RL and the policy gradient
Figure 1: The left block diagram depicts the interactions between the different components of our goal-conditioned hierarchical agent. The diagram suggests that the agent is composed of a low-level policy and a high-level policy that collaborate in order to solve a task. The high-level policy  $\pi^h$  observes the pair  $(s_h, r_h)$ , which denotes the high-level state and reward. It then sends a sub-goal  $g_{sub}$  as an input to the low-level policy  $\pi^l$ .  $\pi^l$  observes the pair  $(s_l, r_l)$ , which encodes the low-level state  $s_l$  and the low-level reward  $r_l$ . To achieve the sub-goal  $g_{sub}$ , the low-level policy  $\pi^l$  interacts with the environment through primitive actions  $a$ . The right diagram illustrates the decomposition of the original MDP  $\mathcal{M}_O$  into the low-level MDP  $\mathcal{M}_l$  and the high-level MDP  $\mathcal{M}_h$ . A detailed description of this decomposition is given in Sec. 2.2
theorem [26] gave birth to a wide range of new RL methods. In contrast, there are few theoretical results in hierarchical RL and many key studies are empirical, e.g. hierarchies of states [8, 10], time [23], or action [28, 22, 2].
To address this gap in the literature, we consider a tabular version of the goal-based approach to HRL [20, 4], and we analyze the induced MDP decomposition to derive a lower bound on the sample complexity of this specific HRL framework. This lower bound allows us to understand when a hierarchical decomposition is beneficial and motivates a new hierarchical Q-learning algorithm that can leverage the hierarchical structure to improve its sample efficiency. In the goal-based HRL framework, a high-level policy and a low-level policy are jointly learned to solve an overarching goal. In such a goal-hierarchical RL system, the high-level policy chooses a sub-goal for the low-level policy, which in turn executes primitive actions to solve the sub-goal (Fig. 1, left diagram). This natural way to break down tasks is universal (i.e., it can be applied to a wide range of tasks) and it induces a decomposition of the original MDP into two sub-MDPs (detailed in Sec. 2.2).
This paper improves our understanding of HRL through the following contributions:
- We provide a lower bound on the sample complexity associated with the hierarchical decomposition (see Sec. 3). This lower bound allows practitioners to quantify the efficiency gain they might obtain from decomposing their task.
- We propose a simple, yet novel, Q-learning-type algorithm for goal-hierarchical RL, inspired by the type of decomposition considered (see Sec. 4).
- We empirically validate the theoretical findings using a synthetic task with hierarchical properties that can be scaled in complexity (see Sec. 5). This evidence confirms that the derived bound is able to successfully identify instances where a hierarchical decomposition could be beneficial (see Sec. 5).
We consider a system where an agent needs to make a sequence of decisions in an uncertain environment to maximise the sum of cumulated rewards. Such problems are modelled as Markov Decision Processes (MDPs) and can be solved by RL algorithms [25]. When a task is too complex, the number of interactions required to learn a near-optimal policy becomes prohibitive. The task complexity typically depends on the difficulty of temporal credit assignment (which is directly related to the episode length) and the size of the state and action spaces [19]. To address this complexity, HRL leverages temporal abstractions [27] and state abstractions [8] to improve sample efficiency when learning an optimal policy. There exists a wide range of HRL frameworks; see [14] for a survey. In this paper, we focus on the goal-conditioned HRL framework [20, 4]. Of the other HRL frameworks, only the options framework [27] and the resulting semi-Markov Decision Process [12, 30, 5, 11] benefit from some theoretical understanding. However, in practice, the goal-conditioned hierarchical
framework presented in Fig. 1 is often preferred. Unlike the options framework, the goal-conditioned HRL framework requires no prior knowledge about the task [14], and its ability to generalize over the goal space when function approximation is used leads to significant performance gains in benchmark tasks [29, 20, 13]. Existing theoretical work [30, 12, 11] on the options framework does not consider the case where all the hierarchy levels are jointly learned. The regret analysis proposed by [30] focuses on the benefit of leveraging repeating sub-structures in hierarchical MDPs. Other regret analyses [12, 11] highlight the efficiency gain of learning with temporally extended actions (such as options). However, they always assume the set of options is known, and the intra-option policies are not learned. Options are composed of an intra-option policy, which governs the agent's behaviour while the corresponding option is executed, making intra-option policies very similar to the low-level policy we consider. However, the goal-conditioned HRL setting considered in this article quantifies the benefits of state abstraction, action abstraction, and time abstraction while jointly learning all levels of the hierarchy (low-level and high-level policy) through interaction with the environment. A detailed description of the option framework and its connection to HRL is available in Appendix B.
For the remainder of this section, we define episodic finite-horizon MDPs and the hierarchical decomposition we consider.
2.1 Episodic Finite-Horizon Markov Decision Process
An episodic finite-horizon MDP is defined by the following tuple:  $\mathcal{M} = \langle S, \mathcal{A}, r, P, p_0, H \rangle$ . Where  $S$  is a finite state space of size  $|S|$  and  $\mathcal{A}$  is a finite action space of size  $|\mathcal{A}|$ . The goal of the task is encoded in a terminal state  $g \in S$ . We assume the reward function  $r(s, g) \in [-a, b]$  (for  $a, b \geq 0$ ) is known  $\forall s \in S, g \in S$ , the reward function penalises each step with a negative reward of at most  $-a$  and reward the completion of the task with a positive reward of  $b$ . The initial state distribution  $p_0$  is a distribution over states that is used to determine in which state an episode starts. The learner interacts with the MDP in episodes of at most  $H$  time steps. The episode's starting state  $s_0 \sim p_0$  is drawn from the initial state distribution. In each time step  $t = 0, \dots, H - 1$ , the learner observes a state  $s_t$  and chooses an action  $a_t$ . Given a state action pair  $(s_t, a_t)$  the next state  $s_{t + 1} \sim P(\cdot | s_t, a_t)$  is drawn from the transition kernel. Eventually, the episode ends because the agent reaches the terminal state or has interacted with the environment for  $H$  time-steps.
The agent's objective is to select actions that maximize the expected return throughout an episode. We typically assume actions are chosen according to a policy,  $a_{t} \sim \pi(s_{t})$ , where  $\pi$  is a function that maps each state and time step pair to a distribution over actions  $\pi: S \times [H - 1] \to \Delta_{A}$ , and  $\Delta_{A}$  is the set of all probability distributions over  $A$  and  $[H]$  is the set of natural numbers up to  $H$ . The agent aims to select a policy  $\pi$  to maximize the sum of expected rewards,  $\mathbb{E}[\sum_{t=1}^{H} r_{t} | a_{t} \sim \pi(s_{t})]$ , where the expectation is over the initial state distribution, the policy and the stochastic transitions. Note that it is usually the case for finite-horizon MDPs that the policy also depends on the current time step. However, to simplify notation, we do not make this relation explicit.
For a given policy  $\pi$ , we define the value function,  $V_{\tau}^{\pi}(s)$ , and the Q-function,  $Q_{\tau}^{\pi}(s,a)$ , at time step  $\tau \in [H - 1]$  as follows:
$$
V _ {\tau} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {t = \tau} ^ {H - 1} r _ {t} | s _ {\tau} = s, a _ {\tau : H - 1} \sim \pi \right], \quad Q _ {\tau} ^ {\pi} (s, a) = \mathbb {E} \left[ \sum_ {t = \tau} ^ {H - 1} r _ {t} | s _ {\tau} = s, a _ {\tau} = a, a _ {\tau + 1: H - 1} \sim \pi \right]
$$
where  $s \in S$  denotes the state,  $a \in \mathcal{A}$  is the action and the notation  $a_{\tau;H-1} \sim \pi$  is used to specify that actions between time step  $\tau$  and time step  $H-1$  were selected using  $\pi$ . The optimal policy  $\pi^*$  is the policy with the highest value function for every time step and every state,  $V_{\tau}^{\pi^*}(s) = V_{\tau}^*(s) = \max_{\pi} V_{\tau}^*(s) \forall \tau \in [H-1], \forall s \in S$ . There is always a deterministic Markov policy that maximizes the total expected reward in a finite-horizon MDP [24].
In this article, we assess the quality of a policy by its expected value at the beginning of an episode. To lighten the notation, we define  $V^{\pi} = \mathbb{E}_{s_0\sim p_0}[V_0^\pi (s)]$  to be the expected value from the beginning of an episode where the expectation is taken over initial states.
2.2 Episodic Finite-Horizon Hierarchical MDP
For a given episodic finite-horizon MDP  $\mathcal{M}_o$ , we assume it can be hierarchically decomposed into a pair of MDPs  $(\mathcal{M}_l, \mathcal{M}_h)$  as illustrated on right diagram of Fig. 1. To avoid ambiguity, we use
the following notation: the subscript  $o$  denotes the original MDP, while subscripts  $l$  and  $h$  denote low-level and high-level MDPs, respectively.
The low-level and high-level MDPs consist of the following tuples  $\mathcal{M}_l = \langle \mathcal{S}_l \times \mathcal{A}_h, \mathcal{A}, r_l, P_l, p_{0,l}, H_l \rangle$  and  $\mathcal{M}_h = \langle \mathcal{S}_h, \mathcal{A}_h, r_h, P_h, p_{0,h}, H_h \rangle$ , respectively. To be a valid hierarchical decomposition, we require that these MDPs satisfy the following set of conditions:
Action space: The low-level action space consists of the set of primitive actions that the agent can use to interact with the environment. It is equivalent to the original MDP action space  $\mathcal{A}$ . The high-level action space  $\mathcal{A}_h$  is the set of the sub-goals the high-level agent can instruct to the low-level agent. We assume that the set of sub-goals encoded in  $\mathcal{A}_h$  is sufficient to solve the task for any state. Note that the set of available actions  $\mathcal{A}_h$  depends on the current high-level state  $s_h$ . To simplify our notation, we do not make this relationship explicit.
State spaces: The low-level state  $s_l$  and the high-level state  $s_h$  contain all necessary information to reconstruct the corresponding state,  $s$ , in the original MDP. States  $s \in S \subset \mathbb{R}^d$  are usually described as multi-dimensional vectors, where each dimension encodes a specific characteristic. For example, a state description can be factored in a tuple  $(s_l, s_h) \in S_l \times S_h$  with a part of the state description that belongs to the low-level MDP and another part to the high-level MDP. Hence, in this work, we consider that any state  $s \in S_o$  can be represented by a tuple  $(s_l, s_h) \in S_l \times S_h$ . Additionally, since the low-level policy is goal-conditioned, its state space also contains the goal description leading to the following state space for the low-level MDP:  $S_l \times \mathcal{A}_h$ , a complete low-level state consists of the concatenation of the low-level state description  $s_l$  and the sub-goal description  $a_h$ .
Initial state distribution: The high-level initial state distribution  $p_{0,h}$  is a restriction of the original state distribution  $p_0$  on  $S_h$ . The low-level initial state distribution  $p_{0,l}(\cdot |s_{h,0})$  is conditioned on the initial high-level state  $s_{h,0}$  and spans the low-level space, ensuring that  $p_0(s) = p_{0,h}(s_h)p_{0,l}(s_l|s_h)$ , where  $s_l$  and  $s_h$  are the decomposition of  $s$ .
Transition functions: The low-level transition function  $P_{l}$  is the restriction of  $P$  on  $S_{l} \times \mathcal{A}_{h}$ . One challenge in HRL is that the high-level transition function,  $P_{h}$ , depends on the low-level policy since the quality of the low-level policy influences the likelihood of reaching a sub-goal state. The high-level transition probability  $P_{h}(s_{h}^{\prime}|s_{h},a_{h},\pi_{l})$  is the probability that the agent transitions to  $s_h^\prime$  given the current high-level state  $s_h$ , the sub-goal  $a_{h}$  and low level policy  $\pi_l$ . Since  $P_{h}$  depends on the low-level policy, it is non-stationary, making the learning task more challenging.
Reward functions: Since the terminal states for the original MDP belong to  $S$  and the sub-goals for the low-level MDP lie in  $S_{l}$  the low-level reward function can be obtained from the original reward function,  $r_{l}(s_{l},g_{sub}) = 2r(s,g)$ , where  $s$  and  $g$  are the reconstruction of the low-level state and the sub-goal in the original MDP, using the current high-level state. The high-level reward function is the sum of rewards obtained by the low level during the sub-episode, where the high-level action plays the role of a sub-goal:  $r_{h}(s_{h},a_{h}) = \sum_{t = 1}^{H_{l}}r_{l}(s_{l,t},a_{h})$ .
Horizons: The original MDP allows an episode to last at most  $H$  steps. Consequently, the horizons of the high-level,  $H_{h}$ , and low-level,  $H_{l}$ , MDPs must satisfy the following equality  $H = H_{h}H_{l}$ .
Note that we can always find a decomposition that satisfies these assumptions; a naive way to decompose any MDP would be to consider a high-level agent whose only action encodes the end goal of the task and a low-level with complete state information (i.e. it does not use state abstraction). While this decomposition is valid, it is not necessarily beneficial. Here, our goal is to identify when a given decomposition is useful, specifically in terms of improvements in the sample efficiency.
We denote by  $\pi_{l}$  a policy interacting with the low-level MDP  $\mathcal{M}_l$ , and  $\pi_h$  a policy interacting with the high-level MDP  $\mathcal{M}_h$ . In goal-conditioned HRL, the low-level policy maps a low-level state and sub-goal pair to an action:  $\pi_l: S_l \times \mathcal{A}_h \to \mathcal{A}_l$  and the high-level policy maps a high-level state to a high-level action:  $\pi_h: S_h \to \mathcal{A}_h$ . Each policy can be evaluated using the corresponding high and low-level value functions  $V_{l}^{\pi_{l}}$  and  $V_{h}^{\pi_{h}}$ . Similar to the non-hierarchical case, we can define optimal high-level and low-level policies as  $\pi_l^* = \operatorname{argmax}_{\pi_l} V_{l}^{\pi_l}$  for the low-level policy and  $\pi_l^* = \operatorname{argmax}_{\pi_h} V_{h}^{\pi_h}$  for the high-level policy. Moreover, as shown below, every pair of policies  $(\pi_l, \pi_h)$  can be combined to produce a policy  $\pi$  that interacts with the original MDP  $\mathcal{M}_o$ .
Definition 2.1. A hierarchical policy consists of a pair  $(\pi_l, \pi_h)$  that can be mapped to a policy  $\pi$  in the original MDP  $\mathcal{M}_o$  as follows:
$$
\pi (a | s) = \pi (a | s _ {l}, s _ {h}) = \sum_ {a _ {h} \in \mathcal {A} _ {h}} \pi_ {h} \left(a _ {h} \mid s _ {h}\right) \pi_ {l} \left(a \mid a _ {h}, s _ {l}\right). \tag {1}
$$
The optimal hierarchical policy is obtained when merging  $(\pi_l^*, \pi_h^*)$ . It is important to note that not all policies  $\pi$  in the original MDP have a corresponding decomposition  $(\pi_l, \pi_h)$ , and in particular, there is no guarantee that the optimal policy in the original MDP can be decomposed.
We aim to understand when a hierarchical decomposition of the MDP allows us to learn a near-optimal policy faster. Therefore, we are interested in evaluating the performance of the combination of  $\pi_{l}$  and  $\pi_{h}$  while they interact with the original MDP  $\mathcal{M}_o$ . To convey the fact that we are evaluating a hierarchical policy in the original MDP, we use the following notation: given a pair of policies  $(\pi_l,\pi_h)$  and their associated policy in the original MDP,  $\pi$ , the value function of the hierarchical policy is denoted by  $V_{o}^{\pi_{l},\pi_{h}} = \mathbb{E}_{s_{0}\sim p_{0}}[V_{o,0}^{\pi}(s_{0})]$ , where the subscript  $o$  is a reminder that we are evaluating a policy on the original MDP  $\mathcal{M}_o$ .
When learning in a decomposed MDP, the learner has to learn two policies, the high-level policy,  $\pi_h$ , and the low-level policy,  $\pi_l$ . This is done in an episodic setting where an episode unfolds as follows. Firstly, the learner observes the initial state and uses the high-level policy to find the most appropriate sub-goal. For the next  $H_l$  time steps, the low-level policy attempts to solve the sub-goal. The low-level agent updates its policy at the end of each low-level step. Once the  $H_l$  time steps are over or if the sub-goal has been reached, the high-level agent observes a new high-level state and can finally perform an update to its policy. The high-level agent instructs a new sub-goal if the overall task is not completed. These interactions are repeated until the task is completed or the horizon  $H$  is reached. We can now think of HRL as two agents interacting with the environment. Often, each agent will try to find the policy that maximizes their value function,  $\max_{\pi_l} V_l^{\pi_l}$  and  $\max_{\pi_h} V_h^{\pi_h}$ .
2.3 Probably-Approximately Correct RL
We aim to find, in as few episodes as possible, a pair of policies  $(\pi_l,\pi_h)$  with a near-optimal value. To formalize this, we introduce the Probably-Approximately Correct (PAC) RL notion. We denote by  $\Delta_{k}$  the sub-optimality gap, that is the difference between the optimal (non-hierarchical) policy  $\pi^{*}$  and the current hierarchical policy  $(\pi_l^k,\pi_h^k)$ :  $\Delta_k\coloneqq V_o^* -V_o^{\pi_l^k,\pi_h^k}$ . Note that both policies are evaluated on the original MDP  $\mathcal{M}_o$ . The PAC guarantee in this paper follows the definition in [6].
Definition 2.2. An algorithm satisfies a PAC bound  $N$  if, for a given input  $\epsilon, \delta > 0$ , it satisfies the following condition for any episodic fixed-horizon MDP: with probability at least  $1 - \delta$ , the algorithm plays policies that are at least  $\epsilon$ -optimal after at most  $N$  episodes. That is, with probability at least  $1 - \delta$ ,  $\max \{k \in \mathbb{N} : \Delta_k > \epsilon\} \leq N$ , where  $N$  is a polynomial that can depend on the properties of the problem instance.
In Section 3, we will bound the sample complexity of HRL algorithms. In this context, the sample complexity refers to the number of episodes,  $N$ , in the original MDP, during which the algorithm may not follow a policy that is at least  $\epsilon$ -optimal with probability at least  $1 - \delta$ .
2.4 Running Example
We consider the following companion example. The original MDP describes the task of solving a maze in a grid-world environment. The state consists of a tuple  $(R, C)$  that indicates in which room,  $R$ , and which cell within that room,  $C$ , the agent is currently in. The reward function incurs a small cost,  $-a$ , at each time step unless the agent reaches the absorbing goal state. Once the goal state is reached, the agent stops receiving penalties and receives a reward of 0 for all the remaining time steps. Mathematically,  $r(s) = -a \mathbb{1}\{s \neq g\}$  where  $g \in S$  is the goal state, and  $\mathbb{1}$  is the indicator function.
We can decompose this MDP as follows. The high-level MDP describes a similar maze, but instead of moving from cell to cell, the agent moves from room to room, so the state is just the current room. The high-level agent aims to find the room sequence that leads to the goal. Hence, at each (high-level) time step, it indicates the most valuable exit the low-level agent should take from the room. As specified in Section 2.2, the high-level reward for a sub-goal is the sum of the rewards accumulated by the low-level agent during that sub-episode. The low-level agent is myopic to other rooms - it only sees the current room and the exit it has to reach, and it receives a penalty of  $-2a$  for each action it takes unless it reaches the sub-goal, in which case it does not receive any penalty. Hence, if  $g_{sub}$  is the sub-goal, it receives reward  $r(s) = -2a\mathbb{1}\{s \neq g_{sub}\}$ .
We will return to this example throughout the paper, but it should be noted that the framework we consider is general enough to be applied to a wide range of tasks. One such example is robotics, where
the low-level agent would be tasked with controlling the joints of the robot to produce movements selected by the high-level policy, whose goal is to perform tasks that require a sequence of distinct movements (i.e. navigational tasks, manipulation tasks or a combination of both).
3 Lower Bound on the Sample Complexity of HRL
It has been proven in [7] that, for any RL algorithm, the number of sample episodes necessary to obtain an  $(\epsilon, \delta)$ -accurate policy (in the original MDP) is lower bounded by:
$$
\mathbb {E} [ N ] = \Omega \left(\frac {| \mathcal {S} | | \mathcal {A} | H ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c}\right)\right), \tag {2}
$$