Title: Sample Complexity of Goal-Conditioned Hierarchical Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Hierarchical Reinforcement Learning

Sample Complexity
Abstract: Hierarchical Reinforcement Learning (HRL) algorithms can perform planning at multiple levels of abstraction. Empirical results have shown that state or temporal abstractions might significantly improve the sample efficiency of algorithms. Yet, we still do not have a complete understanding of the basis of those efficiency gains nor any theoretically grounded design rules. In this paper, we derive a lower bound on the sample complexity for the considered class of goal-conditioned HRL algorithms. The proposed lower bound empowers us to quantify the benefits of hierarchical decomposition and leads to the design of a simple Q-learning-type algorithm that leverages hierarchical decompositions. We empirically validate our theoretical findings by investigating the sample complexity of the proposed hierarchical algorithm on a spectrum of tasks (hierarchical $n$-rooms, Gymnasium's Taxi). The hierarchical $n$-rooms tasks were designed to allow us to dial their complexity over multiple orders of magnitude. Our theory and algorithmic findings provide a step towards answering the foundational question of quantifying the improvement hierarchical decomposition offers over monolithic solutions in reinforcement learning.

Sample Complexity of Goal-Conditioned Hierarchical Reinforcement Learning

Abstract

Hierarchical Reinforcement Learning (HRL) algorithms can perform planning at multiple levels of abstraction. Empirical results have shown that state or temporal abstractions might significantly improve the sample efficiency of algorithms. Yet, we still do not have a complete understanding of the basis of those efficiency gains, nor any theoretically-grounded design rules. In this paper, we derive a lower bound on the sample complexity for the considered class of goal-conditioned HRL algorithms. The proposed lower bound empowers us to quantify the benefits of hierarchical decomposition and leads to the design of a simple Q-learning-type algorithm that leverages hierarchical decompositions. We empirically validate our theoretical findings by investigating the sample complexity of the proposed hierarchical algorithm on a spectrum of tasks (hierarchical n -rooms, Gymnasium's Taxi). The hierarchical n -rooms tasks were designed to allow us to dial their complexity over multiple orders of magnitude. Our theory and algorithmic findings provide a step towards answering the foundational question of quantifying the improvement hierarchical decomposition offers over monolithic solutions in reinforcement learning.

1 Motivation

Hierarchical Reinforcement Learning (HRL) [27, 8, 9, 4] leverages the hierarchical decomposition of a problem to build algorithms that are more sample efficient. While there is significant empirical evidence that hierarchical implementations can drastically improve the sample efficiency of Reinforcement Learning (RL) algorithms [20, 21, 29, 8], there are also cases where temporal abstraction worsens the empirical sample complexity [16]. Therefore, a natural question to ask is: when does HRL lead to improved sample complexity, and how much of an improvement can it provide?

Theoretical work on sample-complexity bound in Machine Learning has been integral to the development of the field. Moreover, theoretical results (e.g. [7, 18, 3, 15, 26]) often uncover interesting principles useful for improving algorithm design. For example, the Q-learning algorithm analysed in [15] improved our understanding of exploration strategies in model-free RL and the policy gradient

Environment

high-level policy

π

h

low-level policy

π

l

(

s

h

,

r

h

)

g

sub

a

(

s

l

, r

l

)

M

O

=

⟨S

,

A

, P, r, p

0

, H

⟩

M

h

=

⟨S

h

,

A

h

, P

h

, r

h

, p

0

,h

, H

h

⟩

M

l

=

⟨S

l

×A

h

,

A

, P

l

, r

l

, p

0

,l

, H

l

⟩

theorem [26] gave birth to a wide range of new RL methods. In contrast, there are few theoretical results in hierarchical RL and many key studies are empirical, e.g. hierarchies of states [8, 10], time [23], or action [28, 22, 2].

To address this gap in the literature, we consider a tabular version of the goal-based approach to HRL [20, 4], and we analyze the induced MDP decomposition to derive a lower bound on the sample complexity of this specific HRL framework. This lower bound allows us to understand when a hierarchical decomposition is beneficial and motivates a new hierarchical Q-learning algorithm that can leverage the hierarchical structure to improve its sample efficiency. In the goal-based HRL framework, a high-level policy and a low-level policy are jointly learned to solve an overarching goal. In such a goal-hierarchical RL system, the high-level policy chooses a sub-goal for the low-level policy, which in turn executes primitive actions to solve the sub-goal (Fig. 1, left diagram). This natural way to break down tasks is universal (i.e., it can be applied to a wide range of tasks) and it induces a decomposition of the original MDP into two sub-MDPs (detailed in Sec. 2.2).

This paper improves our understanding of HRL through the following contributions:

We provide a lower bound on the sample complexity associated with the hierarchical decomposition (see Sec. 3). This lower bound allows practitioners to quantify the efficiency gain they might obtain from decomposing their task.

We propose a simple, yet novel, Q-learning-type algorithm for goal-hierarchical RL, inspired by the type of decomposition considered (see Sec. 4).

We empirically validate the theoretical findings using a synthetic task with hierarchical properties that can be scaled in complexity (see Sec. 5). This evidence confirms that the derived bound is able to successfully identify instances where a hierarchical decomposition could be beneficial (see Sec. 5).

2 Background

We consider a system where an agent needs to make a sequence of decisions in an uncertain environment to maximise the sum of cumulated rewards. Such problems are modelled as Markov Decision Processes (MDPs) and can be solved by RL algorithms [25]. When a task is too complex, the number of interactions required to learn a near-optimal policy becomes prohibitive. The task complexity typically depends on the difficulty of temporal credit assignment (which is directly related to the episode length) and the size of the state and action spaces [19]. To address this complexity, HRL leverages temporal abstractions [27] and state abstractions [8] to improve sample efficiency when learning an optimal policy. There exists a wide range of HRL frameworks; see [14] for a survey. In this paper, we focus on the goal-conditioned HRL framework [20, 4]. Of the other HRL frameworks, only the options framework [27] and the resulting semi-Markov Decision Process [12, 30, 5, 11] benefit from some theoretical understanding. However, in practice, the goal-conditioned hierarchical

framework presented in Fig. 1 is often preferred. Unlike the options framework, the goal-conditioned HRL framework requires no prior knowledge about the task [14], and its ability to generalize over the goal space when function approximation is used leads to significant performance gains in benchmark tasks [29, 20, 13]. Existing theoretical work [30, 12, 11] on the options framework does not consider the case where all the hierarchy levels are jointly learned. The regret analysis proposed by [30] focuses on the benefit of leveraging repeating sub-structures in hierarchical MDPs. Other regret analyses [12, 11] highlight the efficiency gain of learning with temporally extended actions (such as options). However, they always assume the set of options is known, and the intra-option policies are not learned. Options are composed of an intra-option policy, which governs the agent's behaviour while the corresponding option is executed, making intra-option policies very similar to the low-level policy we consider. However, the goal-conditioned HRL setting considered in this article quantifies the benefits of state abstraction, action abstraction, and time abstraction while jointly learning all levels of the hierarchy (low-level and high-level policy) through interaction with the environment. A detailed description of the option framework and its connection to HRL is available in Appendix B.

For the remainder of this section, we define episodic finite-horizon MDPs and the hierarchical decomposition we consider.

2.1 Episodic Finite-Horizon Markov Decision Process

An episodic finite-horizon MDP is defined by the following tuple: M = ⟨S , A , r, P, p 0 , H ⟩ . Where S is a finite state space of size |S| and A is a finite action space of size |A| . The goal of the task is encoded in a terminal state g ∈ S . We assume the reward function r ( s, g ) ∈ [ -a, b ] (for a, b ≥ 0 ) is known ∀ s ∈ S , g ∈ S , the reward function penalises each step with a negative reward of at most -a and reward the completion of the task with a positive reward of b . The initial state distribution p 0 is a distribution over states that is used to determine in which state an episode starts. The learner interacts with the MDP in episodes of at most H time steps. The episode's starting state s 0 ∼ p 0 is drawn from the initial state distribution. In each time step t = 0 , . . . , H -1 , the learner observes a state s t and chooses an action a t . Given a state action pair ( s t , a t ) the next state s t +1 ∼ P ( ·| s t , a t ) is drawn from the transition kernel. Eventually, the episode ends because the agent reaches the terminal state or has interacted with the environment for H time-steps.

The agent's objective is to select actions that maximize the expected return throughout an episode. We typically assume actions are chosen according to a policy, a t ∼ π ( s t ) , where π is a function that maps each state and time step pair to a distribution over actions π : S × [ H -1] → ∆ A , and ∆ A is the set of all probability distributions over A and [ H ] is the set of natural numbers up to H . The agent aims to select a policy π to maximize the sum of expected rewards, E [ ∑ H t =1 r t | a t ∼ π ( s t )] , where the expectation is over the initial state distribution, the policy and the stochastic transitions. Note that it is usually the case for finite-horizon MDPs that the policy also depends on the current time step. However, to simplify notation, we do not make this relation explicit.

For a given policy π , we define the value function, V π τ ( s ) , and the Q-function, Q π τ ( s, a ) , at time step τ ∈ [ H -1] as follows:

where s ∈ S denotes the state, a ∈ A is the action and the notation a τ : H -1 ∼ π is used to specify that actions between time step τ and time step H -1 were selected using π . The optimal policy π ∗ is the policy with the highest value function for every time step and every state, V π ∗ τ ( s ) = V ∗ τ ( s ) = max π V π τ ( s ) ∀ τ ∈ [ H -1] , ∀ s ∈ S . There is always a deterministic Markov policy that maximizes the total expected reward in a finite-horizon MDP [24].

In this article, we assess the quality of a policy by its expected value at the beginning of an episode. To lighten the notation, we define V π = E s 0 ∼ p 0 [ V π 0 ( s )] to be the expected value from the beginning of an episode where the expectation is taken over initial states.

2.2 Episodic Finite-Horizon Hierarchical MDP

For a given episodic finite-horizon MDP M o , we assume it can be hierarchically decomposed into a pair of MDPs ( M l , M h ) as illustrated on right diagram of Fig. 1. To avoid ambiguity, we use

the following notation: the subscript o denotes the original MDP, while subscripts l and h denote low-level and high-level MDPs, respectively.

The low-level and high-level MDPs consist of the following tuples M l = ⟨S l ×A h , A , r l , P l , p 0 ,l , H l ⟩ and M h = ⟨S h , A h , r h , P h , p 0 ,h , H h ⟩ , respectively. To be a valid hierarchical decomposition, we require that these MDPs satisfy the following set of conditions:

Action space: The low-level action space consists of the set of primitive actions that the agent can use to interact with the environment. It is equivalent to the original MDP action space A . The high-level action space A h is the set of the sub-goals the high-level agent can instruct to the low-level agent. We assume that the set of sub-goals encoded in A h is sufficient to solve the task for any state. Note that the set of available actions A h depends on the current high-level state s h . To simplify our notation, we do not make this relationship explicit.

State spaces: The low-level state s l and the high-level state s h contain all necessary information to reconstruct the corresponding state, s , in the original MDP. States s ∈ S ⊂ R d are usually described as multi-dimensional vectors, where each dimension encodes a specific characteristic. For example, a state description can be factored in a tuple ( s l , s h ) ∈ S l ×S h with a part of the state description that belongs to the low-level MDP and another part to the high-level MDP. Hence, in this work, we consider that any state s ∈ S o can be represented by a tuple ( s l , s h ) ∈ S l ×S h . Additionally, since the low-level policy is goal-conditioned, its state space also contains the goal description leading to the following state space for the low-level MDP: S l ×A h , a complete low-level state consists of the concatenation of the low-level state description s l and the sub-goal description a h .

Initial state distribution: The high-level initial state distribution p 0 ,h is a restriction of the original state distribution p 0 on S h . The low-level initial state distribution p 0 ,l ( ·| s h, 0 ) is conditioned on the initial high-level state s h, 0 and spans the low-level space, ensuring that p 0 ( s ) = p 0 ,h ( s h ) p 0 ,l ( s l | s h ) , where s l and s h are the decomposition of s .

Transition functions: The low-level transition function P l is the restriction of P on S l ×A h . One challenge in HRL is that the high-level transition function, P h , depends on the low-level policy since the quality of the low-level policy influences the likelihood of reaching a sub-goal state. The high-level transition probability P h ( s ′ h | s h , a h , π l ) is the probability that the agent transitions to s ′ h given the current high-level state s h , the sub-goal a h and low level policy π l . Since P h depends on the low-level policy, it is non-stationary, making the learning task more challenging.

Reward functions: Since the terminal states for the original MDP belong to S and the sub-goals for the low-level MDP lie in S l the low-level reward function can be obtained from the original reward function, r l ( s l , g sub ) = 2 r ( s, g ) , where s and g are the reconstruction of the low-level state and the sub-goal in the original MDP, using the current high-level state. The high-level reward function is the sum of rewards obtained by the low level during the sub-episode, where the high-level action plays the role of a sub-goal: r h ( s h , a h ) = ∑ H l t =1 r l ( s l,t , a h ) .

Horizons: The original MDP allows an episode to last at most H steps. Consequently, the horizons of the high-level, H h , and low-level, H l , MDPs must satisfy the following equality H = H h H l .

Note that we can always find a decomposition that satisfies these assumptions; a naive way to decompose any MDP would be to consider a high-level agent whose only action encodes the end goal of the task and a low-level with complete state information (i.e. it does not use state abstraction). While this decomposition is valid, it is not necessarily beneficial. Here, our goal is to identify when a given decomposition is useful, specifically in terms of improvements in the sample efficiency.

We denote by π l a policy interacting with the low-level MDP M l , and π h a policy interacting with the high-level MDP M h . In goal-conditioned HRL, the low-level policy maps a low-level state and sub-goal pair to an action: π l : S l × A h → A l and the high-level policy maps a high-level state to a high-level action: π h : S h →A h . Each policy can be evaluated using the corresponding high and low-level value functions V π l l and V π h h . Similar to the non-hierarchical case, we can define optimal high-level and low-level policies as π ∗ l = argmax π l V π l l for the low-level policy and π ∗ l = argmax π h V π h h for the high-level policy. Moreover, as shown below, every pair of policies ( π l , π h ) can be combined to produce a policy π that interacts with the original MDP M o .

Definition 2.1. A hierarchical policy consists of a pair ( π l , π h ) that can be mapped to a policy π in the original MDP M o as follows:

The optimal hierarchical policy is obtained when merging ( π ∗ l , π ∗ h ) . It is important to note that not all policies π in the original MDP have a corresponding decomposition ( π l , π h ) , and in particular, there is no guarantee that the optimal policy in the original MDP can be decomposed.

Weaim to understand when a hierarchical decomposition of the MDP allows us to learn a near-optimal policy faster. Therefore, we are interested in evaluating the performance of the combination of π l and π h while they interact with the original MDP M o . To convey the fact that we are evaluating a hierarchical policy in the original MDP, we use the following notation: given a pair of policies ( π l , π h ) and their associated policy in the original MDP, π , the value function of the hierarchical policy is denoted by V π l ,π h o = E s 0 ∼ p 0 [ V π o, 0 ( s 0 )] , where the subscript o is a reminder that we are evaluating a policy on the original MDP M o .

When learning in a decomposed MDP, the learner has to learn two policies, the high-level policy, π h , and the low-level policy, π l . This is done in an episodic setting where an episode unfolds as follows. Firstly, the learner observes the initial state and uses the high-level policy to find the most appropriate sub-goal. For the next H l time steps, the low-level policy attempts to solve the sub-goal. The low-level agent updates its policy at the end of each low-level step. Once the H l time steps are over or if the sub-goal has been reached, the high-level agent observes a new high-level state and can finally perform an update to its policy. The high-level agent instructs a new sub-goal if the overall task is not completed. These interactions are repeated until the task is completed or the horizon H is reached. We can now think of HRL as two agents interacting with the environment. Often, each agent will try to find the policy that maximizes their value function, max π l V π l l and max π h V π h h .

2.3 Probably-Approximately Correct RL

We aim to find, in as few episodes as possible, a pair of policies ( π l , π h ) with a near-optimal value. To formalize this, we introduce the Probably-Approximately Correct (PAC) RL notion. We denote by ∆ k the sub-optimality gap, that is the difference between the optimal (non-hierarchical) policy π ∗ and the current hierarchical policy ( π k l , π k h ) : ∆ k := V ∗ o -V π k l ,π k h o . Note that both policies are evaluated on the original MDP M o . The PAC guarantee in this paper follows the definition in [6].

Definition 2.2. An algorithm satisfies a PAC bound N if, for a given input ϵ, δ > 0 , it satisfies the following condition for any episodic fixed-horizon MDP: with probability at least 1 -δ , the algorithm plays policies that are at least ϵ -optimal after at most N episodes. That is, with probability at least 1 -δ , max { k ∈ N : ∆ k > ϵ } ≤ N, where N is a polynomial that can depend on the properties of the problem instance.

In Section 3, we will bound the sample complexity of HRL algorithms. In this context, the sample complexity refers to the number of episodes, N , in the original MDP, during which the algorithm may not follow a policy that is at least ϵ -optimal with probability at least 1 -δ .

2.4 Running Example

We consider the following companion example. The original MDP describes the task of solving a maze in a grid-world environment. The state consists of a tuple ( R,C ) that indicates in which room, R , and which cell within that room, C , the agent is currently in. The reward function incurs a small cost, -a , at each time step unless the agent reaches the absorbing goal state. Once the goal state is reached, the agent stops receiving penalties and receives a reward of 0 for all the remaining time steps. Mathematically, r ( s ) = -a { s = g } where g ∈ S is the goal state, and is the indicator function.

̸

̸

✶ We will return to this example throughout the paper, but it should be noted that the framework we consider is general enough to be applied to a wide range of tasks. One such example is robotics, where

✶ ✶ We can decompose this MDP as follows. The high-level MDP describes a similar maze, but instead of moving from cell to cell, the agent moves from room to room, so the state is just the current room. The high-level agent aims to find the room sequence that leads to the goal. Hence, at each (high-level) time step, it indicates the most valuable exit the low-level agent should take from the room. As specified in Section 2.2, the high-level reward for a sub-goal is the sum of the rewards accumulated by the low-level agent during that sub-episode. The low-level agent is myopic to other rooms - it only sees the current room and the exit it has to reach, and it receives a penalty of -2 a for each action it takes unless it reaches the sub-goal, in which case it does not receive any penalty. Hence, if g sub is the sub-goal, it receives reward r ( s ) = -2 a { s = g sub } .

the low-level agent would be tasked with controlling the joints of the robot to produce movements selected by the high-level policy, whose goal is to perform tasks that require a sequence of distinct movements (i.e. navigational tasks, manipulation tasks or a combination of both).

3 Lower Bound on the Sample Complexity of HRL

It has been proven in [7] that, for any RL algorithm, the number of sample episodes necessary to obtain an ( ϵ, δ ) -accurate policy (in the original MDP) is lower bounded by:

where c is a positive constant.

We now extend this result to hierarchical MDPs. Before doing so, it is essential to notice that even the best hierarchical policy (as constructed in Eq. (1)) might be sub-optimal. This is a direct consequence of the goal-conditioned architecture. If, while executing a sub-episode, it appears that another sub-goal becomes more valuable, the architecture proposed does not allow interruptions. The agent will first have to complete the current sub-episode before being able to adapt to the new circumstances. Let V π ∗ l ,π ∗ h o denote the value of the optimal hierarchical policy value function in the original MDP. Then, the sub-optimality gap is larger than the gap between the current policy pair and the optimal hierarchical policy ∆ k = V ∗ o -V π k l ,π k h o ≥ V π ∗ l ,π ∗ h o -V π k l ,π k h o . Therefore, if for some N , V π ∗ l ,π ∗ h o -V π k l ,π k h o ≥ ϵ for at least N episodes, it must also be the case that ∆ k ≥ ϵ for at least N episodes. Hence, N is a lower bound on the number of episodes where the algorithm must follow a sub-optimal policy.

In the following theorem, we lower bound the number of episodes required to learn a pair of policies ( π l , π h ) which are ϵ -accurate with respect to the optimal hierarchical policy ( π ∗ l , π ∗ h ) . By the above argument, this will also be a lower bound on the number of episodes necessary to learn an ϵ -accurate policy with respect to the optimal policy π ∗ .

Theorem 3.1. There exist positive constants c l , c h and δ 0 such that for every δ ∈ (0 , δ 0 ) and for every algorithm A that satisfies a PAC guarantee for ( ϵ, δ ) and outputs a deterministic policy, there is a fixed horizon MDP such that A must interact for

episodes, in the original MDP, until the policy is ( ϵ, δ ) -accurate.

The complete proof is in Appendix A.1. In the following, we highlight the main steps.

Sketch of the proof: An ϵ -accurate pair of policies must satisfy the following inequality, | V π ∗ l ,π ∗ h o -V π l ,π h o | ≤ ϵ . To find a lower bound on the number of episodes N before we obtain an ϵ -accurate pair of policies ( π l , π h ) we used the following steps:

(i) We decompose the objective using the triangle inequality, | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | + | V π ∗ l ,π h o -V π l ,π h o | ≤ ϵ .

(ii) We show that the number of samples required to guarantee | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | ≤ ϵ/ 2 is bounded by Ω ( |S h ||A h | H 2 h ϵ 2 ln ( 1 δ + c h ) )

(iii) We show that the number of samples required to guarantee | V π ∗ l ,π h o -V π l ,π h o | ≤ ϵ/ 2 is bounded by Ω ( |S l ||A H ||A| H 2 l ϵ 2 ln ( 1 δ + c l ) )

Combining these three steps gives us the result in Theorem 3.1; see A.1 for more details.

3.1 Interpretation of the Sample Complexity Bound:

By comparing this lower bound 1 to that in the original MDP, we can identify the problem characteristics that might lead to improved sample efficiency. In general, only one of the two MDP characteristics will dominate the overall sample complexity because of the max operator in the bound in Eq. 3. To maintain the max as small as possible, the complexity should be distributed between the low- and high-level MDP as evenly as possible. We discuss some of these key insights below:

State abstraction: Only one of the two-state space cardinalities will dominate the bound in Eq. 3. This suggests that an efficient decomposition tends to separate the original state space as evenly as possible between the two levels of the hierarchy. Another phenomenon at stake is the low-level re-usability. Due to the state abstraction, the low-level agent can re-use its learned policy in different states (i.e. different states s 1 , s 2 ∈ S whose low-level component s l are the same). We rewrite the |S|

lower bound 3 in terms of the re-usability index κ = |S l | .

Eq. 4 highlights that a large re-usability index improves the sample efficiency.

Temporal abstraction: Similarly, only one of the two-time horizons will dominate the bound, again suggesting a fair repartition of the load. The temporal abstraction (reducing H to H h and H l ) simplifies the credit assignment problem for the high-level and the low-level policies by giving denser feedback. The low-level agent is rewarded for completing sub-tasks that are significantly shorter than the original task, and the high-level trajectory consists of significantly fewer (high-level) steps than a trajectory in the original MDP.

High-level action space: This is the only term that appears on both sides of the max( · , · ) in Eq. 3. This highlights that both the high-level and the low-level benefit from a compact sub-goal representation.

It is interesting to note the contrast between the state space decomposition and the design of the high-level action space. To find efficient state decomposition, the amount of information available at each level must be distributed among each level of the hierarchy. In the case of the sub-goal space, it appears that both levels benefit from a compact representation.

The above discussion highlights properties of the hierarchical decomposition that could improve sample complexity. Note, however, that our bound also shows that a hierarchical decomposition does not always improve the sample efficiency. Indeed, there will be some settings where using a 'bad' hierarchical decomposition does not improve the sample complexity. Our bound can, therefore, provide a sanity check to determine whether a hierarchical decomposition could lead to an improved sample complexity. However, finding an algorithm that achieves this improved sample complexity can still be challenging. Nevertheless, the proposed Q-learning-based hierarchical algorithm empirically demonstrates the potential benefits of leveraging the considered decomposition. In Section 5, we consider several MDP decompositions and empirically validate that when our bound suggests the hierarchical decomposition is beneficial, our algorithm (see Sec. 4) leverages this to achieve lower sample complexity.

4 Stationary Hierarchical Q-Learning

Once we know that we are in an MDP where the hierarchical decomposition could lead to improved sample complexity, the next challenge is to design an algorithm to exploit this. This section proposes the Stationary Hierarchical Q-learning algorithm (SHQL) for this purpose. One of the most challenging aspects of jointly learning a pair of policies is the non-stationarity of the high-level transition dynamics, P h . It was briefly mentioned (in Sec. 2.2) that the high-level transition function, P h , is non-stationary since it depends on the low-level policy, π l with the next high-level state depending on whether π l managed to reach the sub-goal. To address this issue, we leverage the fact that the algorithm knows what a successful sub-episode is, i.e. it knows if the low-level agent managed to

+

steps (in thousands)

0.08

0.16

0.8

8

1.2

0.04

0.08

0.4

0.6

0.00

0

2000

0.00

0

2000

0.0

0

2000

0.0

0

2000

0

2000

episodes

episodes

episodes

episodes

episodes

Q-learning

SHQL

arrive at the desired sub-goal. In the case of off-policy methods, this is typically leveraged with hindsight correction [17]. However, in the considered on-policy setting, we propose that the algorithm only makes an update if the low-level agent behaves reasonably well (i.e. solving the sub-goal). In this way, the algorithm filters all bad examples from the training set, and the behaviour of P h is more stable. Note, however, that the reward function of the highlevel agent remains non-stationary. At first, sub-goals won't be solved optimally, incurring a small reward to the high-level agent. However, the associated reward will increase as the low-level agent learns to solve sub-goals more efficiently. As detailed in the function LowLevelUpdate in Algorithm 1, the low-level agent performs Q-learning updates on the observed low-level transitions and rewards. The high-level agent also performs Q-learning updates, but only on successful transitions, as specified at line 10 of Algorithm 1.

5 Experiments

We now empirically evaluate 2 the impact of the decomposition on various MDPs to validate the lower bound found in Section 3 and evaluate the performance of our proposed SHQL algorithm. To satisfy the assumption of hierarchical structure, the environments considered are a generalization of the four-room problem with an arbitrary number of rooms, called the n -rooms problem. The entire maze is built by arranging an arbitrary number of rooms on a grid. The high-level task would involve

Algorithm 1: Stationary Hierarchical Q-learning (SHQL)

learning the shortest sequence of rooms that lead the agent from the starting position (the top left room) to the goal room (the bottom right room). The low-level task is to learn how to navigate within each room and to reach the instructed hallway. To further modulate the task's difficulty (in addition to the maze size), we vary the room profiles used, as depicted in the rightmost plot of Fig. 4.

The set of MDPs generated by these environments are the following:

The original MDP: This is a standard grid-world MDP, where the state space indicates the cell where the agent is located, and the action space allows the agent to move one cell in any cardinal direction (North, South, East, West). To obtain stochastic environments, each action has a success probability of p success = 4 / 5 . In case of failure, the action will be chosen at random.

The high-level MDP: The high-level state space is restricted to the room where the agent is currently located, and the exact position of the agent within that room is abstracted away. The high-level actions instruct the low-level to reach one of the available hallways. Note that not all rooms have access to the four hallways.

The low-level MDP: The low-level agent only observes the agent's current location within a room and the goal instructed by the high-level agent (one of the reachable hallways). It then uses the primitive action space (the four cardinal directions) to reach the desired hallway. All required code to reproduce the experiments is made available online [1].

5.1 Identical Rooms

We first introduce the experimental setting in its simplest form. The environments considered in this subsection are mazes built by assembling identical rooms without obstacles (i.e. the top room profile in Fig. 4). Fig. 2 illustrates the empirical performance of our SHQL algorithm against Q-learning in the original MDP. As expected for simple mazes (e.g. with 4 or 16 rooms), the hierarchical decomposition does not provide much improvement. Still, as the problems grow more complex, the empirical evaluation suggests a significant improvement in sample efficiency. This is also confirmed by our bound (yellow curve on the rightmost plot of Fig. 4), which highlights that the efficiency gain of HRL is mostly achievable in complex MDPs (i.e. MDPs with large state and action spaces). It is essential to notice that in this experiment, the low-level decomposition remains constant for a given set of room profiles. This is why the benefit of HRL increases with the number of rooms (i.e. the high-level state space) until a plateau is reached. Once the bound is dominated by the high-level MDP, the unchanging complexity of the low-level MDP causes the ratio between the RL bound (Eq. 2) and the high-level part of the HRL bound (Eq. 3), |S||A| H |S h ||A h | H h , to remain constant (even though number of rooms might still grow).

5.2 Different n -rooms & Gymansium Taxi Task

To make the task more challenging, we next increase the number of room profiles used to construct the mazes. As depicted in the rightmost plot of Fig. 4 we considered four different room profiles, each one with a different obstacle in the room. The low-level agent must now learn to navigate multiple

steps (in thousands)

0.08

0.16

1.6

4

8

0.04

0.08

0.8

0.00

0

2000

0.00

0

2000

0.0

0

2000

0

2000

0

5000

episodes

episodes

episodes

episodes

episodes

Q-learning

SHQL

types of rooms to reach the sub-goal sent by the high-level agent. The performance of the algorithms with four different rooms is shown in Fig. 3. The introduction of different room profiles allows us to modulate the complexity of the low-level MDP, in contrast to varying the number of rooms, which only affects the complexity of the high-level MDP. This additional complexity results in a larger state space S l but may also result in a longer horizon H l as the optimal trajectory might require more time to navigate around obstacles to reach the desired hallway successfully. It also becomes evident that, as the number of rooms increases, the hierarchy's benefits become more significant. Nevertheless, comparing Fig. 2 and Fig. 3 we can observe that the introduction of various room layouts has little effect on the Q-learning curve (in red). At the same time, it makes the task slightly more challenging for the HSQL learning curve (in blue), especially when the number of rooms is small since it suffers from the increased complexity of the low-level MDP. But, when the number of rooms is sufficiently large for the high-level MDP complexity to dominate the bound, the benefit of hierarchical decomposition becomes evident. The evolution of the bound ratio (HRL/RL) for the various MDPs considered is shown in the leftmost plot of Fig. 4. It shows that the low-level MDP dominates the bound when the maze consists of a small number of rooms. However, the curves clearly indicate that the expected sample efficiency improves as the high-level MDP becomes more complex (i.e., balancing the complexity between the two levels of the hierarchy). This result is also supported by empirical evidence as illustrated in Figs. 2, 3, 5, and 6. The Gymnasium Taxi environment [9] experiments presented in Appendix A.2.2 further validate our approach and conclusion on an entirely different task.

6 Conclusion

In this work, we analysed the sample complexity of goal-conditioned HRL. To the best of our knowledge, we provide the first result that analyses the decomposition induced by goal-conditioned HRL. In particular, our lower bound offers a valuable tool for practitioners that could help them decide whether they should consider a hierarchical decomposition for their problem. We also designed a novel algorithm that can leverage the hierarchy to improve its sample efficiency and implemented this on a set of hierarchical tasks. These experimental results further emphasizes the usefulness of the proposed bound since our theoretical findings support empirical efficiency gains.

Although this paper has taken a significant first step in bettering our understanding of the benefits of hierarchical decomposition, there is still scope for further work in this area. Three immediate open questions are: (i) whether our lower bound could be refined by explicitly accounting for the interactions between the low-level and the high-level agent, (ii) is it possible to design an algorithm that can theoretically match the proposed lower bound, (iii) the current results only consider the cardinality of the sub-goal space because we assumed that the sub-goal-space was given and that all sub-goals where solvable. Methods that design efficient sub-goal spaces remain largely unexplored and are a critical aspect of the design of HRL algorithms. Moreover, the insights we proposed are framed in a tabular setting and do not yet extend to a continuous setting where function approximation could be leveraged to allow the low-level agent to generalise over sub-goals and to consider setting beyond the tabular case described in this article. Overcoming those limitations is an interesting direction for future work.

Acknowledgments

AR was supported by an EPSRC CASE studentship supported by Shell, and AAF was supported by a UKRI Turing AI Fellowship (EP/V025449/1).

References

A. A. Faisal A. Robert, C. Pike-Burke. Code and resources for running Hierarchical Stationary Q-learning: doi.org/10.6084/m9.figshare.24291172.

Ekaterina Abramova, Luke Dickens, Daniel Kuhn, and Aldo Faisal. Hierarchical, heterogeneous control of non-linear dynamical systems using reinforcement learning. In European Workshop On Reinforcement Learning, at ICML , volume 2012, 2012.

Peter Auer and Ronald Ortner. Online regret bounds for a new reinforcement learning algorithm. In 1st Austrian Cognitive Vision Workshop , pages 35-42. ¨ Osterr. Computer-Ges., 2005.

Benjamin Beyret, Ali Shafti, and A Aldo Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5014-5019. IEEE, 2019.

EmmaBrunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International Conference on Machine Learning , pages 316-324. PMLR, 2014.

Christoph Dann. Strategic Exploration in Reinforcement Learning-New Algorithms and Learning Guarantees . PhD thesis, Google, 2019.

Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning. Advances in Neural Information Processing Systems , 28, 2015.

Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems , 5, 1992.

Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research , 13:227-303, 2000.

Thomas G Dietterich et al. The maxq method for hierarchical reinforcement learning. In International Conference on Machine Learning , volume 98, pages 118-126, 1998.

Ronan Fruit and Alessandro Lazaric. Exploration-exploitation in mdps with options. In Artificial intelligence and statistics , pages 576-584. PMLR, 2017.

Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Emma Brunskill. Regret minimization in mdps with options without prior knowledge. Advances in Neural Information Processing Systems , 30, 2017.

Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning , pages 1851-1860. PMLR, 2018.

Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latr´ e. Hierarchical reinforcement learning: Asurvey and open research challenges. Machine Learning and Knowledge Extraction , 4(1):172221, 2022.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in Neural Information Processing Systems , 31, 2018.

Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In AAMAS (1) , pages 299-306, 2008.

Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948 , 2017.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 , 2022.

Amy McGovern, Richard S Sutton, and Andrew H Fagg. Roles of macro-actions in accelerating reinforcement learning. In Grace Hopper celebration of women in computing , volume 1317, page 15, 1997.

Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems , 31, 2018.

Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618 , 2019.

Marc Pickett and Andrew G Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In International Conference on Machine Learning , volume 19, pages 506-513, 2002.

Doina Precup and Richard S Sutton. Multi-time models for temporally abstract planning. Advances in Neural Information Processing Systems , 10, 1997.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 12, 1999.

Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(12):181-211, 1999.

Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in Neural Information Processing Systems , 29, 2016.

Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning , pages 3540-3549. PMLR, 2017.

Zheng Wen, Doina Precup, Morteza Ibrahimi, Andre Barreto, Benjamin Van Roy, and Satinder Singh. On efficiency in hierarchical reinforcement learning. Advances in Neural Information Processing Systems , 33:6708-6718, 2020.

A Appendix

A.1 Proof of Theorem

Theorem 3.1 states that there exist positive constants c l , c h and δ 0 such that for every δ ∈ (0 , δ 0 ) and for every algorithm A that satisfies a PAC guarantee for ( ϵ, δ ) and outputs a deterministic policy, there is a fixed horizon MDP such that A must collect

episodes until its policy is ( ϵ, δ ) -accurate.

Proof. An ϵ -accurate pair of policies ( π l , π h ) satisfies | V π ∗ l ,π ∗ h o -V π l ,π h o | ≤ ϵ . Note that by the triangle inequality, if | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | + | V π ∗ l ,π h o -V π l ,π h o | ≤ ϵ , then we will have | V π ∗ l ,π ∗ h o -V π l ,π h o | ≤ ϵ . We therefore focus on showing:

(i) the number of samples required to guarantee | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | ≤ ϵ/ 2 is bounded by Ω ( |S h ||A h | H 2 h ϵ 2 ln ( 1 δ + c h ) )

(ii) the number of samples required to guarantee | V π ∗ l ,π h o -V π l ,π h o | ≤ ϵ/ 2 is bounded by Ω ( |S l ||A H ||A| H 2 l ϵ 2 ln ( 1 δ + c l ) )

Then, once we have both (i) and (ii), we know that after

episodes, we will have | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | + | V π ∗ l ,π h o -V π l ,π h o | ≤ ϵ and so | V π ∗ l ,π ∗ h o -V π l ,π h o | ≤ ϵ .

Part (i) Note that only learning the high-level policy when the low-level policy is optimal is equivalent to learning an ϵ -accurate high-level policy interacting with M h with a stationary transition function (since the low-level behaviour is not evolving anymore). Hence we can bound the number of episodes N h required to have: | V ∗ h -V π ∗ l ,π h h | ≤ ϵ , by directly applying Eq. (2) to the high-level MDP to get

To be able to use this result to construct the bound of interest, we need to make sure these results can be translated into the original MDP: | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | ≤ ϵ . In particular, the reward functions are not the same for M o and M h . We defined the original MDP's reward function to penalize each step unless it reaches the end goal, in which case it rewards the agent with a bonus. This reward function was initially defined to take values in the interval [ -a, b ] . Without loss of generality, we shift the reward function so that the rewards are now drawn from the interval [ -a -b, 0] . In this new setting, a successful completion of the task is rewarded with a bonus of 0 . Similarly, the low-level reward function penalizes each step unless it reaches the instructed sub-goal. Lastly, the high-level reward function r h ( s h , a h ) = ∑ H l t =1 r l ( s l,t , a h ) , consists of the sum of the low-level rewards accumulated during each sub-episode. A difference of scale in value function arises because high-level rewards include the bonus the low-level agent receives for completing each sub-goal, whereas the value functions in the original MDP do not include this bonus. To compensate for this difference, the low-level reward is re-scaled with a penalty twice larger; the low-level reward r l now takes values in [ -2 a -2 b, 0] , and, under this specific reward function, the completion of a sub-goal is rewarded by 0 as well. The re-scaling ensures that the accumulated penalties are larger in the high-level MDPs, even for trajectories with a single intermediary step. This guarantees that | V π ∗ l ,π ∗ h o -V π ∗ l ,π h o | ≤ | V ∗ h -V π ∗ l ,π h h | . Hence after E [ N h ] episodes, we have | V ∗ o -V π ∗ l ,π h o | ≤ ϵ

Part (ii) By a similar argument to Part (i), we can bound the number of episodes in the low-level MDP required to obtain an ϵ -optimal low-level policy for a fixed high-level policy π h . In particular, a lower bound on the number of episodes N l required to have | V π ∗ l ,π h l -V π l ,π h l | ≤ ϵ can directly be obtained from Eq. (2):

Ultimately, we are interested in a lower bound on the number of episodes in the original MDP N o . Since a single episode in the original MDP corresponds to several episodes in the low-level MDP, we can divide the number of episodes by a factor H h : E [ N o ] ≥ E [ N l ] H h . However, the difference in episode length between the two MDPs also induces a difference of scale in their value functions. To ensure that the learned policies are ϵ -optimal in the original MDP we need to ensure that | V π ∗ l ,π h l -V π l ,π h l | ≤ ϵ H h , which requires at least H 2 h E [ N l ] low-level episodes. Combining the two arguments, we get that E [ N o ] ≥ E [ N l ] H 2 h H h = E [ N l ] H h . Since we are computing a lower bound on the number of episodes in the original MDP required to learn a near-optimal policy and recognising that H h ≥ 1 , we can conclude that E [ N o ] ≥ E [ N l ] .

This leads us to a lower bound on the number of episodes needed to obtain an ϵ -accurate pair of policies as the one stated in the theorem.

A.2 Additional Experiments

A.2.1 More n -rooms Problems

In the experimental section (Sec. 5), we used several room layouts. In the main paper, we only provide learning curves for mazes that are composed of rooms without any obstacles or mazes that are composed of all the possible room layouts depicted in the rightmost plot of Fig. 4. To complete our experiment, we show below in (Fig. 5 and Fig. 6) the learning curve obtained when mazes are built from two or three different room layouts. These learning curves highlight the same behaviour that we previously discussed. With simple mazes, the low-level MDP strongly dominates the bound but as we increase the high-level MDP complexity (i.e. the number of rooms), the benefit of the hierarchical machinery becomes evident. Note also that those results were used to plot the evolution of the bound ratio in the leftmost plot of Fig. 4.

steps (in thousands)

0.08

0.16

1.6

4

8

0.04

0.08

0.8

4

0.00

0

2000

0.00

0

2000

0.0

0

2000

0

0

2000

0

2000

episodes

episodes

episodes

episodes

episodes

Q-learning

HQL

steps (in thousands)

0.08

0.16

1.6

4

8

0.04

0.08

0.8

2

4

0.00

0

2000

0.00

0

2000

0.0

0

2000

0

2000

0

0

2000

episodes

episodes

episodes

episodes

episodes

Q-learning

HQL

A.2.2 Taxi Environment

Another task we consider is the taxi environment [9]. In this environment, the agent is tasked to pick up a passenger and bring them to a specific location. As illustrated in Fig. 7 the taxi environment is a grid world composed of 25 cells ( 5 × 5 grid). At the beginning of an episode, the passenger is assigned a random initial location, which can only be in one of the four taxi stations depicted by the letters R, G, Y and B in Fig. 7 right plot, and a final destination which is any of the three remaining taxi stations. When visualising the environment, the location of the passenger is denoted by the letter in blue and the final destination by the purple letter. The initial position of the taxi is also randomly sampled, and to complete the task, the taxi will have first to pick up the passenger and then deliver him to his final destination.

This problem can be formalized by a MDP. The state space S encodes the 404 reachable states. While the 25 possible taxi locations, the 4 possible destinations and the 5 potential passenger locations suggest that there are 500 states. The states where the passenger is at the destination but the taxi is somewhere else are not possible and there are 24 ∗ 4 = 96 such configurations. The action space A consists of the six possible actions (move up, down, left, right, pickup, dropoff). The default horizon H considered for this environment is 200 steps. The initial state distribution uniformly chooses the cell in which the taxi starts the episode as well as the passenger's initial location and destination. The reward function assigns a positive bonus (+20) for successfully delivering the passenger, a penalty for illegal action such as an inappropriate dropoff and pickup action (-10) and a small penalty (-1) for each remaining step. The characteristics of the original MDP for this problem are the following |S| = 404 , |A| = 6 and H = 200 .

To apply HSQL, the following decomposition was considered:

(i) High-level MDP: The high-level state space only encodes the current passenger location and the final desired destination, hence |S h | = 20 . The high-level action space is composed of 8 possible subgoals |A h | = 8 . Each sub-goal encodes one of the four special locations and whether the passenger should be picked up or dropped off. Since all possible initial configurations can be completed with only two sub-goals, first a pick-up instruction and then a drop-off instruction, the horizon of the high-level MDP is H h = 2 .

(ii) Low-level MDP: The low-level agent needs to be able to perform all primitive actions so A l = A . Then, the low-level state space encodes the current location of the taxi as well as the instructed sub-goal, yielding to a state space of size |S l | = 200 . Since the trip is evenly divided into two separate instructions, the horizon of the low-level agent is H l = 100 .

By Theorem 3.1, we can observe that the low-level MDP is likely to dominate the bound and to drive the overall complexity of the problem. Ignoring the constant terms in the bound, our theory suggests that small improvement should still be possible. This is confirmed by our empirical results in Fig. 7 (left plot) which shows a significantly steeper learning curve for HSQL compared to Qlearning on the taxi task. In order to make the comparison between the two algorithms as fair as possible, we considered a range of possible hyperparameter values and for each algorithm, we only report the best-performing setting. The hyperparameters considered are the initial exploration rate

Q-learning

R

fsteps

200

D

HSQL

100

0

Y

B

0

500

1000

Numberofepisodes

ϵ ∈ [0 . 1 , 0 . 3 , · · · , 0 . 7 , 0 . 9] and the decay rate δ ∈ [1 -10 -3 , 1 -10 -4 , · · · , 1 -10 -7 ] . Note that we only considered the following decay function ϵ k +1 = ϵ k ∗ δ k +1 , where k denotes the current episode number. The error bars are obtained by running ten different seeds.

B The Option Framework and Goal-Conditioned HRL

As most of the existing theory on HRL [30, 12, 11] has been developed in the option framework, we recall below the definition of that framework and the connection it has with the goal-condition HRL framework considered in this article. An immediate difference is that those previous works consider an infinite horizon setting while we focus on a finite horizon setting. For the remainder of this section, we describe some other significant differences.

Let's first recall that the tuple M = ⟨S , A , p, r, γ ⟩ describe a finite MDP, where S is a finite set of states, A is a finite set of actions, p ( s ′ | s, a ) is the probability of transitioning to the state s ′ given that in state s action a was executed, r ( s, a ) is the reward distribution associated with the state action pair ( s, a ) and, lastly, γ is the discount factor. A policy π : S → A maps each state to an action. An option is a tuple o = {I o , β o , π o } , where I o ⊂ S is the initiation set, i.e. the set of states where the option o can be started, β o : S → [0 , 1] is the probability distribution to terminate in option in a given state and finally π o is the intra-option policy, i.e. the policy followed until the option ends.

Whenever the set of primitive actions A is replaced by a set of options O the original MDP M, becomes a semi-MDP M O = ⟨S O , O , p O , r O , τ O ⟩ . The state space S O ⊆ S is the set of all initial and terminal states. O denotes the set of options and p O ( s ′ , o , s ) is the probability to end option o in state s ′ given that it was started in state s :

where P ( s k = s ′ | s, π o ) is the probability to reach state s ′ in exactly k steps from state s under policy π o . r o ( s, o ) is the distribution of reward cumulated when starting policy π o in state s . Lastly, τ O is the distribution of holding time, i.e. the number of primitive steps executed to reach s ′ from s under policy π o .

It is possible to map the goal-conditioned HRL framework onto the options framework. Sub-goals would correspond to options, the initiation set of a given option o includes all states where the corresponding sub-goal g sub is available, the option will terminate with probability 1 if the goal state has been reached or the H l time steps have been executed and terminate with probability 0 otherwise. Lastly, the intra-option policy π o corresponds to the low-level policy π l ( ·| g sub ) conditioned on the corresponding sub-goal.

From a theoretical standpoint, the key difference between the results in [11, 12, 30] and the ones presented in this article are that they assume the intra-option policies π o to be known. If we do not assume strong prior knowledge about the task, the number of potential intra-option policies will be prohibitive. In this article, we investigate algorithms that do not make assumptions about the low-level policy and jointly learn the low- and high-level policies through interactions with the environment, thus going beyond what has been done for the options framework.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a theoretical analysis of the sample complexity in goal-conditioned hierarchical reinforcement learning (HRL) and establishes a lower bound using hierarchical decomposition to quantify it. Additionally, the paper empirically validates the theoretical results by examining the sample complexity of the proposed hierarchical algorithm on several toy grid-world tasks.\n\nSTRENGTHS:\nI commend the authors for addressing an important theoretical problem in the field of HRL and deriving a lower bound to quantify the sample complexity of goal-conditioned HRL.\n\nWEAKNESSES:\nOne significant aspect that the paper lacks is a thorough theoretical analysis regarding the selection of sub-goal spaces in continuous environments or sets in discrete environments.\n\nQUESTIONS:\nQ1. The selection of the sub-goal space plays a vital role in the efficiency of the HRL algorithm, as different choices of sub-goal spaces can result in varying sample efficiencies, such as in HIRO [1], HRAC [2], HIGL [3], and others [4]. Unfortunately, in Theorem 3.1, the authors do not analyze the impact of different sub-goal spaces on the sample efficiency of HRL. Therefore, I find the theoretical results to be trivial.\n\n[1] Nachum, Ofir, et al. \"Data-efficient hierarchical reinforcement learning.\" Advances in neural information processing systems 31 (2018).\n\n[2] T. Zhang, S. Guo, T. Tan, X. Hu and F. Chen, \"Adjacency Constraint for Efficient Hierarchical Reinforcement Learning,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4152-4166, 1 April 2023.\n\n[3] Junsu Kim, et al. \"Landmark-guided subgoal generation in hierarchical reinforcement learning. \" Advances in Neural Information Processing Systems, 34: 28336–28349, 2021.\n\n[4] Lee, Seungjae, et al. \"DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning.\" Advances in Neural Information Processing Systems 35 (2022): 13668-13678.\n\nQ2. Another limitation of the paper is the absence of a comparison between the proposed method and other existing HRL algorithms in the experimental section. It would be valuable to include such a comparison to provide a more comprehensive evaluation of the proposed approach\n\nLIMITATIONS:\nSee Questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper takes an important step toward quantifying the benefits achieved due to hierarchical decomposition of an MDP by deriving lower bound on sample complexity of goal-conditioned HRL algorithms. The paper also proposes a novel hierarchical Q-learning algorithm that exploits goal-based hierarchical decomposition of an MDP into a high-level and a low-level sub-MDPs and jointly learns their policies. The authors evaluate hierarchical policies for different decompositions on original MDPs to validate when decomposition provides benefits and whether it aligns with the derived bound. \n\nThe paper is well motivated and clearly written. The ideas of the paper to quantify benefits of hierarchical decomposition are novel. The derived theoretical guarantees on the lower bound are sound. I suggest clarifying that the derived bounds only apply to tabular setting of RL i.e. discrete state space problems in the Introduction. The theoretical findings are  backed by empirical results on maze environments of different sizes with convincing insights. The empirical evaluation would benefit from diversification of domains and tasks not restricted to navigation, and an investigation of bounds when using bad and good decompositions for the same environment.\n\nSTRENGTHS:\n(I) The paper provides strong theoretical guarantees on the lower bound of the number of episodes given a decomposition needed to learn an epsilon-accurate hierarchical policy, which also serves as a lower bound to learn an epsilon-accurate optimal policy. \n\n(II) The paper also identifies properties relating to state and temporal abstractions and the size of the high-level action space from the derived bound that can improve sample efficiency.\n\nWEAKNESSES:\n(I) The assumptions regarding the scope of the derived bounds restricted to a tabular setting need to be clarified in the Introduction. \n\n(II) All experiments are on maze environments of different sizes. While the current analysis is convincing for navigation tasks, it will be interesting to see if the benefits of decomposition and the derived bounds align for more diverse domains and tasks that not restricted to just navigation e.g. officeworld [2], taxiworld [3] etc.\n\n(III) It is not clear how the bounds will identify when a decomposition is bad enough and would degrade the performance compared to non-hierarchical algorithms. \n\nMinor errors:\n(I) Line 48: proposes -> propose\n\nQUESTIONS:\n(I) How are the ideas of the Stationary Hierarchical Q-learning to overcome non-stationarity of the high-level policy related to the ideas in [1]? \n\n(II) Can you elaborate what it means to separate the original state space evenly between the two level of hierarchy? \n\n(III) Does the method apply only to dense reward functions?\n\n(IV) Would the bounds identify when a decomposition for an environment would degrade performance of the proposed algorithm compared to Qlearning? \n\nReferences:\n\n[1] Levy, A., Konidaris, G., Platt, R. and Saenko, K., 2017. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948.\n\n[2] Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, pages 2107–2116. PMLR, 2018.\n\n[3] Thomas Dietterich. State abstraction in maxq hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 12, 1999.\n\nLIMITATIONS:\n(Included in the weaknesses)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProvides a sample bound on the complexity of goal-conditioned HRL algorithms based on the two MDPs they are decomposed into, and a Q learning algorithm to leverage these findings. Formulates HRL as a two-level problem, where the upper level passes actions to the lower level policy. The work proves a lower bound on the complexity of the HRL formulation which pivotally scales according to the size of the high level action space and the reusability of the low level space. A new algorithm is introduced which identifies the need for a consistent low level action space, and this method is asssessed in four-room gridworld domains.\n\nSTRENGTHS:\nThis work provides a clean proof with a highly understandable sketch and a strong intuition. Together, this provides an extremely clear and easy-to-read description of the sample complexity of HRL In addition, the theory provides some clear insights into how to understand other HRL work.\n\nThis work provides a simple idea applied to the existing framework of HRL in the description of the high-level training based on low-level performance. It also seems like adding the intuition of the shared upper-level complexity term would be useful for keeping the size of the upper policy action space small (by somehow limiting the goals), which is empirically verified in other HRL work.\n\nWEAKNESSES:\nFigure 1 is difficult to comprehend, somehow managing to be simultaneously overly simple (this is a basic construction of hierarchical RL) and overly complicated (what is the intuition for the equations on the right-hand side?\n\nThis work contrasts against the Options framework in the first paragraph of the background, without specific Ying what the options framework is. As a framework itself, the options framework also makes no assumptions about prior knowledge, no prevents from state abstraction, both statements made about the framework. \n\nThis work spends much of the earlier part justifying the context of the goal-based hierarchy, but it appears that other than the state-based complexity term, there is no strict requirement that the hierarchy be goal based as opposed to simply parameter based. As long as there exists a measure of the performance of the lower-level policy, it seems like the same reasoning would apply. \n\nThe empirical results are somewhat lacking. In particular, while the proof should apply generally to HRL contexts, the work only empirically verifies in maze environments, and maze environments which are constructed to amplify the advantages of the upper-level policy. A different kind of environment such as a mountain car or multiple inverted pendulum would have been interesting, notwithstanding an environment that requires a deep RL method.\n\nQUESTIONS:\nSee the weaknesses section\n\nLIMITATIONS:\nLimited empirical assessment in multiple domains\n\nAdditional evidence of how the terms of the bound translate to empirical results would be insightful", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}