SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[18] Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275, 2022.
[19] Amy McGovern, Richard S Sutton, and Andrew H Fagg. Roles of macro-actions in accelerating reinforcement learning. In Grace Hopper celebration of women in computing, volume 1317, page 15, 1997.
[20] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
[21] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019.
[22] Marc Pickett and Andrew G Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In International Conference on Machine Learning, volume 19, pages 506-513, 2002.
[23] Doina Precup and Richard S Sutton. Multi-time models for temporally abstract planning. Advances in Neural Information Processing Systems, 10, 1997.
[24] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[25] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[26] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[27] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.
[28] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in Neural Information Processing Systems, 29, 2016.
[29] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
[30] Zheng Wen, Doina Precup, Morteza Ibrahimi, Andre Barreto, Benjamin Van Roy, and Satinder Singh. On efficiency in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 33:6708-6718, 2020.
A.1 Proof of Theorem
Theorem 3.1 states that there exist positive constants  $c_{l}$ ,  $c_{h}$  and  $\delta_0$  such that for every  $\delta \in (0, \delta_0)$  and for every algorithm  $A$  that satisfies a PAC guarantee for  $(\epsilon, \delta)$  and outputs a deterministic policy, there is a fixed horizon MDP such that  $A$  must collect
$$
\mathbb {E} [ N _ {e} ] = \Omega \left(\max \left(\frac {| \mathcal {S} _ {l} | | \mathcal {A} _ {h} | | \mathcal {A} | H _ {l} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {l}}\right), \frac {| \mathcal {S} _ {h} | | \mathcal {A} _ {h} | H _ {h} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {h}}\right)\right)\right)
$$
episodes until its policy is  $(\epsilon, \delta)$ -accurate.
Proof. An  $\epsilon$ -accurate pair of policies  $(\pi_l, \pi_h)$  satisfies
$\left|V_{o}^{\pi_{l}^{*}, \pi_{h}^{*}} - V_{o}^{\pi_{l}, \pi_{h}}\right| \leq \epsilon$ . Note that by the triangle inequality, if  $|V_{o}^{\pi_{l}^{*}, \pi_{h}^{*}} - V_{o}^{\pi_{l}^{*}, \pi_{h}}| + |V_{o}^{\pi_{l}^{*}, \pi_{h}} - V_{o}^{\pi_{l}, \pi_{h}}| \leq \epsilon$ , then we will have  $|V_{o}^{\pi_{l}^{*}, \pi_{h}^{*}} - V_{o}^{\pi_{l}, \pi_{h}}| \leq \epsilon$ . We therefore focus on showing:
(i) the number of samples required to guarantee  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| \leq \epsilon / 2$  is bounded by  $\Omega \left( \frac{|S_{h}| |\mathcal{A}_{h}| H_{h}^{2}}{\epsilon^{2}} \ln \left( \frac{1}{\delta + c_{h}} \right) \right)$
(ii) the number of samples required to guarantee  $|V_{o}^{\pi_{l}^{*},\pi_{h}} - V_{o}^{\pi_{l},\pi_{h}}| \leq \epsilon / 2$  is bounded by  $\Omega \left( \frac{|S_{l}||\mathcal{A}_{H}||\mathcal{A}|H_{l}^{2}}{\epsilon^{2}} \ln \left( \frac{1}{\delta + c_{l}} \right) \right)$
Then, once we have both (i) and (ii), we know that after
$$
\Omega \Bigg (\max \left(\frac {| \mathcal {S} _ {l} | | \mathcal {A} _ {h} | | \mathcal {A} | H _ {l} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {l}}\right), \frac {| \mathcal {S} _ {h} | | \mathcal {A} _ {h} | H _ {h} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {h}}\right)\right)
$$
episodes, we will have  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| + |V_{o}^{\pi_{l}^{*},\pi_{h}} - V_{o}^{\pi_{l},\pi_{h}}| \leq \epsilon$  and so  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l},\pi_{h}}| \leq \epsilon$ .
Part (i) Note that only learning the high-level policy when the low-level policy is optimal is equivalent to learning an  $\epsilon$ -accurate high-level policy interacting with  $\mathcal{M}_h$  with a stationary transition function (since the low-level behaviour is not evolving anymore). Hence we can bound the number of episodes  $N_h$  required to have:  $|V_h^* - V_h^{\pi_l^*, \pi_h}| \leq \epsilon$ , by directly applying Eq. (2) to the high-level MDP to get
$$
\mathbb {E} [ N _ {h} ] = \Omega \bigg (\frac {| \mathcal {S} _ {h} | | \mathcal {A} _ {h} | H _ {h} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {h}}\right) \bigg).
$$
To be able to use this result to construct the bound of interest, we need to make sure these results can be translated into the original MDP:  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| \leq \epsilon$ . In particular, the reward functions are not the same for  $\mathcal{M}_o$  and  $\mathcal{M}_h$ . We defined the original MDP's reward function to penalize each step unless it reaches the end goal, in which case it rewards the agent with a bonus. This reward function was initially defined to take values in the interval  $[-a,b]$ . Without loss of generality, we shift the reward function so that the rewards are now drawn from the interval  $[-a - b,0]$ . In this new setting, a successful completion of the task is rewarded with a bonus of 0. Similarly, the low-level reward function penalizes each step unless it reaches the instructed sub-goal. Lastly, the high-level reward function  $r_h(s_h,a_h) = \sum_{t=1}^{H_l} r_l(s_{l,t},a_h)$ , consists of the sum of the low-level rewards accumulated during each sub-episode. A difference of scale in value function arises because high-level rewards include the bonus the low-level agent receives for completing each sub-goal, whereas the value functions in the original MDP do not include this bonus. To compensate for this difference, the low-level reward is re-scaled with a penalty twice larger; the low-level reward  $r_l$  now takes values in  $[-2a - 2b,0]$ , and, under this specific reward function, the completion of a sub-goal is rewarded by 0 as well. The re-scaling ensures that the accumulated penalties are larger in the high-level MDPs, even for trajectories with a single intermediary step. This guarantees that  $|V_{o}^{\pi_{l}^{*},\pi_{h}^{*}} - V_{o}^{\pi_{l}^{*},\pi_{h}}| \leq |V_{h}^{*} - V_{h}^{\pi_{l}^{*},\pi_{h}}|$ . Hence after  $\mathbb{E}[N_h]$  episodes, we have  $|V_{o}^{*} - V_{o}^{\pi_{l}^{*},\pi_{h}}| \leq \epsilon$
Part (ii) By a similar argument to Part (i), we can bound the number of episodes in the low-level MDP required to obtain an  $\epsilon$ -optimal low-level policy for a fixed high-level policy  $\pi_h$ . In particular, a lower bound on the number of episodes  $N_{l}$  required to have  $|V_{l}^{\pi_{l}^{*},\pi_{h}} - V_{l}^{\pi_{l},\pi_{h}}|\leq \epsilon$  can directly be obtained from Eq. (2):
$$
\mathbb {E} [ N _ {l} ] = \Omega \bigg (\frac {| \mathcal {S} _ {l} | | \mathcal {A} _ {h} | | \mathcal {A} | H _ {l} ^ {2}}{\epsilon^ {2}} \ln \left(\frac {1}{\delta + c _ {l}}\right) \bigg).
$$
Ultimately, we are interested in a lower bound on the number of episodes in the original MDP  $N_{o}$ . Since a single episode in the original MDP corresponds to several episodes in the low-level MDP, we can divide the number of episodes by a factor  $H_{h}$ :  $\mathbb{E}[N_{o}]\geq \frac{\mathbb{E}[N_{l}]}{H_{h}}$ . However, the difference in episode length between the two MDPs also induces a difference of scale in their value functions. To ensure that the learned policies are  $\epsilon$ -optimal in the original MDP we need to ensure that  $|V_{l}^{\pi_{l}^{*},\pi_{h}} - V_{l}^{\pi_{l},\pi_{h}}|\leq \frac{\epsilon}{H_{h}}$  which requires at least  $H_{h}^{2}\mathbb{E}[N_{l}]$  low-level episodes. Combining the two arguments, we get that  $\mathbb{E}[N_o]\geq \frac{\mathbb{E}[N_l]H_h^2}{H_h} = \mathbb{E}[N_l]H_h$ . Since we are computing a lower bound on the number of episodes in the original MDP required to learn a near-optimal policy and recognising that  $H_{h}\geq 1$ , we can conclude that  $\mathbb{E}[N_o]\geq \mathbb{E}[N_l]$ .
This leads us to a lower bound on the number of episodes needed to obtain an  $\epsilon$ -accurate pair of policies as the one stated in the theorem.
A.2 Additional Experiments
A.2.1 More  $n$ -rooms Problems
In the experimental section (Sec. 5), we used several room layouts. In the main paper, we only provide learning curves for mazes that are composed of rooms without any obstacles or mazes that are composed of all the possible room layouts depicted in the rightmost plot of Fig. 4. To complete our experiment, we show below in (Fig. 5 and Fig. 6) the learning curve obtained when mazes are built from two or three different room layouts. These learning curves highlight the same behaviour that we previously discussed. With simple mazes, the low-level MDP strongly dominates the bound but as we increase the high-level MDP complexity (i.e. the number of rooms), the benefit of the hierarchical machinery becomes evident. Note also that those results were used to plot the evolution of the bound ratio in the leftmost plot of Fig. 4.
Figure 5: Shows learning curves on various maze sizes with two different room instances; either the room is empty, or it has a U-shape obstacle in it. The agent's performance is measured in the number of steps it requires to solve the task.
Figure 6: Shows learning curves on various maze sizes with three different room layouts; either the room is empty or it has either a U-shape obstacle or the room is stripped with horizontal walls. The performance of the agent is measured in the number of steps it requires to solve the task.
A.2.2 Taxi Environment
Another task we consider is the taxi environment [9]. In this environment, the agent is tasked to pick up a passenger and bring them to a specific location. As illustrated in Fig. 7 the taxi environment is a grid world composed of 25 cells ( $5 \times 5$  grid). At the beginning of an episode, the passenger is assigned a random initial location, which can only be in one of the four taxi stations depicted by the letters R, G, Y and B in Fig. 7 right plot, and a final destination which is any of the three remaining taxi stations. When visualising the environment, the location of the passenger is denoted by the letter in blue and the final destination by the purple letter. The initial position of the taxi is also randomly sampled, and to complete the task, the taxi will have first to pick up the passenger and then deliver him to his final destination.
This problem can be formalized by a MDP. The state space  $S$  encodes the 404 reachable states. While the 25 possible taxi locations, the 4 possible destinations and the 5 potential passenger locations suggest that there are 500 states. The states where the passenger is at the destination but the taxi is somewhere else are not possible and there are  $24 * 4 = 96$  such configurations. The action space  $\mathcal{A}$  consists of the six possible actions (move up, down, left, right, pickup, dropoff). The default horizon  $H$  considered for this environment is 200 steps. The initial state distribution uniformly chooses the cell in which the taxi starts the episode as well as the passenger's initial location and destination. The reward function assigns a positive bonus (+20) for successfully delivering the passenger, a penalty for illegal action such as an inappropriate dropout and pickup action (-10) and a small penalty (-1) for each remaining step. The characteristics of the original MDP for this problem are the following  $|S| = 404$ ,  $|\mathcal{A}| = 6$  and  $H = 200$ .
To apply HSOL, the following decomposition was considered:
(i) High-level MDP: The high-level state space only encodes the current passenger location and the final desired destination, hence  $|S_h| = 20$ . The high-level action space is composed of 8 possible sub-goals  $|\mathcal{A}_h| = 8$ . Each sub-goal encodes one of the four special locations and whether the passenger should be picked up or dropped off. Since all possible initial configurations can be completed with only two sub-goals, first a pick-up instruction and then a drop-off instruction, the horizon of the high-level MDP is  $H_{h} = 2$ .
(ii) Low-level MDP: The low-level agent needs to be able to perform all primitive actions so  $\mathcal{A}_l = \mathcal{A}$ . Then, the low-level state space encodes the current location of the taxi as well as the instructed sub-goal, yielding to a state space of size  $|\mathcal{S}_l| = 200$ . Since the trip is evenly divided into two separate instructions, the horizon of the low-level agent is  $H_l = 100$ .
By Theorem 3.1, we can observe that the low-level MDP is likely to dominate the bound and to drive the overall complexity of the problem. Ignoring the constant terms in the bound, our theory suggests that small improvement should still be possible. This is confirmed by our empirical results in Fig. 7 (left plot) which shows a significantly steeper learning curve for HSQL compared to Q-learning on the taxi task. In order to make the comparison between the two algorithms as fair as possible, we considered a range of possible hyperparameter values and for each algorithm, we only report the best-performing setting. The hyperparameters considered are the initial exploration rate
Figure 7: Left plot shows the learning curves of HSQL and Q-learning in the taxi environment. Right plot illustrates the taxi environment; the agent (yellow square) needs to navigate a grid world to reach the pickup location (encoded by the blue letter) and drop off the passenger at its final destination (encoded as the purple letter). The remaining letters are the possible pickup and drop-off locations. Note that the taxi needs to learn how to navigate the grid world in order to avoid the walls represented by solid lines.
$\epsilon \in [0.1, 0.3, \dots, 0.7, 0.9]$  and the decay rate  $\delta \in [1 - 10^{-3}, 1 - 10 - 4, \dots, 1 - 10^{-7}]$ . Note that we only considered the following decay function  $\epsilon_{k + 1} = \epsilon_k * \delta^{k + 1}$ , where  $k$  denotes the current episode number. The error bars are obtained by running ten different seeds.
B The Option Framework and Goal-Conditioned HRL
As most of the existing theory on HRL [30, 12, 11] has been developed in the option framework, we recall below the definition of that framework and the connection it has with the goal-condition HRL framework considered in this article. An immediate difference is that those previous works consider an infinite horizon setting while we focus on a finite horizon setting. For the remainder of this section, we describe some other significant differences.
Let's first recall that the tuple  $M = \langle \mathcal{S}, \mathcal{A}, p, r, \gamma \rangle$  describes a finite MDP, where  $\mathcal{S}$  is a finite set of states,  $\mathcal{A}$  is a finite set of actions,  $p(s'|s, a)$  is the probability of transitioning to the state  $s'$  given that in state  $s$  action  $a$  was executed,  $r(s, a)$  is the reward distribution associated with the state action pair  $(s, a)$  and, lastly,  $\gamma$  is the discount factor. A policy  $\pi : \mathcal{S} \to \mathcal{A}$  maps each state to an action. An option is a tuple  $o = \{\mathcal{I}_o, \beta_o, \pi_o\}$ , where  $\mathcal{I}_o \subset \mathcal{S}$  is the initiation set, i.e. the set of states where the option  $o$  can be started,  $\beta_o : \mathcal{S} \rightarrow [0,1]$  is the probability distribution to terminate in option in a given state and finally  $\pi_o$  is the intra-option policy, i.e. the policy followed until the option ends.
Whenever the set of primitive actions  $\mathcal{A}$  is replaced by a set of options  $\mathcal{O}$  the original MDP M, becomes a semi-MDP  $M_{\mathcal{O}} = \langle S_{\mathcal{O}},\mathcal{O},p_{\mathcal{O}},r_{\mathcal{O}},\tau_{\mathcal{O}}\rangle$  . The state space  $\mathcal{S}_{\mathcal{O}}\subseteq \mathcal{S}$  is the set of all initial and terminal states.  $\mathcal{O}$  denotes the set of options and  $p_{\mathcal{O}}(s^{\prime},o,s)$  is the probability to end option  $o$  in state  $s^\prime$  given that it was started in state  $s$  ..
$$
p _ {\mathcal {O}} (s ^ {\prime}, o, s) = \sum_ {k = 1} ^ {\infty} P (s _ {k} = s ^ {\prime} | s, \pi_ {o}) \beta_ {o} (s ^ {\prime}),
$$
where  $P(s_{k} = s^{\prime}|s,\pi_{o})$  is the probability to reach state  $s^\prime$  in exactly  $k$  steps from state  $s$  under policy  $\pi_o$ .  $r_o(s,o)$  is the distribution of reward cumulated when starting policy  $\pi_o$  in state  $s$ . Lastly,  $\tau_{\mathcal{O}}$  is the distribution of holding time, i.e. the number of primitive steps executed to reach  $s^\prime$  from  $s$  under policy  $\pi_o$ .
It is possible to map the goal-conditioned HRL framework onto the options framework. Sub-goals would correspond to options, the initiation set of a given option  $o$  includes all states where the corresponding sub-goal  $g_{sub}$  is available, the option will terminate with probability 1 if the goal state has been reached or the  $H_{l}$  time steps have been executed and terminate with probability 0 otherwise. Lastly, the intra-option policy  $\pi_o$  corresponds to the low-level policy  $\pi_l(\cdot | g_{sub})$  conditioned on the corresponding sub-goal.
From a theoretical standpoint, the key difference between the results in [11, 12, 30] and the ones presented in this article are that they assume the intra-option policies  $\pi_{o}$  to be known. If we do not assume strong prior knowledge about the task, the number of potential intra-option policies will be prohibitive. In this article, we investigate algorithms that do not make assumptions about the low-level policy and jointly learn the low- and high-level policies through interactions with the environment, thus going beyond what has been done for the options framework.