SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoyan Hu
xyhu21@cse.cuhk.edu.hk
Ho-fung Leung
ho-fung.leung@outlook.com
The options framework yields empirical success in long-horizon planning problems of reinforcement learning (RL). Recent works show that options improves the sample efficiency in online RL where the learner can actively explores the environment. However, these results are no longer applicable to scenarios where exploring the environment online is risky, e.g., automated driving and healthcare. In this paper, we provide the first analysis of the sample complexity for offline RL with options, where the agent learns from a dataset without further interaction with the environment. We propose the PEssimistic Value Iteration for Learning with Options (PEVIO) algorithm and establish near-optimal suboptimality bounds (with respect to the novel information-theoretic lower bound for offline RL with options) for two popular data-collection procedures, where the first one collects state-option transitions and the second one collects state-action transitions. We show that compared to offline RL with actions, using options not only enjoys a faster finite-time convergence rate (to the optimal value) but also attains a better performance (when either the options are carefully designed or the offline data is limited). Based on these results, we analyze the pros and cons of the data-collection procedures, which may facilitate the selection in practice.
Planning in long-horizon tasks is challenging in reinforcement learning (RL) (Co-Reyes et al., 2018; Eysenbach et al., 2019; Hoang et al., 2021). A line of study proposes to accelerate learning in these tasks using temporally-extended actions (Fikes et al., 1972; Sacerdoti, 1973; Drescher, 1991; Jiang et al., 2019; Nachum et al., 2019; Machado et al., 2021; Erraqabi et al., 2022). One powerful approach is the options framework introduced by Sutton et al. (1999), where the agent interacts with the environment with closed-loop policies called options. Empirical success (Tessler et al., 2017; Vezhnevets et al., 2017) shows that options help achieve sample-efficient performance in long-horizon planning problems.
To provide a theoretical guarantee to the options framework, recent works have focused on the sample complexity of RL with options in the online setting, where the agent continuously explores the environment and learns a hierarchical policy to select options. Brunskill and Li (2014) establish a PAC-like sample complexity of RL with options in the semi-Markov decision processes (SMDPs), where temporally-extended actions are treated as indivisible and unknown units. Later, Fruit and Lazaric (2017) provide the first regret analysis of RL with options under the Markov decision processes (MDPs) framework. While their proposed algorithm attains a sublinear regret, it requires prior knowledge of the environment, which is not usually available in practice. To address this problem, Fruit et al. (2017) propose an algorithm that does not require prior knowledge, yet achieves
a near-optimal regret bound. However, these results are inapplicable to many real-world scenarios where online exploration is not allowed. For example, it has been argued that in healthcare (Gottesman et al., 2019) and automated driving (Shalev-Shwartz et al., 2016), learning in an online manner is risky and costly. In these scenarios, offline learning, where the agent learns a policy from a dataset, is preferred. We note that there is a line of studies on the sample complexity of offline RL with primitive actions only (i.e., without the use of options) (Levine et al., 2020; Fu et al., 2020; Rashidinejad et al., 2021). Unfortunately, to the best of our knowledge, there have been no results reported on the offline RL with options.
In this paper, we make the following contributions. First, we derive a novel information-theoretic lower bound, which generalizes the one for offline learning with actions. Second, we propose the PEssimistic Value Iteration for Learning with Options (PEVIO) algorithm and derive near-optimal suboptimality bounds for two popular data-collection procedures, where the first one collects state-action transitions and the second one collects state-action transitions. More importantly, we show that options facilitate more sample-efficient learning in both the finite-time convergence rate and actual performance. To shed light on offline RL with options in practice, we discuss the pros and cons of both data-collection procedures based on our analysis.
Learning with Options Building upon the theory of semi-Markov decision processes (SMDPs) (Bradtke and Duff, 1994; Mahadevan et al., 1997), Sutton et al. (1999) propose to learn with options. Following their seminal work, learning with options has been widely studied in the function approximation setting (Sorg and Singh, 2010) and hierarchical RL (Igl et al., 2020; Klissarov and Precup, 2021; Wulfmeier et al., 2021). Discovering useful options has also been the subject of extensive research (Stolle and Precup, 2002; Riemer et al., 2018; Mankowitz et al., 2018; Harb et al., 2018; Hiraoka et al., 2019; Bagaria et al., 2021). Despite its empirical success, there have been fairly limited studies on the sample efficiency of learning with options. Brunskill and Li (2014) analyze the sample complexity bound for an RMAX-like algorithm for SMDPs. Fruit and Lazaric (2017) derive the first regret analysis of learning with options. They propose an algorithm that attains sublinear regret in the infinite-horizon average-reward MDP while requiring prior knowledge of the environment. Later, Fruit et al. (2017) remove this requirement.
Offline RL In the offline setting, a dataset that is collected by executing a behavior policy in the environment is provided, and the agent is asked to learn a near-optimal policy using only this dataset. A key challenge in offline RL is the insufficient coverage of the dataset (Wang et al., 2021), which is also known as distributional shift (Chen and Jiang, 2019; Levine et al., 2020). To address this problem, the previous study on sample-efficient learning assumes uniform coverage of the dataset (Liu et al., 2018; Chen and Jiang, 2019; Jiang and Huang, 2020; Yang et al., 2020; Xie and Jiang, 2020; Uehara et al., 2020; Qu and Wierman, 2020; Yin et al., 2021). This assumption is relaxed in recent works by pessimism principle (Xie et al., 2021; Rashidinejad et al., 2021; Jin et al., 2021).
3.1 Episodic MDP with Options
Let  $\Delta(\mathcal{X})$  denote the probability simplex on space  $\mathcal{X}$  and  $[N] := \{1, \dots, N\}$  for any positive integer  $N$ . An episodic MDP with options is a sextuple  $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{O}, H, \mathcal{P}, r)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  the (primitive) action set,  $\mathcal{O}$  the finite set of options,  $H$  the length of each episode,  $\mathcal{P} = \{P_h : \mathcal{S} \times \mathcal{A} \mapsto \Delta(\mathcal{S})\}_{h \in [H]}$  the transition kernel,  $r = \{r_h : \mathcal{S} \times \mathcal{A} \mapsto [0, 1]\}_{h \in [H]}$  the deterministic reward function. We define  $S := |\mathcal{S}|$ ,  $A := |\mathcal{A}|$ , and  $O := |\mathcal{O}|$ . A (Markov) option  $o \in \mathcal{O}$  is a pair  $(\pi^o, \beta^o)$  where  $\pi^o = \{\pi_h^o : \mathcal{S} \mapsto \Delta(\mathcal{A})\}_{h \in [H]}$  is the option's policy and  $\beta^o = \{\beta_h^o : \mathcal{S} \mapsto [0, 1]\}_{h \in [H]}$  is the probability of the option's termination. For convenience, we define  $\beta_{H+1}^o(s) = 1$  for all  $(s, o) \in \mathcal{S} \times \mathcal{O}$ , i.e., any option is terminated after the end of an
<sup>1</sup>Our results can be directly generalized to stochastic rewards.
episode. We assume that the initial state  $s_1$  is fixed. Upon arriving at state  $s_h$  at any timestep  $h \in [H]$ , if  $h = 1$  (at the beginning of an episode), the agent selects option  $o_1 \sim \mu_1(\cdot | s_1)$ , where  $\mu = \{\mu_h : S \mapsto \Delta(\mathcal{O})\}_{h \in [H]}$  is a hierarchical policy to select an option at each state. Otherwise  $(h \geq 2)$ , the agent first terminates option  $o_{h-1}$  with probability  $\beta_h^{o_{h-1}}(s_h)$ . If option  $o_{h-1}$  is terminated, she then selects a new option  $o_h \sim \mu_h(\cdot | s_h)$  according to the hierarchical policy  $\mu$ . If option  $o_{h-1}$  is not terminated, the agent continues to use option  $o_{h-1}$  at timestep  $h$ , i.e.,  $o_h = o_{h-1}$ . After that, the agent takes action  $a_h \sim \pi_h^{o_h}(\cdot | s_h)$ , receives a reward  $r_h := r_h(s_h, a_h)$ , and transits to the next state  $s_{h+1} \sim P_h(\cdot | s_h, a_h)$ . An episode terminates at timestep  $H + 1$ . A special case is that an action  $a$  is an option  $o$ , such that  $\pi_h^o(a|s) = 1$  and  $\beta_h^o(s) = 1$  for any  $(h, s) \in [H] \times S$ . For convenience, we use the notation  $\mathcal{O} = \mathcal{A}$  to represent that each option corresponds to an action, which is the case in RL with primitive actions.
To define the  $Q$ -function and the value function, we introduce some useful notations. Let  $T = \{T_h : \mathcal{S} \times \mathcal{O} \mapsto \Delta(\mathcal{S} \times [H - h + 1])\}_{h \in [H]}$  and  $U = \{U_h : \mathcal{S} \times \mathcal{O} \mapsto [0, H]\}_{h \in [H]}$  denote the option transition function and the option utility function, respectively. Particularly, for any  $(h, s, o) \in [H] \times \mathcal{S} \times \mathcal{O}$ , the option transition function  $T_h(s'|s, o, \tau)$  is the probability that the agent uses option  $o$  at state  $s$  at timestep  $h$ , reaches state  $s'$  at timestep  $h + \tau$  without terminating option  $o$  in these  $\tau$  timesteps, and finally terminates option  $o$  at state  $s'$  at timestep  $h + \tau$ . The option utility function  $U_h(s, o)$  is the expected cumulative reward within timesteps that the option is used without being terminated. Given any arbitrary series of functions  $\{y_h : \mathcal{S} \mapsto \mathbb{R}\}_{h \in [H]}$ , define the operator  $[T_h y_{h + \tau}](s, o) := \sum_{s' \in \mathcal{S}} T_h(s'|s, o, \tau) y_{h + \tau}(s')$  for any  $(s, o, \tau) \in \mathcal{S} \times \mathcal{O} \times [H - h + 1]$ . In the following, we derive the  $Q$ -function and the value function for learning with options. (The detailed proof can be found in Appendix B.)
Theorem 1 (Q-function and value function). For any hierarchical policy  $\mu$  and  $(h,s,o)\in [H]\times S\times \mathcal{O}$ , the Q-function is given by
$$
Q _ {h} ^ {\mu} (s, o) := \mathbb {E} _ {\mu} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, o _ {h} = o \right] = U _ {h} (s, o) + \sum_ {\tau \in [ H - h + 1 ]} \left[ T _ {h} V _ {h + \tau} ^ {\mu} \right] (s, o) \tag {1}
$$
and the value function is given by
$$
V _ {h} ^ {\mu} (s) := \mathbb {E} _ {\mu} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, o _ {h} \sim \mu_ {h} (\cdot | s _ {h}) \right] = \sum_ {o \in \mathcal {O}} \mu_ {h} (o | s) Q _ {h} ^ {\mu} (s, o) \tag {2}
$$
where  $V_{H + 1}^{\mu}(s) = Q_{H + 1}^{\mu}(s,o) = 0$  for any  $(s,o)\in \mathcal{S}\times \mathcal{O}$
Intuitively, the first term  $U_{h}(s,o)$  of the  $Q$ -function is the expected reward within timesteps that option  $o$  is used without being terminated, and the second term  $\sum_{\tau \in [H - h + 1]}[T_hV_{h + \tau}^\mu ](s,o)$  corresponds to the expected reward within timesteps after option  $o$  is terminated and a new option is selected according to  $\mu$ . It can be shown that there exists an optimal (and deterministic) hierarchical policy  $\mu^{*} = \{\mu_{h}^{*}:S\mapsto \mathcal{O}\}_{h\in [H]}$  that attains the optimal value function, i.e.,  $V_{h}^{*}(s) = \sup_{\mu}V_{h}^{\mu}(s)$  for all  $(h,s)\in [H]\times S$  (Sutton et al., 1999).
3.2 Offline RL with Options
We consider learning with options in the offline setting. That is, given a dataset  $\mathcal{D}$  that is collected by an experimenter through interacting with the environment, the algorithm outputs a hierarchical policy  $\widehat{\mu}$ . The sample complexity is measured by the suboptimality, i.e., the shortfall in the value function of the hierarchical policy  $\widehat{\mu}$  compared to that of the optimal hierarchical policy  $\mu^{*}$ , which is given by
$$
\operatorname {S u b O p t} _ {\mathcal {D}} \left(\widehat {\mu}, s _ {1}\right) := V _ {1} ^ {*} \left(s _ {1}\right) - V _ {1} ^ {\widehat {\mu}} \left(s _ {1}\right) \tag {3}
$$
To derive a novel information-theoretic lower bound of  $\mathrm{SubOpt}_{\mathcal{D}}$ , we first define some useful notations. For any hierarchical policy  $\mu$ , we denote by  $\theta^{\mu} = \{\theta_h^\mu : S \times \mathcal{O} \mapsto [0,1]\}_{h \in [H]}$  its state-option
Note that any  $H$ -length episodic MDP with a stochastic initial state is equivalent to an  $(H + 1)$ -length MDP with a fixed initial state  $s_0$ .
3The formal definitions can be found in Appendix A.
occupancy measure. That is,  $\theta_h^\mu (s,o)$  is the probability that the agent selects a particular option  $o$  at state  $s$  at timestep  $h$  (either when  $h = 1$  or when the option  $o_{h - 1}$  used at the timestep  $h - 1$  is terminated) when following the hierarchical policy  $\mu$ . With a slight abuse of the notation, we denote by  $\theta_h^\mu (s)\coloneqq \sum_{o\in \mathcal{O}}\theta_h^\mu (s,o)$  the state occupancy measure for any  $(h,s)\in [H]\times S$ . Further, we define
$$
Z _ {\mathcal {O}} ^ {\mu} := \sum_ {h, s} \theta_ {h} ^ {\mu} (s), \bar {Z} _ {\mathcal {O}} ^ {\mu} := \sum_ {h, s, o} \mathbb {I} \left[ \theta_ {h} ^ {\mu} (s, o) > 0 \right] \tag {4}
$$
where  $\mathbb{I}[\cdot ]$  is the indicator function. Intuitively,  $Z_{\mathcal{O}}^{\mu}$  is the expected number of timesteps to alternate a new option and  $\overline{Z}_{\mathcal{O}}^{\mu}$  is the maximal number of state-option pairs that can be visited, when following the hierarchical policy  $\mu$ . The following proposition shows that options facilitate temporal abstraction and reduction of the state space.
Proposition 1. For any hierarchical policy  $\mu$ , we have that  $Z_{\mathcal{O}}^{\mu} \leq H$ . If  $\mu$  is deterministic, i.e.,  $\mu = \{\mu_h : S \mapsto \mathcal{O}\}_{h \in [H]}$ , we further have that  $\overline{Z}_{\mathcal{O}}^{\mu} \leq HS$ . All the above equalities hold when  $\mathcal{O} = \mathcal{A}$ .
Next, we derive a novel information-theoretic lower bound of  $\mathrm{SubOpt}_{\mathcal{D}}$ . The detailed proof can be found in Appendix C.
Theorem 2 (Information-theoretic lower bound). Let  $\rho = \{\rho_h : S \mapsto \Delta(\mathcal{O})\}_{h \in [H]}$  denote any hierarchical behavior policy to collect the dataset. Define the class of problem instances
$$
\begin{array}{l} \mathcal {M} \left(C ^ {\text {o p t i o n}}, z ^ {*}, \bar {z} ^ {*}\right) := \left\{\left(M, \rho\right): \text {E x i s t s d e t e r m i n i s t i c} \mu^ {*} \text {o f a n e p i s o d i c M D P} M \right. \\ \left. \text {s u c h} \max  _ {h, s, o} \frac {\theta_ {h} ^ {\mu^ {*}} (s , o)}{\theta_ {h} ^ {\rho} (s , o)} \leq C ^ {\text {o p t i o n}}, Z _ {\mathcal {O}} ^ {\mu^ {*}} \leq z ^ {*}, \bar {Z} _ {\mathcal {O}} ^ {\mu^ {*}} \leq \bar {z} ^ {*} \right\}. \\ \end{array}
$$
Suppose that  $C^{\mathrm{option}} \geq 2$ ,  $z^{*} \geq 1$ , and  $\overline{z}^{*} \geq \lfloor z^{*} \rfloor S$ , where  $\lfloor x \rfloor := \max \{ n \in \mathbb{N} : n \leq x \}$  is the largest integer no greater than  $x \in \mathbb{R}$ . Then, there exists an absolute constant  $c_{0}$  such that for any offline algorithm that outputs a hierarchical policy  $\widehat{\mu}$ , if the number of episodes
$$
K \leq \frac {c _ {0} \cdot C ^ {\text {o p t i o n}} H z ^ {*} \bar {z} ^ {*}}{\epsilon^ {2}}
$$
then there exists a problem instance  $(M,\rho)\in \mathcal{M}(C^{\mathrm{option}},z^{*},\overline{z}^{*})$  on which the hierarchical policy  $\widehat{\mu}$  suffers from  $\epsilon$ -suboptimality, that is,
$$
\mathbb {E} _ {M} \left[ \operatorname {S u b O p t} _ {\mathcal {D} _ {1}} (\hat {\mu}, s) \right] \geq \epsilon
$$
where the expectation  $\mathbb{E}_M$  is with respect to the randomness during the execution of  $\rho$  within MDP  $M$ .
Theorem 2 shows that, when dataset  $\mathcal{D}$  sufficiently covers the trajectories induced by  $\mu^{*}$ , i.e.,  $\max_{h,s,o} \theta_h^{\mu^*}(s,o) / \theta_h^\rho(s,o) \leq C^{\text{option}}$ , at least  $\Omega(C^{\text{option}} HZ_{\mathcal{O}}^* \overline{Z}_{\mathcal{O}}^* / \epsilon^2)$  episodes are required to learn an  $\epsilon$ -optimal hierarchical policy from dataset  $\mathcal{D}$ . Note that when  $\mathcal{O} = \mathcal{A}$ , it recovers the lower bound  $\Omega(H^3 SC^* / \epsilon^2)$  for offline RL with primitive actions, where  $C^*$  is the concentrability defined therein.
4 The PEVIO Algorithm
Inspired by the Pessimistic Value Iteration (PEVI) algorithm (Jin et al., 2021), we propose the PEssimistic Value Iteration for Learning with Options (PEVIO) in Algorithm 1. Given a dataset  $\mathcal{D}$  and the corresponding Offline Option Evaluation (OOE) subroutine, whose details are specified in Sections 5.1 and 5.2, PEVIO outputs a hierarchical policy  $\widehat{\mu} = \{\widehat{\mu}_h : S \mapsto \Delta(\mathcal{O})\}_{h \in [H]}$ .
To estimate the  $Q$ -function, given a dataset  $\mathcal{D}$ , PEVIO first constructs  $(\widehat{T},\widehat{U},\Gamma)$  by the OOE subroutine (line 3). Specifically,  $\widehat{T}_h$  and  $\widehat{U}_h$  are the empirical counterparts of  $T_{h}$  and  $U_{h}$  presented in the  $Q$ -function given by Equation (1), respectively. In addition,  $\Gamma$  is a penalty function computed based on dataset  $\mathcal{D}$ . We remark that the OOE subroutine varies when different data-collecting procedures
Algorithm 1 PEssimistic Value Iteration for Learning with Options (PEVIO)
1: Input: Dataset  $\mathcal{D}$  and the corresponding Offline Option Evaluation (OOE) subroutine.  
2: Initialize:  $\widehat{Q}_h(s,o)\gets 0,\widehat{V}_h(s)\gets 0,\widehat{V}_{H + 1}(s)\gets 0$  for any  $(h,s,o)\in [H]\times S\times \mathcal{O}$ .  
3:  $(\widehat{T},\widehat{U},\Gamma)\gets \mathrm{OOE}(\mathcal{D})$ .  
4: for  $h = H,H - 1,\dots ,1$  do  
5: for  $(s,o)\in \mathcal{S}\times \mathcal{O}$  do  
6:  $\overline{Q}_h(s,o)\gets \widehat{U}_h(s,o) + \sum_{\tau = 1}^{H - h + 1}[\widehat{T}_h\widehat{V}_{h + \tau}](s,o) - \Gamma_h(s,o)$ .  
7:  $\widehat{Q}_h(s,o)\gets \max \{0,\min \{\overline{Q}_h(s,o),H - h + 1\}\}$ .  
8: end for  
9: for  $s\in \mathcal{S}$  do  
10:  $\widehat{\mu}_h(\cdot |s)\gets \arg \max_{\mu_h}\langle \widehat{Q}_h(s,\cdot),\mu_h(\cdot |s)\rangle_{\mathcal{O}}$ .  
11:  $\widehat{V}_h(s)\gets \langle \widehat{Q}_h(s,\cdot),\widehat{\mu}_h(\cdot |s)\rangle_{\mathcal{O}}$ .  
12: end for  
13: end for  
14: Output:  $\widehat{\mu} = \{\widehat{\mu}_h\}_{h\in [H]}$ .
are considered and we provide the details in Sections 5.1 and 5.2, respectively. Given  $\widehat{U}_h$ ,  $\widehat{T}_h$ , and  $\Gamma_h$ , the estimated  $Q$ -function  $\widehat{Q}_h$  is the derived (lines 6 and 7). Particularly,  $\overline{Q}_h$  computed in line 6 can be seen as first replacing  $U_h$  and  $T_h$  with their empirical counterparts  $\widehat{U}_h$ ,  $\widehat{T}_h$  in Equation (1), and then subtracting the penalty function  $\Gamma_h$ . Further, a hierarchical policy  $\widehat{\mu}_h$  is constructed greedily with  $\widehat{Q}_h$  (line 10), where  $\langle f(\cdot), g(\cdot) \rangle_{\mathcal{O}} := \sum_{o \in \mathcal{O}} f(o) g(o)$  for any arbitrary functions  $f, g$  defined on  $\mathcal{O}$ . Finally, given  $\widehat{Q}_h$  and  $\widehat{\mu}_h$ , the corresponding estimated value function  $\widehat{V}_h$  is computed (line 11). To analyze the suboptimality of the hierarchical policy  $\widehat{\mu}$  output from PEVIO, we first provide the following definition, which motivates the design of the penalty function  $\Gamma$ .
Definition 1 ( $\xi$ -uncertainty quantifier for dataset  $\mathcal{D}$ ). The penalty function  $\Gamma = \{\Gamma_h : S \times \mathcal{O} \mapsto \mathbb{R}^+\}_{h \in [H]}$  output from the OOE subroutine in Algorithm 1 (line 3) is said to be a  $\xi$ -uncertainty quantifier with respect to  $\mathbb{P}_{\mathcal{D}}$  if the following event
$$
\begin{array}{l} \mathcal {E} = \left\{\left| \widehat {U} _ {h} (s, o) - U _ {h} (s, o) + \sum_ {\tau = 1} ^ {H - h + 1} \left[ \left(\widehat {T} _ {h} - T _ {h}\right) \widehat {V} _ {h + \tau} \right] (s, o) \right| \right. \tag {5} \\ \leq \Gamma_ {h} (s, o) \text {f o r a l l} (h, s, o) \in [ H ] \times \mathcal {S} \times \mathcal {O} \} \\ \end{array}
$$
satisfies that  $\mathbb{P}_{\mathcal{D}}(\mathcal{E}) \geq 1 - \xi$ , where  $\mathbb{P}_{\mathcal{D}}$  is the joint distribution of the data collecting process.
In other words, the penalty function  $\Gamma$  is a  $\xi$ -uncertainty quantifier if it upper bounds the estimation errors in the empirical option transition function  $\widehat{T}$  and the empirical option utility function  $\widehat{U}$ . Next, we show that the suboptimality of  $\widehat{\mu}$  output from PEVIO is upper bounded if  $\Gamma$  is a  $\xi$ -uncertainty quantifier. (The detailed proof can be found in Appendix D.)
Theorem 3 (Suboptimality of learning with options using dataset  $\mathcal{D}$ ). Let  $\widehat{\mu}$  denote the hierarchical policy output by Algorithm 1. Suppose that  $\Gamma = \{\Gamma_h\}_{h\in [H]}$  output from the OOE subroutine is a  $\xi$ -uncertainty quantifier. Conditioned on the successful event  $\mathcal{E}$  defined in Equation (5), which satisfies that  $\mathbb{P}_{\mathcal{D}}(\mathcal{E})\geq 1 - \xi$ , we have that
$$
\operatorname {S u b O p t} _ {\mathcal {D}} (\widehat {\mu}, s _ {1}) \leq 2 \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mu^ {*}} [ \Gamma_ {h} (s _ {h}, o _ {h}) | s _ {1} ] \tag {6}
$$
where  $\mathbb{E}_{\mu^{*}}[g(s_h,o_h)] = \sum_{(s,o)}\theta_h^{\mu^*}(s,o)g(s,o)$  for any  $h\in [H]$  and arbitrary function  $g:\mathcal{S}\times \mathcal{O}\mapsto$ $\mathbb{R}$
Remark 1. Since the temporal structure of learning with options is much more complex than learning with actions, PEVIO is significantly different from the algorithms proposed for offline RL with primitive actions, such as PEVI (Jin et al., 2021) or VI-LCB (Xie et al., 2021), despite sharing a similar intuition. First, in terms of the algorithm design, PEVI and VI-LCB estimate (one-step) transition kernel and reward function to compute the  $Q$ -function of a state-action pair. However, by Equation (1), the  $Q$ -function of a state-option pair depends on multi-step transitions and rewards.
Hence, it is challenging to design the OOE subroutine and analyze the estimated  $(\widehat{T},\widehat{U},\Gamma)$  for options. Indeed, if the dataset contains  $(s,o,u)$  (state-option-utility) tuples, then  $(T,U,\Gamma)$  can be estimated similarly to the case of learning with primitive actions. However, if the dataset contains only  $(s,a,r)$  (state-action-reward) tuples, then it remains elusive to estimate and analyze  $(T,U,\Gamma)$ . Second, in terms of the suboptimality analysis, previous works on offline RL with primitive actions rely on the extended value difference lemma (Cai et al., 2020, Lemma 4.2), which also depends on the one-step temporal structure of actions and cannot be directly applied to our setting. Hence, to derive Theorem 3, it is non-trivial to generalize the extended value difference lemma to the options framework (See Lemma 8 in the Appendices).
5 Data-Collection and Suboptimality Analysis