SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we consider two data-collection procedures that are widely deployed in the options literature. The first one collects state-option-utility tuples (dataset  $\mathcal{D}_1$ ) and similar datasets are utilized in the work of Zhang et al. (2023). The second one collects state-action-reward tuples (dataset  $\mathcal{D}_2$ ) and is studied in a line of works (Ajay et al., 2021; Villecroze et al., 2022; Salter et al., 2022). Intuitively, dataset  $\mathcal{D}_1$  requires smaller storage and enables efficient evaluation of the options, while dataset  $\mathcal{D}_2$  provides richer information on the environment and even facilitates the evaluation of new options. For each dataset, we design the corresponding OOE subroutine and derive a suboptimality bound for the PEVIO algorithm. Based on these results, we further discuss the advantages and the disadvantages of both data-collection procedures, which sheds light on offline RL with options in practice.
5.1 Learning from State-Option Transitions
We consider dataset  $\mathcal{D}_1\coloneqq \{(s_{t_i^k}^k,o_{t_i^k}^k,u_{t_i^k}^k)\}_{i\in [j^k ],k\in [K]}$  consisting of state-option-utility tuples, which is collected by the experimenter's interaction with the environment for  $K$  episodes using a hierarchical behavior policy  $\rho = \{\rho_h:\mathcal{S}\mapsto \Delta (\mathcal{O})\}_{h\in [H]}$ . More precisely, at timestep  $t_i^k$  of the  $k$ th episode, the experimenter selects a new option  $o_{t_i^k}^k$ , uses it for  $(t_{i + 1}^{k} - t_{i}^{k})$  timesteps, collects a cumulative reward of  $u_{t_i^k}^k$  within these  $(t_{i + 1}^{k} - t_{i}^{k})$  timesteps, and finally terminates this option at state  $s_{t_{i + 1}^k}^k$  at timestep  $t_{i + 1}^k$ . For convenience, we define  $t_{j^{k} + 1}^{k}\coloneqq H + 1$  for any  $k\in [K]$ .
Let  $a \vee b \coloneqq \max \{a, b\}$  for any pair of integers  $a, b \in \mathbb{N}$ . When dataset  $\mathcal{D}_1$  is available, the OOE subroutine in Algorithm 1 is given by Subroutine 2. Particularly, Subroutine 2 incorporates the data splitting technique (Xie et al., 2021) (line 2). That is, given dataset  $\mathcal{D}_1$ , the algorithm randomly splits it into  $H$  subdatasets  $\{\mathcal{D}_{1,h}\}_{h \in [H]}$ . Then  $\widehat{T}_h$  and  $\widehat{U}_h$  are constructed using subdataset  $\mathcal{D}_{1,h}$  (lines 4-7).
To derive the suboptimality for  $\widehat{\mu}$  output from PEVIO, we follow the previous study and make a standard assumption on the coverage of dataset  $\mathcal{D}_1$ .
Assumption 1 (Single hierarchical policy concentrability for dataset  $\mathcal{D}_1$ ). The experimenter collects dataset  $\mathcal{D}_1$  by following a hierarchical behavior policy  $\rho = \{\rho_h : S \mapsto \Delta(\mathcal{O})\}_{h \in [H]}$ . There exists some deterministic optimal hierarchical policy  $\mu^*$  such that
$$
C _ {1} ^ {\text {o p t i o n}} := \max  _ {h, s, o} \frac {\theta_ {h} ^ {\mu^ {*}} (s , o)}{\theta_ {h} ^ {\rho} (s , o)} \tag {7}
$$
(with the convention  $0 / 0 = 0$ ) is finite.
In other words, Assumption 1 states that dataset  $\mathcal{D}_1$  sufficiently covers the trajectories of state-option-utility tuples induced by some deterministic optimal hierarchical policy  $\mu^{*}$ . We derive an upper bound of  $\mathrm{SubOpt}_{\mathcal{D}_1}$  in the following theorem. (The detailed proof can be found in Appendix E.)
Theorem 4 (Suboptimality for dataset  $\mathcal{D}_1$ ). Under Assumption 1, with probability at least  $1 - \xi$ , we have that
$$
\operatorname {S u b O p t} _ {\mathcal {D} _ {1}} (\hat {\mu}, s _ {1}) \leq \tilde {O} \left(\sqrt {\frac {C _ {1} ^ {\text {o p t i o n}} H ^ {3} Z _ {\mathcal {O}} ^ {*} \bar {Z} _ {\mathcal {O}} ^ {*}}{K}}\right) \tag {8}
$$
Subroutine 2 Offline Option Evaluation (OOE) for Dataset  $\mathcal{D}_1$
1: Input: Dataset  $\mathcal{D}_1 = \{(s_{t_i^k}^k,o_{t_i^k}^k,u_{t_i^k}^k)\}_{i\in [j^k ],k\in [K]}$
2: Initialize: Randomly split the dataset  $\mathcal{D}$  into  $H$  subdatasets  $\{\mathcal{D}_{1,h}\}_{h\in [H]}$  with  $|\mathcal{D}_{1,h}| = K / H$ . More precisely, let  $l := \{l_h\}_{h\in [H]}$  be a random partition of the set  $[K]$ , where  $l_{h} := \{l_{h,j}\}_{j\in [K / H]} \subset [K]$  is uniformly sampled from  $[K]$  such that  $\cup_{h\in [H]}l_{h} = [K]$  and  $l_{h} \cap l_{h'} = \emptyset$  for any  $h \neq h'$ . Then we have that  $\mathcal{D}_{1,h} = \{(s_{t_i^k}^k, o_{t_i^k}^k, u_{t_i^k}^k)\}_{i \in [j^k], k \in l_h}$  for any  $h \in [H]$ . Let  $n_{h}(s,o) := \sum_{k \in l_{h}} \mathbb{I}[h \in \{t_{i}^{k}\}_{i \in [j^{k}]}, s_{h}^{k} = s, o_{h}^{k} = o]$  denote the number of times that the experimenter selects a particular option  $o$  at state  $s$  at timestep  $h$  in subdataset  $\mathcal{D}_{1,h}$ .
3: for  $(h,s,o)\in [H]\times S\times \mathcal{O}$  do
4: for  $(s',\tau) \in S \times [H - h + 1]$  do
5:  $\widehat{T}_h(s'|s,o,\tau) \gets \frac{\sum_{k\in l_h}\mathbb{I}[h\in\{t_i^k\}_{i\in[j^{k}]},s_h^k=s,o_h^k=o,s_{h+\tau}^k=s']}{1\vee n_h(s,o)}$
6: end for
7:  $\widehat{U}_h(s,o)\gets \frac{\sum_{k\in l_h}\mathbb{I}[h\in\{t_i^k\}_{i\in[j^k]},s_h^k = s,o_h^k = o]u_h^k}{1\vee n_h(s,o)}$
8:  $\Gamma_h(s,o)\gets \tilde{O}\left(\sqrt{\frac{H^2}{n_h(s,o)\lor 1}}\right)$
9: end for
0: Output:  $(\widehat{T} = \{\widehat{T}_h\}_{h\in [H]},\widehat{U} = \{\widehat{U}_h\}_{h\in [H]},\Gamma = \{\Gamma_h\}_{h\in [H]})$
where  $Z_{\mathcal{O}}^{*} \coloneqq Z_{\mathcal{O}}^{\mu^{*}}$  and  $\overline{Z}_{\mathcal{O}}^{*} \coloneqq \overline{Z}_{\mathcal{O}}^{\mu^{*}}$ .
Compared to the lower bound in Theorem 2, suboptimality bound (8) is near-optimal except for an extra factor of  $H$ .<sup>4</sup> More importantly, it shows that learning with options enjoys a faster convergence rate to the optimal value than learning with primitive actions. Recall that the VI-LCB algorithm (Xie et al., 2021) that learns with primitive actions attains the suboptimality bound  $\tilde{O}(\sqrt{H^5SC^* / K})$ , where  $C^*$  is the concentrability defined therein. When ignoring the concentrability parameters, the suboptimality bound (8) is smaller since  $Z_{\mathcal{O}}^* \leq H$  and  $\overline{Z}_{\mathcal{O}}^* \leq HS$ .
Remark 2. While, in the worst case, both  $Z_{\mathcal{O}}^{*}$  and  $\overline{Z}_{\mathcal{O}}^{*}$  can scale with  $H$  and  $HS$ , respectively, we note that in many long-horizon planning problems, they often scale with the number of sub-tasks, which are greatly smaller, especially for tasks that enable temporal abstraction and the reduction of the state space. For example, while the route-planning task of going from City A to City B by transportation takes thousands of primitive actions to finish, it can be efficiently solved by decomposing into the following sub-tasks: (1) going to the airport/train station in City A; (2) taking transportation to City B; and (3) reaching the final destination in City B, for which options are designed. In this case, both  $Z_{\mathcal{O}}^{*}$  and  $\overline{Z}_{\mathcal{O}}^{*} / S$  may only scale as  $o(H)$ . In other words, options facilitate more sample-efficient learning through temporal abstraction, i.e., sticking to an option until a sub-task is finished. Another concrete example is solving a maze, where options are often designed to move agents to bottleneck states (Şimşek and Barto, 2004; Solway et al., 2014; Machado et al., 2017) that connect different densely connected regions of the state space, e.g., doorways. In this case, while the number of option switches may grow proportionally to  $H$ , i.e.,  $Z_{\mathcal{O}}^{*} / H = O(1)$ , the number of states to switch options can be greatly smaller than  $S$ , i.e.,  $\overline{Z}_{\mathcal{O}}^{*} / H = o(s)$ . That is to say, options help improve the sample complexity by the reduction of the state space.
Further, we show that learning with options attains a better performance than learning with primitive actions, when either the options are carefully designed or the offline data is limited.
Corollary 1 (Better performance). Let  $\mathrm{TrueSubOpt}_{\mathcal{D}_1}(\widehat{\mu},s_1)\coloneqq V_1^{*,\mathrm{pri}}(s_1) - V_1^{\widehat{\mu}}(s_1)$ , where  $V^{*,\mathrm{pri}}$  is the optimal value function defined for the primitive actions. Ignoring the concentrability parameters, we have that  $\mathrm{TrueSubOpt}_{\mathcal{D}_1}(\widehat{\mu},s_1)\leq \tilde{O} (\sqrt{H^5SC^* / K})$  attained by the VI-LCB algorithm (Xie et al., 2021), when either the options are carefully designed (i.e.,  $\Delta_{\mathcal{O}}(s_1)\coloneqq V_1^{*,\mathrm{pri}}(s_1) - V_1^* (s_1) = 0$ ) or
<sup>4</sup>We note that the extra factor  $H$  in the suboptimality bound (8) can be reduced by applying the reference-advantage decomposition technique (Xie et al., 2021).
the number  $K$  of trajectories in the dataset is
$$
o \left(\frac {H ^ {3}}{\Delta_ {\mathcal {O}} ^ {2}} \left(\sqrt {H ^ {2} S C ^ {*}} - \sqrt {C _ {1} ^ {\text {o p t i o n}} Z _ {\mathcal {O}} ^ {*} \bar {Z} _ {\mathcal {O}} ^ {*}}\right) _ {+} ^ {2}\right)
$$
where  $(x)_{+} \coloneqq \max \{0, x\}$  for any  $x \in \mathbb{R}$ .
Corollary 1 implies that when data is limited, e.g., in cases where the data collection is highly expensive or risky, learning with options is beneficial since the output hierarchical policy yields a higher value than learning with primitive actions.
5.2 Learning from State-Action Transitions
We consider dataset  $\mathcal{D}_2\coloneqq \{(s_h^k,a_h^k,r_h^k)\}_{h\in [H],k\in [K]}$  consisting of state-action-reward tuples, which is collected by an experimenter's interaction with the environment for  $K$  episodes using any arbitrary behavior policy. That is, the experimenter takes action  $a_h^k$  at state  $s_h^k$  at timestep  $h$  of the  $k$ th episode, receives a reward of  $r_h^k$ , and transits to state  $s_{h + 1}^k$ .
When dataset  $\mathcal{D}_2$  is provided, the OOE subroutine in Algorithm 1 is given by Subroutine 3. Note that one difficulty is that we cannot directly estimate the option transition function and the option utility function from dataset  $\mathcal{D}_2$  as it only includes the information of the primitive actions. Hence, Subroutine 3 first constructs the empirical transition kernel  $\widehat{P} = \{\widehat{P}_h\}_{h\in [H]}$  and the empirical reward function  $\widehat{r} = \{\widehat{r}_h\}_{h\in [H]}$  (lines 4-7), and use them to further construct  $\widehat{T}_h$  and  $\widehat{U}_h$  (lines 8-20). To
Subroutine 3 Offline Option Evaluation (OOE) for Dataset  $\mathcal{D}_2$
1: Input: Dataset  $\mathcal{D}_2 = \{(s_h^k, a_h^k, r_h^k)\}_{h \in [H], k \in [K]}$ .
2: Initialize:  $\widehat{U}_{H + 1}(\cdot)\gets 0$ . Let  $N_{h}(s,a):= \sum_{k = 1}^{K}\mathbb{I}[s_{h}^{k} = s,a_{h}^{k} = a]$  denote the number of visits of state-action pair  $(s,a)$  at timestep  $h$  in dataset  $\mathcal{D}_2$ . Function  $\phi := \{\phi_h:\mathcal{S}\times \mathcal{O}\mapsto \mathbb{R}\}_{h\in [H]}$  is given by Equation (32) in the Appendices.
3: for  $h = H, H - 1, \dots, 1$  do
4: for  $(s,a)\in S\times \mathcal{A}$  do
5:  $\widehat{P}_h(s'|s,a) \gets \frac{\sum_{k\in[K]}\mathbb{I}[s_h^k = s,a_h^k = a,s_{h + 1}^k = s']}{1\vee N_h(s,a)}$  for any  $s^\prime \in \mathcal{S}$
6:  $\widehat{r}_h(s,a)\gets \mathbb{I}[N_h(s,a)\geq 1]r_h(s,a)$
7: end for
8: for  $(s,o,s^{\prime})\in S\times \mathcal{O}\times S$  do
9:  $\widehat{T}_h(s'|s,o,1)\gets \beta_{h + 1}^o (s')\sum_{a\in \mathcal{A}}\pi_h^o (a|s)\widehat{P}_h(s'|s,a)$
10:  $\widehat{T}_h(s'|s,o,1)\gets (1 - \beta_{h + 1}^o (s'))\sum_{a\in \mathcal{A}}\pi_h^o (a|s)\widehat{P}_h(s'|s,a)$
11: for  $l = 2,\dots ,H - h + 1$  do
12:  $\widehat{T}_h(s'|s,o,l) \gets \sum_a \pi_h^o (a|s)\sum_{s'}\widehat{P}_h(s''|s,a)\left(1 - \beta_{h + 1}^o (s'')\right)\widehat{T}_{h + 1}(s'|s'',o,l - 1)$
13:  $\widehat{\overline{T}}_h(s'|s,o,l)\gets \sum_a\pi_h^o (a|s)\sum_{s''}\widehat{P}_h(s''|s,a)\left(1 - \beta_{h + 1}^o (s'')\right)\widehat{\overline{T}}_{h + 1}(s'|s'',o,l - 1)$
14: end for
15: end for
16: end for
17: for  $(h,s,o)\in [H]\times S\times \mathcal{O}$  do
18:  $\Gamma_h(s,o)\gets \tilde{O}\left(\sqrt{\sum_{m = h}^H\sum_{(s,a)\in\mathcal{X}_{h,s,o}^m}\frac{H^3S}{N_m(s,a)\lor 1}} +H\phi_h(s,o)\right)$
19:  $\widehat{U}_h(s,o)\gets \sum_{a\in \mathcal{A}}\pi_h^o (a|s)\widehat{r}_h(s,a) + \sum_{s'\in \mathcal{S}}\widehat{\overline{T}}_h(s'|s,o,1)\widehat{U}_{h + 1}(s',o)$
20: end for
21: Output:  $(\widehat{T} = \{\widehat{T}_h\}_{h\in [H]},\widehat{U} = \{\widehat{U}_h\}_{h\in [H]},\Gamma = \{\Gamma_h\}_{h\in [H]})$
derive the suboptimality, we first define some useful notations. For any  $(h,s,o)\in [H]\times \mathcal{S}\times \mathcal{O}$  and  $h\leq m\leq H$  , we denote by  $\mathcal{X}_{h,s,o}^{m}$  the set of state-action pairs that can be reached at timestep  $m$  by using option  $o$  at state  $s$  and timestep  $h$  without being terminated. Further, let  $d^{\rho}:= \{d_h^\rho :S\times \mathcal{A}\mapsto$
For convenience, we assume that  $\mathcal{X}_{h,s,o}^{m}$  is known prior. When  $\mathcal{X}_{h,s,o}^{m}$  is unknown, it can be replaced by a superset that does not require prior knowledge and our results directly follow (See Remark 3 in Appendix F).
$[0,1]\}_{h\in [H]}$  denote the state-action distribution of the behavior policy  $\rho$  used by the experimenter. That is,  $d_h^\rho (s,a)$  is the probability that the agent takes action  $a$  at state  $s$  at timestep  $h$ . Similarly, we make the following assumption on dataset  $\mathcal{D}_2$ .
Assumption 2 (Single hierarchical policy concentrability for dataset  $\mathcal{D}_2$ ). The experimenter collects dataset  $\mathcal{D}_2$  by following an arbitrary behavior policy  $\rho$ . There exists some deterministic optimal hierarchical policy  $\mu^*$  such that
$$
C _ {2} ^ {\text {o p t i o n}} := \max  _ {h, s, o} \sum_ {h \leq m \leq H, \left(s ^ {\prime}, a ^ {\prime}\right) \in \mathcal {X} _ {h, s, o} ^ {m}} \frac {\theta_ {h} ^ {\mu^ {*}} (s , o)}{d _ {m} ^ {\rho} \left(s ^ {\prime} , a ^ {\prime}\right)} \tag {9}
$$
Intuitively, Assumption 2 states that dataset  $\mathcal{D}_2$  sufficiently covers the trajectories of state-action-reward tuples induced by the optimal hierarchical policy  $\mu^{*}$ . Next, we derive an upper bound of  $\mathrm{SubOpt}_{\mathcal{D}_2}$  under Assumption 2. The detailed proof can be found in Appendix F.
Theorem 5 (Suboptimality for dataset  $\mathcal{D}_2$ ). Under Assumption 2, with probability at least  $1 - \xi$ , we have that
$$
\operatorname {S u b O p t} _ {\mathcal {D} _ {2}} (\widehat {\mu}, s _ {1}) \leq \tilde {O} \left(\sqrt {\frac {C _ {2} ^ {\text {o p t i o n}} H ^ {3} S Z _ {\mathcal {O}} ^ {*} \bar {Z} _ {\mathcal {O}} ^ {*}}{K}} + \frac {H ^ {2} S O C _ {2} ^ {\text {o p t i o n}}}{K}\right) \tag {10}
$$
which translates to
$$
\widetilde {O} \left(\sqrt {\frac {C _ {2} ^ {\text {o p t i o n}} H ^ {3} S Z _ {\mathcal {O}} ^ {*} \overline {{Z}} _ {\mathcal {O}} ^ {*}}{K}}\right)
$$
when  $K$  of dataset  $\mathcal{D}_2$  is sufficiently large, i.e.,  $K \geq \tilde{O}(C_2^{\mathrm{option}}H^5S^9A^2O^2/(Z_{\mathcal{O}}^*\overline{Z}_{\mathcal{O}}^*))$ , where  $Z_{\mathcal{O}}^* = Z_{\mathcal{O}}^{\mu^*}$  and  $\overline{Z}_{\mathcal{O}}^* = \overline{Z}_{\mathcal{O}}^{\mu^*}$ .
While, in general, suboptimality bound (10) does not compare favorably against the suboptimality  $\tilde{O} (\sqrt{H^5SC^* / K})$  attained by the VI-LCB algorithm that learns with primitive actions, we argue that it can be better in long-horizon problems where the horizon  $H$  is much greater than the cardinality of the state space  $S$ .
We analyze the pros and cons of both data-collection procedures, which sheds light on offline RL with options in practice. Compared to  $\mathcal{D}_2$ , dataset  $\mathcal{D}_1$  requires smaller storage and enjoys faster convergence to the optimal value, which is further illustrated as follows.
- Storage: For dataset  $\mathcal{D}_2 = \{(s_h^k, a_h^k, r_h^k)\}_{h \in [H], k \in [K]}$ , the storage is simply  $HK$ . However, for dataset  $\mathcal{D}_1 = \{(s_{t_i^k}^k, o_{t_i^k}^k, u_{t_i^k}^k)\}_{i \in [j^k], k \in [K]}$ , its expected size is  $K \cdot Z_{\mathcal{O}}^\rho \leq HK$ , where  $\rho$  is the hierarchical behavior policy. Therefore, in the case of a small  $Z_{\mathcal{O}}^\rho$ , dataset  $\mathcal{D}_1$  requires much smaller storage than  $\mathcal{D}_2$  (on average).
- Suboptimality: Ignoring the concentrability, the suboptimality bound (10) for dataset  $\mathcal{D}_2$  is worse than the suboptimality bound (8) for dataset  $\mathcal{D}_1$  by a factor of  $\sqrt{S}$ , which is introduced when estimating the option transition function and the option utility function using only the information of the primitive actions.
However, since dataset  $\mathcal{D}_2$  contains more information on the environment than  $\mathcal{D}_1$ , it has a weaker requirement on the behavior (hierarchical) policy and allows the evaluation of new options, which is illustrated as follows.
- Concentrability: Recall that the suboptimality bounds for both datasets build upon the sufficient coverage assumptions, i.e., Assumption 1 for  $\mathcal{D}_1$  and Assumption 2 for  $\mathcal{D}_2$ . While they are generally incomparable (as dataset  $\mathcal{D}_2$  can be collected by an arbitrary behavior policy), we focus on the case that both datasets are collected by the same hierarchical behavior policy  $\rho$ . Particularly, it can be shown that Assumption 2 is weaker than Assumption 1. Indeed, if  $\rho$
covers the trajectories of state-option-utility tuples induced by  $\mu^{*}$  (i.e., Assumption 1 holds), then it must have covered the trajectories of state-action-reward tuples induced by  $\mu^{*}$  (i.e., Assumption 2 holds). However, the opposite does not hold in general and we provide such an example in Appendix H.
- Evaluation of New Options: In the options literature, a popular task is offline option discovery (Ajay et al., 2021; Villecloze et al., 2022), i.e., designing new and useful options from the dataset. Therefore, an important problem is whether these new options can be evaluated through the dataset. We argue that dataset  $\mathcal{D}_2$  yields greater flexibility than  $\mathcal{D}_1$  in this case. Again, we assume that both datasets are collected by the same hierarchical behavior policy  $\rho$ . Unfortunately, one cannot use dataset  $\mathcal{D}_1$  to evaluate any  $(h,s,o)$  that is not visited by  $\rho$ , i.e.,  $\theta_h^\rho(s,o) = 0$ , let along evaluating the new options. However, this is not the case for dataset  $\mathcal{D}_2$ . In fact, any  $(h,s,o)$  can be evaluated if the visiting state-action pairs are also reachable by  $\rho$ , i.e.,  $\sum_{h\leq m\leq H,(s',a')\in S_{h,s,o}^m}1 / d_m^\rho (s',a') < \infty$ . An interesting problem is how to leverage the results in this paper to facilitate offline option discovery, which we shall research in the future work.
In this paper, we provide the first analysis of the sample complexity for offline RL with options. A novel information-theoretic lower bound is established, which generalizes the one for offline RL with actions. We derive near-optimal suboptimality bounds of the PEssimistic Value Iteration for Learning with Options (PEVIO) algorithm for two popular data-collection procedures. Our results show that options facilitate more sample-efficient learning than primitive actions in offline RL in both the finite-time convergence rate to the optimal value and the actual performance.
The work presented in this paper was partially supported by a research grant from the Research Grants Council, Hong Kong, China (RGC Ref. No. CUHK 14206820). Part of the research reported in this paper was done when the second author was with The Chinese University of Hong Kong. The authors are grateful to the anonymous reviewers for their helpful comments.
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. (2021). OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations.
Bagaria, A., Senthil, J., Slivinski, M., and Konidaris, G. (2021). Robustly learning composable options in deep reinforcement learning. In Zhou, Z.-H., editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2161-2169. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Bradtke, S. and Duff, M. (1994). Reinforcement learning methods for continuous-time Markov decision problems. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems, volume 7. MIT Press.
Brunskill, E. and Li, L. (2014). PAC-inspired option discovery in lifelong reinforcement learning. In Xing, E. P. and Jebara, T., editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 316-324, Beijing, China. PMLR.
Cai, Q., Yang, Z., Jin, C., and Wang, Z. (2020). Provably efficient exploration in policy optimization. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1283-1294. PMLR.
Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference
Yin, M., Bai, Y., and Wang, Y.-X. (2021). Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In Banerjee, A. and Fukumizu, K., editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1567-1575. PMLR.
Zhang, F., Jia, C., Li, Y.-C., Yuan, L., Yu, Y., and Zhang, Z. (2023). Discovering generalizable multiagent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations.