Title: Provably (More) Sample-Efficient Offline RL with Options
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning with Options

Offline RL

Provably Efficient RL
Abstract: The options framework yields empirical success in long-horizon planning problems of reinforcement learning (RL). Recent works show that options help improve the sample efficiency in online RL. However, these results are no longer applicable to scenarios where exploring the environment online is risky, e.g., automated driving and healthcare. In this paper, we provide the first analysis of the sample complexity for offline RL with options, where the agent learns from a dataset without further interaction with the environment. We derive a novel information-theoretic lower bound, which generalizes the one for offline learning with actions. We propose the PEssimistic Value Iteration for Learning with Options (PEVIO) algorithm and establish near-optimal suboptimality bounds for two popular data-collection procedures, where the first one collects state-option transitions and the second one collects state-action transitions. We show that compared to offline RL with actions, using options not only enjoys a faster finite-time convergence rate (to the optimal value) but also attains a better performance when either the options are carefully designed or the offline data is limited. Based on these results, we analyze the pros and cons of the data-collection procedures.

Provably (More) Sample-Efficient Offline RL with Options

Abstract

The options framework yields empirical success in long-horizon planning problems of reinforcement learning (RL). Recent works show that options improves the sample efficiency in online RL where the learner can actively explores the environment. However, these results are no longer applicable to scenarios where exploring the environment online is risky, e.g., automated driving and healthcare. In this paper, we provide the first analysis of the sample complexity for offline RL with options, where the agent learns from a dataset without further interaction with the environment. We propose the PE ssimistic V alue I teration for Learning with O ptions (PEVIO) algorithm and establish near-optimal suboptimality bounds (with respect to the novel information-theoretic lower bound for offline RL with options) for two popular data-collection procedures, where the first one collects state-option transitions and the second one collects state-action transitions. We show that compared to offline RL with actions, using options not only enjoys a faster finite-time convergence rate (to the optimal value) but also attains a better performance (when either the options are carefully designed or the offline data is limited). Based on these results, we analyze the pros and cons of the data-collection procedures, which may facilitate the selection in practice.

1 Introduction

Planning in long-horizon tasks is challenging in reinforcement learning (RL) (Co-Reyes et al., 2018; Eysenbach et al., 2019; Hoang et al., 2021). A line of study proposes to accelerate learning in these tasks using temporally-extended actions (Fikes et al., 1972; Sacerdoti, 1973; Drescher, 1991; Jiang et al., 2019; Nachum et al., 2019; Machado et al., 2021; Erraqabi et al., 2022). One powerful approach is the options framework introduced by Sutton et al. (1999), where the agent interacts with the environment with closed-loop policies called options . Empirical success (Tessler et al., 2017; Vezhnevets et al., 2017) shows that options help achieve sample-efficient performance in long-horizon planning problems.

To provide a theoretical guarantee to the options framework, recent works have focused on the sample complexity of RL with options in the online setting, where the agent continuously explores the environment and learns a hierarchical policy to select options. Brunskill and Li (2014) establish a PAC-like sample complexity of RL with options in the semi-Markov decision processes (SMDPs), where temporally-extended actions are treated as indivisible and unknown units. Later, Fruit and Lazaric (2017) provide the first regret analysis of RL with options under the Markov decision processes (MDPs) framework. While their proposed algorithm attains a sublinear regret, it requires prior knowledge of the environment, which is not usually available in practice. To address this problem, Fruit et al. (2017) propose an algorithm that does not require prior knowledge, yet achieves

Ho-fung Leung

a near-optimal regret bound. However, these results are inapplicable to many real-world scenarios where online exploration is not allowed. For example, it has been argued that in healthcare (Gottesman et al., 2019) and automated driving (Shalev-Shwartz et al., 2016), learning in an online manner is risky and costly. In these scenarios, offline learning, where the agent learns a policy from a dataset, is preferred. We note that there is a line of studies on the sample complexity of offline RL with primitive actions only (i.e., without the use of options) (Levine et al., 2020; Fu et al., 2020; Rashidinejad et al., 2021). Unfortunately, to the best of our knowledge, there have been no results reported on the offline RL with options.

In this paper, we make the following contributions. First, we derive a novel information-theoretic lower bound, which generalizes the one for offline learning with actions. Second, we propose the PE ssimistic V alue I teration for Learning with O ptions (PEVIO) algorithm and derive near-optimal suboptimality bounds for two popular data-collection procedures, where the first one collects stateoption transitions and the second one collects state-action transitions. More importantly, we show that options facilitate more sample-efficient learning in both the finite-time convergence rate and actual performance. To shed light on offline RL with options in practice, we discuss the pros and cons of both data-collection procedures based on our analysis.

2 Related Work

Learning with Options Building upon the theory of semi-Markov decision processes (SMDPs) (Bradtke and Duff, 1994; Mahadevan et al., 1997), Sutton et al. (1999) propose to learn with options. Following their seminal work, learning with options has been widely studied in the function approximation setting (Sorg and Singh, 2010) and hierarchical RL (Igl et al., 2020; Klissarov and Precup, 2021; Wulfmeier et al., 2021). Discovering useful options has also been the subject of extensive research (Stolle and Precup, 2002; Riemer et al., 2018; Mankowitz et al., 2018; Harb et al., 2018; Hiraoka et al., 2019; Bagaria et al., 2021). Despite its empirical success, there have been fairly limited studies on the sample efficiency of learning with options. Brunskill and Li (2014) analyze the sample complexity bound for an RMAX-like algorithm for SMDPs. Fruit and Lazaric (2017) derive the first regret analysis of learning with options. They propose an algorithm that attains sublinear regret in the infinite-horizon average-reward MDP while requiring prior knowledge of the environment. Later, Fruit et al. (2017) remove this requirement.

Offline RL In the offline setting, a dataset that is collected by executing a behavior policy in the environment is provided, and the agent is asked to learn a near-optimal policy using only this dataset. A key challenge in offline RL is the insufficient coverage of the dataset (Wang et al., 2021), which is also known as distributional shift (Chen and Jiang, 2019; Levine et al., 2020). To address this problem, the previous study on sample-efficient learning assumes uniform coverage of the dataset (Liu et al., 2018; Chen and Jiang, 2019; Jiang and Huang, 2020; Yang et al., 2020; Xie and Jiang, 2020; Uehara et al., 2020; Qu and Wierman, 2020; Yin et al., 2021). This assumption is relaxed in recent works by pessimism principle (Xie et al., 2021; Rashidinejad et al., 2021; Jin et al., 2021).

3 Preliminaries

3.1 Episodic MDP with Options

Let ∆( X ) denote the probability simplex on space X and [ N ] := { 1 , · · · , N } for any positive integer N . An episodic MDP with options is a sextuple M = ( S , A , O , H, P , r ) , where S is the state space, A the (primitive) action set, O the finite set of options, H the length of each episode, P = { P h : S × A ↦→ ∆( S ) } h ∈ [ H ] the transition kernel, r = { r h : S × A ↦→ [0 , 1] } h ∈ [ H ] the deterministic reward function. 1 We define S := |S| , A := |A| , and O := |O| . A (Markov) option o ∈ O is a pair ( π o , β o ) where π o = { π o h : S ↦→ ∆( A ) } h ∈ [ H ] is the option's policy and β o = { β o h : S ↦→ [0 , 1] } h ∈ [ H ] is the probability of the option's termination . For convenience, we define β o H +1 ( s ) = 1 for all ( s, o ) ∈ S × O , i.e., any option is terminated after the end of an

episode. We assume that the initial state s 1 is fixed . 2 Upon arriving at state s h at any timestep h ∈ [ H ] , if h = 1 (at the beginning of an episode), the agent selects option o 1 ∼ µ 1 ( ·| s 1 ) , where µ = { µ h : S ↦→ ∆( O ) } h ∈ [ H ] is a hierarchical policy to select an option at each state. Otherwise ( h ≥ 2 ), the agent first terminates option o h -1 with probability β o h -1 h ( s h ) . If option o h -1 is terminated, she then selects a new option o h ∼ µ h ( ·| s h ) according to the hierarchical policy µ . If option o h -1 is not terminated, the agent continues to use option o h -1 at timestep h , i.e., o h = o h -1 . After that, the agent takes action a h ∼ π o h h ( ·| s h ) , receives a reward r h := r h ( s h , a h ) , and transits to the next state s h +1 ∼ P h ( ·| s h , a h ) . An episode terminates at timestep H + 1 . A special case is that an action a is an option o , such that π o h ( a | s ) = 1 and β o h ( s ) = 1 for any ( h, s ) ∈ [ H ] ×S . For convenience, we use the notation O = A to represent that each option corresponds to an action, which is the case in RL with primitive actions.

To define the Q -function and the value function, we introduce some useful notations. 3 Let T = { T h : S × O ↦→ ∆( S × [ H -h + 1]) } h ∈ [ H ] and U = { U h : S × O ↦→ [0 , H ] } h ∈ [ H ] denote the option transition function and the option utility function , respectively. Particularly, for any ( h, s, o ) ∈ [ H ] ×S × O , the option transition function T h ( s ′ | s, o, τ ) is the probability that the agent uses option o at state s at timestep h , reaches state s ′ at timestep h + τ without terminating option o in these τ timesteps, and finally terminates option o at state s ′ at timestep h + τ . The option utility function U h ( s, o ) is the expected cumulative reward within timesteps that the option is used without being terminated. Given any arbitrary series of functions { y h : S ↦→ R } h ∈ [ H ] , define the operator [ T h y h + τ ]( s, o ) := ∑ s ′ ∈S T h ( s ′ | s, o, τ ) y h + τ ( s ′ ) for any ( s, o, τ ) ∈ S × O × [ H -h +1] . In the following, we derive the Q -function and the value function for learning with options. (The detailed proof can be found in Appendix B.)

Theorem 1 ( Q -function and value function) . For any hierarchical policy µ and ( h, s, o ) ∈ [ H ] × S × O , the Q -function is given by

and the value function is given by

where V µ H +1 ( s ) = Q µ H +1 ( s, o ) = 0 for any ( s, o ) ∈ S × O .

Intuitively, the first term U h ( s, o ) of the Q -function is the expected reward within timesteps that option o is used without being terminated, and the second term ∑ τ ∈ [ H -h +1] [ T h V µ h + τ ]( s, o ) corresponds to the expected reward within timesteps after option o is terminated and a new option is selected according to µ . It can be shown that there exists an optimal (and deterministic) hierarchical policy µ ∗ = { µ ∗ h : S ↦→ O} h ∈ [ H ] that attains the optimal value function, i.e., V ∗ h ( s ) = sup µ V µ h ( s ) for all ( h, s ) ∈ [ H ] ×S (Sutton et al., 1999).

3.2 Offline RL with Options

We consider learning with options in the offline setting. That is, given a dataset D that is collected by an experimenter through interacting with the environment, the algorithm outputs a hierarchical policy ̂ µ . The sample complexity is measured by the suboptimality , i.e., the shortfall in the value function of the hierarchical policy ̂ µ compared to that of the optimal hierarchical policy µ ∗ , which is given by

To derive a novel information-theoretic lower bound of SubOpt D , we first define some useful notations. For any hierarchical policy µ , we denote by θ µ = { θ µ h : S × O ↦→ [0 , 1] } h ∈ [ H ] its state-option

occupancy measure . That is, θ µ h ( s, o ) is the probability that the agent selects a particular option o at state s at timestep h (either when h = 1 or when the option o h -1 used at the timestep h -1 is terminated) when following the hierarchical policy µ . With a slight abuse of the notation, we denote by θ µ h ( s ) := ∑ o ∈O θ µ h ( s, o ) the state occupancy measure for any ( h, s ) ∈ [ H ] × S . Further, we define

where I [ · ] is the indicator function. Intuitively, Z µ O is the expected number of timesteps to alternate a new option and Z µ O is the maximal number of state-option pairs that can be visited, when following the hierarchical policy µ . The following proposition shows that options facilitate temporal abstraction and reduction of the state space.

Proposition 1. For any hierarchical policy µ , we have that Z µ O ≤ H . If µ is deterministic, i.e., µ = { µ h : S ↦→ O} h ∈ [ H ] , we further have that Z µ O ≤ HS . All the above equalities hold when O = A .

Next, we derive a novel information-theoretic lower bound of SubOpt D . The detailed proof can be found in Appendix C.

Theorem 2 (Information-theoretic lower bound) . Let ρ = { ρ h : S ↦→ ∆( O ) } h ∈ [ H ] denote any hierarchical behavior policy to collect the dataset. Define the class of problem instances

Suppose that C option ≥ 2 , z ∗ ≥ 1 , and z ∗ ≥ ⌊ z ∗ ⌋ S , where ⌊ x ⌋ := max { n ∈ N : n ≤ x } is the largest integer no greater than x ∈ R . Then, there exists an absolute constant c 0 such that for any offline algorithm that outputs a hierarchical policy ̂ µ , if the number of episodes

then there exists a problem instance ( M,ρ ) ∈ M ( C option , z ∗ , z ∗ ) on which the hierarchical policy ̂ µ suffers from ϵ -suboptimality, that is,

where the expectation E M is with respect to the randomness during the execution of ρ within MDP M .

Theorem 2 shows that, when dataset D sufficiently covers the trajectories induced by µ ∗ , i.e., max h,s,o θ µ ∗ h ( s, o ) /θ ρ h ( s, o ) ≤ C option , at least Ω( C option HZ ∗ O Z ∗ O /ϵ 2 ) episodes are required to learn an ϵ -optimal hierarchical policy from dataset D . Note that when O = A , it recovers the lower bound Ω( H 3 SC ∗ /ϵ 2 ) for offline RL with primitive actions, where C ∗ is the concentrability defined therein.

4 The PEVIO Algorithm

Inspired by the Pessimistic Value Iteration (PEVI) algorithm (Jin et al., 2021), we propose the PE ssimistic V alue I teration for Learning with O ptions (PEVIO) in Algorithm 1. Given a dataset D and the corresponding O ffline O ption E valuation (OOE) subroutine, whose details are specified in Sections 5.1 and 5.2, PEVIO outputs a hierarchical policy ̂ µ = { ̂ µ h : S ↦→ ∆( O ) } h ∈ [ H ] .

To estimate the Q -function, given a dataset D , PEVIO first constructs ( ̂ T, ̂ U, Γ) by the OOE subroutine (line 3). Specifically, ̂ T h and ̂ U h are the empirical counterparts of T h and U h presented in the Q -function given by Equation (1), respectively. In addition, Γ is a penalty function computed based on dataset D . We remark that the OOE subroutine varies when different data-collecting procedures

are considered and we provide the details in Sections 5.1 and 5.2, respectively. Given ̂ U h , ̂ T h , and Γ h , the estimated Q -function ̂ Q h is the derived (lines 6 and 7). Particularly, Q h computed in line 6 can be seen as first replacing U h and T h with their empirical counterparts ̂ U h , ̂ T h in Equation (1), and then subtracting the penalty function Γ h . Further, a hierarchical policy ̂ µ h is constructed greedily with ̂ Q h (line 10), where ⟨ f ( · ) , g ( · ) ⟩ O := ∑ o ∈O f ( o ) g ( o ) for any arbitrary functions f, g defined on O . Finally, given ̂ Q h and ̂ µ h , the corresponding estimated value function ̂ V h is computed (line 11). To analyze the suboptimality of the hierarchical policy ̂ µ output from PEVIO, we first provide the the following definition, which motivates the design of the penalty function Γ .

Definition 1 ( ξ -uncertainty quantifier for dataset D ) . The penalty function Γ = { Γ h : S × O ↦→ R + } h ∈ [ H ] output from the OOE subroutine in Algorithm 1 (line 3) is said to be a ξ -uncertainty quantifier with respect to P D if the following event

satisfies that P D ( E ) ≥ 1 -ξ , where P D is the joint distribution of the data collecting process.

In other words, the penalty function Γ is a ξ -uncertainty quantifier if it upper bounds the estimation errors in the empirical option transition function ̂ T and the empirical option utility function ̂ U . Next, we show that the suboptimality of ̂ µ output from PEVIO is upper bounded if Γ is a ξ -uncertainty quantifier. (The detailed proof can be found in Appendix D.)

Theorem 3 (Suboptimality of learning with options using dataset D ) . Let ̂ µ denote the hierarchical policy output by Algorithm 1. Suppose that Γ = { Γ h } h ∈ [ H ] output from the OOE subroutine is a ξ -uncertainty quantifier. Conditioned on the successful event E defined in Equation (5), which satisfies that P D ( E ) ≥ 1 -ξ , we have that

where E µ ∗ [ g ( s h , o h )] = ∑ ( s,o ) θ µ ∗ h ( s, o ) g ( s, o ) for any h ∈ [ H ] and arbitrary function g : S×O ↦→ R .

Remark 1 . Since the temporal structure of learning with options is much more complex than learning with actions, PEVIO is significantly different from the algorithms proposed for offline RL with primitive actions, such as PEVI (Jin et al., 2021) or VI-LCB (Xie et al., 2021), despite sharing a similar intuition. First, in terms of the algorithm design, PEVI and VI-LCB estimate (one-step) transition kernel and reward function to compute the Q -function of a state-action pair. However, by Equation (1), the Q -function of a state-option pair depends on multi-step transitions and rewards.

Hence, it is challenging to design the OOE subroutine and analyze the estimated ( ̂ T, ̂ U, Γ) for options. Indeed, if the dataset contains ( s, o, u ) (state-option-utility) tuples, then ( T, U, Γ) can be estimated similarly to the case of learning with primitive actions. However, if the dataset contains only ( s, a, r ) (state-action-reward) tuples, then it remains elusive to estimate and analyze ( T, U, Γ) . Second, in terms of the suboptimality analysis, previous works on offline RL with primitive actions rely on the extended value difference lemma (Cai et al., 2020, Lemma 4.2), which also depends on the one-step temporal structure of actions and cannot be directly applied to our setting. Hence, to derive Theorem 3, it is non-trivial to generalize the extended value difference lemma to the options framework (See Lemma 8 in the Appendices).

5 Data-Collection and Suboptimality Analysis

In this section, we consider two data-collection procedures that are widely deployed in the options literature. The first one collects state-option-utility tuples (dataset D 1 ) and similar datasets are utilized in the work of Zhang et al. (2023). The second one collects state-action-reward tuples (dataset D 2 ) and is studied in a line of works (Ajay et al., 2021; Villecroze et al., 2022; Salter et al., 2022). Intuitively, dataset D 1 requires smaller storage and enables efficient evaluation of the options, while dataset D 2 provides richer information on the environment and even facilitates the evaluation of new options. For each dataset, we design the corresponding OOE subroutine and derive a suboptimality bound for the PEVIO algorithm. Based on these results, we further discuss the advantages and the disadvantages of both data-collection procedures, which sheds light on offline RL with options in practice.

5.1 Learning from State-Option Transitions

We consider dataset D 1 := { ( s k t k i , o k t k i , u k t k i ) } i ∈ [ j k ] ,k ∈ [ K ] consisting of state-option-utility tuples, which is collected by the experimenter's interaction with the environment for K episodes using a hierarchical behavior policy ρ = { ρ h : S ↦→ ∆( O ) } h ∈ [ H ] . More precisely, at timestep t k i of the k th episode, the experimenter selects a new option o k t k i , uses it for ( t k i +1 -t k i ) timesteps, collects a cumulative reward of u k t k i within these ( t k i +1 -t k i ) timesteps, and finally terminates this option at state s k t k i +1 at timestep t k i +1 . For convenience, we define t k j k +1 := H +1 for any k ∈ [ K ] .

Let a ∨ b := max { a, b } for any pair of integers a, b ∈ N . When dataset D 1 is available, the OOE subroutine in Algorithm 1 is given by Subroutine 2. Particularly, Subroutine 2 incorporates the data splitting technique (Xie et al., 2021) (line 2). That is, given dataset D 1 , the algorithm randomly splits it into H subdatasets {D 1 ,h } h ∈ [ H ] . Then ̂ T h and ̂ U h are constructed using subdataset D 1 ,h (lines 4-7).

To derive the suboptimality for ̂ µ output from PEVIO, we follow the previous study and make a standard assumption on the coverage of dataset D 1 .

Assumption 1 (Single hierarchical policy concentrability for dataset D 1 ) . The experimenter collects dataset D 1 by following a hierarchical behavior policy ρ = { ρ h : S ↦→ ∆( O ) } h ∈ [ H ] . There exists some deterministic optimal hierarchical policy µ ∗ such that

(with the convention 0 / 0 = 0 ) is finite.

In other words, Assumption 1 states that dataset D 1 sufficiently covers the trajectories of state-optionutility tuples induced by some deterministic optimal hierarchical policy µ ∗ . We derive an upper bound of SubOpt D 1 in the following theorem. (The detailed proof can be found in Appendix E.)

Theorem 4 (Suboptimality for dataset D 1 ) . Under Assumption 1, with probability at least 1 -ξ , we have that

Subroutine 2 O ffline O ption E valuation (OOE) for Dataset D 1

1: Input: Dataset D 1 = { ( s k t k i , o k t k i , u k t k i ) } i ∈ [ j k ] ,k ∈ [ K ] .

̸

2: Initialize: Randomly split the dataset D into H subdatasets {D 1 ,h } h ∈ [ H ] with |D 1 ,h | = K/H . More precisely, let l := { l h } h ∈ [ H ] be a random partition of the set [ K ] , where l h := { l h,j } j ∈ [ K/H ] ⊂ [ K ] is uniformly sampled from [ K ] such that ∪ h ∈ [ H ] l h = [ K ] and l h ∩ l h ′ = ∅ for any h = h ′ . Then we have that D 1 ,h = { ( s k t k i , o k t k i , u k t k i ) } i ∈ [ j k ] ,k ∈ l h for any h ∈ [ H ] . Let n h ( s, o ) := ∑ k ∈ l h I [ h ∈ { t k i } i ∈ [ j k ] , s k h = s, o k h = o ] denote the number of times that the experimenter selects a particular option o at state s at timestep h in subdataset D 1 ,h .

3: for ( h, s, o ) ∈ [ H ] ×S × O do

4: for ( s ′ , τ ) ∈ S × [ H -h +1] do

5: ̂ T h ( s ′ | s, o, τ ) ← ∑ k ∈ l h I [ h ∈{ t k i } i ∈ [ j k ] ,s k h = s,o k h = o,s k h + τ = s ′ ] 1 ∨ n h ( s,o )

6: end for

7: ̂ U h ( s, o ) ← ∑ k ∈ l h I [ h ∈{ t k i } i ∈ [ j k ] ,s k h = s,o k h = o ] u k h 1 ∨ n h ( s,o )

8: Γ h ( s, o ) ← ˜ O (√ H 2 n h ( s,o ) ∨ 1 )

9: end for

10: Output: ( ̂ T = { ̂ T h } h ∈ [ H ] , ̂ U = { ̂ U h } h ∈ [ H ] , Γ = { Γ h } h ∈ [ H ] ) .

Compared to the lower bound in Theorem 2, suboptimality bound (8) is near-optimal except for an extra factor of H . 4 More importantly, it shows that learning with options enjoys a faster convergence rate to the optimal value than learning with primitive actions. Recall that the VI-LCB algorithm (Xie et al., 2021) that learns with primitive actions attains the suboptimality bound ˜ O ( √ H 5 SC ∗ /K ) , where C ∗ is the concentrability defined therein. When ignoring the concentrability parameters, the suboptimality bound (8) is smaller since Z ∗ O ≤ H and Z ∗ O ≤ HS .

Remark 2 . While, in the worst case, both Z ∗ O and Z ∗ O can scale with H and HS , respectively, we note that in many long-horizon planning problems, they often scale with the number of sub-tasks , which are greatly smaller, especially for tasks that enables temporal abstraction and the reduction of the state space. For example, while the route-planning task of going from City A to City B by transportation takes thousands of primitive actions to finish, it can be efficiently solved by decomposing into the following sub-tasks: (1) going to the airport/train station in City A; (2) taking transportation to City B; and (3) reaching the final destination in City B, for which options are designed. In this case, both Z ∗ O and Z ∗ O /S may only scale as o ( H ) . In other words, options facilitate more sample-efficient learning through temporal abstraction, i.e., sticking to an option until a sub-task is finished. Another concrete example is solving a maze, where options are often designed to move agents to bottleneck states (¸ Sim¸ sek and Barto, 2004; Solway et al., 2014; Machado et al., 2017) that connect different densely connected regions of the state space, e.g., doorways. In this case, while the number of option switches may grow proportionally to H , i.e., Z ∗ O /H = O (1) , the number of states to switch options can be greatly smaller than S , i.e., Z ∗ O /H = o ( s ) . That is to say, options help improve the sample complexity by the reduction of the state space.

Further, we show that learning with options attains a better performance than learning with primitive actions, when either the options are carefully designed or the offline data is limited.

Corollary 1 (Better performance) . Let TrueSubOpt D 1 ( ̂ µ, s 1 ) := V ∗ , pri 1 ( s 1 ) -V ̂ µ 1 ( s 1 ) , where V ∗ , pri is the optimal value function defined for the primitive actions. Ignoring the concentrability parameters, we have that TrueSubOpt D 1 ( ̂ µ, s 1 ) ≤ ˜ O ( √ H 5 SC ∗ /K ) attained by the VI-LCB algorithm (Xie et al., 2021), when either the options are carefully designed (i.e., ∆ O ( s 1 ) := V ∗ , pri 1 ( s 1 ) -V ∗ 1 ( s 1 ) = 0 ) or

the number K of trajectories in the dataset is where ( x ) := max { 0 , x } for any x ∈ R

Corollary 1 implies that when data is limited, e.g., in cases where the data collection is highly expensive or risky, learning with options is beneficial since the output hierarchical policy yields a higher value than learning with primitive actions.

5.2 Learning from State-Action Transitions

We consider dataset D 2 := { ( s k h , a k h , r k h ) } h ∈ [ H ] ,k ∈ [ K ] consisting of state-action-reward tuples, which is collected by an experimenter's interaction with the environment for K episodes using any arbitrary behavior policy. That is, the experimenter takes action a k h at state s k h at timestep h of the k th episode, receives a reward of r k h , and transits to state s k h +1 .

When dataset D 2 is provided, the OOE subroutine in Algorithm 1 is given by Subroutine 3. Note that one difficulty is that we cannot directly estimate the option transition function and the option utility function from dataset D 2 as it only includes the information of the primitive actions. Hence, Subroutine 3 first constructs the empirical transition kernel ̂ P = { ̂ P h } h ∈ [ H ] and the empirical reward function ̂ r = { ̂ r h } h ∈ [ H ] (lines 4-7), and use them to further construct ̂ T h and ̂ U h (lines 8-20). To

Subroutine 3 O ffline O ption E valuation (OOE) for Dataset D 2

1: Input: Dataset D 2 = { ( s k h , a k h , r k h ) } h ∈ [ H ] ,k ∈ [ K ] . 2: Initialize: ̂ U H +1 ( · ) ← 0 . Let N h ( s, a ) := ∑ K k =1 I [ s k h = s, a k h = a ] denote the number of visits of state-action pair ( s, a ) at timestep h in dataset D 2 . Function ϕ := { ϕ h : S × O ↦→ R } h ∈ [ H ] is given by Equation (32) in the Appendices. 3: for h = H,H -1 , ..., 1 do 4: for ( s, a ) ∈ S × A do 5: ̂ P h ( s ′ | s, a ) ← ∑ k ∈ [ K ] I [ s k h = s,a k h = a,s k h +1 = s ′ ] 1 ∨ N h ( s,a ) for any s ′ ∈ S 6: ̂ r h ( s, a ) ← I [ N h ( s, a ) ≥ 1] r h ( s, a ) 7: end for 8: for ( s, o, s ′ ) ∈ S × O × S do 9: ̂ T h ( s ′ | s, o, 1) ← β o h +1 ( s ′ ) ∑ a ∈A π o h ( a | s ) ̂ P h ( s ′ | s, a ) 10: ̂ T h ( s ′ | s, o, 1) ← (1 -β o h +1 ( s ′ )) ∑ a ∈A π o h ( a | s ) ̂ P h ( s ′ | s, a ) 11: for l = 2 , · · · , H -h +1 do 12: ̂ T h ( s ′ | s, o, l ) ← ∑ a π o h ( a | s ) ∑ s ′′ ̂ P h ( s ′′ | s, a ) ( 1 -β o h +1 ( s ′′ ) ) ̂ T h +1 ( s ′ | s ′′ , o, l -1) 13: ̂ T h ( s ′ | s, o, l ) ← ∑ a π o h ( a | s ) ∑ s ′′ ̂ P h ( s ′′ | s, a ) ( 1 -β o h +1 ( s ′′ ) ) ̂ T h +1 ( s ′ | s ′′ , o, l -1) 14: end for 15: end for 16: end for 17: for ( h, s, o ) ∈ [ H ] ×S × O do 18: Γ h ( s, o ) ← ˜ O ( √ ∑ H m = h ∑ ( s,a ) ∈X m h,s,o H 3 S N m ( s,a ) ∨ 1 + Hϕ h ( s, o ) ) 19: ̂ U h ( s, o ) ← ∑ a ∈A π o h ( a | s ) ̂ r h ( s, a ) + ∑ s ′ ∈S ̂ T h ( s ′ | s, o, 1) ̂ U h +1 ( s ′ , o ) 20: end for 21: Output: ( ̂ T = { ̂ T h } h ∈ [ H ] , ̂ U = { ̂ U h } h ∈ [ H ] , Γ = { Γ h } h ∈ [ H ] ) .

derive the suboptimality, we first define some useful notations. For any ( h, s, o ) ∈ [ H ] ×S × O and h ≤ m ≤ H , we denote by X m h,s,o the set of state-action pairs that can be reached at timestep m by using option o at state s and timestep h without being terminated. 5 Further, let d ρ := { d ρ h : S ×A ↦→

[0 , 1] } h ∈ [ H ] denote the state-action distribution of the behavior policy ρ used by the experimenter. That is, d ρ h ( s, a ) is the probability that the agent takes action a at state s at timestep h . Similarly, we make the following assumption on dataset D 2 .

Assumption 2 (Single hierarchical policy concentrability for dataset D 2 ) . The experimenter collects dataset D 2 by following an arbitrary behavior policy ρ . There exists some deterministic optimal hierarchical policy µ ∗ such that

(with the convention 0 / 0 = 0 ) is finite.

Intuitively, Assumption 2 states that dataset D 2 sufficiently covers the trajectories of state-actionreward tuples induced by the optimal hierarchical policy µ ∗ . Next, we derive an upper bound of SubOpt D 2 under Assumption 2. The detailed proof can be found in Appendix F.

Theorem 5 (Suboptimality for dataset D 2 ) . Under Assumption 2, with probability at least 1 -ξ , we have that which translates to

when K of dataset D 2 is sufficiently large, i.e., K ≥ ˜ O ( C option 2 H 5 S 9 A 2 O 2 / ( Z ∗ O Z ∗ O )) , where Z ∗ O = Z µ ∗ O and Z ∗ O = Z µ ∗ O .

While, in general, suboptimality bound (10) does not compare favorably against the suboptimality ˜ O ( √ H 5 SC ∗ /K ) attained by the VI-LCB algorithm that learns with primitive actions, we argue that it can be better in long-horizon problems where the horizon H is much greater than the cardinality of the state space S .

5.3 Further Discussion

We analyze the pros and cons of both data-collection procedures, which sheds light on offline RL with options in practice. Compared to D 2 , dataset D 1 requires smaller storage and enjoys faster convergence to the optimal value, which is further illustrated as follows.

Storage: For dataset D 2 = { ( s k h , a k h , r k h ) } h ∈ [ H ] ,k ∈ [ K ] , the storage is simply HK . However, for dataset D 1 = { ( s k t k i , o k t k i , u k t k i ) } i ∈ [ j k ] ,k ∈ [ K ] , its expected size is K · Z ρ O ≤ HK , where ρ is the hierarchical behavior policy. Therefore, in the case of a small Z ρ O , dataset D 1 requires much smaller storage than D 2 (on average).

Suboptimality: Ignoring the concentrability, the suboptimality bound (10) for dataset D 2 is worse than the suboptimality bound (8) for dataset D 1 by a factor of √ S , which is introduced when estimating the option transition function and the option utility function using only the information of the primitive actions.

However, since dataset D 2 contains more information on the environment than D 1 , it has a weaker requirement on the behavior (hierarchical) policy and allows the evaluation of new options, which is illustrated as follows.

Concentrability: Recall that the suboptimality bounds for both datasets build upon the sufficient coverage assumptions, i.e., Assumption 1 for D 1 and Assumption 2 for D 2 . While they are generally incomparable (as dataset D 2 can be collected by an arbitrary behavior policy), we focus on the case that both datasets are collected by the same hierarchical behavior policy ρ . Particularly, it can be shown that Assumption 2 is weaker than Assumption 1. Indeed, if ρ

covers the trajectories of state-option-utility tuples induced by µ ∗ (i.e., Assumption 1 holds), then it must have covered the trajectories of state-action-reward tuples induced by µ ∗ (i.e., Assumption 2 holds). However, the opposite does not hold in general and we provide such an example in Appendix H.

Evaluation of New Options: In the options literature, a popular task is offline option discovery (Ajay et al., 2021; Villecroze et al., 2022), i.e., designing new and useful options from the dataset. Therefore, an important problem is whether these new options can be evaluated through the dataset. We argue that dataset D 2 yields greater flexibility than D 1 in this case. Again, we assume that both datasets are collected by the same hierarchical behavior policy ρ . Unfortunately, one cannot use dataset D 1 to evaluate any ( h, s, o ) that is not visited by ρ , i.e., θ ρ h ( s, o ) = 0 , let along evaluating the new options. However, this is not the case for dataset D 2 . In fact, any ( h, s, o ) can be evaluated if the visiting state-action pairs are also reachable by ρ , i.e., ∑ h ≤ m ≤ H, ( s ′ ,a ′ ) ∈S m h,s,o 1 /d ρ m ( s ′ , a ′ ) < ∞ . An interesting problem is how to leverage the results in this paper to facilitate offline option discovery, which we shall research in the future work.

6 Conclusions

In this paper, we provide the first analysis of the sample complexity for offline RL with options. A novel information-theoretic lower bound is established, which generalizes the one for offline RL with actions. We derive near-optimal suboptimality bounds of the PE ssimistic V alue I teration for Learning with O ptions (PEVIO) algorithm for two popular data-collection procedures. Our results show that options facilitate more sample-efficient learning than primitive actions in offline RL in both the finite-time convergence rate to the optimal value and the actual performance.

Acknowledgments and Disclosure of Funding

The work presented in this paper was partially supported by a research grant from the Research Grants Council, Hong Kong, China (RGC Ref. No. CUHK 14206820). Part of the research reported in this paper was done when the second author was with The Chinese University of Hong Kong. The authors are grateful to the anonymous reviewers for their helpful comments.

References

Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. (2021). OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations .

Bagaria, A., Senthil, J., Slivinski, M., and Konidaris, G. (2021). Robustly learning composable options in deep reinforcement learning. In Zhou, Z.-H., editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2161-2169. International Joint Conferences on Artificial Intelligence Organization. Main Track.

Bradtke, S. and Duff, M. (1994). Reinforcement learning methods for continuous-time Markov decision problems. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems , volume 7. MIT Press.

Brunskill, E. and Li, L. (2014). PAC-inspired option discovery in lifelong reinforcement learning. In Xing, E. P. and Jebara, T., editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 316-324, Bejing, China. PMLR.

Cai, Q., Yang, Z., Jin, C., and Wang, Z. (2020). Provably efficient exploration in policy optimization. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1283-1294. PMLR.

Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference

on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1042-1051. PMLR.

Co-Reyes, J., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., and Levine, S. (2018). Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1009-1018. PMLR.

¸ Sim¸ sek, O. and Barto, A. G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning , ICML '04, page 95, New York, NY, USA. Association for Computing Machinery.

Drescher, G. L. (1991). Made-up minds: a constructivist approach to artificial intelligence . MIT press.

Erraqabi, A., Machado., M. C., Zhao, M., Sukhbaatar, S., Lazaric, A., Ludovic, D., and Bengio, Y. (2022). Temporal abstractions-augmented temporally contrastive learning: An alternative to the Laplacian in RL. In Cussens, J. and Zhang, K., editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , volume 180 of Proceedings of Machine Learning Research , pages 641-651. PMLR.

Eysenbach, B., Salakhutdinov, R. R., and Levine, S. (2019). Search on the replay buffer: Bridging planning and reinforcement learning. In Wallach, H., Larochelle, H., Beygelzimer, A., d'AlchéBuc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.

Fikes, R. E., Hart, P. E., and Nilsson, N. J. (1972). Learning and executing generalized robot plans. Artificial Intelligence , 3:251-288.

Fruit, R. and Lazaric, A. (2017). Exploration-Exploitation in MDPs with Options. In Singh, A. and Zhu, J., editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 576-584. PMLR.

Fruit, R., Pirotta, M., Lazaric, A., and Brunskill, E. (2017). Regret minimization in mdps with options without prior knowledge. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning.

Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., and Celi, L. A. (2019). Guidelines for reinforcement learning in healthcare. Nature Medicine , 25(1):16-18.

Harb, J., Bacon, P.-L., Klissarov, M., and Precup, D. (2018). When waiting is not an option: Learning options with a deliberation cost. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1).

Hiraoka, T., Imagawa, T., Mori, T., Onishi, T., and Tsuruoka, Y. (2019). Learning robust options by conditional value at risk optimization. In Wallach, H., Larochelle, H., Beygelzimer, A., d'AlchéBuc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.

Hoang, C., Sohn, S., Choi, J., Carvalho, W., and Lee, H. (2021). Successor feature landmarks for long-horizon goal-conditioned reinforcement learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 26963-26975. Curran Associates, Inc.

Igl, M., Gambardella, A., He, J., Nardelli, N., Siddharth, N., Boehmer, W., and Whiteson, S. (2020). Multitask soft option learning. In Peters, J. and Sontag, D., editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) , volume 124 of Proceedings of Machine Learning Research , pages 969-978. PMLR.

Jiang, N. and Huang, J. (2020). Minimax value interval for off-policy evaluation and policy optimization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 2747-2758. Curran Associates, Inc.

Jiang, Y., Gu, S. S., Murphy, K. P., and Finn, C. (2019). Language as an abstraction for hierarchical deep reinforcement learning. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.

Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. (2020). Learning adversarial Markov decision processes with bandit feedback and unknown transition. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4860-4869. PMLR.

Jin, Y., Yang, Z., and Wang, Z. (2021). Is pessimism provably efficient for offline RL? In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5084-5096. PMLR.

Klissarov, M. and Precup, D. (2021). Flexible option learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 4632-4646. Curran Associates, Inc.

Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems.

Liu, Q., Li, L., Tang, Z., and Zhou, D. (2018). Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.

Machado, M. C., Barreto, A., and Precup, D. (2021). Temporal abstraction in reinforcement learning with the successor representation.

Machado, M. C., Bellemare, M. G., and Bowling, M. (2017). A Laplacian framework for option discovery in reinforcement learning. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2295-2304. PMLR.

Mahadevan, S., Marchalleck, N., Das, T. K., and Gosavi, A. (1997). Self-improving factory simulation using continuous-time average-reward reinforcement learning. In Proc. 14th International Conference on Machine Learning , pages 202-210.

Mankowitz, D., Mann, T., Bacon, P.-L., Precup, D., and Mannor, S. (2018). Learning robust options. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1).

Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. (2019). Why does hierarchy (sometimes) work so well in reinforcement learning?

Qu, G. and Wierman, A. (2020). Finite-time analysis of asynchronous stochastic approximation and q -learning. In Abernethy, J. and Agarwal, S., editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 3185-3205. PMLR.

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 11702-11716. Curran Associates, Inc.

Riemer, M., Liu, M., and Tesauro, G. (2018). Learning abstract options. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.

Sacerdoti, E. D. (1973). Planning in a hierarchy of abstraction spaces. Artif. Intell. , 5:115-135.

Salter, S., Wulfmeier, M., Tirumala, D., Heess, N., Riedmiller, M., Hadsell, R., and Rao, D. (2022). Mo2: Model-based offline options. In Chandar, S., Pascanu, R., and Precup, D., editors, Proceedings of the 1st Conference on Lifelong Learning Agents , volume 199 of Proceedings of Machine Learning Research , pages 902-919. PMLR.

Shalev-Shwartz, S., Shammah, S., and Shashua, A. (2016). Safe, multi-agent, reinforcement learning for autonomous driving.

Solway, A., Diuk, C., Córdova, N., Yee, D., Barto, A. G., Niv, Y., and Botvinick, M. M. (2014). Optimal behavioral hierarchy. PLOS Computational Biology , 10(8):1-10.

Sorg, J. and Singh, S. (2010). Linear options. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1 , pages 31-38.

Stolle, M. and Precup, D. (2002). Learning options in reinforcement learning. In Proceedings of the 5th International Symposium on Abstraction, Reformulation and Approximation , page 212-223, Berlin, Heidelberg. Springer-Verlag.

Sutton, R. S., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112(1):181-211.

Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., and Mannor, S. (2017). A deep hierarchical approach to lifelong learning in minecraft. Proceedings of the AAAI Conference on Artificial Intelligence , 31(1).

Uehara, M., Huang, J., and Jiang, N. (2020). Minimax weight and Q-function learning for off-policy evaluation. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9659-9668. PMLR.

Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. (2017). FeUdal networks for hierarchical reinforcement learning. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3540-3549. PMLR.

Villecroze, V., Braviner, H., Naderian, P., Maddison, C., and Loaiza-Ganem, G. (2022). Bayesian nonparametrics for offline skill discovery. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 22284-22299. PMLR.

Wang, R., Foster, D., and Kakade, S. M. (2021). What are the statistical limits of offline RL with linear function approximation? In International Conference on Learning Representations .

Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdolmaleki, A., Hertweck, T., Neunert, M., Tirumala, D., Siegel, N., Heess, N., and Riedmiller, M. (2021). Data-efficient hindsight off-policy option learning. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11340-11350. PMLR.

Xie, T. and Jiang, N. (2020). Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Peters, J. and Sontag, D., editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) , volume 124 of Proceedings of Machine Learning Research , pages 550-559. PMLR.

Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2021). Policy finetuning: Bridging sampleefficient offline and online reinforcement learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 27395-27407. Curran Associates, Inc.

Yang, M., Nachum, O., Dai, B., Li, L., and Schuurmans, D. (2020). Off-policy evaluation via the regularized lagrangian. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 6551-6561. Curran Associates, Inc.

Yin, M., Bai, Y., and Wang, Y.-X. (2021). Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In Banerjee, A. and Fukumizu, K., editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 1567-1575. PMLR.

Zhang, F., Jia, C., Li, Y .-C., Yuan, L., Yu, Y ., and Zhang, Z. (2023). Discovering generalizable multiagent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the sample complexity of offline reinforcement learning (RL) with options. An information-theoretic lower bound for the suboptimality of the expected return of the policy over options that is learned offline is derived (by constructing hard MDP instances). The main contribution of the paper is the PEVIO algorithm, a version of value iteration for offline learning with options, and its theoretical analysis. The main idea behind PEVIO (borrowed from the PEVI algorithm for offline RL with primitive actions) is to be pessimistic in the face of uncertainty arising from the finite, fixed dataset. This pessimism is implemented by subtracting a penalty function from the value estimates which bounds the error in the estimates for the option transition function and option utilities. Suboptimality bounds are derived for two versions of PEVIO, intended for learning from state-option-utility and state-action-reward triplets. Depending on the frequency of option switches and the optimality of the option set, these bounds show that offline learning with options can be more sample efficient compared to learning with primitive actions.\n\nSTRENGTHS:\nAs offline RL is highly relevant for many applications and hierarchical RL with the options framework promises to speed up learning, a theoretical analysis of the sample complexity offline RL with options is of interest to the RL community. To my knowledge, the results in the paper are novel and relevant. The paper is comprehensive in the sense that it presents both an information-theoretic lower bound and a concrete algorithm with suboptimality bounds. The advantages and disadvantages of storing state-option or state-action transitions are discussed which are relevant for practical applications. The majority of the paper is well written and the presentation is well structured.\n\nWEAKNESSES:\nFrom the perspective of somebody mainly interested in practical algorithms and applications, PEVIO seems to be quite far away from an algorithm that would perform well in practice. In particular the data splitting technique discussed in line 214 seems to effectively throw away a large fraction $\\frac{H – 1}{H}$ of the data as it randomly splits the data into $H$ subsets and estimates $\\hat{T}\\_h$ and $\\hat{U}\\_h$  solely from the subset with index $h$. As a reader, I would be interested in learning whether such a procedure would be required in practice or is merely a tool to facilitate theoretical analysis. When learning from state-action transitions, it is furthermore assumed that the sets $\\mathcal{S}^m_{h, s, o}$ (footnote 5) are known in advance. Moreover, the option transition function and utilities are obtained by first learning a one-step model which would suffer from compounding model errors. \n\nI would like to better understand the claim that PEVIO has near-optimal suboptimality. Equation (8) looks like it has an additional factor of $H^2$ under the square root which is quite significant. In practice, $Z^*_\\mathcal{O}$ and $\\bar{Z}^*_\\mathcal{O}$ can be expected to scale with $H$ as well which results in a factor of $H^5$ under the square root which is identical to $VI-LCB$. This would also mean that Corollary 5.3 would not make sense but that it actually depends on the constants whether (8) or the bound for VI-LCB is better. For these reasons I am not quite convinced that the claim “Therefore, Theorems 5.2 and Corollary 5.3 combine to show that [...], options facilitate more sample-efficient learning than primitive actions in both the convergence rate to the optimal value and the actual performance.” is correct with respect to the convergence rate. \n\nFurthermore, the summation in equation (9) seems to effectively introduce another factor of $H$ which is absorbed into a constant. The requirement on the number of episodes in Theorem 5.5 seems to be quite excessive because it scales with $S^9$ which is quite extreme and could never be satisfied in practice.\nIn general I would expect options to speed things up by a constant factor which is effectively the frequency of option switches. As I do not see why this frequency should automatically decrease on long horizon problems I am not sure if the claim about a faster convergence rate in the conclusion is sufficiently backed.\n\nTypos:\n* In line 116 there is a superfluous space before the comma.\n\nQUESTIONS:\n* I would suggest to explicitly spell out in the text that the reward is assumed to be in the range $[0, 1]$ as it is easy to overlook this assumption when it is absorbed into the formal definition of the reward function.\n* In Subroutine 3, would it also be possible to learn the option model more directly (without relying on a model for primitive actions), perhaps using importance sampling?\n\nLIMITATIONS:\nA dedicated paragraph or section about limitations is lacking at the moment. I feel like this would help the reader to better place the contributions of the paper. In particular a clear discussion of how the suboptimality bounds for PEVIO compare to each other and to VI-LCB with realistic assumptions on the option switching frequency would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze the sample complexity for offline reinforcement learning with options. In particular, the sample complexity is measured by the suboptimality (the shortfall in the value function) of a hierarchical policy compared to the optimal hierarchical policy. The authors propose the pessimistic Value Iteration for Learning with Options algorithm and propose bounds when using two popular data collection methods: one collecting state-option-utility tuples and one collecting state-action-reward tuples. The paper shows that using options can lead to a faster convergence rate (to the value of the optimal hierarchical policy over given options).\n\nSTRENGTHS:\n- The paper investigates an important problem in the HRL literature: how does using options impact the efficiency of offline learning algorithms, considering different data collection approaches. \n- The paper is well-written, drawing parallels with existing work whenever possible.\n- The paper is very thorough with the mathematical formulation, assumptions, and definitions.\n\nWEAKNESSES:\nThe theory is based on the assumption that a set of options is already available, and it focuses on the suboptimality of the hierarchical policy over those options (when compared to the optimal hierarchical policy over those options). Thus, the overall performance of the final hierarchical policy is dependent on the quality of the underlying options (as their policy is not being updated). Also, the intuition behind some of the other assumptions necessary to prove the results presented in the paper could be improved (see questions below).\n\nQUESTIONS:\n- Could the authors please provide more details for how $\\theta_h^\\mu(s,o)$ relates to the policy $\\mu$?\n- There is possibly a type in line 125, $(h, s) \\in [H] \\times \\mathcal O$.\n- Theorem 3 holds “when dataset $\\mathcal D$ sufficiently covers the trajectories induced by $\\mu^*$”. Could the authors provide some intuition for how often this holds, or how one could verify if it holds?\n- Could the authors provide more details about the penalty function $\\Gamma$? For instance, what is it’s intuitive meaning, why does it have the format described in Subroutines 2 and 3?\n- Theorem 4.2 holds if the penalty function is an $\\xi$-uncertainty quantifier. Could the authors provide some intuition for how often this holds, or how one could verify if it holds?\n- How strong are assumptions 5.1 and 5.4? How could one ensure that they are satisfied?\n- How does the magnitude of $C^*$ compares with $C^{option}$? How does the magnitude of these constants generally compare with the other terms in the bounds?\n\nLIMITATIONS:\nThe authors discuss the limitations of the work. No major concerns.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a sample complexity analysis of offline RL with options. A sub-optimality analysis was performed considering two data collection schemes, (D1) state-option-utility collection and (D2) state-action-reward collection for the proposed pessimistic value iteration algorithm (PEVIO). The analysis reveals that (D1) is storage efficient and has a fast convergence rate, while (D2) has weak assumptions and allows flexible evaluation of new options.\n\nSTRENGTHS:\n(S1) Solid theoretical analysis\n\n(S2) Analysis of the Pros and cons of the data-collection procedures\n\nWEAKNESSES:\n(W1) Scope of the theory is not clearly stated.\n\n(W2) Background and discussion on key assumptions seem insufficient.\n\n(W3) Unclear parameter update equation in the proposed Algorithm.\n\nQUESTIONS:\nI think this study is interesting since it draws conclusions about the advantages and disadvantages of the data collection schemes through solid theoretical analysis. However, as mentioned above, it appears the current version of this paper has three weaknesses. I would appreciate it if the authors could answer the following questions.\n\n-Is the scope of the authors' theory and algorithms limited to Tabular MDPs (MDPs with discrete state and action space)? I can guess this from the content, but some clear description is needed.\n-Are there any related works and background on the finite concentrability assumption (Assumption 5.1 and 5.4), which is the central assumption of the analysis? It is helpful if list papers that adopt the same or similar assumptions and/or discuss the approach with different assumptions, etc. That will help to place this study in the context of the research.\n-How to update the penalty function in the Proposed PEVIO algorithm? It only provides the order-level update equation, unlike the (original) PEVI. Please give us some guidelines for implementation even if it is through numerical experiments. \n\nMinor Comment:\n-Citation needed at (L147) and (L217)\n\nLIMITATIONS:\nNo \"Limitations\" section is provided by the author.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis analysis is the study of sample complexity in offline reinforcement learning (RL) with options, highlighting potential risks in online environmental exploration. The paper proposes the innovative PEssimistic Value Iteration for Learning with Options (PEVIO) algorithm and provides near-optimal suboptimality bounds for two common data-collection methods. The research underscores PEVIO's superior performance and faster convergence when optimally designed options are used or offline data is limited. Moreover, the study explores two different data collection methods, providing unique insights into their respective benefits and challenges. Overall, this research offers valuable contributions to the practical implementation and understanding of offline RL with options.\n\nSTRENGTHS:\nThis paper tries to put the innovative exploration of sample complexity in offline reinforcement learning (RL) with options, introducing the novel PEVIO algorithm. Combining  theory with sound empirical evaluations provides robust insights into different data collection methods and their implications. Despite the complexity of the subject, the paper maintains fair clarity. Its practical significance is clear, as it tries to fill a literature gap and holds the potential to influence future RL system design, making it a vital addition to the field of RL.\n\nWEAKNESSES:\nWhile the paper is undoubtedly insightful, further exploration of the influence of option complexity on PEVIO's effectiveness could enhance its contribution. Expanding the experimental setup to include varied environments and offline datasets might be beneficial to validate PEVIO's performance robustly. I think a more thorough discussion of the trade-offs between the two data collection methods could aid practitioners in making more informed choices in diverse application contexts.\n\nQUESTIONS:\nI just want to understand on curiosity - Is there any thought that the work may be helpful or this approach can also be helpful in  Hybrid (Online +Offline RL). Any substantial logic for this?\nAlso, Can the authors discuss potential extensions or improvements of the PEVIO algorithm?\n\nLIMITATIONS:\nAuthor mentioned the limitations in some way Pros and Cons. So, OK. But I would love to understand explicitly about the scope limitation of the algorithm in more detail if there is any additional material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}