Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

2 ) episodes of offline data in the finite-horizon stationary transition setting , where H is the horizon length and dm is the minimal marginal stateaction distribution induced by the behavior policy . This improves over the best known upper bound by a factor of H . Moreover , we establish an informationtheoretic lower bound of Ω ( H2/dm 2 ) which certifies that OPDVR is optimal up to logarithmic factors . Lastly , we show that OPDVR also achieves rate-optimal sample complexity under alternative settings such as the finite-horizon MDPs with non-stationary transitions and the infinite horizon MDPs with discounted rewards . 1 Introduction . Offline reinforcement learning ( offline RL ) aims at learning the near-optimal policy by using a static offline dataset that is collected by a certain behavior policy µ [ Lange et al. , 2012 ] . As offline RL agent has no access to interact with the environment , it is more widely applicable to problems where online interaction is infeasible , e.g . when trials-and-errors are expensive ( robotics , education ) , risky ( autonomous driving ) or even unethical ( healthcare ) [ see , e.g. , a recent survey Levine et al. , 2020 ] . Despite its practical significance , a precise theoretical understanding of offline RL has been lacking . Previous sample complexity bounds for RL has primarily focused on the online setting [ Azar et al. , 2017 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Simchowitz and Jamieson , 2019 , Efroni et al. , 2019 , Cai et al. , 2019 ] or the generative model ( simulator ) setting [ Azar et al. , 2013 , Sidford et al. , 2018a , b , Yang and Wang , 2019 , Agarwal et al. , 2020 , Wainwright , 2019 , Lattimore and Szepesvari , 2019 ] , both of which assuming interactive access to the environment and not applicable to offline RL . On the other hand , the sample complexity of offline RL remains unsettled even for environments with finitely many state and actions , a.k.a , the tabular MDPs ( Markov Decision Processes ) . One major line of work is concerned with the off-policy evaluation ( OPE ) problem [ Li et al. , 2015 , Jiang and Li , 2016 , Liu et al. , 2018 , Kallus and Uehara , 2019a , b , Uehara and Jiang , 2019 , Xie et al. , 2019 , Yin and Wang , 2020 , Duan and Wang , 2020 ] . These works provide sample complexity bounds for evaluating the performance of a fixed policy , and do not imply guarantees for policy optimization . Another line of work studies the sample complexity of offline policy optimization in conjunction with function 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approximation [ Chen and Jiang , 2019 , Xie and Jiang , 2020b , a , Jin et al. , 2020 ] . These results apply to offline RL with general function classes , but when specialized to the tabular setting , they give rather loose sample complexity bounds with suboptimal dependencies on various parameters 1 . The recent work of Yin et al . [ 2021 ] showed that the optimal sample complexity for finding an -optimal policy in offline RL is Õ ( H3/dm 2 ) in the finite-horizon non-stationarysetting ( with matching upper and lower bounds ) , where H is the horizon length and dm is a constant related to the data coverage of the behavior policy in the given MDP . However , the optimal sample complexity in alternative settings such as stationary transition or infinite-horizon settings remains unknown . Further , the Õ ( H3/dm 2 ) sample complexity is achieved by an off-policy evaluation + uniform convergence type algorithm ; other more practical algorithms including ( stochastic ) optimal planning algorithms such as Q-Learning are not well understood in offline RL . Our Contributions In this paper , we propose an algorithm OPDVR ( Off-Policy Doubled Variance Reduction ) for offline reinforcement learning based on an extension of the variance reduction technique initiated in [ Sidford et al. , 2018a , Yang and Wang , 2019 ] . OPDVR performs stochastic ( minibatch style ) value iterations using the available offline data , and can be seen as a version of stochastic optimal planning that interpolates value iteration and Q-learning . Our main contributions are summarized as follows . • We show that OPDVR finds an -optimal policy with high probability using Õ ( H2/dm 2 ) episodes of offline data ( Section 4.1 ) . This improves upon the best known sample complexity by an H factor and to the best of our knowledge is the first that achieves an O ( H2 ) horizon dependence , thus separating the stationary case with the non-stationary case for offline RL . • We establish a sample ( episode ) complexity lower bound Ω ( H2/dm 2 ) for offline RL in the finite-horizon stationary setting ( Theorem 4.2 ) , showing that the sample complexity of OPDVR is optimal up to logarithmic factors . • In the finite-horizon non-stationary setting , and infinite horizon γ-discounted setting , we show that OPDVR achieves Õ ( H3/dm 2 ) sample ( episode ) complexity ( Section 3 ) and Õ ( ( 1−γ ) −3/dm 2 ) sample complexity ( Section 4.2 ) respectively . They are both optimal up to logarithmic factors and our infinite-horizon result improves over the best known results , e.g. , those derived for the fitted Q-iteration style algorithms [ Xie and Jiang , 2020b ] . • On the technical end , our algorithm presents a sharp analysis of offline RL with stationary transitions , and , importantly , the use of the doubling technique to resolve the initialization dependence defect which fails to make the original variance reduction algorithm of [ Sidford et al. , 2018a ] to be optimal , see Appendix F.4 . Running [ Sidford et al. , 2018a ] may not yield the desired accuracy as they stated and our result is robust in preserving the optimality . Related work . There is a large and growing body of work on the theory of offline RL and RL in general . We could not hope to provide a comprehensive survey , thus will instead highlight the few 1See Table 1 for a clear comparison . prior work that we depend upon on the technical level . The variance reduction techniques that we use in this paper builds upon the work of [ Sidford et al. , 2018a ] in the generative model setting , though it is nontrivial in adapting their techniques to the offline setting ; and our two-stage variance reduction appears essential for obtaining optimal rate for > 1 ( see Section 5 and Appendix F.4 for more detailed discussions ) . We also used a fictitious estimator technique that originates from the OPE literature [ Xie et al. , 2019 , Yin and Wang , 2020 ] , but extended it to the stationary-transition case , and to the policy optimization problem . As we mentioned earlier , the optimal sample complexity in offline RL in the tabular MDPs with stationary transitions was not settled . The result of [ Yin et al. , 2021 ] is optimal in the non-stationary case , but is suboptimal by a factor of H in the stationary case . Our lower bound is a variant of the construction of [ Yin et al. , 2021 ] that applies to the stationary case . Other existing work on offline RL has even weaker parameters ( sometimes due to their setting being more general , see details in Table 1 ) . We defer more detailed discussion related to the OPE literature and online RL / generative model literature to Appendix A due to space constraint . 2 Preliminaries . We consider reinforcement learning problems modeled by finite Markov Decision Processes ( MDPs ) ( we focus on the finite-horizon episodic setting , and defer the infinite-horizon discounted setting to Section 4.2 . ) An MDP is denoted by a tuple M = ( S , A , r , T , d1 , H ) , where S and A are the state and action spaces with finite cardinality |S| = S and |A| = A. Pt : S × A × S → [ 0 , 1 ] is the transition kernel with Pt ( s′|s , a ) be the probability of entering state s′ after taking action a at state s. We consider both the stationary and non-stationary transition setting : The stationary transition setting assumes Pt ≡ P is identical for all t ∈ [ H ] , and the non-stationary transition setting allows Pt to be different for different t. rt : S × A → [ 0 , 1 ] is the reward function which we assume to be deterministic2 . d1 is the initial state distribution , and H is the time horizon . A ( non-stationary ) policy π : S → PHA assigns to each state st ∈ S a distribution over actions at each time t. We use dπt ( s , a ) or d π t ( s ) to denote the marginal state-action/state distribution induced by policy π at time t , i.e . dπt ( s ) : = Pπ ( st = s ) and dπt ( s , a ) : = Pπ ( st = s , at = a ) . Q-value and Bellman operator . For any policy π and any fixed time t , the value function V πt ( · ) ∈ RS and Q-value function Qπt ( · , · ) ∈ RS×A , ∀s , a is defined as : V πt ( s ) = E [ ∑H i=t ri ∣∣∣st = s ] , Qπt ( s , a ) = E [ ∑Hi=t ri∣∣∣st , at = s , a ] . For the ease of exposition , we always enumerate Qπ as a column vector and similarly for Pt ( ·|s , a ) . Moreover , for any vector Q ∈ RS×A , the induced value vector and policy is defined in the greedy way : ∀st ∈ S , VQ ( st ) = maxat∈AQ ( st , at ) , πQ ( st ) = argmaxat∈AQ ( st , at ) . Given an MDP , for any vector V ∈ R S and any deterministic policy π , ∀t ∈ [ H ] the Bellman operator T πt : RS → RS is defined as : [ T πt ( V ) ] ( s ) : = r ( s , πt ( s ) ) + P > t ( ·|s , πt ( s ) ) V , and the corresponding Bellman optimality operator Tt : RS → RS , [ Tt ( V ) ] ( s ) : = maxa∈A [ r ( s , a ) + P > t ( ·|s , a ) V ] . Lastly , for a given value function Vt , we define backup function zt ( st , at ) : = P > t ( ·|st , at ) Vt+1 and the one-step variance as σVt+1 ( st , at ) : = Varst+1 [ Vt+1 ( st+1 ) |st , at ] . Offline learning problem . In this paper we investigate the offline learning problem , where we do not have interactive access to the MDP , and can only observe a static dataset D = { ( s ( i ) t , a ( i ) t , r ( i ) t , s ( i ) t+1 ) } t∈ [ H ] i∈ [ n ] .We assume that D is obtained by executing a pre-specified behavior policy µ ( also known as the logging policy ) for n episodes and collecting the trajectories τ ( i ) = ( s ( i ) 1 , a ( i ) 1 , r ( i ) 1 , . . . , s ( i ) H , a ( i ) H , r ( i ) H , s ( i ) H+1 ) , where each episode is rendered in the form : s ( i ) 1 ∼ d1 , a ( i ) t ∼ µt ( ·|s ( i ) t ) , r ( i ) t = r ( s ( i ) t , a ( i ) t ) , and s ( i ) t+1 ∼ Pt ( ·|s ( i ) t , a ( i ) t ) . Given the dataset D , our goal is to find an -optimal policy πout , in the sense that ||V π ? 1 − V πout 1 ||∞ < . Assumption on data coverage Due to the curse of distributional shift , efficient offline RL is only possible under certain data coverage properties for the behavior policy µ . Throughout this paper we assume the following : Assumption 2.1 ( Weak coverage ) . The behavior policy µ satisfies the following : There exists some optimal policy π ? such that dµt′ ( st′ , at′ ) > 0 if there exists t < t ′ such that dπ ? t : t′ ( st′ , at′ |st , at ) > 0 , where dπ ? t : t′ ( st′ , at′ |st , at ) is the conditional multi-step transition probability from step t to t′ . 2This is commonly assumed in the RL literature . The randomness in the reward will only cause a lower order error ( than the randomness in the transition ) for learning . Intuitively , Assumption 2.1 requires µ to “ cover ” certain optimal policy π ? , in the sense that any st′ , at′ is reachable by µ if it is attainable from a previous state-action pair by π ? . It is similar to [ Liu et al. , 2019 , Assumption 1 ] . Note that this is weaker than the standard “ concentrability ” assumption [ Munos , 2003 , Le et al. , 2019 , Chen and Jiang , 2019 ] : Concentrability defines βµ : = supπ∈Π ||dπ ( st , at ) /dµ ( st , at ) ||∞ < ∞ ( cf . [ Le et al. , 2019 , Assumption 1 & Example 4.1 ] ) , which requires the sufficient exploration for tabular case3 since we optimize over all policies ( see Section F.2 for a discussion ) . In contrast , our assumption only requires µ to “ trace ” one single optimal policy.4 With Assumption 2.1 , we define dm : = min t , st , at { dµt ( st , at ) : d µ t ( st , at ) > 0 } , ( 1 ) which is decided by the behavior policy µ and is required by offline learning ( see Theorem G.2 in Yin et al . [ 2021 ] ) . In general , dm is unknown yet we assume dm is known for the moment and will utilize dm in our algorithms . Indeed , in Lemma 5.1 , we show that estimating dm up to a multiplicative factor only requires Õ ( 1/dm ) episodes of offline data ; replacing the exact dm with this estimator suffices for our purpose and , importantly , will not affect our downstream sample complexities .

Summary: The papers presents an algorithms called OPDVR (Off-Policy Double Variance Reduction) based on variance reduction, which is seemingly designed with offline RL in mind, even though the main attribute of offline RL (data constraint and lack of support) is fully missed. The paper contains several theoretical results; their core contribution (among others) is for the stationary case, where transition probabilities are not function of time. They basically present a sample complexity that grows with $H^2$, an improvement over previous results by a factor of $H$. The authors provide similar result for the infinite-horizon (discounted) case as well, and expand their findings for the non-stationary case too.   Main weakness: While this paper is a nice theoretical contribution, discussions on the "practical" impact of the basic assumptions, and how they can be relaxed to become useful in real-world, are missing, [see the comments below].