Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

2 ) episodes of offline data in the finite-horizon stationary transition setting , where H is the horizon length and dm is the minimal marginal stateaction distribution induced by the behavior policy . This improves over the best known upper bound by a factor of H . Moreover , we establish an informationtheoretic lower bound of Ω ( H2/dm 2 ) which certifies that OPDVR is optimal up to logarithmic factors . Lastly , we show that OPDVR also achieves rate-optimal sample complexity under alternative settings such as the finite-horizon MDPs with non-stationary transitions and the infinite horizon MDPs with discounted rewards . 1 Introduction . Offline reinforcement learning ( offline RL ) aims at learning the near-optimal policy by using a static offline dataset that is collected by a certain behavior policy µ [ Lange et al. , 2012 ] . As offline RL agent has no access to interact with the environment , it is more widely applicable to problems where online interaction is infeasible , e.g . when trials-and-errors are expensive ( robotics , education ) , risky ( autonomous driving ) or even unethical ( healthcare ) [ see , e.g. , a recent survey Levine et al. , 2020 ] . Despite its practical significance , a precise theoretical understanding of offline RL has been lacking . Previous sample complexity bounds for RL has primarily focused on the online setting [ Azar et al. , 2017 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Simchowitz and Jamieson , 2019 , Efroni et al. , 2019 , Cai et al. , 2019 ] or the generative model ( simulator ) setting [ Azar et al. , 2013 , Sidford et al. , 2018a , b , Yang and Wang , 2019 , Agarwal et al. , 2020 , Wainwright , 2019 , Lattimore and Szepesvari , 2019 ] , both of which assuming interactive access to the environment and not applicable to offline RL . On the other hand , the sample complexity of offline RL remains unsettled even for environments with finitely many state and actions , a.k.a , the tabular MDPs ( Markov Decision Processes ) . One major line of work is concerned with the off-policy evaluation ( OPE ) problem [ Li et al. , 2015 , Jiang and Li , 2016 , Liu et al. , 2018 , Kallus and Uehara , 2019a , b , Uehara and Jiang , 2019 , Xie et al. , 2019 , Yin and Wang , 2020 , Duan and Wang , 2020 ] . These works provide sample complexity bounds for evaluating the performance of a fixed policy , and do not imply guarantees for policy optimization . Another line of work studies the sample complexity of offline policy optimization in conjunction with function 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approximation [ Chen and Jiang , 2019 , Xie and Jiang , 2020b , a , Jin et al. , 2020 ] . These results apply to offline RL with general function classes , but when specialized to the tabular setting , they give rather loose sample complexity bounds with suboptimal dependencies on various parameters 1 . The recent work of Yin et al . [ 2021 ] showed that the optimal sample complexity for finding an -optimal policy in offline RL is Õ ( H3/dm 2 ) in the finite-horizon non-stationarysetting ( with matching upper and lower bounds ) , where H is the horizon length and dm is a constant related to the data coverage of the behavior policy in the given MDP . However , the optimal sample complexity in alternative settings such as stationary transition or infinite-horizon settings remains unknown . Further , the Õ ( H3/dm 2 ) sample complexity is achieved by an off-policy evaluation + uniform convergence type algorithm ; other more practical algorithms including ( stochastic ) optimal planning algorithms such as Q-Learning are not well understood in offline RL . Our Contributions In this paper , we propose an algorithm OPDVR ( Off-Policy Doubled Variance Reduction ) for offline reinforcement learning based on an extension of the variance reduction technique initiated in [ Sidford et al. , 2018a , Yang and Wang , 2019 ] . OPDVR performs stochastic ( minibatch style ) value iterations using the available offline data , and can be seen as a version of stochastic optimal planning that interpolates value iteration and Q-learning . Our main contributions are summarized as follows . • We show that OPDVR finds an -optimal policy with high probability using Õ ( H2/dm 2 ) episodes of offline data ( Section 4.1 ) . This improves upon the best known sample complexity by an H factor and to the best of our knowledge is the first that achieves an O ( H2 ) horizon dependence , thus separating the stationary case with the non-stationary case for offline RL . • We establish a sample ( episode ) complexity lower bound Ω ( H2/dm 2 ) for offline RL in the finite-horizon stationary setting ( Theorem 4.2 ) , showing that the sample complexity of OPDVR is optimal up to logarithmic factors . • In the finite-horizon non-stationary setting , and infinite horizon γ-discounted setting , we show that OPDVR achieves Õ ( H3/dm 2 ) sample ( episode ) complexity ( Section 3 ) and Õ ( ( 1−γ ) −3/dm 2 ) sample complexity ( Section 4.2 ) respectively . They are both optimal up to logarithmic factors and our infinite-horizon result improves over the best known results , e.g. , those derived for the fitted Q-iteration style algorithms [ Xie and Jiang , 2020b ] . • On the technical end , our algorithm presents a sharp analysis of offline RL with stationary transitions , and , importantly , the use of the doubling technique to resolve the initialization dependence defect which fails to make the original variance reduction algorithm of [ Sidford et al. , 2018a ] to be optimal , see Appendix F.4 . Running [ Sidford et al. , 2018a ] may not yield the desired accuracy as they stated and our result is robust in preserving the optimality . Related work . There is a large and growing body of work on the theory of offline RL and RL in general . We could not hope to provide a comprehensive survey , thus will instead highlight the few 1See Table 1 for a clear comparison . prior work that we depend upon on the technical level . The variance reduction techniques that we use in this paper builds upon the work of [ Sidford et al. , 2018a ] in the generative model setting , though it is nontrivial in adapting their techniques to the offline setting ; and our two-stage variance reduction appears essential for obtaining optimal rate for > 1 ( see Section 5 and Appendix F.4 for more detailed discussions ) . We also used a fictitious estimator technique that originates from the OPE literature [ Xie et al. , 2019 , Yin and Wang , 2020 ] , but extended it to the stationary-transition case , and to the policy optimization problem . As we mentioned earlier , the optimal sample complexity in offline RL in the tabular MDPs with stationary transitions was not settled . The result of [ Yin et al. , 2021 ] is optimal in the non-stationary case , but is suboptimal by a factor of H in the stationary case . Our lower bound is a variant of the construction of [ Yin et al. , 2021 ] that applies to the stationary case . Other existing work on offline RL has even weaker parameters ( sometimes due to their setting being more general , see details in Table 1 ) . We defer more detailed discussion related to the OPE literature and online RL / generative model literature to Appendix A due to space constraint . 2 Preliminaries . We consider reinforcement learning problems modeled by finite Markov Decision Processes ( MDPs ) ( we focus on the finite-horizon episodic setting , and defer the infinite-horizon discounted setting to Section 4.2 . ) An MDP is denoted by a tuple M = ( S , A , r , T , d1 , H ) , where S and A are the state and action spaces with finite cardinality |S| = S and |A| = A. Pt : S × A × S → [ 0 , 1 ] is the transition kernel with Pt ( s′|s , a ) be the probability of entering state s′ after taking action a at state s. We consider both the stationary and non-stationary transition setting : The stationary transition setting assumes Pt ≡ P is identical for all t ∈ [ H ] , and the non-stationary transition setting allows Pt to be different for different t. rt : S × A → [ 0 , 1 ] is the reward function which we assume to be deterministic2 . d1 is the initial state distribution , and H is the time horizon . A ( non-stationary ) policy π : S → PHA assigns to each state st ∈ S a distribution over actions at each time t. We use dπt ( s , a ) or d π t ( s ) to denote the marginal state-action/state distribution induced by policy π at time t , i.e . dπt ( s ) : = Pπ ( st = s ) and dπt ( s , a ) : = Pπ ( st = s , at = a ) . Q-value and Bellman operator . For any policy π and any fixed time t , the value function V πt ( · ) ∈ RS and Q-value function Qπt ( · , · ) ∈ RS×A , ∀s , a is defined as : V πt ( s ) = E [ ∑H i=t ri ∣∣∣st = s ] , Qπt ( s , a ) = E [ ∑Hi=t ri∣∣∣st , at = s , a ] . For the ease of exposition , we always enumerate Qπ as a column vector and similarly for Pt ( ·|s , a ) . Moreover , for any vector Q ∈ RS×A , the induced value vector and policy is defined in the greedy way : ∀st ∈ S , VQ ( st ) = maxat∈AQ ( st , at ) , πQ ( st ) = argmaxat∈AQ ( st , at ) . Given an MDP , for any vector V ∈ R S and any deterministic policy π , ∀t ∈ [ H ] the Bellman operator T πt : RS → RS is defined as : [ T πt ( V ) ] ( s ) : = r ( s , πt ( s ) ) + P > t ( ·|s , πt ( s ) ) V , and the corresponding Bellman optimality operator Tt : RS → RS , [ Tt ( V ) ] ( s ) : = maxa∈A [ r ( s , a ) + P > t ( ·|s , a ) V ] . Lastly , for a given value function Vt , we define backup function zt ( st , at ) : = P > t ( ·|st , at ) Vt+1 and the one-step variance as σVt+1 ( st , at ) : = Varst+1 [ Vt+1 ( st+1 ) |st , at ] . Offline learning problem . In this paper we investigate the offline learning problem , where we do not have interactive access to the MDP , and can only observe a static dataset D = { ( s ( i ) t , a ( i ) t , r ( i ) t , s ( i ) t+1 ) } t∈ [ H ] i∈ [ n ] .We assume that D is obtained by executing a pre-specified behavior policy µ ( also known as the logging policy ) for n episodes and collecting the trajectories τ ( i ) = ( s ( i ) 1 , a ( i ) 1 , r ( i ) 1 , . . . , s ( i ) H , a ( i ) H , r ( i ) H , s ( i ) H+1 ) , where each episode is rendered in the form : s ( i ) 1 ∼ d1 , a ( i ) t ∼ µt ( ·|s ( i ) t ) , r ( i ) t = r ( s ( i ) t , a ( i ) t ) , and s ( i ) t+1 ∼ Pt ( ·|s ( i ) t , a ( i ) t ) . Given the dataset D , our goal is to find an -optimal policy πout , in the sense that ||V π ? 1 − V πout 1 ||∞ < . Assumption on data coverage Due to the curse of distributional shift , efficient offline RL is only possible under certain data coverage properties for the behavior policy µ . Throughout this paper we assume the following : Assumption 2.1 ( Weak coverage ) . The behavior policy µ satisfies the following : There exists some optimal policy π ? such that dµt′ ( st′ , at′ ) > 0 if there exists t < t ′ such that dπ ? t : t′ ( st′ , at′ |st , at ) > 0 , where dπ ? t : t′ ( st′ , at′ |st , at ) is the conditional multi-step transition probability from step t to t′ . 2This is commonly assumed in the RL literature . The randomness in the reward will only cause a lower order error ( than the randomness in the transition ) for learning . Intuitively , Assumption 2.1 requires µ to “ cover ” certain optimal policy π ? , in the sense that any st′ , at′ is reachable by µ if it is attainable from a previous state-action pair by π ? . It is similar to [ Liu et al. , 2019 , Assumption 1 ] . Note that this is weaker than the standard “ concentrability ” assumption [ Munos , 2003 , Le et al. , 2019 , Chen and Jiang , 2019 ] : Concentrability defines βµ : = supπ∈Π ||dπ ( st , at ) /dµ ( st , at ) ||∞ < ∞ ( cf . [ Le et al. , 2019 , Assumption 1 & Example 4.1 ] ) , which requires the sufficient exploration for tabular case3 since we optimize over all policies ( see Section F.2 for a discussion ) . In contrast , our assumption only requires µ to “ trace ” one single optimal policy.4 With Assumption 2.1 , we define dm : = min t , st , at { dµt ( st , at ) : d µ t ( st , at ) > 0 } , ( 1 ) which is decided by the behavior policy µ and is required by offline learning ( see Theorem G.2 in Yin et al . [ 2021 ] ) . In general , dm is unknown yet we assume dm is known for the moment and will utilize dm in our algorithms . Indeed , in Lemma 5.1 , we show that estimating dm up to a multiplicative factor only requires Õ ( 1/dm ) episodes of offline data ; replacing the exact dm with this estimator suffices for our purpose and , importantly , will not affect our downstream sample complexities .

Summary: The paper focuses on the theoretical understanding of offline reinforcement learning. Assuming weak coverage (Assumption 2.1), which appears to be necessary, the authors propose a double variance deduction learning algorithm. The algorithm has two key components: variance reduction (Eq. (2)) that decompose the key estimation of P_t(\cdot|s,a)V_{t+1} into two components that are estimated separately at different stages; and doubling where parameters are instantiated twice. The authors show strong theoretical results of the proposed algorithm: (new) order-optimal sample complexity for the stationary case, order optimal results for the finite-horizon non-stationary setting and improved results for the infinite horizon discounted setting.