Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings 

1 Introduction . Offline reinforcement learning ( offline RL ) targets at learning a reward-maximizing policy in an unknown Markov Decision Process ( MDP ) using a static data generated by running a behavior policy [ Lange et al. , 2012 , Levine et al. , 2020 ] . This framework is widely applicable in applications where online exploration is demanding but historical data are plentiful . Examples include medicine [ Liu et al. , 2017 ] ( safety concerns limit the applicability of unproven treatments but electronic records are abundant ) and autonomous driving [ Codevilla et al. , 2018 ] ( building infrastructure for testing new policy is expensive while collecting data from current setting is almost free ) . Parallel to its practical significance , recently there is a surge of theoretical investigations towards offline RL via two threads : offline policy evaluation ( OPE ) , where the goal is to estimate the value of a target ( fixed ) policy V π [ Jiang and Li , 2016 , Liu et al. , 2018 , Kallus and Uehara , 2020 , 2019 , Uehara and Jiang , 2019 , Nachum et al. , 2019 , Xie et al. , 2019 , Yin and Wang , 2020 , Duan et al. , 2020 , Wang et al. , 2021 , Zhang et al. , 2021a ] and offline ( policy ) learning which intends to output a near-optimal policy [ Chen and Jiang , 2019 , Le et al. , 2019 , Xie and Jiang , 2021 , 2020 , Liu et al. , 2020b , Hao et al. , 2020 , Zanette , 2021 , Jin et al. , 2020c , Hu et al. , 2021 , Yin et al. , 2021b , Rashidinejad et al. , 2021 ] . Yin et al . [ 2021a ] initiates the studies for offline RL from the new perspective of uniform convergence in OPE ( uniform OPE for short ) which unifies OPE and offline learning tasks . Generally speaking , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . given a policy class Π and offline data with n episodes , uniform OPE seeks to coming up with OPE estimators V̂ π1 and Q̂ π 1 satisfy supπ∈Π ||Q̂π1 −Qπ1 ||∞ < . The task is to achieve this with the optimal episode complexity : the “ minimal ” number of episodes n needed as a function of , failure probability δ , the parameters of the MDP as well as the behavior policy µ in the minimax sense . To further motivate the readers why uniform OPE should be considered , we state its relation to offline learning . Indeed , uniform OPE to RL is analogous of uniform convergence of empirical risk in statistical learning [ Vapnik , 2013 ] . In supervised learning , it has been proven that almost all learnable problems are learned by an ( asymptotic ) empirical risk minimizer ( ERM ) [ Shalev-Shwartz et al. , 2010 ] . In offline RL , the natural counterpart is the empirical optimal policy π̂ ? : = argmaxπ V̂ π 1 and with uniform OPE it further ensures π̂ ? is a near-optimal policy for the offline learning via : 0 ≤ Qπ ? 1 −Qπ̂ ? 1 = Q π ? 1 − Q̂π ? 1 + Q̂ π ? 1 − Q̂π̂ ? 1 + Q̂ π̂ ? 1 −Qπ̂ ? 1 ≤ 2 sup π |Qπ1 − Q̂π1 | . ( 1 ) On the policy evaluation side , there is often a need to evaluate the performance of a data-dependent policy . Uniform OPE suffices for this purpose since it will allow us to evaluate policies selected by safe-policy improvements , proximal policy optimization , UCB-style exploration-bonus as well as any heuristic exploration criteria ( please refer to Yin et al . [ 2021a ] and the references therein for further discussions ) . In this paper , we study the uniform OPE problem under the finite horizon stationary MDPs and focus on the model-based approaches . Specifically , we consider two representative class : global policy class Πg ( contains all ( deterministic ) policies ) and local policy class Πl ( contains policies near the empirical optimal one , see Section 2.1 ) . We ask the following question : What is the statistical limit for uniform OPE and what is its connection to optimal offline learning ? We answer the first part by showing the global uniform OPE requires a lower bound of Ω ( H2S/dm 2 ) 1 for the family of model-based approach and the local uniform OPE can achieve Õ ( H2/dm 2 ) minimax rate by the model-based plug-in estimator and this implies optimal offline learning . Importantly , the procedure of the model-based approach via learning π̂ ? through planning over the empirical MDP has a wider range of use in offline RL as it naturally adapts to the challenging tasks like offline task-agnostic learning and offline reward-free learning . See Section 1.2 . 1.1 Related works . Offline reinforcement learning.2 Information-theoretical considerations for offline RL are first proposed for infinite horizon discounted setting via Fitted Q-Iteration ( FQI ) type function approximation algorithms [ Chen and Jiang , 2019 , Le et al. , 2019 , Xie and Jiang , 2021 , 2020 ] which can be traced back to [ Munos , 2003 , Szepesvári and Munos , 2005 , Antos et al. , 2008a , b ] . For the finite horizon case , Yin et al . [ 2021a ] first achieves Õ ( H3/dm 2 ) complexity under nonstationary transition but their results can not further improve in the stationary setting . Recently , Yin et al . [ 2021b ] designs the offline variance reduction algorithm for achieving the optimal Õ ( H2/dm 2 ) rate . Their result is for a specific algorithm that uses data splitting while our results work for any algorithms that returns a nearly empirically optimal policy via uniform convergence . Our results on the offline task-agnostic and the reward-free settings are entirely new . Concurrently , Ren et al . [ 2021 ] considers the horizon-free setting but does not provide uniform convergence guarantee . Model-based approaches with minimaxity . It is known model-based methods are minimax-optimal for online RL with regret Õ ( √ HSAT ) ( e.g . Azar et al . [ 2017 ] , Efroni et al . [ 2019 ] ) . In the generative model setting , Agarwal et al . [ 2020 ] shows model-based approach is still minimax optimal Õ ( ( 1− γ ) −3SA/ 2 ) by using a s-absorbing MDP construction and this model-based technique is later reused for other more general settings ( e.g . Markov games [ Zhang et al. , 2020a ] and linear MDPs [ Cui and Yang , 2020 ] ) and also for overcoming the sample size barrier [ Li et al. , 2020 ] . In offline RL , Yin et al . [ 2021a ] uses the model-based methods to achieve Õ ( H3/dm 2 ) complexity . Task-agnostic and Reward-free problems . The reward-free problem is initiated in the online RL [ Jin et al. , 2020a ] where the agent needs to efficiently explore an MDP environment without using any reward information . It requires high probability guarantee for learning optimal policy for any 1Here dm is the minimal marginal state-action occupancy , see Assumption 2.4 . 2We only provide a short discussion of the most related works due to the space constraint . A detailed discussion can be found in Appendix A. reward function . Later , Kaufmann et al . [ 2020 ] , Menard et al . [ 2020 ] establish the Õ ( H3S2A/ 2 ) complexity and Zhang et al . [ 2020c ] further tightens the dependence to Õ ( H2S2A/ 2 ) . Recently , Zhang et al . [ 2020b ] proposes the task-agnostic setting where one needs to use exploration data to simultaneously learn K tasks and proves an upper bound Õ ( H5SA log ( K ) / 2 ) . However , although these settings remain critical in the offline regime , no statistical result has been derived so far . 1.2 Our contribution . Optimal local uniform OPE . First and foremost , we derive the Õ ( H2/dm 2 ) optimal episode complexity for local uniform OPE ( Theorem 4.1 ) via the model-based method and this implies optimal offline learning with the same rate ( Corollary 4.2 ) ; this result strictly improves upon Yin et al . [ 2021a ] ( Õ ( H3/dm 2 ) ) non-trivially through our new singleton-absorbing MDP technique . Information-theoretical characterization of the global uniform OPE . We characterize the statistical limit for the global uniform convergence by proving a minimax lower bound Ω ( H2S/dm 2 ) ( over all model-based approaches ) ( Theorem 3.1 ) . This result answers the question left by Yin et al . [ 2021a ] that the global uniform OPE is generically harder than the local uniform OPE / offline learning by a factor of S , such a difference will dominate when the state space is exponentially large . Generalize to the new offline settings . Critically , our model-based frameworks naturally generalize to the more challenging settings like task-agnostic and reward-free settings . In particular , we establish the Õ ( H2 log ( K ) /dm 2 ) ( Theorem 5.3 ) and Õ ( H2S/dm 2 ) ( Theorem 5.4 ) complexities for offline task-agnostic learning and offline reward-free learning . Both results are new and optimal . Singleton-absorbing MDP : a sharp analysis tool for episodic stationary transition case . On the technical end , our major contribution is the novel design of singleton-absorbing MDP which handles the data-dependence hurdle encountered in the stationary MDPs . To decouple the data-dependence between P̂s , a and V̂ , Agarwal et al . [ 2020 ] uses a s-absorbing MDP V̂s ( in lieu of V̂ ) of each state for the independence . To control the error propagation between V̂s and V̂ , they use the -net covering such that the value of V̂s traverse the evenly-spaced grids in [ 0 , ( 1− γ ) −1 ] . However , when applied to finite horizon case , the complexity increases as there are H different quantities ( V1 , ... , VH ) and the -nets need to cover the H-dimensional space [ 0 , H ] H . This result in a exponential-H covering number and the metric entropy blows up by a factor H , which yields suboptimal result . In contrast , the singleton-absorbing MDP technique designs a single absorbing MDP that can also control the error propagation sufficiently well . This sharp analysis tool negates the conjecture of Cui and Yang [ 2020 ] that absorbing MDP is not well suitable for finite horizon stationary MDP . Significance : Unifying different offline settings Beyond the study of statistical limit in uniform OPE , this work solves the sample optimality problems for the local uniform OPE , offline task-agnostic and offline reward-free problems . If we take a deeper look , the algorithmic frameworks utilized are all based on the model-based empirical MDP construction and planning . Therefore , as long as we can analyze such framework sharply ( e.g . via novel absorbing-MDP technique ) , then it is hopeful that our techniques can be generalized to tackle more sophisticated settings . On the other hand , things could be more tricky for online RL since the exploration phases need to be specifically designed for each settings and there may not be one general algorithmic pattern that dominates . Our findings reveal the model-based framework is fundamental for offline RL as it subsumes settings like local uniform OPE , offline task-agnostic and offline reward-free learning into the identical learning pattern . Considering these tasks were originally proposed in the online regime under different contexts , such a unified view from the model-based perspective offers a new angle for understanding offline RL .

Summary: This paper studies uniform off-policy evaluation in the episodic setting. In particular, the authors consider a stationary episodic setting. The goal is to evaluate the value function corresponding to a given policy, up to some given accuracy, uniformly among all the policies. The idea is that a uniform off-policy evaluation can be used for off-policy learning directly, due to the equation (1). The authors first characterize a minimax lower bound for global uniform OPE, and they show that this lower bound is proportional to the cardinality of the state space. Next they characterize a local uniform upper bound for plug-in estimator for OPE.  Next, the authors consider offline task-agnostic and offline reward-free learning. They provide the sample complexity of model based methods for these tasks, and they show that these bounds are optimal.