Nearly Horizon-Free Offline Reinforcement Learning

⇣q 1 Kdm ⌘ error bound for the plug-in estimator , which matches the lower bound up to logarithmic factors and does not have additional dependency on poly ( H , S , A , d ) in higher-order term . • For offline policy optimization , we obtain an Õ ⇣q 1 Kdm + min ( S , d ) Kdm ⌘ sub- optimality gap for the empirical optimal policy , which approaches the lower bound up to logarithmic factors and a high-order term , improving upon the best known result by [ 1 ] that has additional poly ( H , S , d ) factors in the main term . To the best of our knowledge , these are the first set of nearly horizon-free bounds for episodic time-homogeneous offline tabular MDP and linear MDP with anchor points . Central to our analysis is a simple yet effective recursion based method to bound a “ total variance ” term in the offline scenarios , which could be of individual interest . 1 Introduction . Reinforcement Learning ( RL ) aims to learn to make sequential decisions to maximize the long-term reward in unknown environments , and has demonstrated success in game-playing [ 2 , 3 ] , robotics [ 4 ] , and automatic algorithm design [ 5 ] . These successes rely on being able to deploy the algorithms in a way that directly interacts with the respective environment , allowing them to improve the policies in a trial-and-error way . However , such direct interactions with real environments can be expensive or even impossible in other real-world applications , e.g. , education [ 6 ] , health and medicine [ 7 , 8 ] , conversational AI [ 9 ] and recommendation systems [ 10 ] . Instead , we are often given a collection of logged experiences generated by potentially multiple and possibly unknown policies in the past . This lack of access to real-time interactions with an environment led to the field of offline reinforcement learning [ 11 ] . Within this , offline policy evaluation ( OPE ) focuses on evaluating a policy , and offline policy optimization ( OPO ) focuses on improving policies ; both rely only upon the given fixed past experiences without any further interactions . OPE and OPO are in general notoriously difficult , as unbiased estimators of the policy value can suffer from exponentially increasing variance in terms of horizon in the worst case [ 12 , 13 ] . To overcome this “ curse of horizon ” in OPE , [ 14 , 15 ] first introduced marginalized importance sampling ( MIS ) based estimators . They showed that if ( 1 ) all of the logged experiences are generated 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from the same behavior policy , and ( 2 ) the behavior policy is known , then the exponential dependency on horizon can be improved to polynomial dependency . Subsequently , [ 16–18 ] showed that the polynomial dependency could be achieved even without assumptions ( 1 ) and ( 2 ) . The basic idea for these MIS-based estimators is estimating the marginal state-action density ratio between the target policy and the empirical data , so as to adjust the distribution mismatch between them . On the algorithmic side , marginal density ratio estimation can be implemented by either plug-in estimators [ 15 , 19 ] , temporal-difference updates [ 20 , 21 ] , or solving a min-max optimization [ 16 , 17 , 22 , 18 ] . These OPE estimators can also be used as one component for OPO , resulting in the algorithms in [ 23–25 ] , which also inherit the polynomial dependency on horizon . In a different but related line of work , [ 26 , 27 ] recently showed that , for the online episodic timehomogeneous tabular Markov Decision Process ( MDP ) that allows for the interactions with environments , the sample complexity only has poly log dependency on the horizon . This motivates us to consider the following question : Can offline reinforcement learning escape from the polynomial dependency on the horizon ? In this paper , we provide an affirmative answer to this question . Specifically , considering the episodic time-homogeneous tabular MDP with S states , A actions and horizon H , or the linear MDP with anchor points and feature dimension d , assuming the total reward for any episode is upper bounded by 1 almost surely , we obtain the following nearly H-free bounds : • For offline policy evaluation ( OPE ) , we show that the plug-in estimator has a finite-sample error of eO ⇣q 1 Kdm ⌘ ( Theorem 1 and 5 ) , where K is the number of episodes and dm is the minimum visiting probability of ( anchor ) state-action pairs , that matches the lower bound up to logarithmic factors . We emphasize that the bound has no additional poly ( H , S , A , d ) dependency in the higher order term , unlike the known results of [ 19 , 24 ] . • For offline policy optimization ( OPO ) , we show that the policy obtained by model-based planning on empirical MDP has a sub-optimality gap of eO ⇣q 1 Kdm + min { S , d } Kdm ⌘ ( Theorem 3 and 6 ) , which matches the lower bound up to logarithmic factors and a high-order term . This also improves upon the best known result from [ 1 ] by removing additional poly ( H , S , d ) factors in the main term . To the best of our knowledge , these are the first set of nearly horizon-free bounds for both OPE and OPO on time-homogeneous tabular MDP and linear MDP with anchor points . To achieve such sharp bounds , we propose a novel recursion based method to bound a “ total variance ” term ( introduced below ) that is broadly emerged in the offline reinforcement learning , which could be of individual interest . Technique Overview . With a sequence of fairly standard steps in the literature , we can bound the error of the plug-in estimator via terms related to the square root of the “ total variance ” ( also known as the Cramer-Rao type lower bound illustrated in [ 13 ] ) qP h2 [ H ] P s , a ⇠⇡ h ( s , a ) VarP ( s , a ) V ⇡ h+1 ( s0 ) where ⇠⇡ is the reaching probability , P is the transition and V ⇡ is the value function under policy ⇡ . ( For a more formal definition , see Section 3. ) . An improper bound of this term will introduce unnecessary dependency on the horizon , either in the main term or in the higher-order term . We instead bound this term with a recursive method , by observing that this “ total variance ” term can be approximately upper bounded by the square root of the “ total variance of value square ” P h2 [ H ] P s , a ⇠⇡ h ( s , a ) VarP ( s , a ) h V ⇡ h+1 ( s0 ) 2i and some remaining term ( see Lemma 1 and Lemma 3 for the detail ) . Applying this argument recursively , we can finally obtain a poly logH upper bound on the total variance , which eventually gets rid of the polynomial dependency on H . The idea of higher order expansion has been investigated in [ 28 , 27 ] . We notice that the recursion introduced in [ 28 ] was designed for the infinite horizon setting , and how to generalize their technique to finite horizon setting is still unclear . Our recursion is conceptually more similar to the recursion in [ 27 ] . However , [ 27 ] considered the online setting , where we only need to bound the error on the visited state-action pairs . For the offline setting , we need to bound the error on every state-action pair that can be touched by the policy ⇡ . This introduces the reaching probability ⇠⇡ in the “ total variance ” term , which we need to deal with using the MDP structure . As a result , our recursion is significantly different from their counterpart , especially in the case of linear MDP . Organization . Our paper is organized as follows : in Section 2 , we review the related literature on offline reinforcement learning , and then briefly introduce the problem we consider in Section 3 . In Section 4 and Section 5 , we show our results of offline policy evaluation and offline policy optimization on tabular MDP correspondingly , and in Section 6 , we show how to generalize our results to linear MDP with anchor points . We finally conclude and discuss our results in Section 7 . 2 Related Work . In this section , we briefly discuss the related literature in three categories , i.e. , offline policy evaluation , offline policy optimization , and horizon-free online reinforcement learning . Notice that , for the setting that assumes an additional generative model , typical model-based algorithms first query equal number of data from each state-action pair , then perform offline policy evaluation/optimization based on the queried data . Thus we view the reinforcement learning with generative model as a special instance of offline reinforcement learning . To make the comparison fair , for method and analysis that do not assume Assumption 1 , we scale the error and sample complexity , by assuming per-step reward is upper bounded by 1 and 1/H under infinite-horizon and finite-horizon setting correspondingly . Offline Policy Evaluation . For OPE in infinite horizon tabular MDP , [ 28 ] showed that plug-in estimator can achieve the error of eO ⇣q 1 dm ( 1 ) ⌘ under Assumption 1 , which matches the lower bound in [ 29 ] up to logarithmic factors . For OPE in finite horizon time-inhomogeneous tabular MDP , [ 19 , 24 ] provided an error bound of eO ⇣q 1 Kdm + p SA Kdm ⌘ under the uniform reward assumption , which matches the lower bound [ 13 ] up to logarithmic factors and an additional higher-order term . We here consider the time-homogeneous MDP , and obtain an error bound of Õ ( q 1 Kdm ) , that does not have the additional p SA in higher-order term , which is different from [ 19 , 24 ] . Beyond the tabular setting , [ 30 ] considered the performance of plug-in estimator with linear function approximation under the assumption of linear MDP without anchor points , and [ 31 , 32 ] provided more detailed analyses on the statistical properties of different kinds of estimators under different assumptions , which are not directly comparable to our work . Recently , there are also works [ e.g . 33 ] focusing on the interval estimation of the policy for practical application . Offline Policy Optimization . Offline policy optimization for infinite horizon MDP can date back to [ 35 ] . [ 28 ] recently showed that a perturbed version of model-based planning can find ✏-optimal policy within eO ⇣ 1 dm ( 1 ) ✏2 ⌘ queries of transitions in infinite horizon tabular MDP when the total reward is upper bounded by 1 , that matches the lower bound up to logarithmic factors . For the finite horizon time-inhomogeneous tabular MDP setting , [ 24 ] showed that model-based planning can identify ✏-optimal policy with eO ⇣ H dm✏ 2 ⌘ episodes , that matches the lower bound for time-inhomogeneous MDP up to logarithmic factors . When it comes to finite horizon time-homogeneous tabular MDP and linrar MDP with anchor points , [ 1 ] provided a eO ⇣ min { H , S , d } dm✏ 2 ⌘ episode complexity for model-based planning , which is min { H , S , d } away from the lower bound . We provide a eO ⇣ 1 dm✏ 2 + min { S , d } dm✏ ⌘ episode complexity , that matches the lower bound up to logarithm factors and a higher order term . A recent work [ 36 ] considered solving the offline policy optimization with model-free Q-learning with variance reduction . Although their algorithm can match the sample complexity lower bound ⌦ ⇣ H 2 dm✏ 2 ⌘ when per-step reward is upper bounded by 1 , model-free algorithms generally need at least ⌦ ( H 2 dm ) episodes of data to finish the algorithm ( a.k.a the sample size barrier in [ 28 ] ) and it is unclear how to translate their results to the setting with total reward upper bounded by 1 . Horizon-Free Online Reinforcement Learning . There are several works obtained nearly horizonfree sample complexity bounds for online reinforcement learning . In the time-homogeneous setting , whether the sample complexity needs to scale polynomially with H was an open problem raised by [ 37 ] . The problem was first addressed by [ 26 ] who proposed an ✏-net over the optimal policies and a simulation-based algorithms to obtain a sample complexity that only scales logarithmically with H , though their dependency on S , A and ✏ is not optimal and their algorithm is not computationally efficient . The sample complexity bound was later substantially improved by [ 27 ] who obtained an eO ⇣ SA ✏2 + S 2 A ✏ ⌘ bound , which nearly matches the contextual bandits ( tabular MDP with H = 1 ) lower bound ⌦ SA ✏2 up to an S 2 A ✏ factor [ 38 ] . Their key ideas are ( 1 ) a new bonus function and ( 2 ) a recursion-based approach to obtain a tight bound on the sample complexity . Such kind of recursion-based approach can not be directly applied to the offline setting , and we develop a novel recursion method to suit the offline scenarios .

Summary: The paper analyzes the plug-in approach to off-policy evaluation and optimization in reinforcement learning, and sharpens some of the sample complexity bounds (assuming sufficiently exploratory data, time-homogenous MDPs). The key novelty is in upper-bounding the variance of estimated values cleverly by using higher-order powers of appropriate values. By doing so, the paper's derived bounds do not directly depend on the horizon of the MDP. One benefit of the proposed analysis is that a very similar line of reasoning is sufficient to derive horizon-independent bounds for both evaluation and optimization, as well as for tabular MDPs as well as linear MDPs with additional structure.