What are the Statistical Limits of Offline RL with Linear Function Approximation?

1 INTRODUCTION . Offline methods ( also known as off-policy methods or batch methods ) are a promising methodology to alleviate the sample complexity burden in challenging reinforcement learning ( RL ) settings , particularly those where sample efficiency is paramount ( Mandel et al. , 2014 ; Gottesman et al. , 2018 ; Wang et al. , 2018 ; Yu et al. , 2019 ) . Off-policy methods are often applied together with function approximation schemes ; such methods take sample transition data and reward values as inputs , and approximate the value of a target policy or the value function of the optimal policy . Indeed , many practical deep RL algorithms find their prototypes in the literature of offline RL . For example , when running on off-policy data ( sometimes termed as “ experience replay ” ) , deep Q-networks ( DQN ) ( Mnih et al. , 2015 ) can be viewed as an analog of Fitted Q-Iteration ( Gordon , 1999 ) with neural networks being the function approximators . More recently , there are an increasing number of both model-free ( Laroche et al. , 2019 ; Fujimoto et al. , 2019 ; Jaques et al. , 2020 ; Kumar et al. , 2019 ; Agarwal et al. , 2020 ) and model-based ( Ross & Bagnell , 2012 ; Kidambi et al. , 2020 ) offline RL methods , with steady improvements in performance ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Wu et al. , 2020 ; Kidambi et al. , 2020 ) . However , despite the importance of these methods , the extent to which data reuse is possible , especially when off-policy methods are combined with function approximation , is not well understood . For example , deepQ-network requires millions of samples to solve certain Atari games ( Mnih et al. , 2015 ) . Also important is that in some safety-critical settings , we seek guarantees when offline- trained policies can be effective ( Thomas , 2014 ; Thomas et al. , 2019 ) . A basic question here is that if there are fundamental statistical limits on such methods , where sample-efficient offline RL is simply not possible without further restrictions on the problem . In the context of supervised learning , it is well-known that empirical risk minimization is sampleefficient if the hypothesis class has bounded complexity . For example , suppose the agent is given a d-dimensional feature extractor , and the ground truth labeling function is a ( realizable ) linear function with respect to the feature mapping . Here , it is well-known that a polynomial number of samples in d suffice for a given target accuracy . Furthermore , in this realizable case , provided the training data has a good feature coverage , then we will have good accuracy against any test distribution.1 In the more challenging offline RL setting , it is unclear if sample-efficient methods are possible , even under analogous assumptions . This is our motivation to consider the following question : What are the statistical limits for offline RL with linear function approximation ? Here , one may hope that value estimation for a given policy is possible in the offline RL setting under the analogous set of assumptions that enable sample-efficient supervised learning , i.e. , 1 ) ( realizability ) the features can perfectly represent the value functions and 2 ) ( good coverage ) the feature covariance matrix of our off-policy data has lower bounded eigenvalues . The extant body of provable methods on offline RL either make representational assumptions that are far stronger than realizability or assume distribution shift conditions that are far stronger than having coverage with regards to the spectrum of the feature covariance matrix of the data distribution . For example , Szepesvári & Munos ( 2005 ) analyze offline RL methods by assuming a representational condition where the features satisfy ( approximate ) closedness under Bellman updates , which is a far stronger representation condition than realizability . Recently , Xie & Jiang ( 2020a ) propose a offline RL algorithm that only requires realizability as the representation condition . However , the algorithm in ( Xie & Jiang , 2020a ) requires a more stringent data distribution condition . Whether it is possible to design a sample-efficient offline RL method under the realizability assumption and a reasonable data coverage assumption — an open problem in ( Chen & Jiang , 2019 ) — is the focus of this work . Our Contributions . Perhaps surprisingly , our main result shows that , under only the above two assumptions , it is information-theoretically not possible to design a sample-efficient algorithm to non-trivially estimate the value of a given policy . The following theorem is an informal version of the result in Section 4 . Theorem 1.1 ( Informal ) . In the offline RL setting , suppose the data distributions have ( polynomially ) lower bounded eigenvalues , and the Q-functions of every policy are linear with respect to a given feature mapping . Any algorithm requires an exponential number of samples in the horizon H to output a non-trivially accurate estimate of the value of any given policy π , with constant probability . This hardness result states that even if the Q-functions of all polices are linear with respect to the given feature mapping , we still require an exponential number of samples to evaluate any given policy . Note that this representation condition is significantly stronger than assuming realizability with regards to only a single target policy ; it assumes realizability for all policies . Regardless , even under this stronger representation condition , it is hard to evaluate any policy , as specified in our hardness result . This result also formalizes a key issue in offline reinforcement learning with function approximation : geometric error amplification . To better illustrate the issue , in Section 5 , we analyze the classical Least-Squares Policy Evaluation ( LSPE ) algorithm under the realizability assumption , which demonstrates how the error propagates as the algorithm proceeds . Here , our analysis shows that , if we only rely on the realizability assumption , then a far more stringent condition is required for sample-efficient offline policy evaluation : the off-policy data distribution must be quite close to the distribution induced by the policy to be evaluated . 1Specifically , if the features have a uniformly bounded norm and if the minimum eigenvalue of the feature covariance matrix of our data is bounded away from 0 , say by 1/poly ( d ) , then we have good accuracy on any test distribution . See Assumption 2 and the comments thereafter . Our results highlight that sample-efficient offline RL is simply not possible unless either the distribution shift condition is sufficiently mild or we have stronger representation conditions that go well beyond realizability . See Section 5 for more details . Furthermore , our hardness result implies an exponential separation on the sample complexity between offline RL and supervised learning , since supervised learning ( which is equivalent to offline RL with H = 1 ) is possible with polynomial number of samples under the same set of assumptions . A few additional points are worth emphasizing with regards to our lower bound construction : • Our results imply that Least-Squares Policy Evaluation ( LSPE , i.e. , using Bellman backups with linear regression ) will fail . Interestingly , while LSPE will provide an unbiased estimator , our results imply that it will have exponential variance in the problem horizon . • Our construction is simple and does not rely on having a large state or action space : the size of the state space is only O ( d ·H ) where d is the feature dimension and H is the planning horizon , and the size of the action space is only is 2 . This stands in contrast to other RL lower bounds , which typically require state spaces that are exponential in the problem horizon ( e.g . see ( Du et al. , 2020 ) ) . • We provide two hard instances , one with a sparse reward ( and stochastic transitions ) and another with deterministic dynamics ( and stochastic rewards ) . These two hard instances jointly imply that both the estimation error on reward values and the estimation error on the transition probabilities could be geometrically amplified in offline RL . • Of possibly broader interest is that our hard instances are , to our knowledge , the first concrete examples showing that geometric error amplification is real in RL problems ( even with realizability ) . While this is a known concern in the analysis of RL algorithms , there have been no concrete examples exhibiting such behavior under only a realizability assumption . 2 RELATED WORK . We now survey prior work on offline RL , largely focusing on theoretical results . We also discuss results on the error amplification issue in RL . Concurrent to this work , Xie & Jiang ( 2020a ) propose a offline RL algorithm under the realizability assumption , which requires stronger distribution shift conditions . We will discuss this work shortly . Existing Algorithms and Analysis . Offline RL with value function approximation is closely related to Approximate Dynamic Programming ( Bertsekas & Tsitsiklis , 1995 ) . Existing work ( Munos , 2003 ; Szepesvári & Munos , 2005 ; Antos et al. , 2008 ; Munos & Szepesvári , 2008 ; Tosatto et al. , 2017 ; Xie & Jiang , 2020b ; Duan & Wang , 2020 ) that analyze the sample complexity of approximate dynamic programming-based approaches usually make the following two categories of assumptions : ( i ) representation conditions that assume the function class approximates the value functions well and ( ii ) distribution shift conditions that assume the given data distribution has sufficient coverage over the state-action space . As mentioned in the introduction , the desired representation condition would be realizability , which only assumes the value function of the policy to be evaluated lies in the function class ( for the case of offline policy evaluation ) or the optimal value function lies in the function class ( for the case of finding near-optimal policies ) , and existing works usually make stronger assumptions . For example , Szepesvári & Munos ( 2005 ) ; Duan & Wang ( 2020 ) assume ( approximate ) closedness under Bellman updates , which is much stronger than realizability . Whether it is possible to design a sample-efficient offline RL method under the realizability assumption and reasonable data coverage assumption , is left as an open problem in ( Chen & Jiang , 2019 ) . To measure the coverage over the state-action space of the given data distribution , existing works assume the concentrability coefficient ( introduced by Munos ( 2003 ) ) to be bounded . The concentrability coefficient , informally speaking , is the largest possible ratio between the probability for a state-action pair ( s , a ) to be visited by a policy , and the probability that ( s , a ) appears on the data distribution . Since we work with linear function approximation in this work , we measure the distribution shift in terms of the spectrum of the feature covariance matrices ( see Assumption 2 ) , which is a well-known sufficient condition in the context of supervised learning and is much more natural for the case of linear function approximation . Concurrent to this work , Xie & Jiang ( 2020a ) propose an algorithm that works under the realizability assumption instead of other stronger representation conditions used in prior work . However , the algorithm in ( Xie & Jiang , 2020a ) requires a much stronger data distribution condition which assumes a stringent version of concentrability coefficient introduced by ( Munos , 2003 ) to be bounded . In contrast , in this work we measure the distribution shift in terms of the spectrum of the feature covariance matrix of the data distribution , which is more natural than the concentrability coefficient for the case of linear function approximation . Recently , there has been great interest in approaching offline policy evaluation ( Precup , 2000 ) via importance sampling . For recent work on this topic , see ( Dudı́k et al. , 2011 ; Mandel et al. , 2014 ; Thomas et al. , 2015 ; Li et al. , 2015 ; Jiang & Li , 2016 ; Thomas & Brunskill , 2016 ; Guo et al. , 2017 ; Wang et al. , 2017 ; Liu et al. , 2018 ; Farajtabar et al. , 2018 ; Xie et al. , 2019 ; Kallus & Uehara , 2019 ; Liu et al. , 2019 ; Uehara & Jiang , 2019 ; Kallus & Uehara , 2020 ; Jiang & Huang , 2020 ; Feng et al. , 2020 ) . Offline policy evaluation with importance sampling incurs exponential variance in the planning horizon when the behavior policy is significantly different from the policy to be evaluated . Bypassing such exponential dependency requires non-trivial function approximation assumptions ( Jiang & Huang , 2020 ; Feng et al. , 2020 ; Liu et al. , 2018 ) . Finally , Kidambi et al . ( 2020 ) provide a model-based offline RL algorithm , with a theoretical analysis based on hitting times . Hardness Results . Algorithm-specific hardness results have been known for a long time in the literature of Approximate Dynamic Programming . See Chapter 4 in ( Van Roy , 1994 ) and also ( Gordon , 1995 ; Tsitsiklis & Van Roy , 1996 ) . These works demonstrate that certain approximate dynamic programming-based methods will diverge on hard cases . However , such hardness results only hold for a restricted class of algorithms , and to demonstrate the fundamental difficulty of offline RL , it is more desirable to obtain information-theoretic lower bounds as initiated by Chen & Jiang ( 2019 ) . Existing ( information-theoretic ) exponential lower bounds ( Krishnamurthy et al. , 2016 ; Sun et al. , 2017 ; Chen & Jiang , 2019 ) usually construct unstructured MDPs with an exponentially large state space . Du et al . ( 2020 ) prove an exponential lower bound under the assumption that the optimal Qfunction is approximately linear . The condition that the optimal Q-function is only approximately linear is crucial for the hardness result in Du et al . ( 2020 ) . The techniques in ( Du et al. , 2020 ) are later generalized to other settings ( Kumar et al. , 2020 ; Wang et al. , 2020 ; Mou et al. , 2020 ) . Error Amplification In RL . Error amplification induced by distribution shift and long planning horizon is a known issue in the theoretical analysis of RL algorithms . See ( Gordon , 1995 ; 1996 ; Munos & Moore , 1999 ; Ormoneit & Sen , 2002 ; Kakade , 2003 ; Zanette et al. , 2019 ) for papers on this topic and additional assumptions that mitigate this issue . Error amplification in offline RL is also observed in empirical works ( see e.g . ( Fujimoto et al. , 2019 ) ) . In this work , we provide the first information-theoretic lower bound showing that geometric error amplification is real in offline RL .

Summary: This paper presents a impossibility result for value-function approximation in batch-mode RL. The chart below puts this work in context. This work essentially shows -- through a constructive example of an MDP -- that the amount of data needed for approximating Q values must increases exponentially with the horizon in episodic RL tasks even if we assume that the Q-values are realizable and that the features gathered by the behavior policy -- that collected the data -- are uncorrelated. This problem arises because the data gathering policy can fail to get data from all states even though the features themselves are uncorrelated.