Asymptotically Exact Error Characterization of Offline Policy Evaluation with Misspecified Linear Models

1 Introduction . We consider the problem of offline data-driven decision optimization , wherein static records of previous interactions between decision makers and the environmental system of interest are given . The possible application areas include autonomous driving vehicles , natural-language dialogue systems , recommender systems , financial portfolio optimization and healthcare treatment optimization . The framework of offline reinforcement learning ( RL ) is one of the promising approaches to this task ( Levine et al. , 2020 ) . In the standard RL , the environment and the decision-making policy are respectively modeled as Markov decision processes ( MDPs ) M and conditional distributions of actions π ( Sutton and Barto , 2018 ) , where each series of consecutive interactions betweenM and π are abstracted as a stochastic sequence of state s , action a and reward r , called an episode . The objective of the offline RL is then formalized as the maximization of the policy value J ( π ) , the expected value of the total reward obtained from a single episode , given a static dataset of previous interactions . The crucial part of the problem is that the dataset is static ; No additional interaction with the environment is allowed . This constraint poses several unique challenges to the problem . First , the policies we are optimizing , i.e. , the target policies , can not be run in the actual environment . Second , the policies used to generate the dataset , i.e. , the behavior policies , are often unknown and may be totally different from the target policies . Consequently , it is even difficult to accurately estimate the value of target policies . This is problematic especially in consideration of real-life applications involving financial costs and healthcare risks . To address the issue of policy value estimation , the problem of offline policy evaluation ( OPE ) have been extensively studied in the literature . A class of OPE algorithms are referred as the direct methods ( DMs ) , in which some characteristics ofM are assumed to be realizable under some 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hypothetical models and J ( π ) is estimated via a direct estimation of such characteristics . For example , in the fitted Q-evaluation ( FQE ) algorithm ( Le et al. , 2019 ) , the policy Q-function is assumed to be well-approximated with a parametric function class and the OPE is reduced to the estimation of its parameters . DMs are known to be empirically effective ( Fu et al. , 2021 ) if such realizability assumptions are satisfied and , more importantly , the converse is also true ( Voloshin et al. , 2019 ) . However , the theoretical understanding of DMs under unrealizability is still in its active development . For example , several authors have recently studied OPE or offline RL under relatively weak or approximate realizability assumptions ( Jin et al. , 2020 ; Xie and Jiang , 2020 ; Wang et al. , 2020 ) and consequently proposing new algorithms . In this paper , we approach the problem of unrealizability in the opposite direction ; we start with an existing OPE method , study its behavior under complete unrealizability and seek for the possibility of regaining its consistency ( i.e. , asymptotically achieving zero errors ) . More specifically , we investigate the properties of a simple DM with linear function approximation , which is equivalent with a number of existing algorithms such as LSTDQ ( Lagoudakis and Parr , 2003 ) , FQE ( Le et al. , 2019 ) with linear function regressors , the marginalized importance sampling estimator ( Yin and Wang , 2020 ) and DualDICE ( Nachum et al. , 2019 ) in tabular settings . In particular , we first characterize the exact asymptotic error of linear DM under as weak assumptions as possible . It turns out the error is governed by an inner product of two approximation residualsRB andRχ , Ĵ ( π ) − J ( π ) ∝ E [ RB ( s , a ) Rχ ( s , a ) ] +O ( 1/ √ n ) , ( informal ) where they are corresponding to the unrealizable components of the value function Qπ and the marginal density ratio ν/µ , respectively . To the best of our knowledge , this is the first to show linear DM is doubly robust against model misspecification , i.e. , consistent if either RB = 0 or Rχ = 0 hold ( Table 1 ) . Leveraging the above result , we also show that a linear DM with the tile-coding function approximation ( Section 8.3.2 , Sutton and Barto ( 2018 ) ) is consistent under surprisingly mild conditions with appropriate tile-size scheduling . The rest of the paper is organized as follows . In Section 2 , we formalize the problem setting as well as the definition of the linear direct estimators . In Section 3 , we present the main results , i.e. , the asymptotic error analysis of the linear direct estimators and a construction of consistent nonparametric estimators as its application . In Section 4 , we discuss related works with comparison to our results . Finally , in Section 5 , we present concluding remarks , limitations and future directions . All the proofs of the propositions and the theorem are relegated to the appendix . See Section F for the proofs of the propositions . For the theorem , we present a proof sketch and the pointer to the full proof . 2 Preliminary . In Section 2.1 , some notational conventions are introduced . The problem of OPE is then formalized in Section 2.2 . Then , Section 2.3 , 2.4 and 2.5 respectively introduce assumptions and definitions on the data-collecting processes , the environmental models and the class of estimators we will examine . 2.1 Basic Notation . We implicitly assume the spaces we encounter in this paper , such as the state space S and the action space A , are equipped with respective metrics and base measures , each of which is a compact subset of either a Euclidean space with the Lebesgue measure , a discrete space with the counting measure or a product of those . We denote by ∫ X f ( x ) dx the integration of function f with respect to the base measure of X . The subscript X may be omitted if it is obvious from the context . This way we can immediately generalize our results to both continuous and discrete spaces . Also , we denote the expectation of function f with respect to probability density p by ⟨f⟩p : = Ex∼p [ f ( x ) ] . Let [ m ] : = { 1 , 2 , . . . , m } denote the set of integers from 1 to m. Let ∥·∥p denote the ℓp-norm for vectors and ∥A∥p→q : = supx ̸=0 ∥Ax∥q / ∥x∥p the operator norms for matrices , with the convention ∥A∥p : = ∥A∥p→p , for all 1 ≤ p , q ≤ ∞ . 2.2 Problem Setup . The goal of OPE is to estimate the value of decision-making strategy based on a static dataset of interactions with the environment of interest , without directly knowing its mechanism . The environment is modeled as a Markov decision process ( MDP ) M≡ ( S , A , p0 , pT , pr ) , where S is the state space , A the action space , p0 ( s ) the initial state probability , pT ( s′|s , a ) the transition probability and pr ( r|s , a ) the [ 0 , 1 ] -valued reward probability density function for s , s′ ∈ S , a ∈ A , r ∈ [ 0 , 1 ] . Here we assume pT and pr are unknown . On the other hand , the decision-making strategy is modeled as a policy , a state-conditional action distribution π ( a|s ) for s ∈ S , a ∈ A . The value of π is measured with the expected cumulative reward J ( π ) : = ∞∑ h=0 γh 〈 Phr̄ 〉 pπ0 , ( 1 ) where γ ∈ [ 0 , 1 ) is the discounting factor , P is the state-transition operator such that ( Pf ) ( s , a ) =∫ f ( s′ , a′ ) pT ( s ′|s , a ) π ( a′|s′ ) ds′ da′ , r̄ ( s , a ) : = ∫ r pr ( r|s , a ) dr is the expected reward function , and pπ0 ( s , a ) : = p0 ( s ) π ( a|s ) is the initial state-action distribution . In particular , ⟨Phr̄⟩pπ0 denotes the expected reward after h transitions starting from pπ0 . The policy value J ( π ) is estimated based on a collection of transition records ξn ≡ ( ξ1 , ... , ξn ) ∈ Dn called an offline dataset , where D : = S ×A× [ 0 , 1 ] × S is the space of transition records and ξi ≡ ( si , ai , ri , s ′ i ) ∈ D , i ∈ [ n ] , is a transition record made of a preceding state-action pair ( si , ai ) , the associated reward ri , and the state after transition s′i . The dataset ξ n is assumed to be an instantiation of the random variables Ξn ≡ ( Ξ1 , ... , Ξn ) , Ξi ≡ ( Si , Ai , Ri , S′i ) , collected with interactions between the environmentM and a query distribution pquery ≡ { pquery ( i ) ( s , a|ξi−1 ) } i∈ [ n ] such that its distribution is given in a conditional fashion , p ( ξi|ξi−1 ) = pquery ( i ) ( si , ai|ξi−1 ) pr ( ri|si , ai ) pT ( s′i|si , ai ) , i ∈ [ n ] . Note that the notion of query distribution is so flexible that it admits ξn to be a union of episodes generated with multiple nonstationary policies and even adversaries on the choice of state-action pairs . Definition 1 ( OPE problem ) . An instance of the offline policy evaluation problem is specified with POPE ≡ ( M , π , γ , pquery ) , where the goal is to estimate the policy value J ( π ) determined by ( M , π , γ ) , given the input data ξn generated with ( M , pquery ) , without knowing any of pT , pr or pquery . 2.3 Assumptions on Data-Collecting Process . To ensure the existence of reasonable estimators for POPE , we pose a condition on the mixing of data-collecting process , i.e. , conditions on pquery . We assume the amounts of mutual dependencies induced by pquery between time-distant transition records are bounded . Assumption 1 ( G∗-mixing dataset ) . There exists a constant G∗ < ∞ such that Ξn is ‘ ϕ ’ -strong mixing with the coefficient g ( h ) satisfying 1 + 2 ∑n h=1 √ g ( h ) ≤ G∗.1 1The symbol ‘ ϕ ’ in the ‘ ϕ ’ -strong mixing has its root in statistics and completely unrelated to the feature mapping . We denote the mixing coefficient by g ( h ) to avoid confusion with the feature mappings . See Definition 17 ( in the appendix ) for the definition of the ‘ ϕ ’ -strong mixing coefficients . Typical examples satisfying Assumption 1 include datasets consisting of multiple short episodes and mixing Markov chains induced by stationary behavior policies . Proposition 1 . The following statements are true . 1 . Assume Ξn consists of multiple independently collected episodes with length bounded by H , ordered in a consecutive manner . Then we have G∗ ≤ 2H − 1 . 2 . Let pquery ( i ) ( s , a|ξi−1 ) = pT ( s|si−1 , ai−1 ) πb ( a|s ) , 1 ≤ i ≤ n , for some stationary behavior policy πb ( a|s ) and assume the resulting Markov chain has a finite mixing time tmix < ∞ . Then we have G∗ ≤ 1 + 7tmix . Note that the definition of G∗-mixing is designed to be more general than these examples . In particular , it is more suitable for our query-distribution framework , which admits adversaries behind the choice of ( si , ai ) -s or dynamically changing behavior policies . Under Assumption 1 , most properties of pquery is characterized with the marginal data density . Definition 2 ( Marginal data density ) . Let µ ( s , a ) be the marginal data density , given by µ ( s , a ) : = 1 n ∑n i=1 E [ pquery ( i ) ( s , a |Ξi−1 ) ] for s ∈ S and a ∈ A . The marginal data density µ quantifies the expected frequency of visitation at each point ( s , a ) ∈ S×A made by the querying process . Thus , roughly speaking , µ ( s , a ) indicates that how likely the point ( s , a ) will be sampled in Ξn .

Summary: This is a theoretical paper which addresses the RL off-policy evaluation with linear function approximation. It proves convergence results of linear function approximations and shows that the direct method (model the dynamics) is asymptotically equivalent to FQE (modeling the value function) and analyzes its convergence properties. Additionally, it proves consistency results for such methods under tile-coding, allowing for consistency (using non-parametric estimation) even under unrealizability of the linear approximators.