Asymptotically Exact Error Characterization of Offline Policy Evaluation with Misspecified Linear Models

1 Introduction . We consider the problem of offline data-driven decision optimization , wherein static records of previous interactions between decision makers and the environmental system of interest are given . The possible application areas include autonomous driving vehicles , natural-language dialogue systems , recommender systems , financial portfolio optimization and healthcare treatment optimization . The framework of offline reinforcement learning ( RL ) is one of the promising approaches to this task ( Levine et al. , 2020 ) . In the standard RL , the environment and the decision-making policy are respectively modeled as Markov decision processes ( MDPs ) M and conditional distributions of actions π ( Sutton and Barto , 2018 ) , where each series of consecutive interactions betweenM and π are abstracted as a stochastic sequence of state s , action a and reward r , called an episode . The objective of the offline RL is then formalized as the maximization of the policy value J ( π ) , the expected value of the total reward obtained from a single episode , given a static dataset of previous interactions . The crucial part of the problem is that the dataset is static ; No additional interaction with the environment is allowed . This constraint poses several unique challenges to the problem . First , the policies we are optimizing , i.e. , the target policies , can not be run in the actual environment . Second , the policies used to generate the dataset , i.e. , the behavior policies , are often unknown and may be totally different from the target policies . Consequently , it is even difficult to accurately estimate the value of target policies . This is problematic especially in consideration of real-life applications involving financial costs and healthcare risks . To address the issue of policy value estimation , the problem of offline policy evaluation ( OPE ) have been extensively studied in the literature . A class of OPE algorithms are referred as the direct methods ( DMs ) , in which some characteristics ofM are assumed to be realizable under some 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hypothetical models and J ( π ) is estimated via a direct estimation of such characteristics . For example , in the fitted Q-evaluation ( FQE ) algorithm ( Le et al. , 2019 ) , the policy Q-function is assumed to be well-approximated with a parametric function class and the OPE is reduced to the estimation of its parameters . DMs are known to be empirically effective ( Fu et al. , 2021 ) if such realizability assumptions are satisfied and , more importantly , the converse is also true ( Voloshin et al. , 2019 ) . However , the theoretical understanding of DMs under unrealizability is still in its active development . For example , several authors have recently studied OPE or offline RL under relatively weak or approximate realizability assumptions ( Jin et al. , 2020 ; Xie and Jiang , 2020 ; Wang et al. , 2020 ) and consequently proposing new algorithms . In this paper , we approach the problem of unrealizability in the opposite direction ; we start with an existing OPE method , study its behavior under complete unrealizability and seek for the possibility of regaining its consistency ( i.e. , asymptotically achieving zero errors ) . More specifically , we investigate the properties of a simple DM with linear function approximation , which is equivalent with a number of existing algorithms such as LSTDQ ( Lagoudakis and Parr , 2003 ) , FQE ( Le et al. , 2019 ) with linear function regressors , the marginalized importance sampling estimator ( Yin and Wang , 2020 ) and DualDICE ( Nachum et al. , 2019 ) in tabular settings . In particular , we first characterize the exact asymptotic error of linear DM under as weak assumptions as possible . It turns out the error is governed by an inner product of two approximation residualsRB andRχ , Ĵ ( π ) − J ( π ) ∝ E [ RB ( s , a ) Rχ ( s , a ) ] +O ( 1/ √ n ) , ( informal ) where they are corresponding to the unrealizable components of the value function Qπ and the marginal density ratio ν/µ , respectively . To the best of our knowledge , this is the first to show linear DM is doubly robust against model misspecification , i.e. , consistent if either RB = 0 or Rχ = 0 hold ( Table 1 ) . Leveraging the above result , we also show that a linear DM with the tile-coding function approximation ( Section 8.3.2 , Sutton and Barto ( 2018 ) ) is consistent under surprisingly mild conditions with appropriate tile-size scheduling . The rest of the paper is organized as follows . In Section 2 , we formalize the problem setting as well as the definition of the linear direct estimators . In Section 3 , we present the main results , i.e. , the asymptotic error analysis of the linear direct estimators and a construction of consistent nonparametric estimators as its application . In Section 4 , we discuss related works with comparison to our results . Finally , in Section 5 , we present concluding remarks , limitations and future directions . All the proofs of the propositions and the theorem are relegated to the appendix . See Section F for the proofs of the propositions . For the theorem , we present a proof sketch and the pointer to the full proof . 2 Preliminary . In Section 2.1 , some notational conventions are introduced . The problem of OPE is then formalized in Section 2.2 . Then , Section 2.3 , 2.4 and 2.5 respectively introduce assumptions and definitions on the data-collecting processes , the environmental models and the class of estimators we will examine . 2.1 Basic Notation . We implicitly assume the spaces we encounter in this paper , such as the state space S and the action space A , are equipped with respective metrics and base measures , each of which is a compact subset of either a Euclidean space with the Lebesgue measure , a discrete space with the counting measure or a product of those . We denote by ∫ X f ( x ) dx the integration of function f with respect to the base measure of X . The subscript X may be omitted if it is obvious from the context . This way we can immediately generalize our results to both continuous and discrete spaces . Also , we denote the expectation of function f with respect to probability density p by ⟨f⟩p : = Ex∼p [ f ( x ) ] . Let [ m ] : = { 1 , 2 , . . . , m } denote the set of integers from 1 to m. Let ∥·∥p denote the ℓp-norm for vectors and ∥A∥p→q : = supx ̸=0 ∥Ax∥q / ∥x∥p the operator norms for matrices , with the convention ∥A∥p : = ∥A∥p→p , for all 1 ≤ p , q ≤ ∞ . 2.2 Problem Setup . The goal of OPE is to estimate the value of decision-making strategy based on a static dataset of interactions with the environment of interest , without directly knowing its mechanism . The environment is modeled as a Markov decision process ( MDP ) M≡ ( S , A , p0 , pT , pr ) , where S is the state space , A the action space , p0 ( s ) the initial state probability , pT ( s′|s , a ) the transition probability and pr ( r|s , a ) the [ 0 , 1 ] -valued reward probability density function for s , s′ ∈ S , a ∈ A , r ∈ [ 0 , 1 ] . Here we assume pT and pr are unknown . On the other hand , the decision-making strategy is modeled as a policy , a state-conditional action distribution π ( a|s ) for s ∈ S , a ∈ A . The value of π is measured with the expected cumulative reward J ( π ) : = ∞∑ h=0 γh 〈 Phr̄ 〉 pπ0 , ( 1 ) where γ ∈ [ 0 , 1 ) is the discounting factor , P is the state-transition operator such that ( Pf ) ( s , a ) =∫ f ( s′ , a′ ) pT ( s ′|s , a ) π ( a′|s′ ) ds′ da′ , r̄ ( s , a ) : = ∫ r pr ( r|s , a ) dr is the expected reward function , and pπ0 ( s , a ) : = p0 ( s ) π ( a|s ) is the initial state-action distribution . In particular , ⟨Phr̄⟩pπ0 denotes the expected reward after h transitions starting from pπ0 . The policy value J ( π ) is estimated based on a collection of transition records ξn ≡ ( ξ1 , ... , ξn ) ∈ Dn called an offline dataset , where D : = S ×A× [ 0 , 1 ] × S is the space of transition records and ξi ≡ ( si , ai , ri , s ′ i ) ∈ D , i ∈ [ n ] , is a transition record made of a preceding state-action pair ( si , ai ) , the associated reward ri , and the state after transition s′i . The dataset ξ n is assumed to be an instantiation of the random variables Ξn ≡ ( Ξ1 , ... , Ξn ) , Ξi ≡ ( Si , Ai , Ri , S′i ) , collected with interactions between the environmentM and a query distribution pquery ≡ { pquery ( i ) ( s , a|ξi−1 ) } i∈ [ n ] such that its distribution is given in a conditional fashion , p ( ξi|ξi−1 ) = pquery ( i ) ( si , ai|ξi−1 ) pr ( ri|si , ai ) pT ( s′i|si , ai ) , i ∈ [ n ] . Note that the notion of query distribution is so flexible that it admits ξn to be a union of episodes generated with multiple nonstationary policies and even adversaries on the choice of state-action pairs . Definition 1 ( OPE problem ) . An instance of the offline policy evaluation problem is specified with POPE ≡ ( M , π , γ , pquery ) , where the goal is to estimate the policy value J ( π ) determined by ( M , π , γ ) , given the input data ξn generated with ( M , pquery ) , without knowing any of pT , pr or pquery . 2.3 Assumptions on Data-Collecting Process . To ensure the existence of reasonable estimators for POPE , we pose a condition on the mixing of data-collecting process , i.e. , conditions on pquery . We assume the amounts of mutual dependencies induced by pquery between time-distant transition records are bounded . Assumption 1 ( G∗-mixing dataset ) . There exists a constant G∗ < ∞ such that Ξn is ‘ ϕ ’ -strong mixing with the coefficient g ( h ) satisfying 1 + 2 ∑n h=1 √ g ( h ) ≤ G∗.1 1The symbol ‘ ϕ ’ in the ‘ ϕ ’ -strong mixing has its root in statistics and completely unrelated to the feature mapping . We denote the mixing coefficient by g ( h ) to avoid confusion with the feature mappings . See Definition 17 ( in the appendix ) for the definition of the ‘ ϕ ’ -strong mixing coefficients . Typical examples satisfying Assumption 1 include datasets consisting of multiple short episodes and mixing Markov chains induced by stationary behavior policies . Proposition 1 . The following statements are true . 1 . Assume Ξn consists of multiple independently collected episodes with length bounded by H , ordered in a consecutive manner . Then we have G∗ ≤ 2H − 1 . 2 . Let pquery ( i ) ( s , a|ξi−1 ) = pT ( s|si−1 , ai−1 ) πb ( a|s ) , 1 ≤ i ≤ n , for some stationary behavior policy πb ( a|s ) and assume the resulting Markov chain has a finite mixing time tmix < ∞ . Then we have G∗ ≤ 1 + 7tmix . Note that the definition of G∗-mixing is designed to be more general than these examples . In particular , it is more suitable for our query-distribution framework , which admits adversaries behind the choice of ( si , ai ) -s or dynamically changing behavior policies . Under Assumption 1 , most properties of pquery is characterized with the marginal data density . Definition 2 ( Marginal data density ) . Let µ ( s , a ) be the marginal data density , given by µ ( s , a ) : = 1 n ∑n i=1 E [ pquery ( i ) ( s , a |Ξi−1 ) ] for s ∈ S and a ∈ A . The marginal data density µ quantifies the expected frequency of visitation at each point ( s , a ) ∈ S×A made by the querying process . Thus , roughly speaking , µ ( s , a ) indicates that how likely the point ( s , a ) will be sampled in Ξn .

Summary: The paper consider off-policy evaluation using linear function approximation under unrealizability setting. Under such more relaxed assumption, they provide characterization of the linear direct method(DM) as an error governed by two approximation residual $\mathcal{R}_B$ and $\mathcal{R}_X$, $$     \hat{J}(\pi) - J(\pi) = \mathbb{E}[\mathcal{R}_B(s,a)\mathcal{R}_X(s,a)] + O(1/\sqrt{n}), $$ with $\mathcal{R}_B$ corresponding to Bellman residual and  $\mathcal{R}_X$ corresponding to residual of marginal density ratio. Leveraing this result, they establish a nonparametric tile-coding estimators  to solve the unrealizability though with a slower convergence.