Variance-Aware Off-Policy Evaluation with Linear Function Approximation

1 Introduction . Reinforcement learning ( RL ) has been a hot spot in both theory and practice in the past decade . Many efficient algorithms have been proposed and theoretically analyzed for finding the optimal policy adopted by an agent to maximize the long-term cumulative rewards . In contrast to online RL where the agent actively interacts with the environment , offline RL ( a.k.a. , batch RL ) [ 25 , 24 ] aims to extract information from past data and use this information to learn the optimal policy . There has been much empirical success of offline RL in various application domains [ 4 , 6 , 37 , 40 , 36 ] . Among various tasks of offline RL , an important task is called off-policy evaluation ( OPE ) , which evaluates the performance of a target policy π given offline data generated by a behavior policy π̄ . Most existing theoretical works on OPE are in the setting of tabular MDPs [ 34 , 26 , 11 , 16 , 45 , 47–49 ] , where the state space S and the action space A are both finite . However , real-world applications often have high-dimensional or even infinite-dimensional state and action spaces , where function approximation is required for computational tractability and generalization . While provably efficient online RL with linear function approximation has been widely studied recently [ 46 , 17 , 50 , 15 , 3 , 54 ] , little work has been done for analyzing OPE with linear function approximation , with one notable exception by Duan et al . [ 10 ] . More specifically , Duan et al . [ 10 ] analyzed a regression-based ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Fitted Q-Iteration method ( FQI-OPE ) that achieves an Õ ( H2 √ ( 1 + d ( π , π̄ ) ) /N ) error for linear MDPs [ 46 , 17 ] , where H is the planning horizon , N is the sample size , and d ( π , π̄ ) represents the distribution shift between the behavior policy and the target policy . They also proved a sample complexity lower bound for a subclass of linear MDPs , for which their algorithm is nearly minimax optimal . However , as we will show later , the H2 dependence is not tight since they discard the useful variance information contained in the offline data . Consequently , their result is only optimal for a small class of MDPs of which the value functions have large variance . The H2 dependence in the sample complexity also makes their algorithm less sample-efficient for long-horizon problems , which is one of the major challenges in RL . Extracting useful information from the data is particularly important for offline RL since the agent can not sample additional data by interacting with the environment , as compared to online RL . In this paper , we propose a new algorithm that incorporates the variance information of the value functions to improve the sample efficiency of OPE . This allows us to achieve a deeper understanding and tighter error bounds of OPE with linear function approximation . In detail , we consider time-inhomogeneous linear MDPs [ 46 , 17 ] where the transition probability and reward function are assumed to be linear functions of a known feature mapping and may vary from stage to stage . The main contributions of this paper are summarized as follows : • We develop VA-OPE ( Variance-Aware Off-Policy Evaluation ) , an algorithm for OPE that effectively utilizes the variance information from the offline data . The core idea behind the proposed algorithm is to calibrate the Bellman residual in the regression by an estimator of the conditional variance of the value functions , such that data points of higher quality can receive larger important weights . • We show that our algorithm achieves Õ ( ∑ h ( v > h Λ −1 h vh ) 1/2/ √ K ) policy evaluation error , where vh is the expectation of the feature vectors under target policy and Λh is the uncentered covariance matrix under behavior policy weighted by the conditional variance of the value function . Our algorithm achieves a tighter error bound and milder dependence on H than FQI-OPE [ 10 ] , and provides a tighter characterization of the distribution shift between the behavior policy and the target policy , which is also verified by extensive numerical experiments . • Our analysis is based on a novel two-step proof technique . In the first step , we use backward induction to establish worst-case uniform convergence2 results for the estimators of the value functions . In the second step , the convergence of OPE estimator is proved by tightening the uniform convergence result based on an average-case analysis . Our proof strategy provides a generic way for analyzing ( weighted ) ridge regression methods that are carried out in a backward and iterated fashion . The analyses in both steps might be of independent interest . Notation We use lower case letters to denote scalars and use lower and upper case boldface letters to denote vectors and matrices respectively . For any vector x ∈ Rd and any positive semi-definite matrix Σ ∈ Rd×d , we denote by ‖x‖2 the Euclidean norm and ‖Σ‖ the operator norm , and define ‖x‖Σ = √ x > Σx . For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } . For any finite set A , we denote by |A| the cardinality of A . For two sequences { an } and { bn } , we write an = O ( bn ) if there exists an absolute constant C such that an ≤ Cbn , and we write an = Ω ( bn ) if there exists an absolute constant C such that an ≥ Cbn . We use Õ ( · ) to further hide the logarithmic factors . 2 Preliminaries . 2.1 Markov Decision Processes . We consider the time-inhomogeneous episodic Markov Decision Process ( MDP ) , which is represented by a tuple M ( S , A , H , { rh } Hh=1 , { Ph } Hh=1 ) . In specific , we denote the state space by S and the action space by A , and H > 0 is the horizon length of each episode . At each stage h ∈ [ H ] , rh : S × A → [ 0 , 1 ] is the reward function , and Ph ( s′|s , a ) is the transition probability function which represents the probability for state s to transit to state s′ given action a . A policy π consists of H mappings { πh } Hh=1 from S to the simplex on A , such that for any ( h , s ) ∈ [ H ] × S , πh ( ·|s ) is a probability distribution over A . Here a policy can be either deterministic ( point mass ) or stochastic . 2By uniform convergence we mean the convergence of the estimated value functions in ` ∞-norm to their true values , which is different from the uniform convergence over all policies in Yin et al . [ 48 ] . For any policy π , we define the associated action-value function Qπh ( s , a ) and value function V π h ( s ) at each stage h ∈ [ H ] as follows : Qπh ( s , a ) = Eπ [ H∑ i=h ri ( si , ai ) ∣∣∣∣sh = s , ah = a ] , V πh ( s ) = ∫ A Qπh ( s , a ) dπh ( a|s ) , ( 2.1 ) where ai ∼ πi ( ·|si ) and si+1 ∼ Pi ( ·|si , ai ) . For any function V : S → R , we introduce the following shorthand notation for the conditional expectation and variance of V : [ PhV ] ( s , a ) = Es′∼Ph ( ·|s , a ) [ V ( s ′ ) ] , [ VhV ] ( s , a ) = [ PhV 2 ] ( s , a ) − ( [ PhV ] ( s , a ) ) 2 . ( 2.2 ) Time-inhomogeneous linear MDPs . We consider a special class of MDPs called linear MDPs [ 46 , 17 ] . Note that most of the existing works on RL with linear function approximation rely on this assumption . Assumption 2.1 . M ( S , A , H , { rh } Hh=1 , { Ph } Hh=1 ) is called a linear MDP with a known feature mapping φ : S × A → Rd , if for any h ∈ [ H ] , there exist γh and µh ∈ Rd , such that for any state-action pair ( s , a ) ∈ S ×A , it holds that Ph ( · | s , a ) = 〈φ ( s , a ) , µh ( · ) 〉 , rh ( s , a ) = 〈φ ( s , a ) , γh〉 . ( 2.3 ) We assume that at any stage h , for any state-action pair ( s , a ) ∈ S ×A , the reward received by the agent is given by r = rh ( s , a ) + h ( s , a ) , where rh ( s , a ) ∈ [ 0 , 1 ] is the expected reward and h ( s , a ) is the random noise . We assume that the noise is zero-mean and independent of anything else . Without loss of generality , we assume that ‖γh‖2 ≤ 1 and ‖φ ( s , a ) ‖2 ≤ 1 for all ( s , a ) ∈ S × A . We also assume that rh ( s , a ) + h ( s , a ) ≤ 1 , | h ( s , a ) | ≤ 1 almost surely and thus Var ( h ( s , a ) ) ≤ 1 for all h ∈ [ H ] and ( s , a ) ∈ S ×A . Moreover , we assume that maxh∈ [ H ] ∥∥∫ S f ( s ) dµh ( s ) ∥∥ 2 ≤ √ d for all bounded function f : S → R such that sups∈S |f ( s ) | ≤ 1 . The above assumption on linear MDPs implies the following proposition for the action-value functions . Proposition 2.2 ( Proposition 2.3 , [ 17 ] ) . For a linear MDP , for any policy π , there exist weights { wπh , h ∈ [ H ] } such that for any ( s , a , h ) ∈ S × A × [ H ] , we have Qπh ( s , a ) = 〈φ ( s , a ) , wπh〉 . Moreover , we have ‖wπh‖2 ≤ 2H √ d for all h ∈ [ H ] . Following this proposition , we may further show that the value functions are also linear functions , but of different features . We define φπh ( s ) = ∫ A φ ( s , a ) dπh ( a|s ) for all s ∈ [ S ] and h ∈ [ H ] . Then by ( 2.1 ) we have V πh ( s ) = ∫ A φ ( s , a ) > wπhdπ ( a|s ) = 〈φπh ( s ) , wπh〉 .

Summary: This paper studies the off-policy evaluation (OPE) problem in reinforcement learning with linear function approximation. They propose to incorporate the variance information of the value function to improve the sample efficiency of OPE. This has better theoretical properties.