Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

1 INTRODUCTION . A key problem in reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) is off-policy policy evaluation : given a fixed target policy of interest , estimating the average reward garnered by an agent that follows the policy , by only using data collected from different behavior policies . This problem is widely encountered in many real-life applications ( e.g. , Murphy et al. , 2001 ; Li et al. , 2011 ; Bottou et al. , 2013 ; Thomas et al. , 2017 ) , where online experiments are expensive and high-quality simulators are difficult to build . It also serves as a key algorithmic component of off-policy policy optimization ( e.g. , Dudı́k et al. , 2011 ; Jiang & Li , 2016 ; Thomas & Brunskill , 2016 ; Liu et al. , 2019b ) . There are two major families of approaches to policy evaluation . The first approach is to build a simulator that mimics the reward and next-state transitions of the real environment ( e.g. , Fonteneau et al. , 2013 ) . While straightforward , this approach strongly relies on the model assumptions in building the simulator , which may invalidate evaluation results . The second approach is to use importance sampling to correct the sampling bias in off-policy data , so that an ( almost ) unbiased estimator can be obtained ( Liu , 2001 ; Strehl et al. , 2010 ; Bottou et al. , 2013 ) . A major limitation , however , is that importance sampling can become inaccurate due to high variance . In particular , most existing IS-based estimators compute the weight as the product of the importance ratios of many steps in the trajectory , causing excessively high variance for problems with long or infinite horizon , yielding a curse of horizon ( Liu et al. , 2018a ) . Recently , Liu et al . ( 2018a ) proposes a new estimator for infinite-horizon off-policy evaluation , which presents significant advantages to standard importance sampling methods . Their method directly estimates the density ratio between the state stationary distributions of the target and behavior policies , instead of the trajectories , thus avoiding exponential blowup of variance in the horizon . While Liu et al. ’ s method shows much promise by significantly reducing the variance , in practice , it may suffer from high bias due to the error or model misspecficiation when estimating the density ratio function . ∗The first two authors contributed equally to this work . In this paper , we develop a doubly robust estimator for infinite horizon off-policy estimation , by integrating Liu et al. ’ s method with information from an additional value function estimation . This significantly reduces the bias of Liu et al. ’ s method once either the density ratio , or the value function estimation is accurate ( hence doubly robust ) . Since Liu et al. ’ s method already promises low variance , our additional bias reduction allows us to achieve significantly better accuracy for practical problems . Technically , our new bias reduction method provides a new angle of double robustness for off-policy evaluation , orthogonal to the existing literature of doubly robust policy evaluation that solely devotes to variance reduction ( Jiang & Li , 2016 ; Thomas & Brunskill , 2016 ; Farajtabar et al. , 2018 ) , mostly based on the idea of control variates ( e.g . Asmussen & Glynn , 2007 ) . Our double robustness for bias reduction is significantly different , and instead yields an intriguing connection with the fundamental primal-dual relations between the density ( ratio ) functions and value functions ( e.g. , Bertsekas , 2000 ; Puterman , 2014 ) . This new perspective can inspire more efficient algorithms for policy evaluation , and lead to unified frameworks for these types of double robustness in future work . 2 BACKGROUND . Infinite Horizon Off-Policy Estimation Let M = 〈S , A , r , T , µ0〉 be a Markov decision process ( MDP ) with state space S , action space A , reward function r , transition probability function T , and initial-state distribution µ0 . A policy π maps states to distributions over A , with π ( a|s ) being the probability of selecting a given s. The average discounted reward for π , with a given discount γ ∈ ( 0 , 1 ) 1 , is defined as Rπ : = lim T→∞ Eτ∼π [ ∑T t=0 γ trt∑T t=0 γ t ] , where τ = { st , at , rt } 0≤t≤T is a trajectory with states , actions , and rewards collected by following policy π in the MDP , starting from s0 ∼ µ0 . Given a set of n trajectories , D = { s ( i ) t , a ( i ) t , r ( i ) t } 1≤i≤n,0≤t≤T , collected under a behavior policy π0 ( a|s ) , the off-policy evaluation problem aims to estimate the average discounted reward Rπ for another target policy π ( a|s ) . Estimation via Value Function The value function for policy π is defined as the expected accumulated discounted future rewards started from a certain state : V π ( s ) = Eτ∼π [ ∑∞ t=0 γ trt|s0 = s ] . We use rπ ( s ) = Ea∼π ( ·|s ) [ r ( s , a ) ] to denote the average reward for state s given policy π . Under the definition , the value function can be seen as a fixed point of the Bellman equation : V π ( s ) = rπ ( s ) + γPπV π ( s ) , PπV π ( s ) : = Ea∼π ( ·|s ) , s′∼T ( ·|s , a ) [ V π ( s′ ) ] , ∀s ∈ S , ( 1 ) where PπV ( s ) is the average of the next value function given the current state s and policy π ; see Appendix A.1 for details . The value function and the expected reward Rπ is related in the following straightforward way Rπ = ( 1− γ ) Es∼µ0 [ V π ( s ) ] , ( 2 ) where the expectation is with respect to the distribution µ0 ( s ) of the initial states s0 at time t. Therefore , given an approximation V̂ of V π , and samples D0 : = { s ( i ) 0 } 1≤i≤n0 drawn from µ0 ( s ) , we can estimate Rπ by R̂πVAL [ V̂ ] = ( 1− γ ) n0 n0∑ i=1 V̂ ( s ( i ) 0 ) . Note that this estimator is off-policy in nature , since it requires no samples from the target policy π. Estimation via State Density Function Denote dπ , t ( · ) as average visitation of st in time step t. The state density function , or the discounted average visitation , is defined as : dπ ( s ) : = lim T→∞ ∑T t=0 γ tdπ , t ( s ) ∑T t=0 γ t = ( 1− γ ) ∞∑ t=0 γtdπ , t ( s ) , 1For average case when γ = 1 , the definition of Rπ is the same . However , the definition of value function is different . We will assume γ < 1 throughout our main paper for simplicity ; for average case check appendix B for more details . where ( 1− γ ) can be viewed as the normalization factor introduced by ∑∞ t=0 γ t. Similar to Bellman equation for value function , the state density function can also be viewed as a fixed point to the following recursive equation ( Liu et al. , 2018a , Lemma 3 ) : dπ ( s ′ ) = ( 1− γ ) µ0 ( s′ ) + γT πdπ ( s′ ) , where T πdπ ( s′ ) : = ∑ s , a T ( s′|s , a ) π ( a|s ) dπ ( s ) . ( 3 ) The operator T π is an adjoint operator of Pπ used in ( 1 ) ; see Appendix A.1 for a discussion . If the density function dπ is known , it provides an alternative way for estimating the expected reward Rπ , by noting that Rπ = Es∼dπ , a∼π ( ·|s ) [ r ( s , a ) ] . ( 4 ) We can see that both density function dπ and value function V π can be used to estimate the expected reward Rπ . We clarify the connection in detail in Appendix A.1 . Off-Policy State Visitation Importance Sampling Equation ( 4 ) can not be directly used for offpolicy estimation , since it involves expectation under the behavior policy π. Liu et al . ( 2018a ) addressed this problem by introducing a change of measures via importance sampling : Rπ = Es∼dπ0 , a∼π0 ( ·|s ) [ wπ/π0 ( s ) π ( a|s ) π0 ( a|s ) r ( s , a ) ] , with wπ/π0 ( s ) = dπ ( s ) dπ0 ( s ) , ( 5 ) where wπ/π0 ( s ) is the density ratio function of dπ and dπ0 . Given an approximation ŵ of wπ/π0 , and samples D = { s ( i ) t , a ( i ) t , r ( i ) t } 1≤i≤n,0≤t≤T collected from policy π0 , we can estimate Rπ as : R̂πSIS [ ŵ ] = 1 Z n∑ i=1 T∑ t=0 γtŵ ( s ( i ) t ) π ( ai|si ) π0 ( ai|si ) ri , Z = n∑ i=1 T∑ t=0 γtŵ ( s ( i ) t ) π ( a ( i ) t |s ( i ) t ) π0 ( a ( i ) t |s ( i ) t ) , ( 6 ) where Z is the normalized constant of the importance weights . 3 DOUBLY ROBUST ESTIMATOR . Doubly robust estimator is first proposed into reinforcement learning community to solve contextual bandit problem by Dudı́k et al . ( 2011 ) as an estimator combining inverse propensity score ( IPS ) estimator and direct method ( DM ) estimator . Jiang & Li ( 2016 ) introduce the idea of doubly robust estimator into off-policy evaluation in reinforcement learning . It incorporates an approximate value function as a control variate to reduce the variance of importance sampling estimator . Inspired by previous works , we propose a new doubly robust estimator based on our infinite horizon off-policy estimator R̂πSIS . 3.1 DOUBLY ROBUST ESTIMATOR FOR INFINITE HORIZON MDP . The value-based estimator R̂πVAL [ V̂ ] and density-ratio-based estimator R̂ π SIS [ ŵ ] are expected to be accurate when V̂ and ŵ are accurate , respectively . Our goal is to combine their advantages , obtaining a doubly robust estimator that is accurate once either V̂ or ŵ or is accurate . To simplify the problem , it is useful to examine the limit of infinite samples , with which R̂πVAL [ V̂ ] and R̂πSIS [ ŵ ] converge to the following limit of expectations : RπSIS [ ŵ ] : = lim n , T→∞ R̂πSIS [ ŵ ] = ∑ s rπ ( s ) dπ0 ( s ) ŵ ( s ) , ( 7 ) RπVAL [ V̂ ] : = lim n0→∞ R̂πVAL [ V̂ ] = ( 1− γ ) ∑ s V̂ ( s ) µ0 ( s ) . ( 8 ) Here and throughout this work , we assume V̂ and ŵ are fixed pre-defined approximations , and only consider the randomness from the data D. A first observation is that we expect to have rπ ≈ V̂ − γPπV̂ by Bellman equation ( 1 ) , when V̂ approximates the true value V π . Plugging this into RπSIS [ ŵ ] in Equation ( 7 ) , we obtain the following “ bridge estimator ” , which incorporates information from both ŵ and V̂ : Rπbridge [ V̂ , ŵ ] = ∑ s ( V̂ ( s ) − γPπV̂ ( s ) ) dπ0 ( s ) ŵ ( s ) , ( 9 ) where operator Pπ is defined in Bellman equation ( 1 ) . The corresponding empirical estimator is defined by R̂πbridge [ V̂ , ŵ ] = n∑ i=1 T−1∑ t=0 ( 1 Z1 γtŵ ( s ( i ) t ) V̂ ( s ( i ) t ) − 1 Z2 γt+1ŵ ( s ( i ) t ) π ( ai|si ) π0 ( ai|si ) V̂ ( s ( i ) t+1 ) ) , ( 10 ) where Z1 = ∑n i=1 ∑T−1 t=0 γ tŵ ( s ( i ) t ) and Z2 = ∑n i=1 ∑T−1 t=0 γ t+1ŵ ( s ( i ) t ) βπ/π0 ( a ( i ) t |s ( i ) t ) are selfnormalized constant of important weights each empirical estimation . However , directly estimating Rπ using the bridge estimator R̂πbridge [ V̂ , ŵ ] yields a poor estimation , because it includes the errors from both ŵ and V̂ and can be “ doubly worse ” . However , we can construct our “ doubly robust ” estimator by canceling Rπbridge [ V̂ , ŵ ] out from R π SIS [ ŵ ] and R π VAL [ V̂ ] : RπDR [ V̂ , ŵ ] = ∑ s rπ ( s ) dπ0 ( s ) ŵ ( s ) ︸ ︷︷ ︸ RπSIS [ ŵ ] + ( 1− γ ) ∑ s V̂ ( s ) µ0 ( s ) ︸ ︷︷ ︸ RπVAL [ V̂ ] − ∑ s ( V̂ ( s ) − γPπV̂ ( s ) ) dπ0 ( s ) ŵ ( s ) ︸ ︷︷ ︸ Rπbridge [ V̂ , ŵ ] . ( 11 ) And its corresponding empirical estimator can be written as : R̂πDR [ V̂ , ŵ ] = R̂ π SIS [ ŵ ] + R̂ π VAL [ V̂ ] − R̂πbridge [ V̂ , ŵ ] . Doubly Robust Bias Reduction The double robustness of R̂πDR [ V̂ , ŵ ] is reflected in the following key theorem , which shows that it is accurate once either V̂ or ŵ is accurate . Theorem 3.1 ( Doubly Robustness ) . Let RπDR [ V̂ , ŵ ] : = limn0 , n , T→∞ R̂πDR [ V̂ , ŵ ] be the limit of R̂πDR when it has infinite samples . Following the definition above , we have RπDR [ V̂ , ŵ ] −Rπ = Es∼dπ0 [ εŵ ( s ) εV̂ ( s ) ] , ( 12 ) where εV̂ and εŵ are errors of V̂ and ŵ , respective , defined as follows εŵ = dπ ( s ) dπ0 ( s ) − ŵ ( s ) , εV̂ ( s ) = V̂ ( s ) − r π ( s ) − γPπV̂ ( s ) . The error εŵ of ŵ is measured by the difference with the true density ratio dπ ( s ) /dπ0 ( s ) , and the error εV̂ of V̂ is measured using the Bellman residual . From the theorem we can see that , if V̂ is exact ( V̂ ≡ V π ) , we have εV̂ ≡ 0 ; if ŵ is exact ( ŵ ≡ dπ/dπ0 ) , we have εŵ ≡ 0 . Therefore , our estimator is consistent ( i.e. , limn , n0→∞ R̂πDR [ V̂ , ŵ ] = Rπ ) if either V̂ or ŵ are exact . The estimator is thus doubly robust in this sense . In contrast , R̂πSIS [ ŵ ] and R̂πVAL [ V̂ ] can be more sensitive to the error of ŵ and V̂ , respectively : RπSIS [ ŵ ] −Rπ = Es∼dπ0 [ εŵ ( s ) r π ( s ) ] , RπVAL [ V̂ ] −Rπ = Es∼dπ0 [ wπ/π0 ( s ) εV̂ ( s ) ] . Variance Analysis Different from the bias reduction , the doubly robust estimator does not guarantee to the reduce the variance over R̂πSIS [ ŵ ] or R̂ π VAL [ V̂ ] in general . However , as we show in the following result , we can break the variance of R̂πDR [ V̂ , ŵ ] into two parts . The first is the variance of R̂πVAL [ V̂ ] which is often relatively small . The second is generally no greater than the variance of R̂πSIS [ ŵ ] , and can be much smaller when V̂ ≈ V π . Moreover , R̂πVAL [ V̂ ] and R̂πSIS [ ŵ ] avoid the curse of horizon by design , so their variances tend to be much smaller than the corresponding estimators that apply IPS correction on the trajectories . Algorithm 1 Infinite Horizon Doubly Robust Estimator Input : Transition data Dπ0 = { s ( i ) t , a ( i ) t , r ( i ) t } 1≤i≤n,0≤t≤T from policy π0 ; D0 = { s ( j ) 0 } 1≤j≤n0 be samples from initial distribution µ0 target policy π , value function estimate V̂ ; density ratio estimate ŵ . Estimation : Use R̂πDR in ( 11 ) to estimate Rπ using sample from D and D0 . Proposition 3.1 ( Variance Analysis ) . Assume R̂πDR [ V̂ , ŵ ] is estimated based on sampleD0 ∼ µ0 and Dπ0 ∼ dπ0 , which we assume to be independent of each other . We have VarD0 , Dπ0 [ R̂πDR [ V̂ , ŵ ] ] = VarD0 [ R̂πVAL [ V̂ ] ] + VarDπ0 [ R̂πres [ V̂ , ŵ ] ] , ( 13 ) with R̂πres [ V̂ , ŵ ] : = R̂ π SIS [ ŵ ] − R̂πbridge [ V̂ , ŵ ] = ÊDπ0 [ ε̂V̂ ( s ) ŵ ( s ) ] and ε̂V̂ ( s ) the TD error of V̂ , ε̂V̂ ( s ) = r̂ π ( s ) − V̂ ( s ) + γP̂πV̂ ( s ) , with r̂π ( s ) = r ( s , a ) π ( a|s ) /π0 ( a|s ) , P̂πV̂ ( s ) = π ( a|s ) /π0 ( a|s ) V̂ ( s′ ) . Therefore , VarDπ0 ( R̂ π res [ V̂ , ŵ ] ) can be small when V̂ is close to the true value V π , or ε̂V̂ ( s ) ≈ 0 . From the proposition we can see that when V̂ is close to the true value V π , the variance of the residual ε̂V̂ may be negligibly small compared to the variance of R̂ π SIS [ ŵ ] . A further comparison of the variances between R̂πres [ V̂ , ŵ ] and R̂ π SIS [ ŵ ] is provided in Appendix A.3 . In the case when the TD error ε̂V̂ is negligible , we have VarD0 , Dπ0 [ R̂πDR [ V̂ , ŵ ] ] ≈ VarD0 [ R̂πVAL [ V̂ ] ] . Typically , the variance of R̂πVAL [ V̂ ] is smaller than R̂ π SIS [ ŵ ] since the variance of importance sampling methods heavily depends on the effective sample size , which is less controllable compared to R̂πVAL [ V̂ ] . Therefore , the variance of our doubly robust estimator may even be smaller than that of R̂πSIS [ ŵ ] in practice . The variance in ( 13 ) is a sum of two terms because of the assumption that samples from µ0 and dπ0 are independent . In practice , they have dependency but it is possible to couple the samples from µ0 and dπ0 in a certain way to further decrease the variance , which we leave as future work . Proposed Algorithm for Off-Policy Evaluation Suppose we have already obtained V̂ and ŵ , as estimations of V π and wπ/π0 , respectively , we can directly use equation ( 11 ) to estimate R π . A detail procedure is described in Algorithm 1 .

Summary: This paper proposes a new algorithm for the off-policy evaluation problem in reinforcement learning. It combines the value function learning method and the stationary distribution ratio estimators. The proposed method achieves double robustness, which means the proposed estimator is consistent as long as the value function or ratio estimator is consistent. Empirical results on some control domains are presented to verify the effectiveness of the algorithm.