Non-asymptotic Confidence Intervals of Off-policy Evaluation:  Primal and Dual Bounds 

1 INTRODUCTION . Off-policy evaluation ( OPE ) seeks to estimate the expected reward of a target policy in reinforcement learnings ( RL ) from observational data collected under different policies ( e.g. , Murphy et al. , 2001 ; Fonteneau et al. , 2013 ; Jiang & Li , 2016 ; Liu et al. , 2018a ) . OPE plays a central role in applying reinforcement learning ( RL ) with only observational data and has found important applications in areas such as medicine , self-driving , where interactive “ on-policy ” data is expensive or even infeasible to collect . A critical challenge in OPE is the uncertainty estimation , as having reliable confidence bounds is essential for making high-stakes decisions . In this work , we aim to tackle this problem by providing non-asymptotic confidence intervals of the expected value of the target policy . Our method allows us to rigorously quantify the uncertainty of the prediction and hence avoid the dangerous case of being overconfident in making costly and/or irreversible decisions . However , off-policy evaluation per se has remained a key technical challenge in the literature ( e.g. , Precup , 2000 ; Thomas & Brunskill , 2016 ; Jiang & Li , 2016 ; Liu et al. , 2018a ) , let alone gaining rigorous confidence estimation of it . This is especially true when 1 ) the underlying RL problem is long or infinite horizon , and 2 ) the data is collected under arbitrary and unknown algorithms ( a.k.a . behavior-agnostic ) . As a consequence , the collected data can exhibit arbitrary dependency structure , which makes constructing rigorous non-asymptotic confidence bounds particularly challenging . Traditionally , the only approach to provide non-asymptotic confidence bounds in OPE is to combine importance sampling ( IS ) with concentration inequalities ( e.g. , Thomas et al. , 2015a ; b ) , which , however , tends to degenerate for long/infinite horizon problems ( Liu et al. , 2018a ) . Furthermore , ∗Equal contribution . neither can this approach be applied to the behavior-agnostic settings , nor can it effectively handle the complicated time dependency structure inside individual trajectories . Instead , it requires to use a large number of independently collected trajectories drawn under known policies . In this work , we provide a practical approach for Behavior-agnostic , Off-policy , Infinite-horizon , Non-asymptotic , Confidence intervals based on arbitrarily Dependent data ( BONDIC ) . Our method is motivated by a recently proposed optimization-based ( or variational ) approach to estimating OPE confidence bounds ( Feng et al. , 2020 ) , which leverages a tail bound of kernel Bellman statistics ( Feng et al. , 2019 ) . Our approach achieves a new bound that is both an order-of-magnitude tighter and computationally efficient than that of Feng et al . ( 2020 ) . Our improvements are based on two pillars 1 ) developing a new primal-dual perspective on the non-asymptotic OPE confidence bounds , which is connected to a body of recent works on infinite-horizon value estimation ( Liu et al. , 2018a ; Nachum et al. , 2019a ; Tang et al. , 2020a ; Mousavi et al. , 2020 ) ; and 2 ) offering a new tight concentration inequality on the kernel Bellman statistics that applies to behavior-agnostic off-policy data with arbitrary dependency between transition pairs . Empirically , we demonstrate that our method can provide reliable and tight bounds on a variety of well-established benchmarks . Related Work Besides the aforementioned approach based on the combination of IS and concentration inequalities ( e.g. , Thomas et al. , 2015a ) , bootstrapping methods have also been widely used in off-policy estimation ( e.g. , White & White , 2010 ; Hanna et al. , 2017 ; Kostrikov & Nachum , 2020 ) . But the latter is limited to asymptotic bounds . Alternatively , Bayesian methods ( e.g . Engel et al. , 2005 ; Ghavamzadeh et al. , 2016a ) offers a different way to estimate the uncertainty in RL , but fails to guarantee frequentist coverage . In addition , Distributed RL ( Bellemare et al. , 2017 ) seeks to quantify the intrinsic uncertainties inside the Markov decision process , which is orthogonal to the estimation of uncertainty that we consider . Our work is built upon the recent advances in behavior-agnostic infinite-horizon OPE , including Liu et al . ( 2018a ) ; Feng et al . ( 2019 ) ; Tang et al . ( 2020a ) ; Mousavi et al . ( 2020 ) , as well as the DICE-family ( e.g. , Nachum et al. , 2019a ; Zhang et al. , 2020a ; Yang et al. , 2020b ) . In particular , our method can be viewed as extending the minimax framework of the infinite-horizon OPE in the infinite data region by Tang et al . ( 2020a ) ; Uehara et al . ( 2020 ) ; Jiang & Huang ( 2020 ) to the non-asymptotic finite sample region . Outline For the rest of the paper , we start with the problem statement in Section 2 , and an overview on the two dual approaches to infinite-horizon OPE that are tightly connected to our method in Section 3 . We then present our main approach in Section 4 and perform empirical studies in Section 5 . The proof and an abundance of additional discussions can be found in Appendix . 2 BACKGROUND , DATA ASSUMPTION , PROBLEM SETTING . Consider an agent acting in an unknown environment . At each time step t , the agent observes the current state st in a state space S , takes an action at ∼ π ( · | st ) in an action space A according to a given policy π ; then , the agent receives a reward rt and the state transits to s′t = st+1 , following an unknown transition/reward distribution ( rt , st+1 ) ∼ P ( · | st , at ) . Assume the initial state s0 is drawn from an known initial distribution D0 . Let γ ∈ ( 0 , 1 ) be a discount factor . In this setting , the expected reward of π is defined as Jπ : = Eπ [ ∑T t=0 γ trt | s0 ∼ D0 ] , which is the expected total discounted rewards when we execute π starting from D0 for T steps . In this work , we consider the infinite-horizon case with T → +∞ . Our goal is to provide an interval estimation of Jπ for a general and challenging setting with significantly released constraints on the data . In particular , we assume the data is behavior-agnostic and off-policy , which means that the data can be collected from multiple experiments , each of which can execute a mix of arbitrary , unknown policies , or even follow a non-fixed policy . More concretely , suppose that the model P is unknown , and we have a set of transition pairs D̂n = ( si , ai , ri , s′i ) n i=1 collected from previous experiments in a sequential order , such that for each data point i , the ( ri , s′i ) is drawn from the model P ( · | si , ai ) , while ( si , ai ) is generated with an arbitrary black box given the previous data points . We formalize both the data assumption and goal as below . Assumption 2.1 ( Data Assumption ) . Assume the data D̂n = ( si , ai , ri , s′i ) ni=1 is drawn from an arbitrary joint distribution , such that for each i = 1 , . . . , n , conditional on D̂ < i : = ( sj , aj , rj , s′j ) j < i ∪ ( si , ai ) , the subsequent local reward and next state ( ri , s′i ) are drawn from P ( · | si , ai ) . Goal Given a confidence level δ ∈ ( 0 , 1 ) , we want to construct an interval [ Ĵ− , Ĵ+ ] ⊂ R based on the data D̂n , such that Pr ( Jπ ∈ [ Ĵ− , Ĵ+ ] ) ≥ 1− δ , where Pr ( · ) is w.r.t . the randomness of the data . The partial ordering on the data points is introduced to accommodate the case that si+1 equals s′j for some j ≤ i . The data assumption only requires that ( ri , s′i ) is generated from P ( · | si , ai ) , and imposes no constraints on how ( si , ai ) is generated . This provides great flexibility in terms of the data collection process . In particular , we do not require ( si , ai ) ni=1 to be independent as always assumed in recent works ( Liu et al. , 2018a ; Mousavi et al. , 2020 ) . A crucial fact is that our data assumption actually implies a martingale structure on the empirical Bellman residual operator of the Q-function , As we will show in Section 4.1 , this enables us to derive a key concentration inequality underpinning our non-asymptotic confidence bounds . Here , we summarize a few notations that will simplify the presentation in the rest of work . First of all , we append each ( si , ai , ri , s′i ) with an action a ′ i ∼ π ( · | s′i ) following s′i . This can be done for free as long as π is given ( See the Remark in Section 3 ) . Also , we write xi = ( si , ai ) , x′i = ( s ′ i , a ′ i ) , and yi = ( x′i , ri ) = ( s ′ i , a ′ i , ri ) . Correspondingly , define X = S ×A to be the state-action space and Y = X ×R . Denote Pπ ( y | x ) = P ( s′ , r | x ) π ( a′ | s′ ) . In this way , the observed data can be written as pairs of { xi , yi } ni=1 , and Assumption 2.1 is equivalent to saying that yi ∼ Pπ ( · | xi ) given D̂ < i , which is similar to a supervised learning setting . We equalize the data D̂n with its empirical measure D̂n = ∑n i=1 δxi , yi/n , where δ is the Delta measure . 3 TWO DUAL APPROACHES TO INFINITE-HORIZON OFF-POLICY ESTIMATION . The deficiency of the traditional IS methods on long-/infinite-horizon RL problems ( a.k.a . the curse of horizon ( Liu et al. , 2018a ) ) has motivated a line of work on developing efficient infinite-horizon value estimation ( e.g. , Liu et al. , 2018a ; Feng et al. , 2019 ; Nachum et al. , 2019a ; Zhang et al. , 2020a ; Mousavi et al. , 2020 ; Tang et al. , 2020a ) . The main idea is to transform the value estimation problem into estimating either the Q-function or the visitation distribution ( or its related density ratio ) of the policy π . This section introduces and reinterprets these two tightly connected methods , which serves to lay out a foundation for our main confidence bounds from a primal and dual perspective . Given a policy π , its Q-function is defined as qπ ( x ) = Eπ [ ∑∞ t=0 γ trt | x0 = x ] , where the expectation is taken when we execute π initialized from a fixed state-action pair ( s0 , a0 ) = x0 = x . Let Dπ , t be the distribution of ( xt , yt ) = ( st , at , s′t , a ′ t , rt ) when executing policy π starting from s0 ∼ D0 for t steps . The visitation distribution of π is defined as Dπ = ∑∞ t=0 γ tDπ , t . Note that Dπ integrates to 1/ ( 1− γ ) , while we treat it as a probability measure in the notation . The expected reward Jπcan be expressed using either qπ or Dπ as follows : Jπ : = Eπ [ ∞∑ t=0 γtrt ] = Er∼Dπ [ r ] = Ex∼Dπ,0 [ qπ ( x ) ] , ( 1 ) where r ∼ Dπ ( resp . x ∼ Dπ,0 ) denotes sampling from the r- ( resp . x- ) marginal distribution of Dπ ( resp . Dπ,0 ) . Eq . ( 1 ) plays a key role in the infinite-horizon value estimation by transforming the estimation of Jπ into estimating either qπ or Dπ . Value Estimation via Q Function Because Dπ,0 ( x ) = D0 ( s ) π ( a|s ) is known , we can estimate Jπ by Ex∼Dπ,0 [ q̂ ( x ) ] with any estimation q̂ of the true Q-function qπ ; the expectation under x ∼ Dπ,0 can be estimated to any accuracy with Monte Carlo . To estimate qπ , we consider the empirical and expected Bellman residual operator : R̂q ( x , y ) = q ( x ) − γq ( x′ ) − r , Rπq ( x ) = Ey∼Pπ ( ·|x ) [ R̂q ( x , y ) ] . ( 2 ) It is well-known that qπ is the unique solution of the Bellman equation Rπq = 0 . Since yi ∼ Pπ ( ·|xi ) for each data point in D̂n , if q = qπ , then R̂q ( xi , yi ) , i = 1 , . . . , n are all zero-mean random variables . Let ω be any function from X to R , then∑i R̂q ( xi , yi ) ω ( xi ) also has zero mean . This motivates the following functional Bellman loss ( Feng et al. , 2019 ; 2020 ; Xie & Jiang , 2020 ) , LW ( q ; D̂n ) : = sup ω∈W { 1 n n∑ i=1 R̂q ( xi , yi ) ω ( xi ) } , ( 3 ) whereW is a set of functions ω : X → R. To ensure that the sup is finite , W is typically set to be an unit ball of some normed function spaceWo , such thatW = { ω ∈ Wo : ‖ω‖Wo ≤ 1 } . Feng et al . ( 2019 ) considers the simple case whenW is taken to be the unit ball K of the reproducing kernel Hilbert space ( RKHS ) with a positive definite kernel k : X × X → R , in which case the loss has a simple closed form solution : LK ( q ; D̂n ) = √√√√ 1 n2 n∑ ij=1 R̂q ( xi , yi ) k ( xi , xj ) R̂q ( xj , yj ) . ( 4 ) Note that the RHS of Eq . ( 4 ) is the square root of the kernel Bellman V-statistics in Feng et al . ( 2019 ) . Feng et al . ( 2019 ) showed that , when the support of data distribution D̂n covers the whole space ( which may require an infinite data size ) and k is an integrally strictly positive definite kernel , LK ( q ; D̂n ) = 0 iff q = qπ . Therefore , one can estimate qπ by minimizing LK ( q , D̂n ) . Remark The empirical Bellman residual operator R̂ can be extended to R̂q ( x , y ) = q ( x ) − r − γ 1m ∑m ` =1 q ( s ′ , a′ ` ) , where { a′ ` } mi=1 are i.i.d . drawn from π ( ·|s′ ) . As m increases , this gives a lower variance estimation of Rπq . If m = +∞ , we have R̂q ( x , y ) = q ( x ) − r − γEa′∼π ( · | s′ ) [ q ( s′ , a′ ) ] , which coincides with the operator used in the expected SARSA ( Sutton & Barto , 1998 ) . In fact , without any modification , all results in this work can be applied to R̂q for any m. Value Estimation via Visitation Distribution Another way to estimate Jπ in Eq . ( 1 ) is to approximate Dπ with a weighted empirical measure of the data ( Liu et al. , 2018a ; Nachum et al. , 2019a ; Mousavi et al. , 2020 ; Zhang et al. , 2020a ) . The key idea is to assign an importance weight ω ( xi ) to each data point xi in D̂n . We can choose the function ω : X → R properly such that Dπ and hence Jπ can be approximated by the ω-weighted empirical measure of D̂n as follows : Jπ ≈ Ĵω : = ED̂ωn [ r ] = 1 n n∑ i=1 ω ( xi ) ri , Dπ ≈ D̂ωn : = 1 n n∑ i=1 ω ( xi ) δxi , yi . ( 5 ) Intuitively , ω can be viewed as the density ratio between Dπ and D̂n , although the empirical measure D̂n may not have well-defined density . Liu et al . ( 2018a ) ; Mousavi et al . ( 2020 ) proposed to estimate ω by minimizing a discrepancy measure between D̂ωn and Dπ . To see this , note that D = Dπ if and only if ∆ ( D , q ) = 0 for any function q , where ∆ ( D , q ) = ED [ γq ( x′ ) − q ( x ) ] − EDπ [ γq ( x′ ) − q ( x ) ] = ED [ γq ( x′ ) − q ( x ) ] + EDπ,0 [ q ( x ) ] , ( 6 ) using the fact that EDπ [ γq ( x′ ) − q ( x ) ] = −EDπ,0 [ q ( x ) ] ( Theorem 1 , Liu et al. , 2018a ) . Also note that the RHS of Eq . ( 6 ) can be practically calculated given any D and q without knowing Dπ . Let Q be a set of functions q : X → R. One can define the following loss for ω : IQ ( ω ; D̂n ) = sup q∈Q { ∆ ( D̂ωn , q ) } . ( 7 ) Similar to LW ( q ; D̂n ) , when Q is a ball in RKHS , IQ ( ω ; D̂n ) also has a bilinear closed form analogous to Eq . ( 4 ) ; see Mousavi et al . ( 2020 ) and Appendix F. As we show in Section 4 , IQ ( ω ; D̂n ) and LW ( q ; D̂n ) are connected to the primal and dual views of our confidence bounds , respectively .

Summary: This work constructs non-asymptotic confidence intervals for off-policy evaluation. This is achieved by assuming that the reward at any given time only depends on the state action pair, leveraging that assumed structure to define the difference between the empirical and estimated bellman residual operators as a Martingale difference sequence. This, in turn, then allows the authors to apply a Hoeffding-like concentration inequality which applies to Hilbert spaces. The authors then provide a derivation of the confidence bounds by considering the divergence between policies. The work improves on the rate of prior work from $O(n^{-\frac{1}{4}})$ to $O(n^{-\frac{1}{2}})$ and allows for estimation without the need of global optimality via the dual formulation, both of which are very nice additions to the literature. Experimental evaluation backs up the authors’ claims, showing very strong performance with respect to prior art.