Universal Off-Policy Evaluation

1 Introduction . Problems requiring sequential decision-making are ubiquitous [ 5 ] . When online experimentation is costly or dangerous , it is essential to conduct off-policy evaluation before deploying a new policy ; that is , one must leverage existing data collected using some policy β ( called a behavior policy ) to evaluate a performance metric of another policy π ( called the evaluation policy ) . For problems with high stakes , such as in terms of health [ 56 ] or financial assets [ 86 ] , it is also crucial to provide high-confidence bounds on the desired performance metric to ensure reliability and safety . Perhaps the most widely studied performance metric in the off-policy setting is the expected return [ 83 ] . However , this metric can be limiting for many problems of interest . Safety-critical applications , such as automated healthcare , require minimizing the chances of risk-prone outcomes , and so performance metrics such as value at risk ( VaR ) or conditional value at risk ( CVaR ) are more appropriate [ 49 , 14 ] . By contrast , applications like online recommendations are subject to noisy data and call for robust metrics like the median and other quantiles [ 2 ] . In order to improve user experiences , applications involving direct human-machine interaction , such as robotics and autonomous driving , focus on minimizing uncertainty in their outcomes and thus use metrics like variance and entropy [ 52 , 84 ] . Recent work in distributional reinforcement learning ( RL ) have also investigated estimating the cumulative distribution of returns [ 7 , 24 ] and its various statistical functionals [ 76 ] . While it may even be beneficial to use all of these different metrics simultaneously to inform better decision-making , even individually estimating and bounding any performance metric , other than mean and variance , in the off-policy setting has remained an open problem . This raises the main question of interest : How do we develop a universal off-policy method—one that can estimate any desired performance metrics and can also provide finite-sample confidence bounds that hold simultaneously with high probability for those metrics ? 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Prior Work : Off-policy methods can be broadly categorized as model-based or model-free [ 83 ] . Model-based methods typically require strong assumptions on the parametric model when statistical guarantees are needed . Further , using model-based approaches to estimate parameters other than the mean can also require estimating the distribution of rewards for every state-action pair in order to obtain the complete return distribution for any policy . By contrast , model-free methods are applicable to a wider variety of settings . Unfortunately , the popular technique of using importance-weighted returns [ 71 ] only corrects for the mean under the off-policy distribution . Recent work by Chandak et al . [ 18 ] provides a specialized extension to only correct for the variance . Outside RL , works in the econometrics and causal inference literature have also considered quantile treatments [ 29 , 99 ] and inferences on counterfactual distributions [ 28 , 20 , 36 ] , but these methods are not developed for sequential decisions and do not provide any high-confidence bounds with guaranteed coverage . Further , they often mandate stationarity , identically distributed data , and full observability ( i.e. , no confounding ) . Existing frequentist high-confidence bounds are not only specifically designed for either the mean or variance , but also hold only individually [ 92 , 45 , 18 ] . Instead of frequentist intervals , a Bayesian posterior distribution over the mean return and various statistics of that distribution can also be obtained [ 105 ] . We are not aware of any method that provides off-policy bounds or even estimates for any parameter of the return , while also handling different domain settings that are crucial for RL related tasks . Therefore , a detailed discussion of existing work is deferred to Appendix C. Contributions : We take the first steps towards a universal off-policy estimator ( UnO ) that estimates and bounds the entire distribution of returns , and then derives estimates and simultaneous bounds for all parameters of interest . With UnO , we make the following contributions : A . For any distributional parameter ( mean , variance , quantiles , entropy , CVaR , CDF , etc . ) , we provide an off-policy method to obtain ( A.1 ) model-free estimators ; ( A.2 ) high-confidence bounds that have guaranteed coverage simultaneously for all parameters and that , perhaps surprisingly , often nearly match or outperform prior bounds specifically designed for the mean and the variance ; and ( A.3 ) approximate bounds using statistical bootstrapping that can often be significantly tighter . B . The above advantages hold for ( B.1 ) fully observable and partially observable ( i.e. , with unobserved confounders ) settings , ( B.2 ) Markovian and non-Markovian settings , and ( B.3 ) settings with stationary , smoothly non-stationary , and discrete distribution shifts in a policy ’ s performance . Limitations : Our method uses importance sampling and thus ( 1 ) Requires knowledge of action probabilities under the behavior policy β , ( 2 ) Any outcome under the evaluation policy should have a sufficient probability of occurring under β , and ( 3 ) Variance of our estimators scales exponentially with the horizon length [ 39 , 57 ] , which may be unavoidable in non-Markovian domains [ 46 ] . Notation : For brevity , we first restrict our focus to the stationary setting . In Section 5 , we discuss how to tackle non-stationarity and distribution shifts . A partially observable Markov decision process ( POMDP ) is a tuple ( S , O , A , P , Ω , R , γ , d0 ) , where S is the set of states , O is the set of observations , A is the set of actions , P is the transition function , Ω is the observation function , R is the reward function , γ ∈ [ 0 , 1 ] is the discount factor , and d0 is the starting state distribution . Although our results extend to the continuous setting , for notational ease , we consider S , A , O , and the set of rewards to be finite . Since the true underlying states are only partially observable , the resulting rewards and transitions from one partially observed state to another are therefore also potentially non-Markovian [ 80 ] . We write St , Ot , At , and Rt to denote random variables for state , observation , action , and reward respectively at time t. Let D be a data set ( Hi ) ni=1 collected using behavior policies ( βi ) ni=1 , where each Hi denotes the observed trajectory ( O0 , A0 , β ( A0|O0 ) , R0 , O1 , ... ) . Notice that an observed trajectory contains β ( At|Ot ) and does not contain the states St , for all t. Let Gi : = ∑T j=0 γ jRj be the return of Hi , where ∀i , Gmin < Gi < Gmax for some finite constants Gmin and Gmax , and T is a finite horizon length . Let Gπ and Hπ be the random variables for returns and complete trajectories under any policy π , respectively . Since the set of observations , actions , and rewards are finite , and T is finite , the total number of possible trajectories is finite . Let X be the finite set of returns corresponding to these trajectories . Let Hπ be the set of all possible trajectories for any policy π . Sometimes , to make the dependence explicit , we write g ( h ) to denote the return of trajectory h. Further , to ensure that samples in D are informative , we make a standard assumption that any outcome under π has sufficient probability of occurring under β ( see Appendix B.1 for further discussion of assumptions in general ) , Assumption 1 . The set D contains independent ( not necessarily identically distributed ) observed trajectories generated using ( βi ) ni=1 , such that for some ( unknown ) ε > 0 , ( βi ( a|o ) < ε ) =⇒ ( π ( a|o ) = 0 ) , for all o ∈ O , a ∈ A , and i ∈ { 1 , 2 , ... , n } . 2 Idea Summary . For the desired universal method , instead of considering each parameter individually , we suggest estimating the entire cumulative distribution function ( CDF ) of returns first : ∀ν ∈ R , Fπ ( ν ) : = Pr ( Gπ ≤ ν ) . Any distributional parameter , ψ ( Fπ ) , can then be estimated from the estimate of Fπ . However , we only have off-policy data from a behavior policy β , and the typical use of importance sampling [ 71 ] only corrects for the mean return . To overcome this , we propose an estimator F̂n that uses importance sampling from the perspective of the CDF to correct for the entire distribution of returns . The CDF estimate , F̂n , is then used to obtain a plug-in estimator ψ ( F̂n ) for any distributional parameter ψ ( Fπ ) . Next , we show that this CDF-centric perspective provides the additional advantage that , if we can compute a 1− δ confidence band F : R→ 2R such that Pr ( ∀ν ∈ R , Pr ( Gπ ≤ ν ) ∈ F ( ν ) ) ≥ 1− δ , then a 1− δ upper ( or lower ) high-confidence bound on any parameter , ψ ( Fπ ) , can be obtained by searching for a function F that maximizes ( or minimizes ) ψ ( F ) and ∀ν ∈ R has F ( ν ) ∈ F ( ν ) . 3 UnO : Universal Off-Policy Estimator . In the on-policy setting , one approach for estimating any parameter of returns , Gπ , might be to first estimate its cumulative distribution Fπ and then use that to estimate its parameter ψ ( Fπ ) . However , doing this in the off-policy setting requires additional consideration as the entire distribution of the observed returns needs to be adjusted to estimate Fπ since the data is collected using behavior policies that can be different from the evaluation policy π . We begin by observing that ∀ν ∈ R , Fπ ( ν ) can be expanded using the fact that the probability that the return Gπ equals x is the sum of the probabilities of the trajectories Hπ whose return equals x , Fπ ( ν ) = Pr ( Gπ ≤ ν ) = ∑ x∈X , x≤ν Pr ( Gπ = x ) = ∑ x∈X , x≤ν ( ∑ h∈Hπ Pr ( Hπ = h ) 1 { g ( h ) =x } ) , ( 1 ) where 1A = 1 if A is true and 0 otherwise . Now , observing that the indicator function can be one for at most a single value less than ν as g ( h ) is a deterministic scalar given h , ( 1 ) can be expressed as , Fπ ( ν ) = ∑ h∈Hπ Pr ( Hπ = h ) ∑ x∈X , x≤ν 1 { g ( h ) =x } = ∑ h∈Hπ Pr ( Hπ = h ) ( 1 { g ( h ) ≤ν } ) , where the red color is used to highlight changes . Now , from Assumption 1 as ∀β , Hπ ⊆Hβ ,1 Fπ ( ν ) = ∑ h∈Hβ Pr ( Hπ = h ) ( 1 { g ( h ) ≤ν } ) = ∑ h∈Hβ Pr ( Hβ = h ) Pr ( Hπ = h ) Pr ( Hβ = h ) ( 1 { g ( h ) ≤ν } ) . ( 2 ) The form of Fπ ( ν ) in ( 2 ) is beneficial as it suggests a way to not only perform off-policy corrections for one specific parameter , as in prior works [ 71 , 18 ] , but for the entire cumulative distribution function ( CDF ) of return Gπ . Formally , let ρi : = ∏T j=0 π ( Aj |Oj ) βi ( Aj |Oj ) denote the importance ratio for Hi , which is equal to Pr ( Hπ = h ) /Pr ( Hβ = h ) ( see Appendix D ) . Then , based on ( 2 ) , we propose the following non-parametric and model-free estimator for Fπ . ∀ν ∈ R , F̂n ( ν ) : = 1 n n∑ i=1 ρi1 { Gi≤ν } . ( 3 ) 1Results can be extended to hybrid probability measures using Radon-Nikodym derivatives . Figure 1 provides intuition for ( 3 ) . In the following theorem , we establish that this estimator , F̂n , is unbiased and not only pointwise consistent , but also a uniformly consistent estimator of Fπ , even when the data D is collected using multiple behavior policies ( βi ) ni=1 . The proof ( deferred to Appendix D ) also illustrates that by using knowledge of action probabilities under the behavior policies , no additional adjustments ( e.g. , front-door or backdoor [ 70 ] ) are required by F̂n to estimate Fπ , even when the domain is non-Markovian or has partial observability ( confounders ) . Theorem 1 . Under Assumption 1 , F̂n is an unbiased and uniformly consistent estimator of Fπ , ∀ν ∈ R , ED [ F̂n ( ν ) ] = Fπ ( ν ) , sup ν∈R ∣∣∣F̂n ( ν ) − Fπ ( ν ) ∣∣∣ a.s.−→ 0 . Remark 1 . Notice that the value of F̂n ( ν ) can be more than one , even though Fπ ( ν ) can not have a value greater than one for any ν ∈ R. This is an expected property of estimators based on importance sampling ( IS ) . For example , the IS estimates of expected return during off-policy mean estimation can be smaller or larger than the smallest and largest possible return when ρ > 1 . Having an estimator F̂n of Fπ , any parameter ψ ( Fπ ) can now be estimated using ψ ( F̂n ) . However , some parameters like the mean µπ , variance σ2π , and entropy Hπ , are naturally defined using the probability distribution dFπ instead of the cumulative distribution Fπ . Similarly , parameters like the α-quantile Qαπ and inter-quantile range ( which provide tail-robust measures for the mean and deviation from the mean ) and conditional value at risk CVaRαπ ( which is a tail-sensitive measure ) are defined using the inverse CDF F−1π ( α ) . Therefore , let ( G ( i ) ) n i=1 be the order statistics for samples ( Gi ) n i=1 and G ( 0 ) : = Gmin . Then , we define the off-policy estimator of the inverse CDF for all α ∈ [ 0 , 1 ] , and the probability distribution estimator dF̂n as , F̂−1n ( α ) : = min { g ∈ ( G ( i ) ) ni=1 ∣∣∣F̂n ( g ) ≥ α } , dF̂n ( G ( i ) ) : = F̂n ( G ( i ) ) − F̂n ( G ( i−1 ) ) , ( 4 ) where dF̂n ( ν ) : = 0 if ν 6= G ( i ) for any i ∈ ( 1 , . . . , n ) . Using ( 4 ) , we now define off-policy estimators for parameters like the mean , variance , quantiles , and CVaR ( see Appendix E.1 for more details on these ) . This procedure can be generalized to any other parameter of Fπ for which a sample estimator ψ ( F̂n ) can be directly created using F̂n as a plug-in estimator for Fπ . µπ ( F̂n ) : = n∑ i=1 dF̂n ( G ( i ) ) G ( i ) , σ2π ( F̂n ) : = n∑ i=1 dF̂n ( G ( i ) ) ( G ( i ) − µπ ( F̂n ) ) 2 , Qαπ ( F̂n ) : = F̂ −1 n ( α ) , CVaR α π ( F̂n ) : = 1 α n∑ i=1 dF̂n ( G ( i ) ) G ( i ) 1 { G ( i ) ≤Qαπ ( F̂n ) } . Remark 2 . Let Hi be the observed trajectory for the Gi that gets mapped to G ( i ) when computing the order statistics . Note that dF̂n ( G ( i ) ) equals ρi/n for this Hi . This implies that the estimator for the mean , µπ ( F̂n ) , reduces exactly to the existing full-trajectory-based IS estimator [ 71 ] . Notice that many parameters and their sample estimates discussed above are nonlinear in Fπ and F̂n , respectively ( the mean is one exception ) . Therefore , even though F̂n is an unbiased estimator of Fπ , the sample estimator , ψ ( F̂n ) , may be a biased estimator of ψ ( Fπ ) . This is expected behavior because even in the on-policy setting it is not possible to get unbiased estimates of some parameters ( e.g. , standard deviation ) , and UnO reduces to the on-policy setting when π = β . However , perhaps surprisingly , we establish in the following section that even when ψ ( F̂n ) is a biased estimator of ψ ( Fπ ) , high-confidence upper and lower bounds can still be computed for both Fπ and ψ ( Fπ ) .

Summary: The paper proposes an algorithm for estimating any parameter of the distribution of returns in an off-policy manner, and derives confidence intervals for the quantities of interest. The paper also shows how the proposed algorithm can be extended to deal with many issues commonly faced when applying RL to real world problems (nonstationarity, non-markovian observations, etc.).