Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies

1 INTRODUCTION . In many real-world decision-making scenarios , evaluating a novel policy by directly executing it in the environment is generally costly and can even be downright risky . Examples include evaluating a recommendation policy ( Swaminathan et al. , 2017 ; Zheng et al. , 2018 ) , a treatment policy ( Hirano et al. , 2003 ; Murphy et al. , 2001 ) , and a traffic light control policy ( Van der Pol & Oliehoek , 2016 ) . Off-policy policy evaluation methods ( OPPE ) utilize a set of previously-collected trajectories ( for example , website interaction logs , patient trajectories , or robot trajectories ) to estimate the value of a novel decision-making policy without interacting with the environment ( Precup et al. , 2001 ; Dudı́k et al. , 2011 ) . For many reinforcement learning applications , the value of the decision is defined in a long- or infinite-horizon , which makes OPPE more challenging . The state-of-the-art methods for infinite-horizon off-policy policy evaluation rely on learning ( discounted ) state stationary distribution corrections or ratios . In particular , for each state in the environments , these methods estimate the likelihood ratio of the long-term probability measure for the state to be visited in a trajectory generated by the target policy , normalized by the probability measure generated by the behavior policy . This approach can effectively avoid the exponentially high variance compared to the more classic importance sampling ( IS ) estimation methods ( pre ; Dudı́k et al. , 2011 ; Hirano et al. , 2003 ; Wang et al. , 2017 ; Murphy et al. , 2001 ) , especially for infinite-horizon policy evaluation ( Liu et al. , 2018 ; Nachum et al. , 2019 ; Hallak & Mannor , 2017 ) . However , learning state stationary distribution requires detailed information on distributions of the behavior policy , and we call them policy-aware methods . As a consequence , policy-aware methods are difficult to apply ∗Corresponding author . On leave from College of Computing , Georgia Institute of Technology . 2Equal Contribution . when off-policy data are pre-generated by multiple behavior policies or when the behavior policy ’ s form is unknown . To address this issue , Nachum et al . ( 2019 ) proposes a policy-agnostic method , DualDice , which learns the joint state-action stationary distribution correction that is much higher dimension , and therefore needs more model parameters than the state stationary distribution . Besides , there is no theoretic comparison between policy-aware and policy-agnostic methods . In this paper , we propose a OPPE method with behavior policy learning , EMP ( estimated mixture policy ) for infinite-horizon off-policy policy evaluation with multiple known or unknown behavior policies . We call EMP a partially policy-agnostic method in the sense that , EMP does not require any information on each “ physical ” behavior policy , instead , it utilizes some aggregated information of the behavior policies learned from data . In detail , EMP includes a pre-estimation step using certain parametric model to learn a “ virtual ” policy ( we call it the mixture policy and formally define it in Section 4 ) . Hence , its performance depends on the accuracy of mixture policy estimation . Like the method in Liu et al . ( 2018 ) , EMP obtain OPPE also via learning the state stationary distribution correction , so it remains computationally cheap and is scalable in terms of the number of behavior policies . Besides , inspired by Hanna et al . ( 2019 ) , we provide a theoretic guarantee that EMP yields smaller mean square error ( MSE ) than the policy-aware methods in stationary distribution corrections learning , even in the single-behavior policy setting . On the other hand , compared to DualDice , EMP learns the state stationary distribution correction of smaller dimension , more importantly the estimation of the mixture policy can be considered as an inductive bias as far as the stationary distribution correction is concerned , and hence could achieve better performance when the pre-estimation is not expensive . In addition , we propose an ad-hoc improvement of EMP , whose theoretical analysis is left for future studies . EMP is compared with both policy-aware and policy-agnostic methods in a set of continuous and discrete control tasks and shows significant improvement . 2 BACKGROUND AND RELATED WORK . We first introduce the general setting of OPPE in infinite horizon . Then we review two families of OPPE methods , based on importance sampling ( IS ) and stationary distribution correction learning , respectively . 2.1 INFINITE-HORIZON OFF-POLICY POLICY EVALUATION . We consider a Markov Decision Process ( MDP ) and our goal is to estimate the infinite-horizon average reward . The environment is specified by a tupleM = 〈S , A , R , T 〉 , consisting of a state space , an action space , a reward function , and a transition probability function . A policy π interacts with the environment iteratively , starting with an initial state s0 . At step n = 0 , 1 , ... , the policy produces a distribution π ( ·|sn ) over the actions A , from which an action an is sampled and applied to the environment . The environment stochastically produces a scalar reward r ( sn , an ) and a next state sn+1 ∼ T ( ·|sn , an ) . The infinite-horizon average reward under policy π is Rπ = lim N→∞ 1 N + 1 N∑ n=0 Eπ [ r ( sn , an ) ] . Without gathering new data , off-policy policy evaluation ( OPPE ) considers the problem of estimating the expected reward of a target policy π via pre-collected state-action-reward tuples from policies that are different from π , which are called behavior policies . In our paper , we consider the general setting that the data are generated by multiple behavior policies πj ( j = 1 , .. , m ) . Most OPPE literature has focused on the single-behavior-policy case where m = 1 . In this case , we denote the behavior policy by π0 to distinguish from the multiple-behavior-policy case . Roughly speaking , most OPPE methods can be grouped into two categories : importance-sampling ( IS ) based OPPE and stationary-distribution-correction based OPPE . 2.2 IMPORTANCE SAMPLING POLICY EVALUATION . As for short-horizon off-policy policy evaluation , importance sampling policy evaluation ( IS ) methods ( Precup et al. , 2001 ; Dudı́k et al. , 2011 ; Swaminathan et al. , 2017 ; Precup et al. , 2000 ; Horvitz & Thompson , 1952 ) have shown promising empirical results . The main idea of importance sampling based OPPE is using importance weighting π/πj to correct the mismatch between the target policy π and the behavior policy πj that generates the trajectory . One key element in our EMP method are inspired by importance sampling literature . Li et al . ( 2015 ) and Hanna et al . ( 2019 ) show that using estimated behavior policy in the importance weighting can reduce the mean square error ( MSE ) . EMP also uses estimated policy , but there are two key difference between EMP and the previous works : ( 1 ) EMP is not an IS-based method , it involves a min-max problem ; ( 2 ) EMP focuses on multiple-behavior-policy setting while these papers have focused on single-behavior setting . 2.3 POLICY EVALUATION VIA LEARNING STATIONARY DISTRIBUTION CORRECTION . The state-of-the-art methods for long-horizon off-policy policy evaluation are stationary-distributioncorrection based ( Liu et al. , 2018 ; Nachum et al. , 2019 ; Hallak & Mannor , 2017 ) . Let dπ0 ( s ) and dπ ( s ) be the stationary distribution of state s under the behavior policy π0 and target policy π respectively . The main idea of such methods is directly applying importance weighting by ω = dπ/dπ0 on the stationary state-visitation distributions to avoid the exploding variance suffered from IS , and estimate the average reward as Rπ = E ( s , a ) ∼dπ [ r ( s , a ) ] = E ( s , a ) ∼dπ0 [ ω ( s ) · π ( a|s ) π0 ( a|s ) r ( s , a ) ] . ( 1 ) For example , Liu et al . ( 2018 ) uses min-max approach to estimate ω directly from the data . This class of methods require exact knowledge of behavior policy π0 and are not straightforward to apply in multiple-behavior-policy setting . Recently , Nachum et al . ( 2019 ) proposed DualDice to overcome such limitation by learning the state-action stationary distribution correction ω ( s , a ) = dπ ( s ) π ( a|s ) /dπ0 ( s ) π0 ( a|s ) . 3 SINGLE BEHAVIOR POLICY ESTIMATION . When the behavior policy π0 in ( 1 ) is unknown , a natural idea is to estimate it from data . In this section , we focus on the standard case where the data are generated by a single behavior policy so that estimation of behavior policy is more straightforward . We first breifly review the method introduced by Liu et al . ( 2018 ) in Section 3.1 , which we shall refer as the BCH method in the rest of the paper , to explain the min-max problem formulation of the stationary distribution correction learning task . In Section 3.2 , we show that behavior policy estimation is beneficial in two aspects . First , it extends the stationary distribution correction method to settings where behavior policy is unknown . Second , even when the behavior policy is known with exact values , we prove that the stationary distribution correction learned using behavior policy estimation has smaller MSE than that using exact values . Later , we will extend this behavior policy estimation idea to more general multiple-behavior-policy cases in Section 4 . 3.1 LEARNING STATIONARY DISTRIBUTION CORRECTION WITH EXACT BEHAVIOR POLICY . Assume the data , consisting of state-action-next-state tuples , are generated by a single behavior policy π0 , i.e . D = { ( sn , an , s′n ) : n = 1 , 2 , ... , N } . Recall that dπ0 and dπ are the stationary state distribution under the behavior and target policy respectively , and ω = dπ/dπ0 is the stationary distribution correction . In the rest of Section 3 , by slight notation abuse , we also denote dπ ( s , a ) = dπ ( s ) π ( a|s ) , dπ0 ( s , a ) = dπ0 ( s ) π0 ( a|s ) and dπ0 ( s , a , s′ ) = dπ0 ( s ) π0 ( a|s ) T ( s′|a , s ) . We briefly review the BCH method proposed by Liu et al . ( 2018 ) . As dπ ( s ) is the stationary distribution of sn as n→∞ under policy π , it follows that : dπ ( s ′ ) = ∑ s , a dπ ( s ) π ( a|s ) T ( s′|s , a ) = ∑ s , a ω ( s ) π ( a|s ) π0 ( a|s ) dπ0 ( s ) π0 ( a|s ) T ( s′|a , s ) , ∀s′ . Therefore , for any function f : S → R , ∑ s′ ω ( s′ ) dπ0 ( s ′ ) f ( s′ ) = ∑ s , a , s′ ω ( s ) π ( a|s ) π0 ( a|s ) dπ0 ( s ) π0 ( a|s ) T ( s′|a , s ) f ( s′ ) . Recall that dπ0 ( s , a , s ′ ) = dπ0 ( s ) π0 ( a|s ) T ( s′|a , s ) , so ω and the data sample satisfy the following equation E ( s , a , s′ ) ∼dπ0 [ ( ω ( s′ ) − ω ( s ) π ( a|s ) π0 ( a|s ) ) f ( s′ ) ] = 0 , ∀f . BCH solves the above equation via the following min-max problem : min ω max f E ( s , a , s′ ) ∼dπ0 [ ( ω ( s′ ) − ω ( s ) π ( a|s ) π0 ( a|s ) ) f ( s′ ) ] 2 , ( 2 ) and use kernel method to solve ω . The derivation of kernel method is put in Appendix A .

Summary:   The main contribution of this paper is the development of estimated mixture policy (EMP), which takes ideas from the new off-policy policy evaluation infinite horizon estimators (i.e. Liu) and from a recent development in more traditional importance weight approaches using regression importance sampling (i.e. Hannah). This new method provides a nice extension of Liu's algorithm to many policies, and to when the policy is unknown. The paper provides some nice analysis inspired by Hannah. Finally, they provide empirical results.