Off-policy Bandits with Deficient Support

1 INTRODUCTION . Many interactive systems ( e.g. , voice assistants , recommender systems , ad placement ) can be modeled as contextual bandit problems ( Langford & Zhang , 2008 ) . In particular , each user request provides a context ( e.g. , user profile , query ) for which the system selects an action ( e.g. , recommended product , presented ad ) and receives a reward ( e.g. , purchase , click ) . Such contextual-bandit data is logged in large quantities as a by-product of normal system operation ( Li et al. , 2011 ; 2015 ; Joachims et al. , 2017 ) , making it an attractive and low-cost source of training data . With terabytes of such log data readily available in many online systems , a range of algorithms have been proposed for batch learning from such logged contextual-bandit feedback ( Strehl et al. , 2011 ; Dudı́k et al. , 2011 ; Swaminathan & Joachims , 2015a ; Thomas & Brunskill , 2016 ; Farajtabar et al. , 2018 ; Su et al. , 2019 ; London & Sandler , 2019 ) . However , as we will argue below , these algorithms require an assumption about the log data that makes them unsuitable for many real-world applications . This assumption is typically referred to as the positivity or support assumption , and it is required by the Empirical Risk Minimization ( ERM ) objective that these algorithms optimize . Specifically , unlike in online learning for contextual bandits ( Williams , 1992 ; Agarwal et al. , 2014 ) , batch learning from bandit feedback ( BLBF ) operates in the off-policy setting . During off-policy learning , the algorithm has to address the counterfactual question of how much reward each policy in the policy space would have received , if it had been used instead of the logging policy . To this effect , virtually all state-of-the-art off-policy learning methods for contextual-bandit problems rely on counterfactual estimators ( Bottou et al. , 2013 ; Dudı́k et al. , 2011 ; Swaminathan & Joachims , 2015a ; Thomas & Brunskill , 2016 ; Farajtabar et al. , 2018 ; Su et al. , 2019 ) that employ inverse propensity score ( IPS ) weighting to get an unbiased ERM objective . Unlike regression-based direct-modeling ( DM ) approaches that are often hampered by bias from model-misspecification , IPS allows a controllable bias-variance trade-off through clipping and other variance-regularization techniques ( Strehl et al. , 2011 ; Swaminathan & Joachims , 2015a ; London & Sandler , 2019 ) . Unfortunately , IPS and its variance-control mechanisms break down when the logging policy does not have full support – meaning that some actions have zero probability of being selected under the logging policy . In this case IPS can be highly biased . Full support is an unreasonable assumption in many real-world systems , especially when the action space is large and many actions have poor rewards . For example , in a recommender system with a large catalog ( e.g . movies , music ) , it may be that less than 10 % of the actions have support under the logging policy . We will show that existing learning algorithms can fail catastrophically on such support deficient data . In this paper , we develop new off-policy contextual-bandit algorithms that are specifically designed to deal with support deficient log data . Since support deficiency translates into blind spots where we do not have any knowledge about the rewards , accounting for these blind spots as part of learning is crucial for robust learning . We approach this problem from three perspectives . First , we explore restricting the action space to those actions that have support under the logging policy . Second , we explore imputation methods that extrapolate estimated rewards to those blind spots . And , third , we restrict the policy space to only those policies that have limited exposure to the blind spots . To make the latter approach computationally tractable , we define a new measure of Support Divergence between policies , show how it can be estimated efficiently without closed-form knowledge of the logging policy , and how it can be used as a constraint on the policy space . We analyze the statistical and computational properties of all three approaches and perform an extensive empirical evaluation . We find that restricting the policy space is particularly effective , since it is computationally efficient , empirically effective at learning good policies , and convenient to use in practice . 2 RELATED WORK . Most prior works on BLBF can be classified into two different approaches . The first – called Direct Model ( DM ) – is based on a reduction to supervised learning , where a regression estimate is trained to predict rewards ( Beygelzimer & Langford , 2009 ) . To derive a policy , the action with the highest predicted reward is chosen . A drawback of this simple approach is the bias that results from misspecification of the regression model . Since regression models are often substantially misspecified for real-world data , the DM approach often does not work well empirically . The second approach is based on policy learning via ERM with a counterfactual risk estimator . Inverse propensity score ( IPS ) weighting is one of the most popular estimators to be used as empirical risk . However , policy learning algorithms based on IPS and related estimators ( Strehl et al. , 2011 ; Swaminathan & Joachims , 2015a ; b ; Thomas & Brunskill , 2016 ; London & Sandler , 2019 ) require the assumption that the logging policy has full support for every policy in the policy space . One exception is the work of Liu et al . ( 2019 ) . They relax the assumption to the existence of an optimal policy such that the logging policy covers the support of this optimal policy . However , this is an untestable assumption that does not provide guarantees for real-world applications . Our work proposes three approaches to addressing off-policy learning with support deficiency . First , our conservative extrapolation method is related to the method proposed by Liu et al . ( 2019 ) . They focus on the correction of the state distribution by defining an augmented MDP , and pessimistic imputation is used to get an estimate for policy-gradient learning . Second , our method of restricting the policy space uses a surrogate for the support divergence of two policies that was previously used as control variate in the SNIPS estimator ( Swaminathan & Joachims , 2015b ) . It also appeared in the Lagrangian formulation of the BanditNet objective ( Joachims et al. , 2018 ) and in the gradient update in REINFORCE algorithm ( Williams , 1992 ) . This connection gives interesting new insight that the baselines used in policy-gradient algorithms not only help to reduce variance in gradients ( Greensmith et al. , 2004 ) , but that they also connect to the problem of support deficiency in the off-policy setting . 3 OFF-POLICY LEARNING WITH DEFICIENT SUPPORT . We start by formally defining the problem of learning a contextual-bandit policy in the BLBF setting . Input to the policy are contexts x ∈ X drawn i.i.d from a fixed but unknown distribution P ( X ) . Given context x , the system executes a possibly stochastic policy π ( Y|x ) that selects an action y ∈ Y . For this context and action pair , the system observes a reward r ∈ [ rmin , rmax ] from P ( r|x , y ) . Given a space of policies Π , the reward of any policy π ∈ Π is defined as R ( π ) = E x E y∼π ( y|x ) E r∼P ( r|x , y ) [ r ] . ( 1 ) In the BLBF setting , the learning algorithm is given a dataset D : = { xi , yi , ri , π0 ( yi|xi ) } ni=1 of past system interactions which consists of context-action-reward-propensity tuples . The propensity π0 ( yi|xi ) is the probability of selecting action yi for context xi under the policy π0 that was used to log the data . We call π0 the logging policy , and we will discuss desired conditions on the stochasticity of π0 in the following . The goal of off-policy learning is to exploit the information in the logged data D to find a policy π̂ ∈ Π that has high reward R ( π̂ ) . Analogous to the ERM principle in supervised learning , off-policy learning algorithms typically optimize a counterfactual estimate R̂ ( π ) of R ( π ) as the training objective ( Li et al. , 2011 ; 2015 ; Bottou et al. , 2013 ; Swaminathan & Joachims , 2015a ) . π̂ = arg max π∈Π [ R̂ ( π ) ] ( 2 ) For conciseness , we ignore additional regularization terms in the objective ( Swaminathan & Joachims , 2015a ) , since they are irrelevant to the main point of this paper . As counterfactual estimator R̂ ( π ) , most algorithms rely on some form of IPS weighting ( Strehl et al. , 2011 ; Dudı́k et al. , 2011 ; Swaminathan & Joachims , 2015a ; b ; Wang et al. , 2017 ; Su et al. , 2019 ) to correct the distribution mismatch between the logging policy π0 and each target policy π ∈ Π. R̂IPS ( π ) = 1 n n∑ i=1 π ( yi|xi ) π0 ( yi|xi ) ri . ( 3 ) A crucial condition for the effectiveness of the IPS estimator ( and similar estimators ) is that the logging policy π0 assigns non-zero probability to all actions that have non-zero probability under the target policy π we aim to evaluate . This condition is known as positivity or full support , and it is defined as follows . Definition 1 ( Full support ) . The logging policy π0 is said to have full support for π when π0 ( y|x ) > 0 for all actions y ∈ Y and contexts x ∈ X for which π ( y|x ) > 0 . It is known that the IPS estimator is unbiased , ED [ R̂IPS ( π ) ] = R ( π ) , if the logging policy π0 has full support for π ( Li et al. , 2011 ) . To ensure unbiased ERM , algorithms that use the IPS estimator require that the logging policy π0 has full support for all policies π ∈ Π in the policy space . For sufficiently rich policy spaces , like deep-networks fw ( x , y ) with softmax outputs of the form πw ( y|x ) = exp ( fw ( x , y ) ) ∑ y′∈Y exp ( fw ( x , y ′ ) ) , ( 4 ) this means that the logging policy π0 needs to assign non-zero probability to every action y in every context x . This is a strong condition that is not feasible in many real-world systems , especially if the action space is large and many actions have poor reward . If the support requirement is violated , ERM learning can fail catastrophically . We will show below that the underlying reason is bias , not excessive variance that could be remedied through clipping or variance regularization ( Strehl et al. , 2011 ; Swaminathan & Joachims , 2015a ) . To quantify how support deficient a logging policy is , we denote the set of unsupported actions for context x under π0 as U ( x , π0 ) : = { y ∈ Y|π0 ( y|x ) = 0 } . The bias of the IPS estimator is then characterized by the expected reward on the unsupported actions . Proposition 1 . Given contexts x ∼ P ( X ) and logging policy π0 ( Y|x ) , the bias of R̂IPS for target policy π ( Y|x ) is equal to the expected reward on the unsupported action sets , i.e. , bias ( π|π0 ) = Ex [ − ∑ y∈U ( x , π0 ) π ( y|x ) δ ( x , y ) ] . The proof is in Appendix A.1 . From Proposition 1 , it is clear that support deficient log data can drastically mislead ERM learning . To quantify the effect of support deficiency on ERM , we define the support divergence between a logging policy π0 and a target policy π as follows . Definition 2 ( Support Divergence ) . For contexts x ∼ P ( X ) and any corresponding pair of target policy π and logging policy π0 , the Support Divergence is defined as DX ( π|π0 ) : = E x∼P ( X ) ∑ y∈U ( x , π0 ) π ( y|x ) . ( 5 ) With this definition in hand , we can quantify the effect of support deficiency on ERM learning for a policy space Π under logging policy π0 . Theorem 1 . For any given hypothesis space Π with logging policy π0 ∈ Π , there exists a reward distribution Pr with support in [ rmin , rmax ] such that in the limit of infinite training data , ERM using IPS over the logged data D ∼ P ( X ) × π0 ( ·|X ) × Pr can select a policy π̂ ∈ arg maxπ∈Π ED [ R̂IPS ( π ) ] that is at least ( rmax − rmin ) maxπ∈ΠDX ( π|π0 ) suboptimal . The proof is in Appendix A.2 . To illustrate the theorem , consider a problem with rewards r ∈ [ −1 , 0 ] . Furthermore , consider a policy space Π that contains a good policy πg with R ( πg ) = −0.1 and a bad policy πb with R ( πb ) = −0.7 . If policy πb has support divergence DX ( πb|π0 ) = 0.6 or larger , then ERM may return the bad πb instead of πg even with infinite amounts of training data . Note that it is sufficient to merely have one policy in Π that has large support deficiency to achieve this suboptimality . It is therefore crucial to control the support divergence DX ( π|π0 ) uniformly over all π ∈ Π , or to account for the suboptimality it can induce . To this effect , we explore three approaches in the following .

Summary: This paper talks about the problem of off-policy or batch learning in the contextual bandit setting without the complete support assumption. This problem setting is very realistic and encountered in most problems, especially in temporally extended settings, such as reinforcement learning. They compare three approaches for the same: restricting action selection, learning extrapolated reward models, and by restricting the policy class. They derive a SNIPS style estimator for the support constraint in the final approach. The approach with restricting the policy class demonstrates decent empirical results although the direct method is very much comparable.