Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning

1 Introduction . The availability of historically collected data is a common scenario in many real-world decisionmaking problems , including medical treatments [ e.g. , 17 , 67 ] , recommendation systems [ e.g. , 33 , 16 ] , personalized advertising [ e.g. , 3 , 60 ] , finance [ e.g. , 43 ] , and industrial robot control [ e.g. , 27 , 26 ] . Historical data can be leveraged to address two classes of problems . First , given data collected with a behavioral policy , we want to estimate the performance of a different target policy . This problem is known as off-policy evaluation [ Off-PE , 21 ] . Second , we want to employ the available data to improve the performance of a baseline policy . This latter problem is named off-policy learning [ OffPL 14 ] . Off-policy methods are studied by both the reinforcement learning [ RL , 58 ] and contextual multi-armed bandit [ CMAB , 30 ] communities . Given its intrinsic simplicity compared to RL , offpolicy methods are nowadays well understood in the CMAB framework [ e.g. , 44 , 1 , 14 , 64 ] . Among them , the doubly robust estimator [ DR , 14 ] is one of the most promising off-policy methods for CMABs . DR combines a direct method ( DM ) , in which the reward is estimated from historical data via regression , with an importance sampling [ IS , 46 ] control variate . IS plays a crucial role in the off-policy methods and counterfactual reasoning . However , IS tends to exhibit problematic behavior for general distributions . This is formalized by its heavy-tailed properties [ 40 ] , which prevent the application of exponential concentration bounds [ 4 ] . To cope with this issue , typically , corrections are performed on the importance weight including truncation [ 23 ] and self-normalization [ SN , 46 ] , among the most popular . Significant results have recently been derived for both techniques [ 47 , 29 , 39 ] . Nevertheless , we believe that the widespread use of IS calls for a better theoretical understanding of its properties and for the design of principled weight corrections . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Defining the desirable properties of an off-policy estimator is a non-trivial task . Some works employed the mean squared error ( MSE ) as an index of the estimator quality [ 34 , 64 ] . However , controlling the MSE , while effectively capturing the bias-variance trade-off , does not provide any guarantee on the concentration properties of the estimator , which might still display a heavy-tailed behavior [ 37 ] . For this reason , we believe that a more suitable approach is to require that the estimator deviations concentrate at a subgaussian rate [ 12 ] . Subgaussianity implicitly controls the tail behavior and leads to tight exponential concentration inequalities . Unlike MSE , the probabilistic requirements are nonasymptotic ( finite-sample ) , from which guarantees on the MSE can be derived . While subgaussianity can be considered a satisfactory requirement for Off-PE , additional properties are advisable when switching to Off-PL . In particular , the differentiability w.r.t . the target policy parameters is desirable whenever Off-PL is carried out via gradient optimization . For instance , weight truncation , as presented in [ 47 ] , allows achieving subgaussianity but leads to a non-differentiable objective . Consequently , the optimization phase requires additional care , which sometimes leads to computationally heavy discretizations [ 47 ] . On the contrary , the SN estimator is differentiable in the target policy , but fails to achieve subgaussian concentration for general distributions . In this paper , we take a step towards a better understanding of IS . After having introduced the necessary background ( Section 2 ) , we derive an anticoncentration bound for the mean estimation with vanilla IS . We show that polynomial concentration ( Chebychev ’ s inequality ) is tight in this setting ( Section 3 ) . This result formalizes the intuition behind the undesired behavior of these estimators for general distributions . Hence , we propose a class of importance weight corrections , based on the notion of power mean ( Section 4 ) . The rationale behind these corrections is to “ smoothly shrink ” the weights towards the mean , with different intensities . In this way , we mitigate the heavy-tailed behavior and , in the meantime , we exert control over the induced bias . Then , we derive bounds on the bias and variance that allow obtaining an exponential concentration inequality and , under certain conditions , subgaussian concentration ( Section 5 ) . Furthermore , the smooth transformation allows preserving the differentiability in the target policy , unlike some existing transformations , like weight truncation . To the best of our knowledge , this is the first IS correction that preserves the differentiablity and is proved to be subgaussian . This correction , however , requires knowledge of a distributional divergence between the target and behavioral policies , which may be unknown or difficult to compute . To this end , we introduce an approach to empirically estimate the correction parameter , preserving the desirable concentration properties ( Section 6 ) . After providing a comparative review of the literature ( Section 7 ) , we present an experimental study comparing our approach with traditional and modern off-policy baselines on synthetic domains and in the CMAB framework ( Section 8 ) . The proofs of the results presented in the main paper can be found in Appendix A . A preliminary version of this work was presented at the “ Workshop on Reinforcement Learning Theory ” of ICML 2021 [ 42 ] .1 2 Preliminaries . We start introducing the background about probability , importance sampling and contextual bandits . Probability We denote with PpYq the set of probability measures over a pY , FYq . Let P PPpYq , f : YÑR be a function , and µn be an estimator for the mean µ “ Ey „ P rfpyqs obtained with n i.i.d . samples . Suppose that with probability 1´δ it holds that |µn´µ|ď a gpn , δq . For βą0 , we say that µn admits : ( i ) polynomial concentration if gpn , δq “ O ` 1 { pnδqβ ˘ ; ( ii ) exponential concentration if gpδq “ O `` logp1 { δq { n ˘β˘ ; ( iii ) subgaussian concentration if ( ii ) holds with β “ 1 [ 37 ] . These cases correspond to Chebyshev ’ s , Bernstein ’ s , and Höeffding ’ s inequalities respectively [ 4 ] . Importance Sampling Let P , QPPpYq admitting p and q as density functions , if P ! Q , i.e. , P is absolutely continuous w.r.t . Q , for any αPp1,2s , we introduce the integral : IαpP } Qq “ ş Y ppyq αqpyq1´αdy . If P “ Q a.s. ( almost surely ) then IαpP } Qq “ 1 . IαpP } Qq allows defining several divergences , like Rényi [ 51 ] : pα´1q´1 logIαpP } Qq . Let f : YÑR be a function , ( vanilla ) importance sampling [ IS , 46 ] allows estimating the expectation of f under the target distribution P , i.e. , µ “ Ey „ P rfpyqs , using i.i.d . samples tyiuiPrns collected with the behavioral distribution Q : pµn “ 1 n ÿ iPrns ωpyiqfpyiq , where ωpyq “ ppyq qpyq , @ yPY . 1https : //lyang36.github.io/icml2021_rltheory/camera_ready/7.pdf . It is well-known that pµn is unbiased , i.e. , Eyi „ Qrpµns “ µ [ 46 ] . If f is bounded , the variance of the estimator can be upper-bounded as Varyi „ Qrpµnsď 1n } f } 8I2pP } Qq [ 40 ] . More in general , the integral IαpP } Qq represents the α-moment of the importance weight ωpyq under Q. Contextual Bandits A contextual multi-armed bandit [ CMAB , 30 ] is represented by the tuple C “ pX , A , ρ , pq , where X is the set of contexts , A is the finite set of actions ( or arms ) , ρPPpX q is the context distribution , and p : X ˆAÑPpRq is the reward distribution . The agent ’ s behavior is encoded by a policy π : XÑPpAq . At each round tPN , the agent observes a context xt „ ρ , plays an action at „ πp¨|xtq , gets the reward rt „ pp¨|xt , atq and the system moves on to the next round . The value of a policy π is given by vpπq “ ş X ρpxq ř aPAπpa|xq ş R ppr|x , aqrdrdx . We denote with rpx , aq “ ş R ppr|x , aqrdr the reward function . A policy π ˚ is optimal if it maximizes the value function , i.e. , π˚ PargmaxπPΠ vpπq , where Π “ tπ : XÑPpAqu is the set of all policies . Let D “ tpxt , at , rtqutPrns be a dataset of samples collected in a CMAB with a behavioral policy πb PΠ . The off-policy evaluation [ Off-PE , 21 ] problem consists in estimating the value function vpπeq of a target policy πe PΠ using the samples in D. The off-policy learning [ Off-PL , 14 ] problem consists in estimating an optimal policy π˚ PΠ using the samples in D. The simplest approach to address the Off-PE/Off-PL problem is to learn from D an estimate prpx , aq of the reward function rpx , aq via regression . This approach is known as direct method ( DM ) and its properties heavily depend on the quality of the estimate pr . Another approach is to simply apply IS to reweight the samples of D , leading to the inverse propensity scoring [ IS , 17 ] estimator . The two approaches are combined in the doubly-robust [ DR , 14 ] estimator , in which the DM estimate is corrected with an IS control variate to reduce the variance using the estimated reward pr ( see also Table 12 in Appendix D ) . 3 Anticoncentration of Vanilla Importance Sampling . In this section , we analyze the intrinsic limitations of the vanilla IS . It is well-known that under the assumption that for some αPp1,2s the divergence IαpP } Qq is finite and f is bounded , the vanilla IS estimator pµn admits polynomial concentration , i.e. , with probability at least 1´δ:2 |pµn´µ|ď } f } 8 ˆ 22´αIαpP } Qq δnα´1 ˙ 1 α . ( 1 ) We now show that the concentration in Equation ( 1 ) is tight , by deriving an anticoncentration bound for |pµn´µ| ; then , we discuss its implications and compare it with previous works . Theorem 3.1 . There exist two distributions P , QPPpYq with P ! Q and a bounded measurable function f : YÑR such that for every αPp1,2s and δPp0 , e´1q if něδemax 1 , pIαpP } Qq´1q 1 α´1 ( , with probability at least δ it holds that : |pµn´µ|ě } f } 8 ˆ IαpP } Qq´1 δnα´1 ˙ 1 α ˆ 1´ eδ n ˙ n´1 α . First of all , we note the polynomial dependence on the confidence level δ . The bound is vacuous when IαpP } Qq “ 1 , i.e. , when P “ Q a.s. , since in an on-policy setting and , being the function f bounded , subgaussian concentration bounds ( like Höeffding ’ s inequality ) hold . In particular , for α “ 2 , n and I2pP } Qq sufficiently large , the bound has order O ` a I2pP } Qq { pδnq ˘ , matching Chebyshev ’ s and the existing concentration inequalities for vanilla importance sampling [ 40 , 41 ] . Our result is of independent interest and applies for general distributions . Previous works considered the MAB [ 34 ] and CMAB [ 64 ] settings proving minimax lower bounds in mean squared error ( MSE ) Ey „ Qrppµn´µq2s . These results differ from ours in several respects . First , we focus on a specific estimator , the vanilla one , while those result are minimax . Second , they provide lower bounds to the MSE , while we focus on the deviations in probability.3 From our probabilistic result , it is immediate to derive an MSE guarantee ( Corollary A.1 of Appendix A.1 ) . Finally , they assume that the second moment of the importance weight I2pP } Qq is finite , whereas our result allows considering scenarios in which only moments of order αă2 are finite . 2The original result [ 41 , Theorem 2 ] was limited to α “ 2 and based on Cantelli ’ s inequality which approaches Chebyshev ’ s when δÑ0 . See Theorem A.1 in Appendix A.1 , for a proof of Equation ( 1 ) . 3As noted in [ 36 ] , when the estimator is not well-concentrated around its mean ( e.g. , in presence of heavy tails ) , the MSE is not adequate to capture the error and high-probability bounds are more advisable .

Summary: This paper considers a power mean correction term for off-policy evaluation and learning. The resulting estimator has a subgaussian concentration rate, improving on other variants of importance sampling (which has a polynomial concentration rate). The authors bound the variance and bias of the resulting estimator. These bounds are functions of a tuned parameter $\lambda$, where higher values of lambda increase the bias but reduce the variance giving practitioners the ability to tune the properties to their problem. The estimator also has other beneficial properties: the correction term is differentiable, and the free parameter $\lambda$ can be tuned through data.