Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning

1 Introduction . The availability of historically collected data is a common scenario in many real-world decisionmaking problems , including medical treatments [ e.g. , 17 , 67 ] , recommendation systems [ e.g. , 33 , 16 ] , personalized advertising [ e.g. , 3 , 60 ] , finance [ e.g. , 43 ] , and industrial robot control [ e.g. , 27 , 26 ] . Historical data can be leveraged to address two classes of problems . First , given data collected with a behavioral policy , we want to estimate the performance of a different target policy . This problem is known as off-policy evaluation [ Off-PE , 21 ] . Second , we want to employ the available data to improve the performance of a baseline policy . This latter problem is named off-policy learning [ OffPL 14 ] . Off-policy methods are studied by both the reinforcement learning [ RL , 58 ] and contextual multi-armed bandit [ CMAB , 30 ] communities . Given its intrinsic simplicity compared to RL , offpolicy methods are nowadays well understood in the CMAB framework [ e.g. , 44 , 1 , 14 , 64 ] . Among them , the doubly robust estimator [ DR , 14 ] is one of the most promising off-policy methods for CMABs . DR combines a direct method ( DM ) , in which the reward is estimated from historical data via regression , with an importance sampling [ IS , 46 ] control variate . IS plays a crucial role in the off-policy methods and counterfactual reasoning . However , IS tends to exhibit problematic behavior for general distributions . This is formalized by its heavy-tailed properties [ 40 ] , which prevent the application of exponential concentration bounds [ 4 ] . To cope with this issue , typically , corrections are performed on the importance weight including truncation [ 23 ] and self-normalization [ SN , 46 ] , among the most popular . Significant results have recently been derived for both techniques [ 47 , 29 , 39 ] . Nevertheless , we believe that the widespread use of IS calls for a better theoretical understanding of its properties and for the design of principled weight corrections . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Defining the desirable properties of an off-policy estimator is a non-trivial task . Some works employed the mean squared error ( MSE ) as an index of the estimator quality [ 34 , 64 ] . However , controlling the MSE , while effectively capturing the bias-variance trade-off , does not provide any guarantee on the concentration properties of the estimator , which might still display a heavy-tailed behavior [ 37 ] . For this reason , we believe that a more suitable approach is to require that the estimator deviations concentrate at a subgaussian rate [ 12 ] . Subgaussianity implicitly controls the tail behavior and leads to tight exponential concentration inequalities . Unlike MSE , the probabilistic requirements are nonasymptotic ( finite-sample ) , from which guarantees on the MSE can be derived . While subgaussianity can be considered a satisfactory requirement for Off-PE , additional properties are advisable when switching to Off-PL . In particular , the differentiability w.r.t . the target policy parameters is desirable whenever Off-PL is carried out via gradient optimization . For instance , weight truncation , as presented in [ 47 ] , allows achieving subgaussianity but leads to a non-differentiable objective . Consequently , the optimization phase requires additional care , which sometimes leads to computationally heavy discretizations [ 47 ] . On the contrary , the SN estimator is differentiable in the target policy , but fails to achieve subgaussian concentration for general distributions . In this paper , we take a step towards a better understanding of IS . After having introduced the necessary background ( Section 2 ) , we derive an anticoncentration bound for the mean estimation with vanilla IS . We show that polynomial concentration ( Chebychev ’ s inequality ) is tight in this setting ( Section 3 ) . This result formalizes the intuition behind the undesired behavior of these estimators for general distributions . Hence , we propose a class of importance weight corrections , based on the notion of power mean ( Section 4 ) . The rationale behind these corrections is to “ smoothly shrink ” the weights towards the mean , with different intensities . In this way , we mitigate the heavy-tailed behavior and , in the meantime , we exert control over the induced bias . Then , we derive bounds on the bias and variance that allow obtaining an exponential concentration inequality and , under certain conditions , subgaussian concentration ( Section 5 ) . Furthermore , the smooth transformation allows preserving the differentiability in the target policy , unlike some existing transformations , like weight truncation . To the best of our knowledge , this is the first IS correction that preserves the differentiablity and is proved to be subgaussian . This correction , however , requires knowledge of a distributional divergence between the target and behavioral policies , which may be unknown or difficult to compute . To this end , we introduce an approach to empirically estimate the correction parameter , preserving the desirable concentration properties ( Section 6 ) . After providing a comparative review of the literature ( Section 7 ) , we present an experimental study comparing our approach with traditional and modern off-policy baselines on synthetic domains and in the CMAB framework ( Section 8 ) . The proofs of the results presented in the main paper can be found in Appendix A . A preliminary version of this work was presented at the “ Workshop on Reinforcement Learning Theory ” of ICML 2021 [ 42 ] .1 2 Preliminaries . We start introducing the background about probability , importance sampling and contextual bandits . Probability We denote with PpYq the set of probability measures over a pY , FYq . Let P PPpYq , f : YÑR be a function , and µn be an estimator for the mean µ “ Ey „ P rfpyqs obtained with n i.i.d . samples . Suppose that with probability 1´δ it holds that |µn´µ|ď a gpn , δq . For βą0 , we say that µn admits : ( i ) polynomial concentration if gpn , δq “ O ` 1 { pnδqβ ˘ ; ( ii ) exponential concentration if gpδq “ O `` logp1 { δq { n ˘β˘ ; ( iii ) subgaussian concentration if ( ii ) holds with β “ 1 [ 37 ] . These cases correspond to Chebyshev ’ s , Bernstein ’ s , and Höeffding ’ s inequalities respectively [ 4 ] . Importance Sampling Let P , QPPpYq admitting p and q as density functions , if P ! Q , i.e. , P is absolutely continuous w.r.t . Q , for any αPp1,2s , we introduce the integral : IαpP } Qq “ ş Y ppyq αqpyq1´αdy . If P “ Q a.s. ( almost surely ) then IαpP } Qq “ 1 . IαpP } Qq allows defining several divergences , like Rényi [ 51 ] : pα´1q´1 logIαpP } Qq . Let f : YÑR be a function , ( vanilla ) importance sampling [ IS , 46 ] allows estimating the expectation of f under the target distribution P , i.e. , µ “ Ey „ P rfpyqs , using i.i.d . samples tyiuiPrns collected with the behavioral distribution Q : pµn “ 1 n ÿ iPrns ωpyiqfpyiq , where ωpyq “ ppyq qpyq , @ yPY . 1https : //lyang36.github.io/icml2021_rltheory/camera_ready/7.pdf . It is well-known that pµn is unbiased , i.e. , Eyi „ Qrpµns “ µ [ 46 ] . If f is bounded , the variance of the estimator can be upper-bounded as Varyi „ Qrpµnsď 1n } f } 8I2pP } Qq [ 40 ] . More in general , the integral IαpP } Qq represents the α-moment of the importance weight ωpyq under Q. Contextual Bandits A contextual multi-armed bandit [ CMAB , 30 ] is represented by the tuple C “ pX , A , ρ , pq , where X is the set of contexts , A is the finite set of actions ( or arms ) , ρPPpX q is the context distribution , and p : X ˆAÑPpRq is the reward distribution . The agent ’ s behavior is encoded by a policy π : XÑPpAq . At each round tPN , the agent observes a context xt „ ρ , plays an action at „ πp¨|xtq , gets the reward rt „ pp¨|xt , atq and the system moves on to the next round . The value of a policy π is given by vpπq “ ş X ρpxq ř aPAπpa|xq ş R ppr|x , aqrdrdx . We denote with rpx , aq “ ş R ppr|x , aqrdr the reward function . A policy π ˚ is optimal if it maximizes the value function , i.e. , π˚ PargmaxπPΠ vpπq , where Π “ tπ : XÑPpAqu is the set of all policies . Let D “ tpxt , at , rtqutPrns be a dataset of samples collected in a CMAB with a behavioral policy πb PΠ . The off-policy evaluation [ Off-PE , 21 ] problem consists in estimating the value function vpπeq of a target policy πe PΠ using the samples in D. The off-policy learning [ Off-PL , 14 ] problem consists in estimating an optimal policy π˚ PΠ using the samples in D. The simplest approach to address the Off-PE/Off-PL problem is to learn from D an estimate prpx , aq of the reward function rpx , aq via regression . This approach is known as direct method ( DM ) and its properties heavily depend on the quality of the estimate pr . Another approach is to simply apply IS to reweight the samples of D , leading to the inverse propensity scoring [ IS , 17 ] estimator . The two approaches are combined in the doubly-robust [ DR , 14 ] estimator , in which the DM estimate is corrected with an IS control variate to reduce the variance using the estimated reward pr ( see also Table 12 in Appendix D ) . 3 Anticoncentration of Vanilla Importance Sampling . In this section , we analyze the intrinsic limitations of the vanilla IS . It is well-known that under the assumption that for some αPp1,2s the divergence IαpP } Qq is finite and f is bounded , the vanilla IS estimator pµn admits polynomial concentration , i.e. , with probability at least 1´δ:2 |pµn´µ|ď } f } 8 ˆ 22´αIαpP } Qq δnα´1 ˙ 1 α . ( 1 ) We now show that the concentration in Equation ( 1 ) is tight , by deriving an anticoncentration bound for |pµn´µ| ; then , we discuss its implications and compare it with previous works . Theorem 3.1 . There exist two distributions P , QPPpYq with P ! Q and a bounded measurable function f : YÑR such that for every αPp1,2s and δPp0 , e´1q if něδemax 1 , pIαpP } Qq´1q 1 α´1 ( , with probability at least δ it holds that : |pµn´µ|ě } f } 8 ˆ IαpP } Qq´1 δnα´1 ˙ 1 α ˆ 1´ eδ n ˙ n´1 α . First of all , we note the polynomial dependence on the confidence level δ . The bound is vacuous when IαpP } Qq “ 1 , i.e. , when P “ Q a.s. , since in an on-policy setting and , being the function f bounded , subgaussian concentration bounds ( like Höeffding ’ s inequality ) hold . In particular , for α “ 2 , n and I2pP } Qq sufficiently large , the bound has order O ` a I2pP } Qq { pδnq ˘ , matching Chebyshev ’ s and the existing concentration inequalities for vanilla importance sampling [ 40 , 41 ] . Our result is of independent interest and applies for general distributions . Previous works considered the MAB [ 34 ] and CMAB [ 64 ] settings proving minimax lower bounds in mean squared error ( MSE ) Ey „ Qrppµn´µq2s . These results differ from ours in several respects . First , we focus on a specific estimator , the vanilla one , while those result are minimax . Second , they provide lower bounds to the MSE , while we focus on the deviations in probability.3 From our probabilistic result , it is immediate to derive an MSE guarantee ( Corollary A.1 of Appendix A.1 ) . Finally , they assume that the second moment of the importance weight I2pP } Qq is finite , whereas our result allows considering scenarios in which only moments of order αă2 are finite . 2The original result [ 41 , Theorem 2 ] was limited to α “ 2 and based on Cantelli ’ s inequality which approaches Chebyshev ’ s when δÑ0 . See Theorem A.1 in Appendix A.1 , for a proof of Equation ( 1 ) . 3As noted in [ 36 ] , when the estimator is not well-concentrated around its mean ( e.g. , in presence of heavy tails ) , the MSE is not adequate to capture the error and high-probability bounds are more advisable .

Summary: The paper addresses off-policy evaluation. It proposes to transform importance weights using the function $\frac{\omega(y)}{1-\lambda + \lambda \omega(y)}$, where $\omega(y) = p(y)/q(y)$ is the standard importance weight. The estimator which uses these transformed weights is analyzed both theoretically and empirically. The obtained estimator is unique in having both subgaussian concentration rate and being differentiable. The estimator depends on the hyperparameter $\lambda$, for which the authors give an estimation procedure that only depends on observable quantities.