Control Variates for Slate Off-Policy Evaluation

1 Introduction . Online services ( news , music and video streaming , social media , e-commerce , app stores , etc . ) serve content that often takes the form of a combinatorial , high-dimensional slate , where each slot on the slate can take multiple values . For example , a personalized news service can have separate slots for local news , sports , politics , etc. , with multiple news items from which to select in each slot ( Li et al. , 2010 ) ; or a video streaming service may organize the contents of the landing homepage of a user as a multi-slot slate , where each slot can contain shows from a given category ( Gomez-Uribe and Hunt , 2015 ) . Typically , the offered content is personalized via machine learning and A/B testing . However , the number of eligible items per slot may be in the hundreds or thousands , which makes the testing and optimization of personalized policies a challenging task . An alternative to A/B testing and online learning is off-policy evaluation ( OPE ) . In OPE we use historical data collected by a past policy in order to evaluate a new candidate policy . This is primarily an estimation problem , often grounded in causal assumptions about the data-generating process and its connections to a counterfactual one ( Horvitz and Thompson , 1952 ; Robins and Rotnitzky , 1995 ; Dudík et al. , 2011 ; Bottou et al. , 2013 ; Athey and Wager , 2017 ; Bibaut et al. , 2019 ; Kallus and Uehara , 2019 ) . However , even with plentiful off-policy data , handling a combinatorial action space can be challenging : Unbiased importance sampling ( IS ) ( Horvitz and Thompson , 1952 ) suffers variance on the scale of the cardinality of the action space under a uniform logging policy and deterministic target policy . To address this , Swaminathan et al . ( 2017 ) have proposed the pseudoinverse ( PI ) estimator , which significantly attenuates the variance and remains unbiased for decomposable reward structures . In this paper , we discuss how to further reduce this variance using a control variates approach ( Glynn and Szechtman , 2002 ) . We demonstrate that the self-normalized PI ( wPI ) estimator , which Swaminathan et al . ( 2017 ) found to be superior to PI , is asymptotically equivalent to adding a control variate . But , the choice of control variate is not optimal . We instead show how to choose optimal ∗The author was with Netflix when this work was concluded . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . control variates , and even how to do so without incurring any bias . We provide strong empirical evidence based on the MSLR-WEB30K data from the Microsoft Learning to Rank Challenge ( Qin and Liu , 2013 ) that confirms the improvement of our approach over the PI and wPI estimators . 1.1 Related work . Much of the existing OPE work involves the use of doubly-robust estimators and variants , which add control variates to the standard IS estimator ( Robins and Rotnitzky , 1995 ; Dudík et al. , 2011 ; Thomas and Brunskill , 2016 ; Farajtabar et al. , 2018 ; Bibaut et al. , 2019 ; Kallus and Uehara , 2019 ; Su et al. , 2020 ) . Directly optimizing control variates for OPE has been suggested earlier ( Farajtabar et al. , 2018 ; Vlassis et al. , 2019 ) but not for problems involving high-dimensional slates . Combinatorial bandits and variants such as semi-bandits have been studied by Cesa-Bianchi and Lugosi ( 2012 ) and Kveton et al . ( 2015 ) and are also covered in the book of Lattimore and Szepesvári ( 2020 ) . Earlier approaches to OPE for contextual combinatorial bandits have been restricted to small slate problems ( Li et al. , 2011 ) or relied on accurate parametric models of slate rewards ( Chapelle and Zhang , 2009 ; Guo et al. , 2009 ; Filippi et al. , 2010 ) . Li et al . ( 2015 ) and Wang et al . ( 2016 ) have suggested the use of partial matches between the logging and target actions in order to mitigate the variance explosion . Swaminathan et al . ( 2017 ) have proposed the PI estimator , a version of which we study in this paper . Under factored policies , the PI estimator is similar to an estimator proposed by Li et al . ( 2018 ) for OPE of ranking policies with user click models . Swaminathan and Joachims ( 2015 ) and Joachims et al . ( 2018 ) have developed estimators for OPE under the framework of counterfactual risk minimization and discuss approaches for offline policy optimization . Agarwal et al . ( 2018 ) have developed an OPE method for rankers , when only the relative value of two given rankers is of interest . Chen et al . ( 2019 ) have developed an approach for offline policy gradients , when slates are unordered lists of items ( of random size ) . McInerney et al . ( 2020 ) and Lopez et al . ( 2021 ) have studied the problem of slate OPE with semi-bandit feedback ( one observed reward per slot ) , under different assumptions on the structure of the rewards and the contextual policies . In constrast to the latter two works , and similar to Swaminathan et al . ( 2017 ) and Su et al . ( 2020 ) , we study the slate OPE problem when only a single , slate-level reward is observed . Su et al . ( 2020 ) achieve variance reduction by clipping propensities , whereas we achieve variance reduction by optimizing control variates . Applications of slate OPE in real-world problems include Sar Shalom et al . ( 2016 ) who apply OPE to a large scale real world recommender system that handles purchases from tens of millions of users , Hill et al . ( 2017 ) who present a method to optimize a message that is composed of separated widgets ( slots ) such as title text , offer details , image , etc , and McInerney et al . ( 2020 ) who apply OPE to a slate problem involving sequences of items such as music playlists . 2 Setup and Notation . We consider contextual slate bandits where each slate has K slots . We will use [ K ] to denote the set { 1 , 2 , . . . , K } . The available actions in slot k are [ dk ] = { 1 , . . . , dk } . Thus , the set of available slates has cardinality ∏K k=1 dk . The data consists of n independent and identically distributed ( iid ) triplets ( Xi , Ai , Ri ) , i = 1 , . . . , n , representing the observed context , slate taken , and resulting reward , respectively . Here , Ai = ( Ai1 , . . . , AiK ) where Aik ∈ [ dk ] . We use ( X , A , R ) to refer to a generic draw of one of the above triplets , where X ∼ P ( X ) , A ∼ µ ( · | X ) , R ∼ P ( · | A , X ) . The distribution µ ( a | x ) = P ( A = a | X = x ) is called the logging policy as the data can be seen as the logs generated by executing this policy . Probabilities P , expectations E , and ( co ) variances without subscripts indicating otherwise are understood to be with respect to this distribution . For any function f ( x , a , r ) we will write En [ f ( X , A , R ) ] = 1n ∑n i=1 f ( Xi , Ai , Ri ) . We will also follow the convention that random , data-driven objects have the form ·̂n ( such as ŵn or θ̂n ) . In OPE , we are given a fixed policy π ( a | x ) and are interested in estimating its average reward Eπ [ R ] using data collected under a logging policy µ ( a | x ) . Here Eπ refers to expectation under the distribution X ∼ P ( X ) , A ∼ π ( · | X ) , R ∼ P ( · | A , X ) . The quantity Eπ [ R ] can be interpreted as a counterfactual given that P ( · | A = a , X ) is equal to the distribution of the potential reward of slate a given context X , for every a ; or , in other words , that P ( · | A , X ) represents a structural model . This assumption , known as unconfoundedness or ignorability , is only needed for giving a causal interpretation to Eπ [ R ] but not for estimation . The classical IS estimator for the problem of estimating Eπ [ R ] is En [ π ( A|X ) µ ( A|X ) R ] , which is unbiased but can suffer unacceptably large variance when A is high-dimensional and µ provides coverage of all slates , as is necessary for the identification of arbitrary policies . Generally , we can not do much better than IS in the worst case ( Wang et al. , 2017 ) , unless we impose additional assumptions about the structure of actions or rewards . In this paper , we will focus on the case in which the logging policy is factored , that is , µ satisfies µ ( a | x ) = ∏K k=1 µk ( ak | x ) . This generally holds if µ is given by a per-slot -greedy policy or is the uniform distribution , both are common cases in practice . We define the following slot-level density ratios and their sum Yk = π ( Ak | X ) µk ( Ak | X ) , G = 1 + K∑ k=1 ( Yk − 1 ) . ( 1 ) Then , we have the following reformulation result for Eπ [ R ] . ( A succinct proof of this result , as well as all our novel results , can be found in Section 9 of this paper and in the Supplementary Text . ) Lemma 1 ( Reformulation under additive rewards ( Swaminathan et al. , 2017 ) ) . Under a factored µ , Eπ [ R ] = E [ GR ] whenever E [ R | A , X ] = ∑K k=1 φk ( Ak , X ) for some ( latent ) functions φk ( ak , x ) . Motivated by this result , in this paper we focus on estimating the target parameter θ = E [ GR ] . Crucially , lemma 1 implies that under additive rewards , the variance of an estimator of θ need only suffer the size of the slot-level density ratios ( namely , ∑K k=1 d 2 k for uniform exploration and deterministic evaluation ) rather than the possibly astronomical size of the slate density ratio appearing in the IS estimator ( namely , ∏K k=1 d 2 k under the same settings ) . Moreover , lemma 1 shows that µ need not even cover every slate combination – only marginally every action per slot . In the rest of the paper we will not assume any special structure for E [ R | A , X ] ; our estimation results on θ will hold irrespective of whether θ coincides with Eπ [ R ] . In the experiments we will show results for both additive and non-additive rewards . In the factored policy setting , the PI and wPI estimators of Swaminathan et al . ( 2017 ) are , respectively , θ̂PIn = En [ GR ] , θ̂wPIn = En [ GR ] En [ G ] . The PI estimator is unbiased for estimating θ by construction . While wPI may be biased , it can further reduce PI ’ s variance .

Summary: This paper studies off policy evaluation in batch contextual bandits for multi dimensional action aka "slates". The paper studies the estimation of a certain target parameter which is equivalent to the expected reward under the key assumptions of (1) a target logging policy and (2) a decomposition of the conditional reward in terms of additive latent functions of the context and the action dimension. The authors consider control variates formed by linear combinations of the mean shifted importance weight ratios of each action dimension and show that a certain biased low variance previous estimator (wPI) can be recovered within this class. They also further derive minimum variance unbiased estimators from this class along with experimental evidence to support the theory.