Control Variates for Slate Off-Policy Evaluation

1 Introduction . Online services ( news , music and video streaming , social media , e-commerce , app stores , etc . ) serve content that often takes the form of a combinatorial , high-dimensional slate , where each slot on the slate can take multiple values . For example , a personalized news service can have separate slots for local news , sports , politics , etc. , with multiple news items from which to select in each slot ( Li et al. , 2010 ) ; or a video streaming service may organize the contents of the landing homepage of a user as a multi-slot slate , where each slot can contain shows from a given category ( Gomez-Uribe and Hunt , 2015 ) . Typically , the offered content is personalized via machine learning and A/B testing . However , the number of eligible items per slot may be in the hundreds or thousands , which makes the testing and optimization of personalized policies a challenging task . An alternative to A/B testing and online learning is off-policy evaluation ( OPE ) . In OPE we use historical data collected by a past policy in order to evaluate a new candidate policy . This is primarily an estimation problem , often grounded in causal assumptions about the data-generating process and its connections to a counterfactual one ( Horvitz and Thompson , 1952 ; Robins and Rotnitzky , 1995 ; Dudík et al. , 2011 ; Bottou et al. , 2013 ; Athey and Wager , 2017 ; Bibaut et al. , 2019 ; Kallus and Uehara , 2019 ) . However , even with plentiful off-policy data , handling a combinatorial action space can be challenging : Unbiased importance sampling ( IS ) ( Horvitz and Thompson , 1952 ) suffers variance on the scale of the cardinality of the action space under a uniform logging policy and deterministic target policy . To address this , Swaminathan et al . ( 2017 ) have proposed the pseudoinverse ( PI ) estimator , which significantly attenuates the variance and remains unbiased for decomposable reward structures . In this paper , we discuss how to further reduce this variance using a control variates approach ( Glynn and Szechtman , 2002 ) . We demonstrate that the self-normalized PI ( wPI ) estimator , which Swaminathan et al . ( 2017 ) found to be superior to PI , is asymptotically equivalent to adding a control variate . But , the choice of control variate is not optimal . We instead show how to choose optimal ∗The author was with Netflix when this work was concluded . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . control variates , and even how to do so without incurring any bias . We provide strong empirical evidence based on the MSLR-WEB30K data from the Microsoft Learning to Rank Challenge ( Qin and Liu , 2013 ) that confirms the improvement of our approach over the PI and wPI estimators . 1.1 Related work . Much of the existing OPE work involves the use of doubly-robust estimators and variants , which add control variates to the standard IS estimator ( Robins and Rotnitzky , 1995 ; Dudík et al. , 2011 ; Thomas and Brunskill , 2016 ; Farajtabar et al. , 2018 ; Bibaut et al. , 2019 ; Kallus and Uehara , 2019 ; Su et al. , 2020 ) . Directly optimizing control variates for OPE has been suggested earlier ( Farajtabar et al. , 2018 ; Vlassis et al. , 2019 ) but not for problems involving high-dimensional slates . Combinatorial bandits and variants such as semi-bandits have been studied by Cesa-Bianchi and Lugosi ( 2012 ) and Kveton et al . ( 2015 ) and are also covered in the book of Lattimore and Szepesvári ( 2020 ) . Earlier approaches to OPE for contextual combinatorial bandits have been restricted to small slate problems ( Li et al. , 2011 ) or relied on accurate parametric models of slate rewards ( Chapelle and Zhang , 2009 ; Guo et al. , 2009 ; Filippi et al. , 2010 ) . Li et al . ( 2015 ) and Wang et al . ( 2016 ) have suggested the use of partial matches between the logging and target actions in order to mitigate the variance explosion . Swaminathan et al . ( 2017 ) have proposed the PI estimator , a version of which we study in this paper . Under factored policies , the PI estimator is similar to an estimator proposed by Li et al . ( 2018 ) for OPE of ranking policies with user click models . Swaminathan and Joachims ( 2015 ) and Joachims et al . ( 2018 ) have developed estimators for OPE under the framework of counterfactual risk minimization and discuss approaches for offline policy optimization . Agarwal et al . ( 2018 ) have developed an OPE method for rankers , when only the relative value of two given rankers is of interest . Chen et al . ( 2019 ) have developed an approach for offline policy gradients , when slates are unordered lists of items ( of random size ) . McInerney et al . ( 2020 ) and Lopez et al . ( 2021 ) have studied the problem of slate OPE with semi-bandit feedback ( one observed reward per slot ) , under different assumptions on the structure of the rewards and the contextual policies . In constrast to the latter two works , and similar to Swaminathan et al . ( 2017 ) and Su et al . ( 2020 ) , we study the slate OPE problem when only a single , slate-level reward is observed . Su et al . ( 2020 ) achieve variance reduction by clipping propensities , whereas we achieve variance reduction by optimizing control variates . Applications of slate OPE in real-world problems include Sar Shalom et al . ( 2016 ) who apply OPE to a large scale real world recommender system that handles purchases from tens of millions of users , Hill et al . ( 2017 ) who present a method to optimize a message that is composed of separated widgets ( slots ) such as title text , offer details , image , etc , and McInerney et al . ( 2020 ) who apply OPE to a slate problem involving sequences of items such as music playlists . 2 Setup and Notation . We consider contextual slate bandits where each slate has K slots . We will use [ K ] to denote the set { 1 , 2 , . . . , K } . The available actions in slot k are [ dk ] = { 1 , . . . , dk } . Thus , the set of available slates has cardinality ∏K k=1 dk . The data consists of n independent and identically distributed ( iid ) triplets ( Xi , Ai , Ri ) , i = 1 , . . . , n , representing the observed context , slate taken , and resulting reward , respectively . Here , Ai = ( Ai1 , . . . , AiK ) where Aik ∈ [ dk ] . We use ( X , A , R ) to refer to a generic draw of one of the above triplets , where X ∼ P ( X ) , A ∼ µ ( · | X ) , R ∼ P ( · | A , X ) . The distribution µ ( a | x ) = P ( A = a | X = x ) is called the logging policy as the data can be seen as the logs generated by executing this policy . Probabilities P , expectations E , and ( co ) variances without subscripts indicating otherwise are understood to be with respect to this distribution . For any function f ( x , a , r ) we will write En [ f ( X , A , R ) ] = 1n ∑n i=1 f ( Xi , Ai , Ri ) . We will also follow the convention that random , data-driven objects have the form ·̂n ( such as ŵn or θ̂n ) . In OPE , we are given a fixed policy π ( a | x ) and are interested in estimating its average reward Eπ [ R ] using data collected under a logging policy µ ( a | x ) . Here Eπ refers to expectation under the distribution X ∼ P ( X ) , A ∼ π ( · | X ) , R ∼ P ( · | A , X ) . The quantity Eπ [ R ] can be interpreted as a counterfactual given that P ( · | A = a , X ) is equal to the distribution of the potential reward of slate a given context X , for every a ; or , in other words , that P ( · | A , X ) represents a structural model . This assumption , known as unconfoundedness or ignorability , is only needed for giving a causal interpretation to Eπ [ R ] but not for estimation . The classical IS estimator for the problem of estimating Eπ [ R ] is En [ π ( A|X ) µ ( A|X ) R ] , which is unbiased but can suffer unacceptably large variance when A is high-dimensional and µ provides coverage of all slates , as is necessary for the identification of arbitrary policies . Generally , we can not do much better than IS in the worst case ( Wang et al. , 2017 ) , unless we impose additional assumptions about the structure of actions or rewards . In this paper , we will focus on the case in which the logging policy is factored , that is , µ satisfies µ ( a | x ) = ∏K k=1 µk ( ak | x ) . This generally holds if µ is given by a per-slot -greedy policy or is the uniform distribution , both are common cases in practice . We define the following slot-level density ratios and their sum Yk = π ( Ak | X ) µk ( Ak | X ) , G = 1 + K∑ k=1 ( Yk − 1 ) . ( 1 ) Then , we have the following reformulation result for Eπ [ R ] . ( A succinct proof of this result , as well as all our novel results , can be found in Section 9 of this paper and in the Supplementary Text . ) Lemma 1 ( Reformulation under additive rewards ( Swaminathan et al. , 2017 ) ) . Under a factored µ , Eπ [ R ] = E [ GR ] whenever E [ R | A , X ] = ∑K k=1 φk ( Ak , X ) for some ( latent ) functions φk ( ak , x ) . Motivated by this result , in this paper we focus on estimating the target parameter θ = E [ GR ] . Crucially , lemma 1 implies that under additive rewards , the variance of an estimator of θ need only suffer the size of the slot-level density ratios ( namely , ∑K k=1 d 2 k for uniform exploration and deterministic evaluation ) rather than the possibly astronomical size of the slate density ratio appearing in the IS estimator ( namely , ∏K k=1 d 2 k under the same settings ) . Moreover , lemma 1 shows that µ need not even cover every slate combination – only marginally every action per slot . In the rest of the paper we will not assume any special structure for E [ R | A , X ] ; our estimation results on θ will hold irrespective of whether θ coincides with Eπ [ R ] . In the experiments we will show results for both additive and non-additive rewards . In the factored policy setting , the PI and wPI estimators of Swaminathan et al . ( 2017 ) are , respectively , θ̂PIn = En [ GR ] , θ̂wPIn = En [ GR ] En [ G ] . The PI estimator is unbiased for estimating θ by construction . While wPI may be biased , it can further reduce PI ’ s variance .

Summary: The paper discusses a family of control variates for the problem of slate off-policy evaluation, under the assumption that the logging policy is a factored policy. For the defined class of estimators, the optimal weights are derived and improvements over unweighted estimators are quantified. Using a 3-way sample split the optimal weights are learned. Asymptotic statistics of the estimators are studied.