Control Variates for Slate Off-Policy Evaluation
Authors: Nikos Vlassis
Ashok Chandrashekar
Fernando Amat
Nathan Kallus
Venue: NeurIPS 2021 Poster (2021)
Keywords: off-policy evaluation
combinatorial actions
slate bandits
control variates

Abstract: We study the problem of off-policy evaluation from batched contextual bandit data with multidimensional actions, often termed slates. The problem is common to recommender systems and user-interface optimization, and it is particularly challenging because of the combinatorially-sized action space. Swaminathan et al. (2017) have proposed the pseudoinverse (PI) estimator under the assumption that the conditional mean rewards are additive in actions. Using control variates, we consider a large class of unbiased estimators that includes as specific cases the PI estimator and (asymptotically) its self-normalized variant. By optimizing over this class, we obtain new estimators with risk improvement guarantees over both the PI and the self-normalized PI estimators. Experiments with real-world recommender data as well as synthetic data validate these improvements in practice.

Control Variates for Slate Off-Policy Evaluation
Nikos Vlassis Netflix
Ashok Chandrashekar"
WarnerMediia
Fermando Amat Gil Netflix
Nathan Kallus Cornell University and Netflix
Abstract
We study the problem of off- off-policy evaluation from batched contextual bandit data with multidimensional actions, often termed slates. The problem common to recommender systems and user-interface optimization, and it is particularly challenging because of the combinatorially-sizee action space. Swaminathan al. (2017) have proposed the pscudoinverse (PI) estimator under the assumption that the conditional mean rewards are additive actions. Using control variates, we consider large class of unbiased estimators that includes specific cases the PI estimator and (asymptotically) its self- normalzedd variant. By optimizing over this class, obtain estimators with risk improvement guarantees both the PI and the self normaiezdl estimators. Experiments with real- world recommender data well synthetic data validate these improvements in practice.
Introduction
Online services (news, music and video streaming, social media, e-commerce, app stores, etc.) serve content that often takes the form of a combinatorial, high-dimensional slate, where each slot on the slate can take multiple values. For example, personalized news service can have separate slots for local news, sports, politics, etc., with multiple news items from which to select in each slot (Li et al., 2010); or video streaming service may organize the contents of the landing homepage of a user as multi-slot slate, where each slot can contain shows from a given category (Gomez- Uribe and Hunt, 2015). Typically, the offered content is personalized via machine learning and A/B testing. However, the number of eligible items per slot may be in the hundreds or thousands, which makes the testing and optimization of personalized policies a challenging task.
An alternative to A/B testing and online learning is off-policy evaluation (OPE). In OPE we use historical data collected by a past policy in order to evaluate new candidate policy. This is primarily an estimation problem, often grounded in causal assumptions about the data gettraaiennginngggg process and its connections to counterfactual one (Horvitz and Thompson, 1952; Robins and Rotnitzky, 1995; Dudfk et al., 2011; Bottou et al., 2013; Athey and Wager, 2017; Bibaut et al., 2019; Kallus and Uchara, 2019). However, even with plentiful off- data, handling combinatorial action space can be challenging: Unbiased importance sampling (IS) (Horvitz and Thompson, 1952) suffers variance on the scale of the cardinality of the action space under uniform logging policy and deterministic target policy. To address this, Swaminathan et al. (2017) have proposed the pseudoinverse (PI) estimator, which significantly attenuates the variance and remains unbiased for decomposable reward structures. In this paper, we discuss how to further reduce this variance using control variates approach (Glynn and Szechtman, 2002). We demonstrate that the self-normalized Swaminathan al. (2017) found to be superior to PI, is asymptotically equivalent to adding control variate. But, the choice of control variate not optimal. We instead show how to choose optimal
(wPI) estimator, which
'The author with Netflix when this work concluded.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

control variates, and even how to do so without incurring any bias. We provide strong empirical evidence based on the and Liu, 2013) that confirms the improvement of our approach over the Pl and wPI estimators.
data from the Microsoft Learning to Rank Challenge (Qin
1.1 Related work
Much of the existing OPE work involves the use of doubly-robuse estimators and variants, which add control variates to the standard IS estimator (Robins and Rotnitzky, 1995; Dudík et al., 2011; Thomas and Brunskill, 2016; Farajtabar et al., 2018; Bibaut et al., 2019; Kallus and Uchara, 2019; Su et al., 2020). Directly optimizing control variates for OPE has been suggested earlier (Farajtabar et al., 2018; Vlassis et al., 2019) but not for problems involving high-dimensional slates.
Combinatorial bandits and variants such as semi- semi-bandin bandits have been studied by Desa-Bianchi and Lugosi (2012) and Kveton et al. (2015) and are also covered in the book of Latiinore and Szepesvári (2020). Earlier approaches to OPE for contextual combinatorial bandits have been restricted to small slate problems (Li et al., 201 1) or relied on accurate parametric models of slate rewards (Chapelle and Zhang, 2009; Guo et al., 2009; Filippi et al., 2010). Li et al. (2015) and Wang et al. (2016) have suggested the use of partial matches between the logging and target actions in order to mitigate the variance explosion. Swaminathan et al. (2017) have proposed the PI estimator, a version of which we study in this paper. Under factored policies, the PI estimator is similar to an estimator proposed by Li et al. (2018) for OPE of ranking policies with user click models. Swaminathan and Joachims (2015) and Joachims et al. (2018) have developed estimators for OPE under the framework of counterfactual risk minimization and discuss approaches for offline policy optimization. Agarwal et al. (201 8) have developed an OPE method for rankers, when only the relative value of two given rankers is of interest. Chen et al. (2019) have developed an approach for offline policy gradients, when slates are unordered lists of items (of random size). McInerney et al. (2020) and Lopez et al. (2021) have studied the problem of slate OPE with semi-bandit feedback (one observed reward per slot), under different assumptions on the structure of the rewards and the contextual policies. In constrast to the latter two works, and similar to Swaminathan et al. (2017) and Su et al. (2020), we study the slate OPE problem when only a single, slate-level reward is observed. Su et al. (2020) achieve variance reduction by clipping propensities, whereas we achieve variance reduction by optimizing control variates.
Applications of slate OPE in real-world problems include Sar Shalom et al. (2016) who apply OPE to large scale real world recommender system that handles purchases from tens of millions of users, Hill et al. (2017) who present a method to optimize a message that is composed of separated widgets (slots) such title text, offer details, image, etc, and McInerney et al. (2020) who apply OPE to slate problem involving sequences of items such as music playlists.
2 Sctup and Notation
We consider contextual slate bandits where each slate has K slots. We will use [K] to denote the set 21, .,, K.. The available actions in slot & are [dy] = {L.... J. Thus, the set of available slates has cardinality I dg. The data consists of independent and identically distributed (iid) triplets (X,, A; R2), = 1, ....., representing the observed context, slate taken, and resulting reward, respectively. Here, A, = (Aa...., Ax)) where A.e. E [dz). We use (X, A, R) to refer to a generic draw of one of the above triplets, where X ~ PP PXX). A ~ yl- X), R ~ TP A, X). The distribution y(a| x) = PA a x= 2)) is called the logging policy as the data can be seen as the logs generated by executing this policy. Probabilities P. expectations E, and (co)variances without subscripts indicating otherwise are understood to be with respect to this distribution. For any function f(z a, r) we will write (x, A, R)) _ Em [X,, A,, R.). We will also follow the convention that random, data-driven objects have the form 'n (such as wú or Ó,).
In OPE, we are given fixed policy (a z) and are interested in estimating its average reward E, [R] using data collected under a logging policy ula x). Here E, refers to expectation under the distribution X ~ P (X), A ~ π[ X), R ~ (- X,).. The quantity E,, [R] can be interpreted as a counterfactual given that P (. A = a, X) is equal to the distribution of the potential reward of slate given context X, for every or, in other words, that TP (. A, X) represents structural model. This assumption, known as unconfoundedness or ignorability, is only needed for giving a causal interpretation to E,, [R] but not for estimation.
2

---

The classical IS estimator for the problem of estimating [,] is E,, [AXI ((AAI)) RJ, which is unbiased but suffer unacceptably large variance when A is high-dimensional and 1s provides coverage of all slates, as is necessary for the identification of arbitrary policies. Generally, we cannot do much better than IS in the worst case (Wang al., 2017), unless we impose additional assumptions about the structure of actions or rewards.
In this paper, we will focus on the case in which the logging policy is factored, that is, μ satisfies y(a 2) IT (4 (ax x:). This generally holds if j is given by a per-slot c-greedy policy or is the uniform distribution, both are common cases in practice.
We define the following slot-level density ratios and their sum
aAAk X) (xx X)'
Y&
G-1+ 111+-(Y-11
Then, we have the following reformulation result for [x,[R) (A succinct proof of this result, as well all our novel results, can be found in Section 9 of this paper and in the Supplementary Text.)
Lemma (Reformulation under additive rewards (Swaminathan et al., 2017)). Under a factored p. E, [R] = E[ whenever E[R| A, = E O% (Ax, ) for some (latent) functions on (ax, x).
Motivated by this result, in this paper we focus estimating the target parameter 0 E.G.R. Crucially, lemma implies that under additive rewards, the variance of an estimator of 0 need only suffer the size of the slot-level density ratios (namely, Σ k= di for uniform exploration and deterministic evaluation) rather than the possibly astronomical size of the slate density ratio appearing in the IS estimator (namely, IIx de under the same settings). Moreover, lemma I shows that jr need not even cover every slate combination only marginally every action per slot. In the rest of the paper we will not assume any special structure for E R A, X]; our estimation results on will hold irrespective of whether & coincides with E [R]. In the experiments we will show results for both additive and non-additive rewards.
In the factored policy setting, the PI and wPI estimators of Swaminathan et al. (2017) are, respectively,
E5, [GR]
0 = E. [GR),
E2, G
The Pl estimator is unbiased for estimating 0 by construction. While wPI may be biased, it can further reduce Pl's variance.
3 A Class of Estimators
We now construct a class of unbiased estimators that includes PI and (almost) wPI. We will then propose to choose optimal estimators in this class and show how they can be approximated.
For any fixed tuple of weights w = (Wi,.... ..k) E RR we define the estimator
@(() = E, [w],
P. GR
we(Yr - 1).
Here, each Y2 1 acts as control variate (Glynn and Szechtman, 2002). We slightly overload notation and for the case wi = _..- WK- = B E IR we will write
((3) E.
I' = GR B(G-
which corresponds using single control variate G 1. Note that 0ct is included in the estimator
class: When 3 = 0, we have Tu = GR and 6 = (00) We further define V,, = Var(Tw). is straightforward show that, for any fixed w, the estimator o(") is unbiased (for any n) and asymptotically normal.
Lemma 2 (Unbiasedness and asymptotic normality of )("))), For any fixed w, we have E(O(1>) = E[T w| = 8, ( 0) +a NO, V...
3

---

While PI is in this class and has variance Vo, wPI is not actually member of this class, but it is asympiotically equivalent to a member. To see this, we first need to understand the asymptotics of ol"h) when we plug in a random, data-driven w denoted wn. It turns out (by Slutsky's theorem) that, if ih converges to some fixed vector w, the asymptotic behavior of ((.h)) is the same plugging in the fixed w:
Lemma (Asymptotics of plug-in w). Suppose tú, -?p w. for some fixed w. Then
v vn((((0>) 0) - N(0, Vie).
We can now establish the following corollary showing that wPI is asymptotically equivalent to 0!" that is, member of our class of estimators using 0 (the unknown target estimand).
Lemma 4 (Asymptotics of wPl). We have
vu((nn - 0) ~2,((0
4 Optimal Control Variates
In the previous section we showed that both PI and wPI are, or are asymptotically equivalent to, members of the class of estimators o(w), But the class is more general than these two, so it behooves us to try to find an optimal member.
4.1 Optimal Single Control Variate
We first focus on the case Wi = ... WK 8, that is, using a single control variate G - 1. Both PI and wPI fall in this category with B and B 0, respectively, as we showed above. Let
vt inl VB3 BER
represent the minimal variance in the class of single-control-vaiiae estimators. By construction, v Vi = Var(GR) and Vt S Vo, the right-hand sides being the (asymptotic) variance of and wPI, respectively. Our next result derives this optimum.
Lemma (Optimal single control variate). We have
(B(G² 0)2
vt = =3 = Vo
Σx Var(Yk)
where
8r EGR - Σn Var(Yz)
Notice that generally 3 * 0. That is, while wPI is asymptotically equivalent control-varie estimate, it is not generally optimal. In the degenerate case where R L G (e.g., constant reward) then we have 3
Our results, nonetheless, immediately suggest how to obtain this optimum asymptotically, using a feasible estimator.
Lemma 6 (Achieving optimal single control variate). Compute a data-driven B3 as
E,, G -1]]
3
Σx Fun - -122]
Then,
((b) - +o ~00,
That is, asymptotically, o(,7) general.
least as good as either Pl and wPI. The improvement positive, in

---

Lemma 7. The improvement of ( over PI is
Vo vt= E|GR( - 1)]²
E[ - 2 0,
and the improvement over wPI is
V/8 vi E 1)
+| E 1) 2 0.
Optimal Multiple Control Variates
We now turn attention to optimally choosing weights on all K control variates. Let
V' inf Va-
(3)
WERK
By construction, V S Vt < min(Vu, Vo), that is, it is smaller than equal the asymptotic variances of PI, wPI, and the optimal
estimator. Empirically, we find that the first inequality can have small gap while the second a large one; that is, optimizing control variates provides substantive improvement over and wPI, but much of the improvement is often captured by optimizing single control variate in certain highly symmetric settings. See sections 6 and 7.
Lemma 8 (Optimal multiple control variates). We have
BJGR(Y2 1)]² Var(Yk)
V V.L
where
EIGR(Y2 1)] Var(Yx)
Again, if R 1l Ys, have wy 0 for all Generally, wy vary over l and equal to 0.
As in the single control-variate case above, we can obtain the above optimum asymptotically using feasible estimator.
Lemma (Achieving optimal multiple control variates). Compute data-driven weights we
En En,RR((Y, 1)]
En. [V, -1)2]
Then,
v -a -NO,VVV.
Lemma 10. The improvement of ((0))) over Pl is
EIRR(Y& 1)
V
2 0. EY, 1)
The improvement over the single control variate g(
EGGY(Y, - 1))²
E 1)]²
- V* Σ
E| - 1 20 0.
B 1)2]
This improvement is of course no smaller than the improvement over wPI, which is no better than the optimal single control variate (asymptoticallyy. The improvement again collapses zero the special case of constant rewards.
5

---

5 Achieving Optimality Without Suffering Bias
Although the above estimators have asymptotically optimal variance among control variates, they may incur finite-sample bias due to the estimation of control variate weights. We can avoid this using a three-way cross-fitting. The estimator we construct in this section will be both finite-sample unbiased and have optimal variance asymptotically.
The estimator we propose is as follows:
1. Split the data randomly into three even folds, Do. D1, D2.
2. For j = 0, 2, compute (اا =
En, (O,]GR((2))) Eo, 179 where ED, refers to the sample average
only over Dj-
3. Set
D2j G+1 mod 3) ED, [Yx 1]. t
= - Σ
J-0,1,
Lemma 11 (Unbiased estimator with optimal asymptotic variance). We have
E[ON] 0, vn(0n- 0) - V(),
While using just two folds would suffice to eliminate bias, ensuring that each data point is inde- pendent from the data used to fit its weight, this would not suffice for ensuring we get the optimal variance. For this,
crucial that we use three folds so as to eliminate covariance as well. Having three folds ensures that, given any two data points and i', either is disjoint from the data used to fit the weights of i' or vice versa. That is, for any two folds j and j', we always have either j ++1$1 (mod 3) j' # j + (mod 3).
In summary, lemma 11 establishes that we can completely avoid bias even finite samples. We could in fact adapt our cross-fitting procedure to obtain alternative estimators that are both sampee unbiased and match the asymptotic variance of the single-control variate version or even wPI: for that we would need only change the weights computed in step 2. We note, however, that the variance characterization in all of our results is asymptotic and we do not currently have corresponding finite-sample results.
6 Experiments on real data
We have benchmarked the proposed estimators on the publicly available dataset R-WBB3KKK from the Microsoft Learning to Rank Challenge (Qin and Liu, 2013). This is labeled dataset that contains about 31k user queries, each providing up to 1251 labeled documents (the labels are relevance scores from 0 to 4). The queries form the contexts of our OPE problem. In order to be able to provide a head-to-head comparison with the results reported by Swaminathan et al. (2017), we have closely followed the experimental protocol of the latter (with minor diffcrences, explained next) in order to generate contextual slate bandit instances from the data. (Additional details about the in the Supplementary Text.)
data and the experimental protocol that followed can be found
Each qucry-document pair in the dataset is annotated with 'title' and 'body' features. As pre- processing step, we first use all data to train a regression tree that predicts document scores from the 'title" features of query-document pairs. We then use this regressor as a standard greedy ranker to extract, for each query, the top-M predicted documents (with M E {10, 50, 1003). Finally, we discard all queries that have less than M judged documents. (This latter step experimental pipeline of Swaminathan et al., 2017. We include it here because it facilitates sampling with replacement, which is required in our experiments. Even for M 100, the resulting dataset contains a sizable number (18k+) of unique queries, and running the PI and wPI estimators against these data attained essentially the same performance as with the unfiltered dataset.)
not present in the
The above procedure defines the sets A(x) of 'allowed' documents per context x. each problem instance, we train a new predictor (using decision trees or lasso) constrained on the sets A(x). and use it as a greedy ranker to extract, for each query, the top- K predicted documents (with K E (5, 10, 30)). This mapping defines deterministic target policy IT (z) for our OPE problem: each K- -izee ranking

---

SDCG, Meda KHS
1DOG, M-33 JC-00
NDOG, W-100 X-10
A,CG, NH-000
192211
2.0
-10
-10
190 N-10, snb
Emi No50 Ke1D
Fmn N-100 1-10
CT%, M-350 Cx110
0.5
Inging
0.3
logpng
FDW
P0W
M05
-1.0
-20
-15
Sumplor
Figure 1: Benchmarking the proposed PICVs estimator against PI and wPI on the MSLR WEB30K data, using tree-based regression model for the target policy. Here we vary M, the number of available actions (documents) per slot, K, the number of slots (size of ranked lists), and the choice of metric that defines slate-level reward. Missing values for wPT (for sample size 1k) are due to excessive variance caused by the presence of outliers. See text for details.
defines a K-slot slate, with each slot having cardinality M. Note that each document can appear only once in the slates mapped by π xxx). The logging policy (u(x) is uniform, and samples K documents with replacement from the set (xx), for each context x. (In Swaminathan et al., 2017, documents were sampled without replacement by the logging policy.) To create each logged dataset, we sampled the x uniformly from the set of all non-filtered queries, up the desired sample size (which, in Swaminathan et al., 2017, could end up reusing some queries multiple times).
We report results for PI, wPI, and the optimal single control variate estimator 0!.0t) from lemma 6, denoted PICVs. The multiple control variate estimator from lemma 9 obtained near-identical MSE as the reported PICVs, so we omit from the plots. (The similar performance of the single vs multiple control variate estimators in this problem is owed to the symmetry between slots; for example, each having the same number of actions. In the next section we show an example where the two estimators exhibit different behavior.) We evaluate each estimator using (log) root mean square error (RMSE) as a function of sample size. We estimate MSE and standard errors of log(RMSE) (the latter computed via the delta method on log (MS))) using 300 independent runs for each setting. As in Swaminathan et al. (2017), we repeat the protocol for a number of different experimental conditions, varying the values of M, K, the regression model for the target policy (tree-based or lasso), and the choice of metric. We tested two metrics, NDCG (additive) and ERR (non-additive); see Swaminathan et al. (2017) for definitions. The version of NDCG that we used is tailored to rankings with replacements: It only differs to standard NDCG in that the denominator is the DCG of slate that is formed by the globally most relevant document (for the given 2) replicated over all K slots of the slate (capturing the fact that documents are sampled with replacement in 14). Note that this version of NDCG is additive over slots, which is a requirement for PI (and, asymptotically, for our PICVs estimator) to be unbiased when the estimand is the value of the target policy. The code for these experiments is publicly available at https://github.comm www..////oaooaaaaaaaaaaaaaeaeoee
In Fig. I we show the results for the NDCG and ERR metrics when using tree-based predictor to define the target policy. (See Supplementary Text for results when using lasso.) We observe that the proposed estimator PICVs dominates both PI and wPI, in both metries, with notable improvements for small sample sizes relative to M.
For relatively small sample sizes, wPI would often return abnormally large estimates (values up to four orders of magnitude larger than the median). This can happen when the denominator of wPI gets, by chance, close to zero. This in an artifact of wPI being defined as the ratio of estimates and
7

---

can lead to very high variance- even higher than Pl's. Our proposed PICVs is not prone to such outliers and the ensuing variance, and is robust uniformly over all the sample sizes tested.
While wPI seems to converge to PICVs for large sample sizes, this need not be the case in general. Lemma 4 shows that wPI has asymptotic variance Vo. that is, wPI is equivalent to using the weight 3 = O on the single control variate. Since, by definition, V/ 2 v Vg-. it is not true in general that wPI should converge to PICVs, at least in the sense of having similar asymptotic distributions, unless certain special conditions hold so that these variances are equal. The estimators do appear similar for large sample sizes in Fig. due to the nature of the particular dataset and the reported setup (choice of metric, target policy, and value of K); see Supplementary Text for different problem settings, where we can observe a uniform gap between PICV and (w)PI.
Our theoretical results support the observation that the MSE of the various estimators in Fig. 1 wil asymptotically appear as parallel curves that never catch up. Note that the plots in Fig. 1 are on a log-log scale (log(MSE) vs log(#t samples)). Our theory predicts that, for large enough sample sizes n, the MSE of the different estimators will be V/n for different values of V. Since log(V/n) = (V) log(n), we expect all methods to eventually appear linear in the log-log scale with the very same slope (hence parallel), but different heights given by the corresponding V (hence never catch up). The fact that PICVs is always at the bottom is consistent with lemma 6, which shows that PICVs attains the smallest asymptotic variance among all single-control control-variate estimators.
7 Experiments on synthetic data: The gap between PICV single and multi
The results on the WEBBKK data do not demonstrate any substantial differentiation between the single and multi variants of the proposed PICV estimators. This is mainly due to symmetry between slots, which need not always be the case. In this section we show results from synthetic simulations on a non-contextual slate bandit problem, using a reward model that is skewed toward the first slot of the slate and toward the target policy (which picks action 0 from each slot). This setting reveals a differential behavior of the single vs the multi PICV variants. Here we assume Bernoulli slate-level rewards, and the slate-level Bernoulli rates p(a) are given by the weighted sum
paa) (0.5))1 6 (a,) 0.01 Ex=z (x(()))
(4)
where the slot-level actions are σą .11,...,uu ). for k- 1,.... K. Under this model, slot-l actions that are closer to action 1 are conferring more impact to the slate-level reward, and hence we expect the latter be more correlated with the target policy. This allows us examine if there any differential behavior between the single (PICVs) and the multi (PICVm) variants.
In each simulation experiment, eq. (4), where, for each 0g, the value of Φu (ax) is drawn from a Gaussian distribution AittributionnN((0 (/002/ 0.01). The parameters of each simulated instance are the number N of logged slates, the number of slots K, and the slot action cardinality tuple D = of the i-th slot. For each sampled tensor, we generate 300 datasets by drawing slates a μ(-) using a uniform logging policy μι(a) and drawing Bernoulli rewards from the corresponding p(a). We use each dataset evaluate deterministic policy (a) I[(a ..., 0)) (w.l.o.g.) using each of the candidate estimators. Since we know the ground truth, we can compute the MSE of
generate T 20 random reward tensors p(a) 1] using
where d; is the number of available actions
each estimator. We report (log) average root mean square error (RMSE), with standard errors
(almost negligible). over the T tensors. The code for these experiments is publicly available at wtwp:///wb..oooeeemamaaaaaaaa//a
The results are shown in Fig. 2 and demonstrate a differentiation between PICVs and PICVm, with the multi variant obtaining lower RMSE by virtue of leveraging more control variates. In the Supplementary Text we show the analogous plot for larger sample sizes.
8 Conclusions and Discussion
We studied the problem of off-policy evaluation for slate bandits. We considered a class of unbiased estimators that included Pl and (asymptotically) wPI, constructed using a control variate approach. This strongly suggested trying to obtain the minimal variance in this class. We showed how to do so
8

---

M=10, K=2
Mm20, Km2
Mm50, K=2
-0.8
-0.8
-0.9
PI
WPI
MCVs
MCVS
-1.0
P.CV
-1.0
-1.0
10g (gg10/MMS))
-1.1
log lgg00AM5E)
-1.1
0g100RMSE)
-1.2
-1.2
-1.3
-13
-13
R
wPI
-1.5
-15
-1.5
MCWV
-00
1000
2000
4000
-00
1000
2000
4000
-1.800
1000
2000
4000
samples
sampless
samppes
Figure 2: Comparing the estimators in a small scale simulation setting, demonstrating a differentiated behavior between the single and multi variants of PICV. See text for details. Missing values for wPI are due to excessive variance caused by the presence of outliers.
asymptotically, and even how to do so without incurring any additional bias in finite samples. Our new estimators led to gains both simulations and real-data experiments.
Future Work. An interesting avenue for future research is how to meld this approach with doubly- robust (DR) style centering. Su et al. (2020) consider the DR-PI estimator given by adding to the PI estimator the (unweighted) control variate E[Gf (X, A) EεeTTe [d.] =(a (X,
In particular, they consider f (ar, a) being an estimate of B[R A = a, X = x]. But unlike the standard non-additive OPE case, it is not clear that it would be efficient to consider only this control variate (Kallus and Uehara, 2019). In particular, restricting the statistical model to have additive rewards make standard semiparametric efficiency analyses difficult, and the efficient influence function is unknown and likely has no analytic form. Again, we can consider range of centered control variates, Σx f (X, As)] - E T(a X)fx(X, a+)l). Letting (a, ax) = recovers our estimator class 0% However, there may be benefit to using (I,, a)-deenneent control variates. Efficiency considerations suggest letting fa estimate E[R A = dg, X = x). but the (sub)optimality of such in the additive-rewards model requires further investigation. A promising direction forward to consider a parametric family, fo E Fe. and optimize the choice of parameter to minimize an empirical variance estimate, in the spirit of more robust doubly robust estimation (Farajtabar et al., 2018). As long as F& has nice complexity, we can obtain the best-in-class variance asymptotically. Characterizing this precisely and investigating the value of this empirically remains future work.
Societal Impacts. In general, having accurate methods for performing OPE can allow decision makers evaluate potentially unsafe decision making policies, and decide whether they may lead to improved societal outcomes, without having to risk the potentially negative consequences of trialling such policies. Nonetheless, as discussed above, OPE with combinatorial actions is nearly hopeless without making additional assumptions. These, however, can fail to hold and lead to biased estimates, which needs to taken into account when considering potential harms of proposed policy. Another potential risk, which applies to OPE in general, is that the data used may not accurately reflect the diversity of the population that the policy will actually be deployed on. In this case, resulting policy value estimates may be much more accurate for individuals who are well represented in the data, compared with individuals who are not. This may bias downstream decision making towards policies that best serve those who are well represented in the data, possibly at the cost of those who are not well represented. Such biases must also be taken into account in any OPE analysis.
Proofs of the main results
Here we provide suceinet proofs for all results, except for lemma whose proof is quite involved and long and is therefore deferred to the Supplementary Text.

---

Proof of lemma 1. Using the definition of G from eq. (I), we have
K
E|G X
Σ
Yu on (Ax, X) Σ 1- K Ex; on (AR,
j&k
Taking expectation A, the last term cancels (when is factored) since B
X
[(de <(a; X) = 1, and the first term is [ X]. Further taking expectation over X
gives the result.
Proof of lemma 2. Since E| X] = we have E[T... X gives the first statement. The second statement is immediate from the central limit theorem (CL.T). n
E[GR| X].
Iterated expectations
Proof of lemma 3. Set 02 En From CLT, vn(ūn - 0) ta N0, V.). Next, note that
Vn(ln - 00,) Σ (in, Wn,k) VnB, .Yx - 1].
Since Vnl... (Y - -2 (O, Var(Y))) by CLT and wn,k Wn.k +pp O by assumption, we have by Slutsky's theorem that each of the K terms converges in distribution to the constant O, and therefore also converges in probability to the constant O. Hence, vn 0m) +pp O. Applying Slutsky's theorem again establishes the claim of the lemma.
Proof of lemma 4. Since En [..G +p and E., [G] +t, 1, the continuous mapping theorem gives GWPi +p Next, note that we can rewrite
E GRR E, F, [
E, = R, [GR] OUPIE, G 1] (.R Σ
guPt Em Yv
E,, G
and invoking lemma 3 completes the proof.
Proof of lemma 5. We have
Va = Var(GR) - 2 - 1)] + B - 1)
which is convex 8. Differentiating, setting to zero, and solving for B, we get 3 = -[(G-1)41 For the numerator, have BJGR(G- E - G R] For the denominator, we have E[ 1) Var(G) = Σ> Var(Ys). Combining yields the claim of the lemma.
B
Proof of lemma 6. This is direct from the weak law of large numbers and lemmas 3 and 5.
Proof of lemma 7. This follows by algebra and noting that vt > max Vo) since both 3 = 0 and are feasible in eq. (2).
Proof of lemma 8. Let = (Yi 1,,-1,,.. I...., YK - 1). have
V. Var(GR) 2w 2w E[G + w wTE[CCT]..
which is convex w. Differentiating, setting to zero, and solving for w gives w" E[CCT] BEGGRC) We have Cov(Yr, Yv) 0 whenever le # W. Hence we simplify and obtain the result.
Proof of lemma 9. This is direct from the weak law of large numbers and lemmas 3 and 8.
Proof of lemma 10. This follows by algebra and noting that V* 2 max(Vo. Vt) since both w1 WK 0 and wi wk 3" are feasible in eq. (3).
10

---

Acknowledgments
We want thank Adith Swaminathan for helping with the MSLR-WEB30L data and code, and for many useful discussions,
References
Agarwal, A., Wang, X., Li, C., Bendersky, M., and Najork, M. (2018), Offline comparison of ranking functions using randomized data. arXiv preprint arXiv: arXiv: 180000555..
Athey, S. and Wager, S. (2017). Efficient policy learning. arXiv preprint arXiv:1702.02899
Bibaut, A., Malenica, I., Vlassis, N., and Van Der Laan, M. (2019). More efficient off off-policy evaluation through regularized targeted learning. In International Conference on Machine Learning. pages 654 663. PMLR.
Bottou, L.,, Peters, J., Quinonero-Candela. J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., and Snelson, E. (2013). Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(65):3207-3261
Cesa-Bianchi, N. and Lugosi, G. (2012). Combinatorial bandits. Journal of Computer and System
Sciences, 78(5):1404-144 1422.
Chapelle, O. and Zhang, Y. (2009). A dynamic Bayesian network click model for web search ranking. In Proceedings of the 18th international conference World wide web, pages 1-10.
Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., and Chi, E. H. (2019). Top-k off-policy correction for a reinforce recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 456-464.
Dudík, M., Langford, J., and Li, L.. (2011). Doubly robust policy evaluation and learning. In Pro- ceedings of the 28th International Conference on International Conference on Machine Learning, pages 1097-1104.
Farajtabar, M., Chow, Y., and Ghavamzadeh, M. (2018). More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456. PMLR.
Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. NIPS, volume 23, pages 586-594.
Glynn, P. W. and Szechtman, R. (2002). Some new perspectives on the method of control variates. In Monte Carlo and Quasi-Monte Carlo Methods 2000, pages 27-49. Springer.
Uribe, C. A. and Hunt, N. (2015). The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), 6(4):1-19.
Guo, F., Liu, C., Kannan, A., Minka, T., Taylor, M., Wang, Y.-M., and Faloutsos, C. (2009). Click chain model in web search. In Proceedings of the 18th international conference on World wide web, pages 11-20.
Hill, N., Nassif, H., Liu, Y., Iyer, A., and Vishwanathan, S. (2017). An efficient bandit algorithm for realtime multivariate optimization. In Proceedings of the 23rd ACM SICKDD International Conference on Knowledge Discovery and Data Mining, pages 1813-1821.
Horvitz, D. G. and Thompson, D. J. (1952). generalization of sampling without replacement from finite universe. Journal of the American statistical Association, 47((60):663-66 685.
Joachims, T., Swaminathan, A., and de Rijke, M. (2018). Deep learning with logged bandit feedhack. In International Conference on Learning Representations.
Kallus, N. and Uehara, M. (2019). Intrinsically efficient, stable, and bounded off-policy evaluation for reinforcement learning. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché-Bue, F., Fox, E., and Garnett, R., editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc

---

Kveton, B., Wen, Z., Ashkan, A., and Szepesvari, C. (2015). Tight regret bounds for stochastic In Artificial Intelligence and Statistics, pages 535-543, PMLR.
combinatorial
Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms. Cambridge University Press.
Li, L.,, Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bande approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, pages 661-670.
Li, L.,, Chu, W., Langford, J., and Wang, X. (2011). Unbiased offline evaluation of contextual -aandittullbandi-- based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 297-306.
Li, L., Kim, Y., and Zitouni, (2015). Toward predicting the outcome of an A/B experiment for search relevance. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 37-46.
Li, S., Abdasi-Yadkori, Y., Kveton, B., Muthukrishnan, S., Vinay, V., and Wen, Z. (2018). Offline evaluation of ranking policies with click models. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1685-1694.
Lopez, R., Dhillon, I. S., and Jordan, M. I. (2021). Learning from eXtreme bandit feedback. Proc. Association for the Advancement of Artificial Intelligence, pages 7732-8740.
McInerney, J., Brost, B., Chandar, P., Mehrotra, R., and Carterette, B. (2020). Counterfactual evaluation of slate recommendations with sequential reward interactions. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1779-1788.
Qin, T. and Liu, T. (2013). Introducing LETOR 4.0 datasets. CoRR, abs//330..n
Robins, J. M. and Rotnitzky, A. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 922429)
Sar Shalom, O., Koenigstein, N., Paquet, U., and Vanchinathan, H. P. (2016). Beyond collaborative filtering: The list recommendation problem. In Proceedings of the 25th international conference on world wide web, pages 63-72.
Su, Y., Dimakopoulou, M., Krishnamurthy, A., and Dudík, M. (2020). Doubly robust off-policy evaluation with shrinkage. In International Conference Machine Learning, pages 9167-9176. PMLR.
Swaminathan, A. and Joachims, T. (2015). Counterfactual risk minimization: Learning from logged bandit feedback. International Conference on Machine Learning, pages 814-823, PMLR.
Swaminathan, A., Krishnamurthy, A., Agarwal, A., Dudik, M., Langford. J., Jose, D., and Zitouni. I. (2017). Off-policy evaluation for slate recommendation. In Guyon, I., Luxburg, U. V., Bengio. S.. Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Thomas, P. and Brunskill, E. (2016). Data-efficient off-policy policy evaluation for reinforcement learning, In International Conference Machine Learning, pages 2139-2148. PMLR.
Vlassis, N., Bibaut, A., Dimakopoulou, M., and Jebara, T. (2019). On the design of estimators for bandit off-policy evaluation. In International Conference on Machine Learning, pages 6468-6476. PMLR.
Wang, X., Bendersky, M., Metzler, D., and Najork, M. (2016). Learning to rank with selection bias in personal search. In Proceedings the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 115-124.
Wang. Y.-.., Agarwal, A., and Dudik, M. (2017). Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning, pages 3589-3597. PMLR.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 8.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 9 and Supplementary Text.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include the code and instructions for all experiments as URL.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Supplementary Text.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Sections and 7.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Supplementary Text.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section 6.
(b) Did you mention the license of the assets? [Yes] See Supplementary Text.
(c) Did you include any assets either in the supplemental material
URL? [Yes]
We include the code for all experiments as URL.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We are using publicly available data.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Supplementary Text.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "q0YZTBGBld", "reviewer": "Reviewer_EZVP", "summary": "This paper studies off policy evaluation in batch contextual bandits for multi dimensional action aka \"slates\". The paper studies the estimation of a certain target parameter which is equivalent to the expected reward under the key assumptions of (1) a target logging policy and (2) a decomposition of the conditional reward in terms of additive latent functions of the context and the action dimension. The authors consider control variates formed by linear combinations of the mean shifted importance weight ratios of each action dimension and show that a certain biased low variance previous estimator (wPI) can be recovered within this class. They also further derive minimum variance unbiased estimators from this class along with experimental evidence to support the theory. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has novel results with clear exposition and technically sound proofs. Extensive experimental evidence is provided to demonstrate improvements of the new estimator over existing alternatives on several benchmark metrics. The authors also theoretically explore optimal estimators in the general class of weighted control variates considered and use a synthetic example to drill down into the somewhat surprising observation in the real data benchmarks that a single weight optimized version performs as well as the version with separate weights. \n\nThe assumptions needed could be considered technically strong (factored $\\mu$ and additive rewards), but may be reasonable for certain settings. The authors claim that most of the estimation results hold generally without the additive decomposition of conditional reward; while this is technically true, it seems like the target parameter $E[GR]$ has significance primarily when this assumption holds. So this seems like a somewhat moot point.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is primarily a theory/ statistical techniques paper and only indirectly relevant for societal impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "q-CUqF4BZdU", "reviewer": "Reviewer_AtDc", "summary": "The paper proposes a new off-policy evaluation estimator for slate recommendation. The main idea focuses on a specific setup of Swaminathan et al, 2017, where the target / behavior policy are factorized. The new estimator generalizes the pseudo-inverse estimator by Swaminathan et al, 2017 with control variates, discusses asymptotic variance property of PI, self-normalized PI (wPI), and compares them empirically, showing that the new estimator based on minimizing the variance using control variate, achieves the best performance.\n\n===== Updated post rebuttal ====\n\nThe authors have addressed my main technical concerns. I made quite some suggestions on the paper, such as adding some additional clarifications on the experiment results to the paper. If the authors are able to make such adjustment in the final manuscript, I am happy to stick with my original evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "======== Originality ==========\n\nVariance reduction using control variate is not a new idea for variance reduction in off-policy evaluation. However, I find that the connection between wPI and PI + control variate in the asymptotic case, to be rather interesting and original.\n\n======== Quality ==========\n\nThe theoretical results of the paper seem to derive from fairly straightforward deduction, however, I still find them to be quite insightful and interesting. The empirical gains are a bit marginal.\n\n========= Clarity ==========\n\nThe paper is clearly written in its theoretical part, but might require some further clarification in the empirical section.\n\n========= Significance =========\n\nThis paper seems to make a new contribution to slate recommendation, by unifying a few ideas and suggesting new methods using control variates. I do have a few detailed tech questions below.\n\n========== Detailed questions =========\n\n1. The analysis is mostly based on asymptotics. And since pretty much all estimators are consistent, when sample sizes approach infinity, only the variance matters. However, overall, one might find such characterizations to be rather weak, given that arguably the main contribution of the paper is theoretical. Have the authors attempted finite-sample bias and variance analysis of variance estimators? In particular, for wPI, where there is a finite sample bias, I think it would be of interest to characterize the bias when the sample size is finite.\n\n2. Once again, I find the connection between control variate + PI being equivalent to wPI to be quite interesting. The equivalence, however, is in the asymptotic case. It would be nice to have a finite sample characterization of the bias of wPI. Can the authors comment on this?\n\n3. In Sec 5, a new estimator with unbiased mean and asymptotically optimal variance property is suggested. If I understand correctly, the intuition is to partition the data set into K folds and compute the control variate coefficient of one fold using other K-1 folds. This preserves the unbiasedness of the overall estimator. I think it might be better to present the general K fold case (instead of K=3 in the paper), and discuss the effect of K. \n\nIt would also be nice to have some more discussion on the intuition of this approach, before diving into the pseudo code.\n\n4. What happens in Fig 1 when the sample size is 10 or 100? I wonder if the bias of wPI would be pretty hurtful and so is the variance of PI.\n\n5. In Fig 2 middle and right plot, it seems that wPI performs worse than PI for pretty much all sample size. Is this contradictory to the claim that wPI has lower asymptotic variance? \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author discusses some limitations and has included a societal impact section.", "review_text": "", "extra_scores": {"ethical_concerns": "No clear ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_0QYnKHMqAn", "reviewer": "Reviewer_apca", "summary": "The paper addresses the problem of bandit slate off-policy evaluation, and introduces a family of estimators using control variates which effectively trade-off the bias and variance for these algorithms. The authors show their family of estimators generalizes two commonly used estimators and provide experimental evidence of the ability of their method to out perform these estimators on real and synthetic datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is overall a well written solid paper. The idea of generalizing equation (1) by adding control parameters in the summation term in G seems like an obvious next step in such line of research (hence not an earth shattering innovation), but the execution of this  idea, and the resulting theoretical results is performed well and elegantly. The optimal estimators derived are very straightforward to compute and therefore I see potential for the method to be adopted in future works. The experiments are not exhaustive, but certainly enough to demonstrate the merit of the method. Additionally, the results in section 7 do a good job demonstrating the additional benefit of slate dependent variates.\n\nA few questions\\comments to the authors:\n- I think my main confusion stems from how lemma 4 interplays with the optimal CV found in section 4. If I understand correctly, since \\beta^* should converge to a fixed point, then asymptotically, PICV should always converge to wPI (which in turn converges to PI). Is this true? It seems to happen in Figure 1 and not in Figure 2. Could Lemma 4 be used to estimate the number of samples at which PCVI and wPI become very close given V_w, and see if that number roughly corresponds to where the estimators meet in Fig 1 and show it for Fig 2 as well (assuming in Fig 2 it doesn't happen for an unreasonably large number of samples, but it is odd that the PCVI curves are parallel to the wPI curves and the wPI curves never catch up.\n- It is unclear exactly which estimator is used in the experiments - invoking Lemma 6 in line 208 makes it seem like it's the vanilla estimator, while the parentheses in line 223 makes it seem like the cross-fitted estimator of section 5. Either way it would be interesting to see a comparison between the two estimators to get a sense of how bad the variance is when using the vanilla estimator.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors properly adressed the societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~4 hours"}, "extra_text": {}}, {"review_id": "Gknq7RcPG81", "reviewer": "Reviewer_nq6H", "summary": "The paper discusses a family of control variates for the problem of slate off-policy evaluation, under the assumption that the logging policy is a factored policy. For the defined class of estimators, the optimal weights are derived and improvements over unweighted estimators are quantified. Using a 3-way sample split the optimal weights are learned. Asymptotic statistics of the estimators are studied.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is pretty clearly written and the contributions are novel. I do not have any major complaints.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies off-policy evaluation with combinatorial action spaces, referred to as "slate recommendation," and developed control-variate approaches that tradeoff bias and variance and provide a more complete story of the estimator landscape in this space. In addition, experimental results are provided, which help bridge the gap between theory and practice. 

Overall, the paper is clearly written and well executed and the reviewers found few weaknesses. As such, we recommend acceptance. Congrats!
Decision: Accept (Poster)