Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies off policy evaluation in batch contextual bandits for multi dimensional action aka "slates". The paper studies the estimation of a certain target parameter which is equivalent to the expected reward under the key assumptions of (1) a target logging policy and (2) a decomposition of the conditional reward in terms of additive latent functions of the context and the action dimension. The authors consider control variates formed by linear combinations of the mean shifted importance weight ratios of each action dimension and show that a certain biased low variance previous estimator (wPI) can be recovered within this class. They also further derive minimum variance unbiased estimators from this class along with experimental evidence to support the theory. 
main_review: The paper has novel results with clear exposition and technically sound proofs. Extensive experimental evidence is provided to demonstrate improvements of the new estimator over existing alternatives on several benchmark metrics. The authors also theoretically explore optimal estimators in the general class of weighted control variates considered and use a synthetic example to drill down into the somewhat surprising observation in the real data benchmarks that a single weight optimized version performs as well as the version with separate weights. 

The assumptions needed could be considered technically strong (factored $\mu$ and additive rewards), but may be reasonable for certain settings. The authors claim that most of the estimation results hold generally without the additive decomposition of conditional reward; while this is technically true, it seems like the target parameter $E[GR]$ has significance primarily when this assumption holds. So this seems like a somewhat moot point.


limitations_and_societal_impact: This paper is primarily a theory/ statistical techniques paper and only indirectly relevant for societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper discusses a family of control variates for the problem of slate off-policy evaluation, under the assumption that the logging policy is a factored policy. For the defined class of estimators, the optimal weights are derived and improvements over unweighted estimators are quantified. Using a 3-way sample split the optimal weights are learned. Asymptotic statistics of the estimators are studied.
main_review: The paper is pretty clearly written and the contributions are novel. I do not have any major complaints.
limitations_and_societal_impact: See above
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the problem of bandit slate off-policy evaluation, and introduces a family of estimators using control variates which effectively trade-off the bias and variance for these algorithms. The authors show their family of estimators generalizes two commonly used estimators and provide experimental evidence of the ability of their method to out perform these estimators on real and synthetic datasets.
main_review: This is overall a well written solid paper. The idea of generalizing equation (1) by adding control parameters in the summation term in G seems like an obvious next step in such line of research (hence not an earth shattering innovation), but the execution of this  idea, and the resulting theoretical results is performed well and elegantly. The optimal estimators derived are very straightforward to compute and therefore I see potential for the method to be adopted in future works. The experiments are not exhaustive, but certainly enough to demonstrate the merit of the method. Additionally, the results in section 7 do a good job demonstrating the additional benefit of slate dependent variates.

A few questions\comments to the authors:
- I think my main confusion stems from how lemma 4 interplays with the optimal CV found in section 4. If I understand correctly, since \beta^* should converge to a fixed point, then asymptotically, PICV should always converge to wPI (which in turn converges to PI). Is this true? It seems to happen in Figure 1 and not in Figure 2. Could Lemma 4 be used to estimate the number of samples at which PCVI and wPI become very close given V_w, and see if that number roughly corresponds to where the estimators meet in Fig 1 and show it for Fig 2 as well (assuming in Fig 2 it doesn't happen for an unreasonably large number of samples, but it is odd that the PCVI curves are parallel to the wPI curves and the wPI curves never catch up.
- It is unclear exactly which estimator is used in the experiments - invoking Lemma 6 in line 208 makes it seem like it's the vanilla estimator, while the parentheses in line 223 makes it seem like the cross-fitted estimator of section 5. Either way it would be interesting to see a comparison between the two estimators to get a sense of how bad the variance is when using the vanilla estimator.

limitations_and_societal_impact: The authors properly adressed the societal impacts of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new off-policy evaluation estimator for slate recommendation. The main idea focuses on a specific setup of Swaminathan et al, 2017, where the target / behavior policy are factorized. The new estimator generalizes the pseudo-inverse estimator by Swaminathan et al, 2017 with control variates, discusses asymptotic variance property of PI, self-normalized PI (wPI), and compares them empirically, showing that the new estimator based on minimizing the variance using control variate, achieves the best performance.

===== Updated post rebuttal ====

The authors have addressed my main technical concerns. I made quite some suggestions on the paper, such as adding some additional clarifications on the experiment results to the paper. If the authors are able to make such adjustment in the final manuscript, I am happy to stick with my original evaluation.
main_review: ======== Originality ==========

Variance reduction using control variate is not a new idea for variance reduction in off-policy evaluation. However, I find that the connection between wPI and PI + control variate in the asymptotic case, to be rather interesting and original.

======== Quality ==========

The theoretical results of the paper seem to derive from fairly straightforward deduction, however, I still find them to be quite insightful and interesting. The empirical gains are a bit marginal.

========= Clarity ==========

The paper is clearly written in its theoretical part, but might require some further clarification in the empirical section.

========= Significance =========

This paper seems to make a new contribution to slate recommendation, by unifying a few ideas and suggesting new methods using control variates. I do have a few detailed tech questions below.

========== Detailed questions =========

1. The analysis is mostly based on asymptotics. And since pretty much all estimators are consistent, when sample sizes approach infinity, only the variance matters. However, overall, one might find such characterizations to be rather weak, given that arguably the main contribution of the paper is theoretical. Have the authors attempted finite-sample bias and variance analysis of variance estimators? In particular, for wPI, where there is a finite sample bias, I think it would be of interest to characterize the bias when the sample size is finite.

2. Once again, I find the connection between control variate + PI being equivalent to wPI to be quite interesting. The equivalence, however, is in the asymptotic case. It would be nice to have a finite sample characterization of the bias of wPI. Can the authors comment on this?

3. In Sec 5, a new estimator with unbiased mean and asymptotically optimal variance property is suggested. If I understand correctly, the intuition is to partition the data set into K folds and compute the control variate coefficient of one fold using other K-1 folds. This preserves the unbiasedness of the overall estimator. I think it might be better to present the general K fold case (instead of K=3 in the paper), and discuss the effect of K. 

It would also be nice to have some more discussion on the intuition of this approach, before diving into the pseudo code.

4. What happens in Fig 1 when the sample size is 10 or 100? I wonder if the bias of wPI would be pretty hurtful and so is the variance of PI.

5. In Fig 2 middle and right plot, it seems that wPI performs worse than PI for pretty much all sample size. Is this contradictory to the claim that wPI has lower asymptotic variance? 



limitations_and_societal_impact: The author discusses some limitations and has included a societal impact section.
review_title: null

Meta-review: The paper studies off-policy evaluation with combinatorial action spaces, referred to as "slate recommendation," and developed control-variate approaches that tradeoff bias and variance and provide a more complete story of the estimator landscape in this space. In addition, experimental results are provided, which help bridge the gap between theory and practice. 

Overall, the paper is clearly written and well executed and the reviewers found few weaknesses. As such, we recommend acceptance. Congrats!
Decision: Accept (Poster)