Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new estimator for off-policy evaluation and learning: the estimator modifies original importance sampling by adding a weight correction term controlled by one additional hyper-parameter. The paper analyzed the theoretical properties of the proposed estimator and used experiments to show the effectiveness of the proposed approach.
main_review: + The proposed estimator in harmonic correction (s=-1) is novel and simple its design. 
+ The paper is well organized and easy to follow overall
+ The theoretical property of the harmonic correction part is sound and clearly discussed
- There are a lot of different variations of the power-mean correction of IS dicussed in Sec 4. I am wondering what properties they might have. If only harmonic mean version achieves subgaussian rate, and all later experiments only use harmonic mean version, this part feels a little disconnected to the main text of the paper.
- About the bound in Theorem 5.1 and Eq. 2, I am wondering this seems only to upper bound the difference, rather than the absolute difference. If the left hand side difference is smaller than zero, and the right hand side is positive, the bound holds automatically. Could it be possible that the proposed estimator always underestimates the $\mu$, thus always achieve the proposed bound? In my opinion, this bound doesn't help too much in our understanding.
- The motivation of subgaussian concentration is not very obvious for readers without enough context. I think the paper would benefit a lot by showing the practical/ theoretical benefit of achieving a subgaussian concentration bound.
limitations_and_societal_impact: The paper address the limitations and societal impact well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an off-policy evaluation estimator that is sub-Gaussian and differentiable. They provide an anti-concentration bound of the vanilla inverse propensity score weighting estimator, which implies that it is not sub-Gaussian. They propose a correction term on the inverse propensity score that shrinks the score. They show that the proposed estimator is sub-Gaussian under some conditions, and is differentiable with respect to the target policy parameters. They provide some empirical study to show that the proposed estimator is competitive with existing estimators. 
main_review: The paper is well-structured and easy to follow. 

The theoretical study of the concentration property of off-policy evaluation estimators could help better understand the behavior of these estimators. I like Table 2 in the paper which summarizes some theoretical properties of some model-free estimators. The paper presents new anti-concentration bound on the vanilla IPS estimator, and they show their proposed estimator is sub-Gaussian. 

The data-driven approach for selecting \lambda makes the proposed estimator more applicable, and is theoretically grounded by concentration bound in the paper. 

Since the paper is focused on concentration behaviors, I am wondering if the paper could show their concentration behaviors. I am curious whether IS, SN-IS, IS-OS are heavy-tailed empirically. 

It would be great if the paper could discuss the concentration property of the model-based estimators, since it seems that these estimators perform much better than model-free estimators in both evaluation and learning. In particular, the discussion of the effect of the correction weight on the doubly robust estimator seems missing in the paper. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper considers a power mean correction term for off-policy evaluation and learning. The resulting estimator has a subgaussian concentration rate, improving on other variants of importance sampling (which has a polynomial concentration rate). The authors bound the variance and bias of the resulting estimator. These bounds are functions of a tuned parameter $\lambda$, where higher values of lambda increase the bias but reduce the variance giving practitioners the ability to tune the properties to their problem. The estimator also has other beneficial properties: the correction term is differentiable, and the free parameter $\lambda$ can be tuned through data.
main_review: Overall, I think the paper is well written and provides a really interesting algorithm with interesting analysis of the new estimator and of vanilla importance-sampling. The work is well situated in the literature, although my breadth of knowledge of the CMAB literature is limited. I've also lightly checked parts of the theory for correctness, but have not thoroughly checked all the proofs. While the theoretical sections are quite dense, I think the authors do a good job giving readers the appropriate context to wade through the theory. The empirical comparisons also seem quite robust and provide comparisons to several baselines, which has been expanded from prior submissions (as they report). I have very few comments overall, and think the paper is a strong accept assuming there are no significant errors in the theory.

- More runs would make the empirical results better.
- Line 637: I think you mean just $\leq$ instead of just $\leq =$ in the equation.
limitations_and_societal_impact: I believe the authors do a good job detailing their algorithms. This is an algorithmic paper, so there is little to not in societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper addresses off-policy evaluation. It proposes to transform importance weights using the function $\frac{\omega(y)}{1-\lambda + \lambda \omega(y)}$, where $\omega(y) = p(y)/q(y)$ is the standard importance weight. The estimator which uses these transformed weights is analyzed both theoretically and empirically. The obtained estimator is unique in having both subgaussian concentration rate and being differentiable. The estimator depends on the hyperparameter $\lambda$, for which the authors give an estimation procedure that only depends on observable quantities.
main_review: The paper is original and high-quality.  It is clearly written - I feel I understand the main message even though I am a practitioner and do not come from this subcommunity. The contribution is significant and well within the scope of NeurIPS.

As a practitioner, I find it hard to comment on the proofs, but I have a couple of questions to the authors.
1. You introduce the importance weight transformation (your definition 4.1) for arbitrary values $s$ but then the paper focusses on using $s=-1$ only. Why is that? I feel the paper should either use $s\neq-1$ at least once in practice or else not discuss it.
2. You address the contextual bandit setting. How hard would it be to generalise the result to off-policy evaluation for MDPs? Do you have a hunch about whether the same weight transformation would work? Of course, I am not expecting a proof (the paper already has a lot of material), but a couple of sentences of discussion would be nice.
3. In the contextual bandit comprehensive experiment (as described in section B.1.2 in the appendix), you tried two values of alpha for the behavioral policy: $\alpha_b \in \{ 0.8, 0.9\}$. Was there a particular reason you did not try more extreme values (closer to one)? Such evaluation would be useful because it makes the task harder (the regular, untransformed importance weights can become quite large).
4. In Table 4, the most successful approaches for large n are those that plug the proposed weight transform into the doubly robust estimator. However, theorems 5.1 and 6.1 talk about "regular" IS, not about doubly robust estimation. Is there a theoretical justification for combining doubly robust estimation with the proposed importance weight transform?

Very minor points:
- Line 81: "more in general" => "more generally"
- Line 120: "result" => "results"
- Line 772 in the appendix: missing word, probably "samples"
limitations_and_societal_impact: The authors do not discuss societal impact, arguing that the paper is theoretical. I agree with that assessment: off-policy evaluation can be misused but the potential for such misuse is same for this paper as for any other research on off-policy methods.
review_title: null

Meta-review: Importance sampling (IS, aka IPS) is an oft used and oft studied technique for counterfactual reasoning. While it is unbiased (under a "full support" assumption), it can have very large variance when the target policy's distribution is substantially different from that of the data collection (i.e., logging) policy, and this can have an adverse effect on policy evaluation and learning. This paper analyzes the concentration properties of the IS estimator, proving a new "anti-concentration bound," which shows that reward estimated by IS can be bounded away from the true expected reward with probability $\geq \delta$. The paper then proposes a power-mean transformation of the importance weights, and proves that the resulting estimator has subgaussian concentration. Experiments with the new estimator round out the paper, and the results show that the transformed importance weights generally outperform the "vanilla" importance weights.

The reviews agree that this is an interesting, novel analysis, and that it is well written and easy to follow. There were some concerns, but they have mostly been answered by the authors' responses. One unresolved comment (from Reviews 8Vhk and 9nhQ) is that the case for subgaussianity would be more convincing with more empirical evidence that it helps in practice. The reviewers were puzzled why the model-based baselines, which don't exhibit subgaussian tails, outperformed the proposed methods in some experiments.

Regardless, this is a strong paper that deserves to be accepted. Based on the novelty and relevance of the topic, I would go as far as to nominate it for a spotlight talk.
Decision: Accept (Spotlight)