Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The work studies offline evaluation of policies assessed in terms of risk measures in a contextual bandit setting. It leverages the fact that a variety of risk measures are represented as some function of cumulative distribution function (CDF) of policy rewards with known Lipschitz constants. The work provides two consistent estimators of CDF, bounds their error from true CDF in sup norm, and use these to bound the risk measures. In experiments on two realistic datasets, the estimators are shown to have low error.
main_review: The work provides a general way of performing high-confidence off-policy value estimation which is applicable to a large class of evaluation metrics. These metrics, characterised as being Lipschitz functionals of CDF, are understudied in the machine learning literature and are practically relevant for policy evaluation in high-risk applications. The proposed approach that starts from bounding errors in estimating CDF and carrying this over to bounding errors in risk measure-based value is novel. As a result, the method guarantees good value estimates for any set of risk measures simultaneously, given they satisfy Lipschitz property. Experimental evaluation adequately verifies the theoretical results on realistic datasets.

The advantage of the approach is its generality for all Lipschitz risk measures. But familiarity with applications of such measures and their relative merits is assumed. Although many examples of such measures are presented (beyond mean, variance, and CVaR), some more details on strengths of each and applications in which they are appropriate will help to provide more guidance on use of such risk measures by the machine learning community.
Experimental evaluation does not seem to be suited to ideal settings where risk measures are useful, e.g. continuous-valued rewards with a skewed distribution and a large range (max-min is high). For binary rewards, as in the experiments, Lipschitz constant determined by max reward D is relatively small. While classification datasets are easier to convert to a bandit task, alternatives based on semi-synthetic datasets with simulated continuous-valued outcomes could be explored. This may provide understanding of usefulness of the confidence intervals when Lipschitz constants are large.


Questions I will like the authors to address:

Does the approach assume non-negative rewards (for applying Lemma 4.1)? Is this without any loss of generality?

How does the bounds from the presented approach compare with existing ones for mean, variance, and CVaR?

Does the experiments provide a sense of how tight are the confidence intervals for different risk measures and datasets?

In case of unknown behaviour policy, doubly robust estimators have another advantage (in addition to being consistent under misspecification of either action or reward models) of having root n convergence rates under mild misspecification. Does this property follows as well for the proposed doubly robust estimator, both for CDF and policy value?
limitations_and_societal_impact: Limitations of the analysis are mentioned. Please discuss limitations of the current experiments to evaluate methods across settings relevant to risk measures.

Suggestions to improve presentation:

Please add more detailed comparison of contributions between the concurrent work of Chandak et al. [9] and this work, highlighting the distinction between risk measures considered, guarantees for estimators (in more detail), importance of Lipschitz property to the results, and experimental comparison, if possible.

Please consider adding baselines specialised to estimating mean (e.g. https://www.cs.utexas.edu/~sniekum/classes/RL-F17/papers/HCOPE.pdf), variance, and CVaR. Is the CDF based approach competitive in terms of error and interval width?

Please consider plotting percent coverage and width of the confidence intervals. The intervals in Figures 1 and 2 for different methods are not easily distinguishable, so it hard to visually verify coverage and reduction in interval width with number of samples. For example, width of DR is not visible in Figure 2.

Please explicitly write the assumption making the problem a contextual bandit, i.e. independent draws from context distribution, in contrast to the full RL problem.

Please consider including experiments on datasets where risk measures-based evaluation is well motivated for example medical applications. Some potential datasets are antimicrobial resistance http://clinicalml.org/data/amr-dataset/, or vasopressor and fluid use in ICU https://github.com/matthieukomorowski/AI_Clinician, https://github.com/clinicalml/gumbel-max-scm).


Minor:

Line 35 ‘off-hig’
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes off-policy risk assessment(OPRA), a framework that estimate a risk functional
through first estimate CDF and second plugin estimates for any risks functional that
satisfies Lipschitz properties.
The new proposed framework is elegant and can simultaneously estimate a broad class 
of objectives including CVar, variance, mean-variance and many others.
The paper also provides theoretical convergence analysis for the estimator.
main_review: 
I am satisfied with the novelty and the quality of the paper, and I think the paper is 
well written and inspiring. I would recommend acceptance of the submission.

**Novelty:**
The paper provides the first DKW-style proof of uniform convergence of weighted empirical CDF
if the importance ratio is bounded.
The proof looks standard but is new and I think it is an important complement to 
DKW-style inequalities.

**Quality:**
The proof is sound to me.
And I have the same concern of theorem 5.2 as the authors also mention in line 271.
The error for DR does not depend on $\hat{G} - G$ and cannot improve over IS also seems
weird to me. 
A better bound on that would definitely improves the paper's quality.

**Clarity:**
The paper is well written and inspiring. 
The background on Lipschitz risk functionals bring up a wide range of risk functionals
that we can directly utilized given the estimation of the CDF.
And the main proof in Section 5 discusses two important estimator: IS and DR, 
both are provided with sufficient background the theoretical results.


**Minor:**
In line 229, 'DI' should replace with 'DM'.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper studies the problem of risk assessment with offline data. Unlike traditional work that focuses on the offline evaluation of policies, this paper introduces a new class of risk functionals (of the CDF) as the estimands and generalizes the IS estimator and AIPW estimator for the estimation of the CDF (and the risk functionals).
main_review: 1. Novelty: 

my main concern for this work is that it seems to be a straightforward combination of the well-known properties of the IS estimator and the AIPW estimator and a new class of estimands. The authors introduce a new class of estimands that are (Lipschitz) functionals of the CDF, so to estimate these estimands well it suffices to estimate the CDFs well. Establishing the concentration results for CDF estimates is also a straightforward adaptation of the results in the offline policy evaluation literature. Therefore, I feel there is a lack of technical novelty. Please correct me if I missed something.

2.  Stronger results: 

Continuing with the above comment, I think the paper can be improved if stronger results can be shown. For example, is it possible to provide (asymptotic) confidence intervals for the estimands? Is it possible to establish optimality for the current results?

3. Minor: 

Line 125: “its value” should be “the difference between the function values evaluated at Z and Z’”?

Line 158: the definition of u^+(z) should depend on c as well?

Line 351: might be improved

limitations_and_societal_impact: Discussion on potential negative societal impact should be added.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Previous OPE studies mostly are focused on the overall expected return. This paper studies off-policy evaluation for a more broad class of objectives, i.e., the Lipschitz risk functionals, including CVaR, variance, mean-variance, etc. The main idea is to first use an IPS/DR estimate to estimate the target policy's CDF, then use it as a plug-in estimator to any functionals that satisfy the Lipschitz property. Theoretically, an $O(1/\sqrt{n})$ error bound is derived for the Lipschitz estimate. Empirically, the performance of the plug-in estimator (based on different CDF estimates) is examined.

OPE is crucial to estimate the performance of any new policy while online A/B testing or experimentation might be infeasible. Most prior works on OPE focus on efficient estimation of the expected reward of the policy, while this paper studies the estimation problem under more general metrics, which may be of high importance in real-world applications when people care about different functionals, such as CVaR in finance. 
main_review: Originality/Significance:

I like the problem this paper studies since it has some practical applicability and the method is easy to use with a solid statistical guarantee. I will view the significance level as high, as this more broad class of objectives characterizes lots of useful and popular metrics people care about. Regarding to the originality, I think it is a medium-low. The method proposed is based on CDF estimate and then a plug-in in any risk functionals, this kind of approach has been used in the previous literature as well, though not explicitly stated, see the algorithms [1]. Though this paper discusses its difference with a concurrent work [2], there is still some literature that utilizes this kind of approach [1], it would be great if the authors could comment on this.

Quality:

The quality of this paper is good. The motivation is clear, and the assumption of Lipschitz risk functional is well-motivated and justified. I like Section 4.1, but it might be shortened, and leave some space for experiments. The method is clearly introduced. Theoretically, this paper gives the first uniform concentration for CDF estimates in bandits, which also directly translates to a $O(1/\sqrt{n})$ rate for the Lipschitz objective. For the empirical study side, it seems all the examinations are about, first estimating CDF using different approaches, then a direct plug-in. I am wondering if there any other baselines in the literature that says do OPE for CVaR, variance, etc. If not, it would be great to state them clearly. If yes, it would be great to include them as baselines. I would view the current experiments as ablation studies, rather than a comparison with existing methods. Some specific comments:

[1]. How to estimate the Lipschitz constant when it is not obvious from the context? Or currently, it seems some estimates are not tight?
[2]. It seems the requirement of the Lipschitz property is to derive the uniform convergence rate theoretically. I am wondering how this method (probably empirically) applies to some metrics that not satisfy this requirement, say VaR? It would be great to comment on the necessity of this assumption.

[3]. Some typos: 
line 6, it would be great to define CPT when use it the first time; 
line 52, is scales;  
line 35, off-hig evaluation; 
line 90, steps generates; 

Clarity:
I enjoy reading the paper and it writes well!


ref: [1]. Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy
       [2]. Universal off-policy evaluation
limitations_and_societal_impact: Yes, they are discussed in Section 8.
review_title: null

Meta-review: This paper considers the off-policy Lipschitz risk functional estimation in contextual bandit setting. The authors considered plug-in estimator based on the CDF-estimator, with guarantees from DKW-inequality. The paper is well-written: the motivation is clear and the logic flow is easy to follow.  The method is also novel. 

As the reviewers suggested, the experiment part is relatively weak. Please consider to add the extra empirical study into final version. 

Another issue is that there are some important related work missing, e.g., [1, 2]. Therefore, the method is not well-positioned in literature. Please consider the comparison with [1, 2] and add the table in the response to Reviewer 8N59. 

[1] Faury, Louis, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. "Distributionally robust counterfactual risk minimization." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, pp. 3850-3857. 2020.

[2] Karampatziakis, Nikos, John Langford, and Paul Mineiro. "Empirical likelihood for contextual bandits." arXiv preprint arXiv:1906.03323 (2019).
Decision: Accept (Poster)