Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to characterize, at validation time, the extreme deviation of a predictor and possibility using this characterization to select among a subset of these predictors, e.g., by adjusting a threshold. The deviation is measured through notions such as value-at-risk, i.e., the value of a given high quantile. The method consists of lower bounding the CDF of the risk with high probability. Experiments are given to illustrate the use of the technique.
strength_and_weaknesses: ### Strengths

* Characterizing the performance of a predictor in the extremes, in addition to in average, can help mitigate risks. This is particularly important for life-threatening or society-disturbing applications of machine learning. This paper addresses this directly by formulating it as a characterization and control of the quantiles of the risk.

* The general methodology provided to derive CDF bounds based on goodness-of-fit statistics provides many useful insight. In particular, the benefits of finer bounds deeper in the tail are evident, especially for integral-type quantile metrics where inaccuracies accumulate. In general, the paper opens up opportunities to bring more ideas from statistics into machine learning.

 * The paper is organized and written very well.

### Weaknesses

* The use of quantiles to control the risk of predictors is not in itself new, as it is shared by many uncertainty quantification techniques. Order statistics always naturally show up in this context. The technical contributions are also quite straightforward, since connections between CDFs and quantiles through inversion is standard. However, this is counterbalanced by the clarity of combining the many ideas, and the emphasis on some of the new insights, like the use of the truncated Berk-Jones statistics to get finer bounds in the tail.

* The significance of the contributions is a bit debatable. It is not clear how much better the novel aspects of this methodology help. For example, my first insight when I started reading the paper was to just use the empirical process lower bounds, i.e., the Dvortesky-Kiefer-Wolfowitz-Massart inequality, by capping it left and right. It tourns out the paper considers this as one of the baselines, and it does perform mostly equivalently to the best proposed approach. I think examples where the new contributions show more striking gains (which I believe exist, since these should do much better in the tail) can help bring out more of the significance of this work. For instance, it is worth elaborating more on when integral metrics like CVaR are most appropriate, since this is where the new stuff shines.

* Generally, it feels that too much of the core contributions currently resides in the appendix. This may be okay for proofs, but when it comes to experimental results, they should be in the main text (namely I’m talking about the fairness experiments results.)

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The significance could be emphasized further, though I believe it is there (see weaknesses for details). The results appear to be easily reproducible.
summary_of_the_review: The paper bridges many interesting ideas, even if not majorly novel. The significance of the results is a bit weak, as some straightforward approaches can compete with it. That said, the paper is an insightful contribution to the efforts of characterizing and mitigating risks of predictors in machine learning, and may be worthwhile to share with the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method to bound a notion of risk that is measured in terms of quantiles, called "quantile-based risk measures (QBRM)". For a given predictor, a distribution over loss values is induced, and this paper shows that a lower bound of the CDF of this distribution serves as an upper bound of any QBRM of this predictor. They further show that it's possible to uniformly lower bound the CDF of the loss distribution for a set of predictors, which further serves as an upper bound to any QBRM for all predictors, especially the optimized predictor. The CDF bound can be derived via various minimum-type test statistics. Empirical results compare the tightness of the bounds produced by a suite of test statistics, various QBRM's, and demonstrate the guarantees are generally met.
strength_and_weaknesses: The paper is coherent, well-written and addresses an interesting and seemingly novel problem of bounding quantile-based measures of risk (as opposed to the usual definition of risk, which takes the expectation). The arguments made in developing the proposed method seems sound. 

Regarding the empirical results, however, I'm not sure if it tells a convincing case for the usefulness of the bounds.
First off, there's not much of a comparison to be done here, since the proposed method of bounding QBRM's is applied with all methods (i.e. test statistics), and they all seem to have close to 0 violation of their guarantees. The newly proposed Truncated-BJ statistic does tend to produce tighter bounds. What was the alpha value used? Is 0% violation actually expected, or are all of the bounds too loose? Also, it's also difficult to tell how significant the difference is between the upper bound and the actual values for each metric. Can you scan different values of alpha? Lastly, if using a specific test statistic and applying the proposed QBRM bound leads to a bound that is way too loose, would it be considered a fault of the test statistic or the bounding method? 

clarity,_quality,_novelty_and_reproducibility: The overall clarity, and quality is good. However, some figures are hard to read: can you make the axes and legend bigger in Figure 2? And maybe different contrasting colors could be used for Figure 3 left, since green and blue both appear muddled with short boxes.
summary_of_the_review: While I think this paper addresses an interesting and relevant problem and proposes a sound method, it's also a little difficult to tell the significance of the empirical results. I am looking forward to the author's response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new framework for obtaining bounds on quantiles of the loss distribution of a given predictor. The method uses a lower bound on the CDF to obtain high-confidence bounds on the quantile loss profiles of the predictor. 
strength_and_weaknesses: Strength
The framework produced by the paper generalizes exisitng known frameworks for quantifying uncertainty in classifier predictions. The paper is quite well written and easy to follow, and the arguments are supported by both theoretical and empirical results.  

Weakness
While the paper doe sa good job of laying the foundations of the framework, it does not stand out on from either the theoretical or the empirical perspective.
- From a theoretical standpoint, most of the theoreems (1, 2, 3) are simple extensions of known results on CDFs and concentration inequalities. There is no discussion on what the challenges were in proving these results, and while they are essential to establish the framework, they do not provide new novel insights. 
- From the empirical side of things, this framework can be quite impactful. Currently, the paper focusses on results for COCO object detection and UCI datasets. There is very little difference in the actual performance when using any of the methods in the literature (Table 2, 3) - why is this the case? 
- Quantile losses are typically used when data is very noisy (say finance time series) or when there are robustness issues to be dealt with. It looks like some of the methods might be more useful in those scenarios and trying out the proposed methods for such applications would strengthen the paper. 

clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is well written overall.

Quality + Novelty 
The framework proposed is novel, but the accompanying theoretical results lack novelty. 

Reproducibility
The authors mention that they will release code on github.
summary_of_the_review: The proposed framework for quantile loss uncertainty estimation is quite nice, but the paper needs to be updated to showcase the empirical utility of this framework.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a framework for measuring and optimizing for quantile-based risk functions, which go beyond the standard practice of measuring and optimizing for the average (expected) loss function. The framework allows one to optimize for any quantity expressed as an integral over the quantile function of the loss; common examples include Value-at-Risk (VaR), Conditional Value-at-Risk (CVaR), and VaR-Interval. These quantile-based risk functions tend to be used to measure something closer to the worst-case loss, rather than the average loss. This is useful when planning to guard against tail risk in a setting where an ML model is deployed.

The paper introduces a novel way to define a lower confidence bound (LCB) on the loss function's cumulative distribution function (CDF), which equivalently produces an upper confidence bound on the loss function's quantile function (the quantile function is the inverse CDF), which produces an upper confidence bound on any quantile-based risk function.
strength_and_weaknesses: Strength:
* The technique is widely applicable for any quantile-based risk function and any set of black-box ML models that produce prediction functions.

Weaknesses:
* The 95% confidence bound on the CDF displayed in Figure 2 looks very conservative, given 1000 samples. I wonder whether there are alternate methods to produce tighter bounds on the CDF.
* Some of the bounds in the empirical section appear to be quite loose, with a 1.5 to 2x gap between the bound and the actual metric (e.g., Mean and CVaR in Table 2, and both results in Table 3). Whether this has practical implications would depend on whether the technique is being used to produce tight bounds on the metrics, or only to select the best model among several candidates.
clarity,_quality,_novelty_and_reproducibility: No concerns. The paper is relatively easy to follow and seems to make novel contributions to an area of study that receives relatively little attention from the ML community today (or at least the part of it where I work), but arguably deserves more. Perhaps it is better-known in the finance and medical fields, where worst-case outcomes are relatively more important than average-case.
summary_of_the_review: Overall, I like this area of study, and I find the proposed technique to be interesting and potentially useful. I like that the technique can be used on any set of pre-trained, black-box models, and that it can be used to select the model that optimizes a given quantile-based functions of a loss. However, that the bounds that are produced seem quite loose in some cases, which makes me question whether practitioners would actually trust that the method is giving useful (tight) bounds.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) The paper shows how to provide mathematical guarantees about the tail of the distribution of losses incurred when making predictions, i.e., just the worst predictions.

(b) High quality mathematical work, clear relevance to fairness in ML applications.

(c) Reviewers who are experts are lukewarm about the mathematical novelty. The actual computational benefit is small (Table 2). 

An ICML 2022 workshop contained the paper "VaR-Control: Bounding the Probability of High-Loss Predictions" by Jake Snell, Thomas Zollo and Richard Zemel.

Comment from the AC: The application to fairness needs more emphasis. The nursery school dataset is from the 1980s and the meaning of "convenient" for financial status is unclear. Moreover, it is unclear what the fairness goal is here; maybe poorer families should be prioritized? Or maybe richer families must be prioritized in order for the nursery school to be viable financially? (The authors' university has the luxury of doing need-blind admissions, but other colleges are less privileged.)
justification_for_why_not_higher_score: Reviewers have not given the submission very high scores.
justification_for_why_not_lower_score: Very solid research with clear potential benefit for follow-on research and ultimately applications.
Decision: Accept: poster