Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary:   This paper augments the CSS and OvA loss functions used for learning to defer such that they
  no longer underfit the base model, particularly in problems with many possible labels. This is
  done through the introduction of two post-hoc estimators which can used to take a calibrated
  model where there is zero cost in consulting an expert and adapt it for the setting where
  there is a positive cost for deferring to an expert. There are both theoretical and experimental
  results to support the contributions.

strengths_and_weaknesses:   Originality:

  The particular algorithms introduced in the paper appear to be novel. Post-hoc procedures such
  as this one do occur for uncertainty calibration, but this is a novel application of that approach.

  
  Technical Quality:
  
  The technical quality is very good with the experiments well-thought out and providing very compelling
  results. Since this work is meant to used on top of CSS or OvA it really needed to be compared with
  those methods for accuracy or loss. The proofs for calibration of the loss function were also clear.
  Some of the results are left a little open-ended. When calibrating with a rejector loss there are many
  to choose from and there is not much insight as to why one loss should be preferred over another.
  
  Clarity:

  The entire presentation of this paper is written very clearly and easy to follow. Both in the main presentation
  and in the appendix it was easy to understand what was being done in each experiments and the proofs
  was easy to follow as well.

  Significance:

  This work is very similar to the loss of Verma and Nalisnick and while it is presented as a generalisation
  of that work it's not immediately clear that it's a significant contribution by itself. I nevertheless suspect
  the post-hoc methods introduced in the paper would be useful for others to build on top of.

questions: What rejector losses should be preferred for their settings?
limitations: The authors do not adequately address the limitations and societal impact of the work. While I don't think there is significant ethical or negative societal risk with this work, the paper would have been strengthened by some discussion of the limitations of the presented methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of jointly learning a classifier and a rejector where the rejector can choose whether the classifier or the human should predict, this problem is denoted as learning to defer and the authors tackle the problem when the cost of deferring to the expert is a constant $c_0$ plus the misclassification error of the expert. The authors show that previous state-of-the-art approaches fail when $c_0>0$ by making a connection to label smoothing. They then propose two training strategies for the rejector that combine previous surrogate approaches with new machinery. They evaluate their method on synthetic and semi-synthetic data (CIFAR-10 with simulated experts) and show that they outperform previous methods. 
strengths_and_weaknesses: Originality: The paper has novel analysis of previous surrogate approaches for learning to defer (L2D) when the cost of deferral has an additional penalty $c_0$. The paper proposes a new method to handle this case which leverages previous methods, namely OvA, and adds an extra term in the loss function to form it. This new surrogate has limited novelty and doesn't address the case where $c_0$ is potentially a function of X, a more realistic extension. The related work is very well cited.

Quality: I checked the proofs for the theoretical results and the claims in the papers and they are sound. The empirical results seem reasonable, but no code was provided to verify the results. Furthermore, there are more baselines in the literature that should have been performed. Most notably, the Confidence baseline, where one trains 1) a classifier using cross entropy, 2) a model for expert error, and then 3) defers if classifier error is higher than expert error + $c_0$. One important concern, is that the baselines of Mozannar and Sontag and that of OvA optimize for the cost sensitive loss, while the results show in the experiments  accuracy (without the penalty c_0).  Furthermore, the paper only has experiments with synthetic human experts which is not sufficient to back up claims for human-AI performance.

Clarity: Please make section titles and the title have each word start with a uppercase letter. The paper is well written and easy to follow. 

Significance: I think the paper makes an important contribution in showing that previous surrogate approaches may underfit when $c_0>0$. However, one concern is that the paper doesn't show that the method optimizes for the objective in equation 1, but only that the baselines underfit the target. I think if the authors can answer the questions below in the positive, then my overall evaluation may be raised substantially.  
questions: Why are Figure 5 and Figure 6 showing performance on accuracy instead of on the objective in equation 1?  The baselines optimize equation 1 instead of human-AI accuracy, and it is not fair to show only results of accuracy, but one must show results on the loss in equation 1. Furthermore, you must show that the approach does better on the loss in equation 1 and on accuracy. 

What is the performance on Figure 5 of cSS and OvA when you set $c_0=0$ and you just vary the coverage by changing the deferral threshold ?
limitations: The authors address the limitations adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
The paper addresses the problem of learning to defer, in which a machine learning model learns when to defer the decision on an instance to an expert based on an estimate of the expert's cost. Existing state-of-the-art solutions are conceived for settings in which the expert cost is simply her probability of error, while real-world settings typically involve a fixed cost for querying the expert. In the latter setting, existing solutions underperform because of an oversmoothing of the prediction probabilities. The proposed solution is a couple of post-hoc strategies that prevent oversmoothing and achieve high performance in the setting involving fixed costs.
strengths_and_weaknesses: PROs:

The paper identifies a structural limitation of recent solutions that seriously affects their applicability to real-world scenarios.

The proposed solution is theoretically grounded and easy to implement.

A careful experimental evaluation confirms the substantial advantage of the proposed post-hoc approaches wrt sota solutions.

CONs:

The solution is a post-hoc strategy, in which the base model (and the
expert cost estimator, in the thresholding case) is trained in a
simpler setting (zero fixed cost), and the aggregate model is adjusted
post-hoc on the desired fixed cost c0. This two-step strategy is
usually suboptimal, as the model is not optimized for its working
conditions. 


questions: My main question concerns the suboptimality of two-step learning strategies. Is it possible to say something on whether and how this affects your post-hoc estimator strategies? 
limitations: There is not much discussion on the limitations of the proposed solution. My suggestion is to add something on the limitations of a two-step strategy, possibly hinting at future directions (or open problems) for a fully joint learning setting that also prevents oversmoothing.

AFTER REBUTTAL:
The authors better highlighted how the suboptimality of the two step procedure is (for threshold correction) or could be (for rejector training) quantified and possibly addressed.
review_title: null

Meta-review: metareview: This paper proposed a novel framework for “learning to defer” (L2D), which decides when to defer the decision on an instance to an expert based on an estimate of the expert's cost. The key results include identifying the “failure mode” of existing L2D approaches, a novel post-doc estimation procedure for model calibration, and thorough experiments showing that the proposed algorithm works well when compared to SOTA. During the rebuttal phase, the authors included additional experiments which make the empirical performance more convincing (e.g., baselines + post-hoc thresholding, and additional baseline results as suggested by Reviewer xmUy). Other than several clarity issues, there were no critical concerns in the reviews. 

There are valuable suggestions in the reviews, including improving the clarity when introducing key concepts and notations in the main text, and providing details of the experimental setting and results. The authors are strongly encouraged to address the concerns raised in the reviews when preparing a revision of this paper.

confidence: Certain
Decision: Accept