Post-hoc estimators for learning to defer to an expert
Authors: Harikrishna Narasimhan
Wittawat Jitkrittum
Aditya Krishna Menon
Ankit Singh Rawat
Sanjiv Kumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: learning to defer
adaptive inference

Abstract: Many practical settings allow a learner to defer predictions to one or more costly experts. For example, the learning to defer paradigm allows a learner to defer to a human expert, at some monetary cost. Similarly, the adaptive inference paradigm allows a base model to defer to one or more large models, at some computational cost. The goal in these settings is to learn classification and deferral mechanisms to optimise a suitable accuracy-cost tradeoff. To achieve this, a central issue studied in prior work is the design of a coherent loss function for both mechanisms. In this work, we demonstrate that existing losses have two subtle limitations: they can encourage underfitting when there is a high cost of deferring, and the deferral function can have a weak dependence on the base model predictions. To resolve these issues, we propose a post-hoc training scheme: we train a deferral function on top of a base model, with the objective of predicting to defer when the base model's error probability exceeds the cost of the expert model. This may be viewed as applying a partial surrogate to the ideal deferral loss, which can lead to a tighter approximation and thus better performance. Empirically, we verify the efficacy of post-hoc training on benchmarks for learning to defer and adaptive inference.

Post-hoc Estimators for Learning to Defer to an Expert
Harikrishna Narasimhan Google Research, Mountain View hmarasinhan@google.ccmm
Wittawat Jitkrittum Google Research, New York wi wittawat@@ooggee.com
Aditya Krishna Menon Google Research, New York adi tyalmenon@google..com
Ankit Singh Rawat Google Research, New York anki talkitsraatt@gogle.ccom
Sanjiv Kumar Google Research, New York sanjivk@googles
Abstract
Many practical settings allow a classifier defer predictions one or more costly experts. For example, the learning to defer paradigm allows a classifier to defer to human expert, at some monetary cost. Similarly, the adaptive inference paradigm allows a base model to defer to one or more large models, at some computational cost. The goal in these settings is to learn classification and deferral mechanisms to optimise suitable accuracy-cos: tradeoff. To achieve this, a central issue studied in prior work the design coherent loss function for both mechanisms. In this work, demonstrate that existing losses can underfit the training when there is a non-trivial deferral cost, owing to an implicit application a high level label smoothing. To resolve this, we propose two post-huc estimators that fit deferral function on top of a base model, either by threshold correction, or by learning when the base model's errоr rate exceeds the cost of deferring to the expert. Both approaches are equipped with theoretical guarantees, and empirically yield effective accuracy-cos: tradcoff's on learning to defer and adaptive inference benchmarks.
Introduction
Supervised classification conventionally considers learning a single model with good average-case test performance. However, many practical settings allow for choosing amongst multiple models to classify given sample. For cxample, in the learning defer (L2D) paradigm [18], base classifier may be used in conjunction with a human expert. Specifically, the classifier has the option of deferring to the expert, at the expense of incurring some additional cost (e.g., monetary expense). The goal is to learn a classifier and deferral function achieving a suitable accuracy-cost tradeoff. This problem has received surge of interest [26, 19, 22, 34] given the increasing of machine learning in high-stakes decision making. Interestingly, similar considerations are also found in the problem of adaptive inference for resource-constrained prediction [13]: here, base model can choose defer to one or more experts, which are themselves learning models with higher capacity and inference cost. Similar to L.2,, the goal is to achieve a suitable tradeoff between accuracy and (inference) cost.
key problem in such settings is deciding when the base model should defer its prediction to the expert(s). An intuitive strategy is to defer when the model has low prediction confidence (c.g., classification margin). While theoretically grounded [8], this strategy can under-perform when the base model has low capacity [9]. An alternate strategy is to explicitly formulate and optimise a joint loss function for the base classifier and deferral function. This has been followed with success in the L2D literature, particularly with the cost-sensitive softmax cross-entropy loss (CSS) of Mozannar and Sontag [19], and one-versus-all (OvA) loss of Verma and Nalisnick [34].
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

65
train
training samples e
20.0
Fixed deferral cost Co
test
0.0
Accuracy (96)
55
15.0
0.01
50
0.02
10.0
0.03
40
5.0
35
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2
0.4
0.6
0.8 1.0
Fixed deferral cost Co
Normalised entropy
Figure 1: Illustration of underfitting of the cost-sensitive softmax cross-entropy (CSS) loss of Mozannar and Sontag [19]. On CIFAR-100, consider learning defer setting comprising ResNet-8 base model and ResNet-32 "expert" heep. assume a cost Cepp y) = cb 1yy A hesp (a)) deferring to the expert (see $2 for details on notation), where the fixed cost co is varied from [0. 1]. For each cu, we train the base model using the CSS loss (4), and report the resulting accuracy. For co >> 0, the base model exhibits underfitting. evidenced by significant degradation in the training accuracy. In $3.1, trace this behaviour the loss applying high level of label smoothing [31] to incorrect labels. Consequently, the entropy of the hase model probabilities stcadily increase with co (right panel).
While the above losses have proven effective, they are not without limitations. Indeed, both focus on the setting where the cost of deferring is only given by the probability that the expert makes mistake on a sample; this does not consider the additional fixed cost of querying an expert in the first place. As shall see ($3), incorporating any non-zero fixed cost makes these losses prone underfitting; cf. Figure 1. This behaviour is owing to the fact that the losses implicitly apply high level of label smoothing. The setting of non-zero fixed cost particularly important in adaptive inference settings, where querying an expert inherently involves paying an increased computational cost.
In this work, resolve this issue via deferral schemes based on post-hoc estimators. In nutshell, we follow a two-step procedure wherein we fit a deferral function on top of a base model, either by threshold correction, or by learning to predict when the base model's error rate exceeds the deferral cost. This allows us to build on top of the standard (zero fixed cost) CSS and OvA solutions, resulting in an empirically effective strategy with theoretical guarantees. In sum, our contributions are:
we identify limitations in prior losses for L2D, particularly when the cost of deferring to an expert involves non-zero fixed cost in addition to the probability of the expert making mistake ($3) (ii) we propose new post-hoe estimators for learning deferral function, which overcome the above limitation while enjoying theoretical guarantees ($4, Theorem 2, Theorem 4)
(iii) empirically show that the new post-hoc estimators work well on synthetic and real-world benchmark datasets for both L2D and adaptive inference settings ($5).
In the course of our analysis, we additionally generalise the OvA loss to reliably account for general deferral costs ((10), Lemma I), and to reliably estimate label probabilities practical settings (12).
2
Background and Notation
Our interest is in learning settings where a classifier is equipped with the option of deferring to an expert model. For concreteness, we shall focus our exposition on the learning to defer paradigm.
2.1 Multi-class Classification
Fix an instance space X and label space y = [L]
and let be a distribution over X x y. Given training sample S = {(I,,, (n) ]NE[N] drawn from P. multi-class classification seeks classifier h: X + y with low misclassification error R(h) P(u * h(x)). We may parameterise h as hı(x) = argmax Syy (xx) for a scorer f: - R4, Such scorer may be learned by minimising the empirical surrogate risk R(f)) 1 EnE(N ((xn. Un. f(xx)) for some loss f: Xx X + R,, such as the softmax cross-entropy f(x. y, fxx)) = log Eyey efv((e) (x).
For binary classification problems with = 2, common to enforce f2((1) = - ((()) and set ((z, y, f())) (z)) for some margin loss o: IR + R,, such 6(2) = loggl1 + ²)

---

2.2 Learning to Defer (L2D) Problem
In the learning to defer (L2D) problem [18], one seeks a classifier that can either make a standard prediction in y via some base model, defer its prediction to an experi model heep: X - y, at the expense of incurring sample-dependent cost Cesp(r,)) y) > O. Typically, Gesp takes into account both the fixed cost of deferring to the expert, which we will assume to be constant E [0, 1] (e.g.. normalised monetary cost), and an estimate of the probability that the expert makes a mistake on the and Geap (r, = + 1 (v * (z)) The choice Geup (I,, y) corresponds to learning with a reject option or abstention [8, 2, 9, 6].
example. Canonical examples include (1) (2,, y)
Given suitable (z y), the learning problem may be formalised as follows. Consider classifier h: X + y U { equipped with a "defer" option L. Our goal is to minimise
Raer (h) E [1 * h(x)) 1 (z) *1) Cosp (z. v) 1 1l((x) =1)]
(1)
Intuitively, when h chooses not to defer, we incur the usual misclassification error; otherwise, we incur the cost Cesp (Ir v) of invoking the expert hexp. One cast (1) an instantiation of cost-sensitiive learning [11]; see Appendix B. We remark that Okati et al. [22] considered a slightly different setting, wherein one enforces hard constraint the expert cost Cesp(ar, y)- We focus on soft additive penalty in (1), but we may handle hard constraints via standard Lagrangian theory; see Appendix G.
In practice, it is useful to parameterise h in terms of scorer f: X + g with label logits ( }. and "defer" logit (z). We then predict via
(xx))= = argmax (x) if max [x(x) > (T);;
T((x) = otherwise.
(2)
In practice, we may parameterise / via neural network logits: for any v' E (z w. (ar)
for weights wy and embedding 4b This parameterisation induces implicit sharing amongst all logits.
One conceptually useful quantity is the Bayes-optimal h argming. x- (her,h), which is
(argmax P(y | x) 1
max Ply
x))
< E (cesp(x, v)]
h²(x)= =
DEY
y
y/2
(3)
else.
Intuitively, we defer iff the expected cost of querying the expert is smaller than the expected error of the base model. For the constant cost function cesp (r,y), y), this is known as Chow's 's rule [8].
2.3 Losses for Learning to Defer
We review some popular recent strategies to learn f (and thus h) below.
Confidence thresholding. For Geep (ar, y) = cp. one strategy is to learn Ji,..., Fr. by minimising the softmax cross-entropy, and fix f = co- This confidence thresholding approach [21] mimics the Bayes-optimal classifier (3). A related series of approaches [27, 38, 1] make deferral decisions by comparing the base model confidence with the confidence in the expert being correct. When the base model has low-capacity, such approaches may be sub-optimal as the base model may sub-optimally allocate its finite capacity by classifying examples that the expert can easily predict.
Cost-sensitive softmax cross-entropy (CSS). To overcome the limitations of confidence thresholding. it is desirable explicitly learn f- well. To this end, Mozannar and Sontag [19] proposed a cost- sensitive softmax cross-entropy (CSS) loss: for a constant upper bound Cmax 2 on the classification costs, this can be written as:
less(r: f(x))
Σ (Cmmx-11 - 1 # 1/)) (x)) (Cruxx Ceep (1) y)) (x)( (4)
y'Ey
where (z) = exp(f exp(fo(a)///eeuul exp(fy (z)) is the softmax distribution over the standard and the defer label. For Ceap (rr v) = co + 1 (y heep (x)), we have Cmax = co- The loss less is Fisher-consistent [19, Proposition 1]. with Bayes-optimal prediction
(Vy' E Y) py (I) Cmax - 1111P(1 x)) P1 (x) x Cmax - BellrCCopp CZ,pppp (2, y]].
(5)
3

---

The original loss of Mozannar and Sontag [19] uses slightly tighter formulation, with an upper bound Cmax (z, .y 2 that sample-dependent Consequently, we can set Cmux (z. y) max(1, Cesp(2, y)}.
For simplicity, we use constant upper bound for all the losses in this paper.
One-versus-all (OvA). Despite the Fisher consistency of fcss, Verma and Nalisnick [34] observed that empirically, the loss may not produce calibrated estimates of the expert's error probability. Verma and Nalisnick thus proposed a one-versus-all (OvA) loss for the cost function Coep (2, y) = (3 o hexp (x)):
lova v, [(x)) =(()) +
Σ ((-Iv(x)) O'EY (v)
(6)
(1 y)) (x)) Capla y) (x)
Here, o: R R. is binary proper composite loss [28]. c.g., d(2) = log(1++ +) Such losses have an associated inverse link function sb R- + [0, 1]. and produce Bayes-optimal scorer
(Vy' y) (f (x)) Pwy' [x)
(ff(())) - Byle (cexp(r, 9)}-
Intuitively, both (4) and (6) seek to have the first L logits (ff( Fz. (x)) estimate Ply 2), and the reject logit fi (ar) model the cost of querying the expert. With this, the highest scoring logit provides us with an estimate of the Bayes-optimal deferral decision (3).
3 Limitations of Existing Learning to Defer Losses
We now study existing losses for learning to defer more carefully, and show that they may underfit in an important practical setting. Specifically, recall that canonical choice of deferral cost function in (I) is (xxp(z,)) y) = 1 (y * hxxp(z)) co, for constant 2 0. Here, co represents a fixed cost of querying the expert, which is independent of whether the expert misclassifies the sample.
The setting of non-zero fixed cost co > is of practical interest; e.g.. this is intrinsic to adaptive inference settings, wherein querying the expert model fundamentally involves increasing a computational overhead (see $4.3 for more discussion). Thus, an ideal surrogate loss for (1) should be performant for co > O. Unfortunately, we now show that this does not hold for the CSS and OvA losses.
3.1 Underfitting of Cost-Sensitive Softmax Cross-Entte Cross-Entropy when co
The cost-sensitive softmax eross-entropy (CSS) is in principle applicable for any deferral cost Gesp(1: y). and in particular C2sp(2r y) Co + 1yy F h=s)(z)) for any co 2 0. However, the focus in Mozannar and Sontag [19] was the case co = O. In fact, we now show that when co > O, the loss can underfit. To this, note that the tightest possible maximal cost Cmax + cg. With this choice, (4) becomes
(ass(a, u. f(x)) = log (P, + co Σ - loggpe (x)) - 1 = hx)) log (z)) (8)
When co = O, the second term vanishes, and we have a conic combination of the log-loss on the true and deferral labels. When co > 0. the second term is active, and applies form of label smoothing [31]: we allow for all labels y' E y to be potential "positives" for æ, with a weight of co E [0, 1]. Put differently, we treat the ground-truth label distribution as mixture of one-hot distribution on y. and a uniform distribution over all labels. As co + 1 we apply a large amount of smoothing on labels other than y: this can be problematic. Concretely, consider x where the expert gives the wrong prediction, i.e., P(y- = heep (x)) = O. Then, the Bayes-optimal solution (5) will be
(Vy' 4) (2) (P(s// x)) +
L.
p (z)
Observe that for co > O, as L increases, the range of values in 5* shrinks: indeed, the gap between the optimal probability for true label y and a competing label y/ will be (()L). This small probability margin makes learning challenging: given a finite sample, it will be easy for the model to mistakenly swap the true label y with some competing label y'. Thus, in settings where L is moderately large, and where there is a non-zero fixed deferral cost Co > 0, Less may underperform. Such issues have also been observed [7] for similar loss developed for label noise [25].
As an illustration, we consider in Figure learning to defer setting comprising ResNet-8 base model and a ResNet-32 "expert" on CIFAR- 100. We assume deferral cost (a, y) = co co+12 +ly * hes (x)).

---

where is varied in [0, 1]. For each cu. we train the base model using (4), and report the train and test accuracy. As cj increases, the base model shows significant training accuracy degradation, i.e. underfits. The high level of label smoothing plays role: in the right panel, we see that the entropy of the model's predicted distribution over labels increases with co. (The entropy is normalised by log to lie in [0, 1].) See Appendix I for additional plots, and Appendix 1.3 for results when Coxp( v) co-
worth pointing out that the original loss of Mozannar and Sontag [19], which uses a tighter sample-dependent upper bound, will not have the label smoothing term on the samples that the expert predicts correctly, but will still underfit on the samples that the expert wrong on.
3.2 Underfitting of One-Versus-Al Loss when Co 0
The one- one-versus-all (OvA) loss of Verma and Nalisnick [34] was presented for the specific case of co O; as stated, the loss is not applicable when co > O. We may however extend this loss to general Cesp(2, y) with upper bound Cmax 2 (see Appendix D for a more general derivation):
(o. f(x))
Cuax (xf( + Σ ((emax - 1) () (x)) + ((-))(
V'
+ (Cmax - (x, 8)) ((f((=)) Casp y) o( ((-F1(x))
(10)
each Φy (x). As with OvA, this loss results in coherent Bayes-optimal solution when Φ is a strictly proper composite loss [5, 28]. Recall that such losses are characterised by an inverse link function eh: IR + 11 such as the logistic loss <(=) = log(1 + + with sigmoid inverse link vb(v) = (1 + exp(-oo)
Compared (6), have additional weighted
Lemma 1. Suppose ④ is a strictly proper composite loss with inverse link function th. Then, the one-versus-all loss in (10) is calibrated for the learning to defer risk (1), with Bayes-optimal scorer
(Ayy E ((F
Cmax
(1y x))
((F((z))
Cmax
Eyla [Ceup y)l
Gmax
Cmax
Unlike the optimal scorer for the CSS loss, here Evel (0 (x)) 11. Nonetheless, the highest scoring logit agrees with the Bayes-optimal classifier (3) for learning to defer.
Reviewing (10), see that when Cuax 1, we apply a form of label smoothing by additional terms of the form En' o(fgr (z)), i.e., we effectively treat all y' * as potential positives. As with the cost-sensitive softmax entropy, this can result in underfitting: recall that with Cexp(T; (I, y) co + 1y of hexp(z)), we have an upper bound Cmax + co L. We empirically confirm in Appendix I.2 that we indeed observe analogous underfitting behaviour as per the CSS loss in Figure 1.
The above shows that the CSS and OvA losses have subtle limitation when co > 0. This is not in conflict with the cisset-consistency of these losses: the latter establishes that the asymptotic minimiser of these losses agrees with the Bayes-optime classifier (3). Our analysis explicates that the precise form of these Bayes-optimal solutions may not be amenable to learning from a finite sample.
Since both losses seamlessly handle co = 0, natural question is whether one can leverage this solution to help guide the one for Co > 0. We now present post-approaches that do precisely this.
Post-hoe Approaches to Learning to Defer
We now present two simple post-hoc strategies for learning to defer, which build on the CSS and OvA solutions for co via threshold correction and re-training the deferral function.
4.1 Post-hoe Threshold Correction
Our first host-hoc strategy (Figure 2) exploits the form of the Bayes-optimal classifier in (3): to mimic
this classifier for the cost function (2,,(2,,)) y) = 1 3 hesp (xss(()) (x)) + co. it suffices to obtain estimates of Pyy 2) and the expected deferral cost E[Cosp(I, 8)] = F * hea((x)) + cu- Consequently, suppose we learn models n: X + [0, 1]" and e: X + [0, 1] estimating Pyy x)) and Pyy * hz)(()) respectively. Then, for any co 0,
approximate (3) via the classifier
h(z) argmax Ty if max my(x)) e(x) + co: 7((x) =1 else.
(11)
5

---

Training
Minimise S0vA (12) with
Logits
na(2) Ever exp(y(())
(xp))(
=(x))
Classity vin (1l) for any ou
sample
Deter logit
f(a) = ((()
model training
Compute statistics
Final classifier
Figure 2: Summary of post-hoc threshold correction procedure ($4.1).
Intuitively, this procedure ought to perform well when T(x) is close to ny(x) = [P(y >))eey, and (x) is close P(y & hesp(a)), The following excess risk bound for formalises this intuition.
Theorem 2. Pick any w: X + [0. 11th and c: X + [0, 1]. Let h be the corresponding classifier in (11). Then, the excess risk of h compared to the Bayes-optimal classifier h² (3) is: Rher(h) Reter (h*)
Craax IE [1l(())) [1)] + 2 [[I((y + hesp (x)) <
Crucially, observe that we can learn π and once, and then sweep across different co values (11). Thus, we attempt to leverage the solutions for either the CSS or OvA losses with co = 0- recalling that these do not exhibit underfitting, unlike when co > 0 - and then simply vary co post-hoc. Indeed, both losses' Bayes-optimal solutions (5), (7) for co = 0 suggest that a transform of .i.....fa provide estimates of Ply| z), while a transform of fi provides an estimate of (P # hesp(2))
Unfortunately, in practice, both losses are potentially suboptimal to learn x, c. For the CSS loss Verma and Nalisnick [34] showed does not provide calibrated estimates of P o hand, while the OvA loss provides calibrated estimates of P(y & hexp((r) (x)). ) its probability estimates of (P(y 2) may underperform when L is large (see Appendix J). We overcome this with a simple solution: we estimate via the softmax cross-entropy, and e via the OvA loss. This is achieved via hybrid softmax cross-entropy plus OvA loss (SOvA):
On the other
P. (x) ) + (Emax Cxxplz, v)) o(f((x)) + Cocp ( ((-fi(x)), (12) Esova (I, y, f(z)) = - loggo,
where (x) = exp(f,(x))/ Eye ex)( Following the Bayes-optimal solution, we set (x) (pi(z).... Pz())) (z)) and c(z) = (ff((r)) (z)). As with the CSS and OvA losses, we may parameterise (x) = w. (x) for shared Φ and thus induce information sharing amongst all logits.
The above post-hoc approach crucially relies estimating the expert's error rate. A potentially simpler task is learning to predict whether is beneficial to invoke the expert, i.e., whether the expert's error is lower than that of the base model. We consider an alternate strategy that implements this.
4.2 Post-hoe Rejector Training
Our second post-hoc strategy revisits the fundamental risk (1) underpinning the learning to defer problem. While defined in terms of a classifier h: X + y U {L), we may rewrite this in terms of a base classifier h: + and rejector r: X + R, with the latter denoting the confidence deferring:
Baer h, r) E(z,z) (1(((x) < 0) (x, v) + 1(r(x) > 0) Cesp(x,, v)).
(13)
where Cmod( y) = 1yy + h(x)). Given a scorer f: X + R++1 we may set h(x) = argmax, S (x) and r(x) fz( max (x). Both CSS and OvA construct surro- gate losses that are applied jointly over all logits surrogate loss for both the base classifier h and rejector r. As established earlier, the solution obtained from these losses reasonable under fixed deferral cost co = O, but tends to suffer when Co > 0.
S1: equally, they construct joint
We may resolve this issue via a two-step procedure (see Figure 3): in the first step, we obtain standard logits (z:), .............................. (x) by minimising either the CSS or OvA loss with co 0. In the second step, we train only the deferral logit, by constructing partial surrogate for only the rejector r(æ) in (13). Concretely, for a binary surrogate loss o:
R,, we may define the following surrogate for (x, y, r(x); h) = amod (I, y) ((((x) + cap( ((-r(())))
(14)
and minimize the resulting partial surrogate risk over r:: R (h, = T::=EElE ((e v, rxx]
(15)
With this loss, co only plays role in the Capp y) weighting term, and thus only influences the deferral decision; it does not induce any underfitting of the standard logits fi....,fe ft. Observe also

---

Logits fu....f
Defer
Training
((rm Bol) n-1
Minimise CSS loss (8) with Or 0
Minimise rejector loss (14) for any 6o
logit
Classify via (2)
sample
(Frozen)
Base model training
Post-hoc training
Final classifier
Figure 3: Summary of post-hoe training procedure ($4.2).
that (2, y) will involve the frozen logits ft....: :......f f obtained from the first stage. We may further leverage these logits by parameterising r(z) = fi m for (z), and optimising only the (x) term. In practice, exploiting the probabilitiess from the first stage model can offer a closer approximation to the Bayes-optimal classifier; see Appendix F for more discussion.
Theoretically, the surrogate loss in (14) provides calibrated deferral rule for any base classifier h. For strictly proper composite & with inverse link sb= IR + [0, 1], the optimal rejector is as follows.
E (and((r)))
Lemma 3. The Bayes-optimal rejector for the loss (14) is (b(r"())) (2)a()))))
uia + (x.))))))
Note that when the first stage logits agree with the Bayes-optimal solution, Eyja (Cmma y)] 1 - maxyey (y æ), and so deferring when (x)) > 0.5 produces the optimal decision.
We next provide bound an excess risk bound for a classifier learned using the post-hoc training procedure, under the assumption that the surrogate loss e used is classification calibrated [3].
Theorem 4. Suppose & is classification calibrated and Cexp ( (x. y) E [Cmin Cmaxx) for some Cmin > 0. Denote the misclassffccation risk for classifier h by Rerr (h) = E [1 & h()))]. Let be the rejector obtained by minimizing the partial surrogate risk in for base classifier h. Then the excess L2D risk for the resulting classifier (h, f) is bounded in terms of the excess surrogate risk for the rejector and the excess misclassification risk for the base classifier h. (he
(h, h:X 6:(--1)
min
< (Ranchh
min Rd (he((h.))) + (ho - min Rore (h), r
m:x
for some increasing function V R4+ + R+ with (0) = O.
4.3 Discussion and Extensions
Relation to existing work. Post-hoc threshold correction generalises Chow's rule [8] for learning to reject, wherein Goep ( y) = Cg- It also generalises the approaches of Raghu et al. [27]. Wilder et al. [37], Bansal et al. [1] for learning to defer: these assume Co O, and use separate models to estimate (2) and e(z). By contrast, we estimate these quantities with a single neural model, by minimising the SOvA loss in (12) with for (x) = w. +
(x) for y' E ( As with the standard OvA loss, this induces implicit information sharing between the defer logit f1 and the standard logits JI,..., Jz: thus, the estimates of the expert error probability will be influenced by the base model. Note that the confidence approach of Raghu et al. [27] shown to be outperformed by the CSS and OvA losses in Mozannar and Sontag [19], Verma and Nalisnick [34]; our proposed threshold correction further achieves competitive superior performance to these losses ($5).
Verma and Nalisnick [34, Appendix E] considered a post-hoc scheme for the setting where there is a hard constraint on the expert cost. Here, one trains a probabilistic model f using the OvA loss, and defers on the samples with the largest values p (x) - max (x)). While derived for different setting, this is similar to (1)), as varying the cost effectively changes the fraction of samples that are deferred. Note however that we use the SOvA rather than OvA loss to obtain 5,e, which can have notable impact performance when L is large (see Appendix J).
Cortes al. [9] proposed joint surrogate over f for (13) in the special case of constant Casp(ar, y) = Co. and scalar scorer f: X -, IR. for binary labels. By contrast, the surrogate loss (14) handles multi-class f. generic cost functions Cexps and provides partial surrogate over only r.
Adaptive inference and multiple experts. Post-hoc training is naturally amenable to settings where there multiple experts. Concretely, suppose there are K experts with logits fz)) f(k)
7

---

85
Chou
C55
Ouk
Posthot Training
60
Posther Thredhlddngg
-6 -4
-2 0
10
20
35
40 50
Samples coferred expert (%)
Figure 4: Results on synthetic dataset (left panel) comprising two subgroups, each with positive (+) and negative (x) samples. The subgroup of concentric circles corresponds to latent variable a = 1, while the subgroup of adjacent circles corresponds to = 0. The base learner is a lincar classificr on the raw features (I1,, 22), and query expert that trains classifier classifiers using different losses, for varying fixed deferral cost co. Posthoc training is competitive superior the cost-sensitive softmax (CSS) and one-versus-all loss (OvA) (right panel).
quadratic features. We train
and our goal is learn which expert to employ for a given sample. One may learn multiple deferral functions / (x)..... [(x)(x) (I)) in a post-hoc manner, and choose the smallest k with [1 (x) < O.
This setting arises naturally in adaptive inference [13], where goal is to reduce the inference cost of a learning system by allowing for variable compute across samples: intuitively, we would like to spend less computation "easy" samples. To achieve this, one may construct several models ("experts") of increasing inference cost, and defer to a suitable expert. Such models include early-exit classification heads at intermediate nodes of network [24, 32, 36, 15, 29], sequential cascades of models of increasing complexity [4, 30, 35]. Conceptually, adaptive inference involves the same core problem as learning to defer: any given intermediate model, one has the option defer prediction to the next model, at the expense of increasing the overall computation cost. Interestingly, similar ideas post-hoc training have been explored in adaptive inference [33]. Compared to this work, we present a unified view of the learning to defer and adaptive inference problems, demonstrate limitations of existing losses for the former, and leverage generic logits f1...., fL from any base classifier.
Experimental Results
We now present empirical results illustrating the efficacy of both our proposed posthoc estimators.
5.1 Results on Synthetic Data
We begin with a synthetic problem where X C R2. We consider a distribution P(z, y) Ene 10. P(z, y, a), where E f0, 1) denotes some latent subgroup. Figure shows sam- ple draw from Pa, y): in a nutshell, P(x,y y a 0) is a mixture of isotropic Gaussians, while P(a, y a = 1) comprises two concentric circles. See Appendix H.1 for precise specification. We consider learners that have access to a sample ((T,, yx)
MN } from P(x, y), and to expert model predic- tions (I.)) The expert is taken to be linear classifier that is trained on only samples with using quadratic features: i.e., for (2), 22), we construct (xx) = (æ), 22; 2f. 22. xn Ø2).
We set the expert deferral cost function to be Cesp (2, y) = co + 1 (y * (x))
We train linear classifiers using confidence thresholding based on constant deferral cost co (Chow), the CSS and OvA losses, and our proposed post-hoc approaches. For each learner, we vary the fixed deferral cost co from {0.0, 0.05, 0.10, ... 0.50). ). For each setting, compute the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert. We conduct 250 independent trials, each with different draw of the training sample and expert predictions.
We highlight few trends from the results Figure 4. First, Chow strongly underperforms post-hoc thresholding when many samples are deferred. This is because the latter does not consider the expert's s error (which is high on a = 1) when deferring. Second, post-hoc training is competitive or superior to CSS and OvA at most operating points. Third, post-hoc thresholding underperforms post-hoc training. This is by construction, as challenging to reliably estimate IP(y)/ x)) with a linear model. However, we shall now see that on real-world benchmarks, thresholding is far more competitive.
8

---

87
60
60
85
Accuracy (%6
550
Accuracy (%k)
Accurocy
84
40
83
40
82
20
C55
OuA.
81
30
Post hoc Training
hoc Threshokcnng
80
10
20
30
0
20 40
60
0.0
10.0 12.5 Samples deferred to expert (36)
2.5
5.0
7.5
Samples deferred to expert (%)
Samples deferred to expert (%)
Figure 5: Results CIFAR-10 (left). CIFAR- 100 (middle), and ImageNet (right) in learning to defer setting, where base model is allowed to defer to "specialist" expert. The latter is trained on only samples from the first classes on CIFAR-10, 10, the 50 classes from the first 10 "coarse labels" on CIFAR-100 and only classes corresponding to the "dog", synset on ImageNet. Our posthoc schemes offers gains over the existing cost-sensitive softmax (CSS) and one-versus-all (OvA) at most operating points, particularly when only few samples deferred the expert (corresponding to fixed deferral cost co > 0).
5.2 Results on Real-World Data
We now report results on the CIFAR-IO, CIFAR-100 [16), and ImageNet [10] datasets.
Learning to defer. Inspired by Mozannar and Sontag [19, Section 6.2]. we consider a setting where there is a "specialist" expert: train expert model only samples with label belonging to some subset yuub Intuitively, this model "specialises" to only samples from Ysub- We then consider base model that is trained all samples, but is allowed defer to the "specialist" expert. On CIFAR-10 10 and CIFAR-100, we use a ResNet-8 base model and ResNet-56 expert; on ImageNet, we use MobileNet-v2 base model and an EfficientNet-BOO expert. The label subset ysub comprises the first 5 labels on CIFAR-IO, 10, the 50 labels corresponding to the first 10 "coarse labels" on CIFAR-100, and all labels corresponding to the "dog" synset ImageNet.
We consider a cost (x, 3) = co + 1yy * heep (z)) of deferring to the expert model. Per Figures and 3, we apply posthoc threshold correction on top of the SOvA solution with co 0, and posthoc training on top of the CSS solution with co = O. Figure 5 compares all approaches as we vary the fixed cost co. reporting the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert. Our posthoc estimators offer gains over the existing approaches at most operating points. We particularly see gains in the regime where only small fraction of samples deferred to the expert, which corresponds to larger values of co. Between the two post-hoc methods, we see that the threshold correction approach tends have (slight) edge. This suggests that these benchmarks, it is possible to obtain sufficiently reliable estimates of IP(y x) and Ply & hz)((z)).
We reiterate that when eo is large, the existing CSS and OvA losses tend to degrade owing to underfitting. Indeed, their performance in this regime is significantly lower than simply using the student model trained with the softmax cross-entropy (corresponding to Chow with 0% of samples deferred). Note also that in comparison to the synthetic dataset, here L is larger and thus further exacerbates underfitting Finally, as in the synthetic dataset, Chow strongly underperforms when large fraction of samples are deferred to the expert; this is because it does not take into account the expert's 's error rate, and thus potentially defers when the expert performs akin to random guessing.
Adaptive inference. We conclude with results on the related problem of adaptive inference. We consider a setting where there is a cascade of two models of differing inference costs, as measured in floating point operations (FLOPs), and we seek to learn a deferral function that intelligently forwards small subset of samples to the latter model. This allows for an inference scheme with favourable accuracy-compute tradcoff. Compared to the previous section, we now train the experts on all labels.
We report results on CIFAR- 100 and ImageNet, with the former using ResNet-8 8 and ResNet-56 models, and the latter using a MobileNet-v2 v2 and EfficientNet model. As before, we consider the deferral cost function Coxp (2. y) = cu + ( * h=)(())). Figure 6 compares the FLOPs versus the overall accuracy of the base model plus expert. Here, the CSS and OvA methods again degrade in the low FLOPPs (high op) regime. However, the Chow baseline is more competitive, since the expert models tend to be highly accurate on all samples. Our posthoe estimators are seen to strongly outperform CSS and OvA, while being competitive with slightly superior to Chow.

---

Accuracy (%)
Chow
Accuracy
70
CSS
50
OvA
Post-hoc Training
65
40
Post-hoc Thresholding
7.0
7.5
FLOPs
1e8
FLOPs
1eB
Figure 6: Results CIFAR- 100 (left), and ImageNet (right) in adaptive inference setting, where computa- tionally cheap base model allowed to defer more expensive expert. Our posthoc schemes offers gains over the existing cost-sensitive softmax (CSS) one- one-versus-all (OvA) most operating points, particularly in the low-FLOPs regime (corresponding to fixed deferral cost cu > 0).
6 Discussion and Future Work
While post-hoc training resolves issues identified with previous joint surrogate losses, is not without limitation: it relies crucially reliable base classifiers from the first phase. As formalised in Theorem 2, any degradation in performance for the latter is also transferred to the final post-hoc predictor. As this may be further compounded with estimation errors the second phase, it natural to ask whether there may be other joint surrogates that do not exhibit underfitting. One candidate the cost-sensitive loss of Lee et al. [17], which Fisher-consistent and is suitable when co > 0. This loss imposes the constraint that Σgenu(L) (s(x) = O, and takes the form
((z, y. f(x)) = -yvey(( [ [ fv) + cxxp(x,)) 3) f1 One may show that
under this loss, the gap between the Bayes-optial scores for the highest scoring label and any competing label is 0 (1 + 4.). and thus is not adversely affected as L increases. As qualifying comment, the loss poses challenges due to being non-differentiable, and requiring constraints on the logits. Nonetheless, further study of the viability of this loss would be worthwhile.
Acknowledgment. We thank Hussein Mozannar for pointing out that the original cost-sensitive softmax cross-entropy loss of Mozannar and Sontag [19] uses slightly tighter formulation than the one we consider, and will not underfit on samples that the expert predicts correctly.
References
[1] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S. Weld. Is the most accurate AI the best teammate? optimizing Al for teamwork. In Thirty- AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 11405-11414, AAAI Press, 2021.
[2] Peter L. Bartlett and Marten H. Wegkamp. Classification with reject option using a hinge loss. Journal of Machine Learning Research,
2008.
[3] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
[4] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for fast test-time prediction. International Conference on Machine Learning, 2017.
[5] Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Technical report, UPenn, 2005.
[6] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings Machine Learning Research, pages 1507-1517. PMLR, 18-24 Jul 2021.
10

---

[7] Yu- TunTing Chou, Gang Niu. Hsuan- Tien Lin, and Masashi Sugiyama. Unbiased risk estimators can mislead: A case study of learning with complementary labels. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1929-1938. PMLR, 2020.
[8] C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1))41148) 1970. doi: 10.1 1..1009/TTT197 1054406.
[9] Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Learning with rejection. In ALT, 2016.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[11] Charles Elkan. The foundations of cost-sensitive learning. In In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 973-978, 2001.
[12] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv:
2017.
[13] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. CoRR, abs/2102.04906, 2021.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IFFE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[15] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinherger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018.
[16] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[17] Yoonkyung Lee, Yi Lin, and Grace Wahba. Multicategory support vector machines. Journal of the American Statistical Association, 99(465):67-81 2004. doi: 10.1 0098/06611404400000000
[18] David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 6150-6160 Red Hook, NY, USA, 2018. Curran Associates Inc.
[19] Hussein Mozannar and David Sontag. Consistent estimators for learning defer to expert. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7076-7087. PMLR, 13-18 Jul 2020.
[20] Harikrishna Narasimhan, Harish Ramaswamy, Aadirupa Saha, and Shivani Agarwal. Consistent multiclass algorithms for complex performance measures. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages PMLR.
Lille, France, 07-09 Jul 2015.
[21] Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2582-2592, 2019.
[22] Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. Advances in Neural Information Processing Systems, 34, 2021.

---

[23] Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Z. Liu, Jasper Snoek, and Balaji Lak- shminarayanan. Revisiting one-vs-all classifiers for predictive uncertainty and out-of- distribution detection in neural networks. CoRR, abs/2007.05134 2020. URL https: //arxiv.org. http//bbs/20007/ 05134.
051114
[24] Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy-efficient and enhanced pattern recognition. In Proceedings of the 2016 Conference Design, Automation & Test in Europe, DATE '1, page 475-480, San Jose, CA, USA, 2016.
EDA Consortium. ISBN 9783981537062.
[25] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: a loss correction approach. In Computer Vision and Puttern Recognition (CVPR), pages 2233-2241, 2017.
[26] Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv: 1903.1220.,0 2019.
[27] Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning, pages 5281-52999 PMLR, 2019.
[28] Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research, 1:2387-24222, 2010.
[29] Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. The right tool for the job: Matching model and instance complexities. In Proc. of ACL. 2020.
[30] Matthew Streeter. Approximation algorithms for cascading prediction models. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4752-4760. PMLR, 10-15 Jul 2018.
[31] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818-2826, 2016.
[32] Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Cancún, Mexico, December 4-8, 2016, pages 2464 2469. IEEE, 2016.
[33] Kirill Trapeznikov and Venkatesh Saligrama. Supervised sequential classification under budget constraints. In Carlos M. Carvalho and Pradeep Ravikumar, editors, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pages 581-589, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR.
[34] Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. arXiv preprint arXiv:2002.03673, 2022.
[35] Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovitz-Atiaa, and Elad Eban. Wisdom committees: An overlooked approach to faster and more accurate models. In International Conference on Learning Representations, 2022.
[36] Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK cascades: Fast deep learning by learning not to overthink. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 580-590. AUAI Press, 2018
[37] Bryan Wilder, Eric Horvitz, and Fee Kamar. Learning to complement humans. CoRR, 2020.
12

---

[38] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20, 2021. ISBN 9780999241165
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] The paper
primarily analytical in nature: it identifies mathematical and empirical limitations of existing methods, and proposes new methods for a problem where a model can query an "expert"s.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including thcoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] The newly proposed methods in this paper involve simple post-hoc operations over existing models. Thus, they do not add significant computational overhead.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're re
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "r8g3I0lnCLX", "reviewer": "Reviewer_3cKz", "summary": "  This paper augments the CSS and OvA loss functions used for learning to defer such that they\n  no longer underfit the base model, particularly in problems with many possible labels. This is\n  done through the introduction of two post-hoc estimators which can used to take a calibrated\n  model where there is zero cost in consulting an expert and adapt it for the setting where\n  there is a positive cost for deferring to an expert. There are both theoretical and experimental\n  results to support the contributions.\n", "questions": "What rejector losses should be preferred for their settings?", "limitations": "The authors do not adequately address the limitations and societal impact of the work. While I don't think there is significant ethical or negative societal risk with this work, the paper would have been strengthened by some discussion of the limitations of the presented methods.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "  Originality:\n\n  The particular algorithms introduced in the paper appear to be novel. Post-hoc procedures such\n  as this one do occur for uncertainty calibration, but this is a novel application of that approach.\n\n  \n  Technical Quality:\n  \n  The technical quality is very good with the experiments well-thought out and providing very compelling\n  results. Since this work is meant to used on top of CSS or OvA it really needed to be compared with\n  those methods for accuracy or loss. The proofs for calibration of the loss function were also clear.\n  Some of the results are left a little open-ended. When calibrating with a rejector loss there are many\n  to choose from and there is not much insight as to why one loss should be preferred over another.\n  \n  Clarity:\n\n  The entire presentation of this paper is written very clearly and easy to follow. Both in the main presentation\n  and in the appendix it was easy to understand what was being done in each experiments and the proofs\n  was easy to follow as well.\n\n  Significance:\n\n  This work is very similar to the loss of Verma and Nalisnick and while it is presented as a generalisation\n  of that work it's not immediately clear that it's a significant contribution by itself. I nevertheless suspect\n  the post-hoc methods introduced in the paper would be useful for others to build on top of.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YqxX4eSubzs", "reviewer": "Reviewer_xmUy", "summary": "This paper tackles the problem of jointly learning a classifier and a rejector where the rejector can choose whether the classifier or the human should predict, this problem is denoted as learning to defer and the authors tackle the problem when the cost of deferring to the expert is a constant $c_0$ plus the misclassification error of the expert. The authors show that previous state-of-the-art approaches fail when $c_0>0$ by making a connection to label smoothing. They then propose two training strategies for the rejector that combine previous surrogate approaches with new machinery. They evaluate their method on synthetic and semi-synthetic data (CIFAR-10 with simulated experts) and show that they outperform previous methods. ", "questions": "Why are Figure 5 and Figure 6 showing performance on accuracy instead of on the objective in equation 1?  The baselines optimize equation 1 instead of human-AI accuracy, and it is not fair to show only results of accuracy, but one must show results on the loss in equation 1. Furthermore, you must show that the approach does better on the loss in equation 1 and on accuracy. \n\nWhat is the performance on Figure 5 of cSS and OvA when you set $c_0=0$ and you just vary the coverage by changing the deferral threshold ?", "limitations": "The authors address the limitations adequately. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The paper has novel analysis of previous surrogate approaches for learning to defer (L2D) when the cost of deferral has an additional penalty $c_0$. The paper proposes a new method to handle this case which leverages previous methods, namely OvA, and adds an extra term in the loss function to form it. This new surrogate has limited novelty and doesn't address the case where $c_0$ is potentially a function of X, a more realistic extension. The related work is very well cited.\n\nQuality: I checked the proofs for the theoretical results and the claims in the papers and they are sound. The empirical results seem reasonable, but no code was provided to verify the results. Furthermore, there are more baselines in the literature that should have been performed. Most notably, the Confidence baseline, where one trains 1) a classifier using cross entropy, 2) a model for expert error, and then 3) defers if classifier error is higher than expert error + $c_0$. One important concern, is that the baselines of Mozannar and Sontag and that of OvA optimize for the cost sensitive loss, while the results show in the experiments  accuracy (without the penalty c_0).  Furthermore, the paper only has experiments with synthetic human experts which is not sufficient to back up claims for human-AI performance.\n\nClarity: Please make section titles and the title have each word start with a uppercase letter. The paper is well written and easy to follow. \n\nSignificance: I think the paper makes an important contribution in showing that previous surrogate approaches may underfit when $c_0>0$. However, one concern is that the paper doesn't show that the method optimizes for the objective in equation 1, but only that the baselines underfit the target. I think if the authors can answer the questions below in the positive, then my overall evaluation may be raised substantially.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-MpvnwbLXl", "reviewer": "Reviewer_sKXi", "summary": "\nThe paper addresses the problem of learning to defer, in which a machine learning model learns when to defer the decision on an instance to an expert based on an estimate of the expert's cost. Existing state-of-the-art solutions are conceived for settings in which the expert cost is simply her probability of error, while real-world settings typically involve a fixed cost for querying the expert. In the latter setting, existing solutions underperform because of an oversmoothing of the prediction probabilities. The proposed solution is a couple of post-hoc strategies that prevent oversmoothing and achieve high performance in the setting involving fixed costs.", "questions": "My main question concerns the suboptimality of two-step learning strategies. Is it possible to say something on whether and how this affects your post-hoc estimator strategies? ", "limitations": "There is not much discussion on the limitations of the proposed solution. My suggestion is to add something on the limitations of a two-step strategy, possibly hinting at future directions (or open problems) for a fully joint learning setting that also prevents oversmoothing.\n\nAFTER REBUTTAL:\nThe authors better highlighted how the suboptimality of the two step procedure is (for threshold correction) or could be (for rejector training) quantified and possibly addressed. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PROs:\n\nThe paper identifies a structural limitation of recent solutions that seriously affects their applicability to real-world scenarios.\n\nThe proposed solution is theoretically grounded and easy to implement.\n\nA careful experimental evaluation confirms the substantial advantage of the proposed post-hoc approaches wrt sota solutions.\n\nCONs:\n\nThe solution is a post-hoc strategy, in which the base model (and the\nexpert cost estimator, in the thresholding case) is trained in a\nsimpler setting (zero fixed cost), and the aggregate model is adjusted\npost-hoc on the desired fixed cost c0. This two-step strategy is\nusually suboptimal, as the model is not optimized for its working\nconditions. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept