Differentiable Learning Under Triage
Authors: Nastaran Okati
Abir De
Manuel Gomez Rodriguez
Venue: NeurIPS 2021 Poster (2021)
Keywords: learning under triage
human-ai collaboration
differentiable learning

Abstract: Multiple lines of evidence suggest that predictive models may benefit from algorithmic triage. Under algorithmic triage, a predictive model does not predict all instances but instead defers some of them to human experts. However, the interplay between the prediction accuracy of the model and the human experts under algorithmic triage is not well understood. In this work, we start by formally characterizing under which circumstances a predictive model may benefit from algorithmic triage. In doing so, we also demonstrate that models trained for full automation may be suboptimal under triage. Then, given any model and desired level of triage, we show that the optimal triage policy is a deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on a per-instance level. Building upon these results, we introduce a practical gradient-based algorithm that is guaranteed to find a sequence of predictive models and triage policies of increasing performance. Experiments on a wide variety of supervised learning tasks using synthetic and real data from two important applications---content moderation and scientific discovery---illustrate our theoretical results and show that the models and triage policies provided by our gradient-based algorithm outperform those provided by several competitive baselines.

Differentiable Learning Under Triage
Nastaran Okati MPI for Software Systems nastaran@mpi swapppi-sss.... org
Abir De IIT Bombay abir@cse. www.iiittt..........
Manuel Gomez-Rodriguez MPI Software Systems hanuelgr@mph org
Abstract
Multiple lines of evidence suggest that predictive models may benefit from algo- rithmic triage. Under algorithmic triage, a predictive model does not predict all instances but instead defers some of them to human experts. However, the interplay between the prediction accuracy the model and the human experts under algorith- mic triage is not well understood. In this work, start by formally characterizing under which circumstances a predictive model may benefit from algorithmic triage. In doing so, also demonstrate that models trained for full automation may be suboptimal under triage. Then, given any model and desired level of triage, we show that the optimal triage policy is deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on per-instance level. Building upon these results, we introduce practical gradient- based algorithm that is guaranteed to find sequence of predictive models and triage policies of increasing performance. Experiments on a wide variety of supervised learning tasks using synthetic and real data from two important applications- content moderation and scientific discovery- iilustrate our theoretical results and show that the models and triage policies provided by our algorithm outperform those provided by several competitive baselines.
Introduction
In recent years, there has been raising interest new learning paradigm which seeks the de- velopment predictive models that operate under diffcrent automation levels-models that take decisions for given fraction of instances and leave the remaining ones human experts. This new paradigm has been so far referred to as learning under algorithmic triage [I) learning under human assistance (2 3 learning to complement humans 4 (5, and learning defer to expert (6]. Here, one does not only has to find a predictive model but also triage policy which determines who predicts each instance.
The motivation that underpins learning under algorithmic triage is the observation that, while there are high-stake tasks where predictive models have matched, even surpassed, the average performance of human experts [8] [9, they are still less accurate than human experts on some instances, where they make far more errors than average [ The main promise is that, by working together, human experts and predictive models are likely to achieve a considerably better performance than each of While the above mentioned work has shown some success at
them would achieve their fulfilling this promise, the interplay between the predictive accuracy of a predictive model and its human counterpart under algorithmic triage
well understood.
One of the main challenges learning under algorithmic triage that, each potential triage policy, there is an optimal predictive model, however, the triage policy is also something one seeks to optimize, as first noted by De et al. [2]. In this context, previous work on learning under algorithmic triage can be naturally differentiated into two lines of work. The first line of work has developed rather general heuristic algorithms that do not enjoy theoretical guarantees [ 4 5). The second has developed algorithms with theoretical guarantees
6), however, they have focused more
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

restrictive settings. More specifically, De et al. 2 focus on ridge regression and support vector machines and reduce the problem to the maximization of approximately submodular functions and Mozannar and Sontag [6] view the problem from the perspective of cost sensitive learning and introduce a convex and consistent surrogate loss of the objective function they consider.
Our contributions. Our starting point is a theoretical investigation of the interplay between the prediction accuracy of supervised learning models and human experts under algorithmic triage. By doing so, we hope to better inform the design of general purpose techniques for training differentiable models under algorithmic triage. Our investigation yields the following insights:
I. To find the optimal triage policy and predictive model, we need to take into account the amount of human expert disagreement, or expert uncertainty, on per-instance level.
II. We identify under which circumstances a predictive model that is optimal under full automation may be suboptimal under a desired level of triage.
III. Given any predictive model and desired level of triage, the optimal triage policy is deterministic threshold rule in which triage decisions are derived deterministically by thresholding the difference between the model and human errors on a per-instance level.
Building on the above insights, we introduce practical gradient- based algorithm that finds sequence of predictive models and triage policies of increasing performance subject constraint on the maximum level of triage. We apply our gradient-based algorithm in a wide variety of supervised learning tasks using both synthetic and real-world data from two important ooplications-conteee moderation and scientific discovery. Our experiments illustrate our theoretical results and show that the models and triage policies provided by our algorithm outperform those provided by several competitive baselines
Further related work. Our work is also related to the areas of learning defer and active learning. In learning to defer, the goal is to design machine learning models that are able to defer predictions [1O]- 18]. Most previous work focuses supervised learning and design classifiers that learn to defer either by considering the defer action as an additional label value or by training an independent classifier to decide about deferred decisions. However, in this line of work, there are human experts who make predictions whenever the classifiers defer them, in contrast with the literature on learning under algorithmic triage [166 they just pay constant cost every time they defer predictions. Moreover, the classifiers are trained to predict the labels of all samples in the training set, as in full automation. In active learning, the goal is to find which subset of samples one should label so that model trained on these samples predicts accurately any sample at test time [19]26]. In contrast, in our work, the trained model only needs predict accurately a fraction of samples picked by the triage policy at test time and rely on human experts to predict the remaining samples.
2 Supervised Learning under Triage
Let X' C R. be the feature domain, )' be the label domain, and assume features and labels are sampled from a ground truth distribution P(x, y) = P(y) x). Moreover, let ő = h(x) be the label predictions provided by a human expert and assume they are sampled from a distribution P(h x). which models the disagreements amongst experts [27] Then, in supervised learning under triage, one needs to find:
(i) triage policy Txx) X' + {0, 1}. which determines who predicts each feature vector-a supervised learning model (T(x) = 0) or a human expert (T(() = 1);
(ii) predictive model m(x) X' + Vr which needs to provide label predictions ÿ = m(x) for those feature vectors x for which π (x)) 0.
Here, similarly as in standard supervised learning, we look for the triage policy and the predictive model that result into the most accurate label predictions by minimizing loss function { v). More formally, let II be the set of all triage policies, then, given a hypothesis class of predictive models. M,
"ur code and data are available httppp///githubbbbbbbbb http://hhhw..ccoototttctc/eeaaaa
Lew.eeeerrnnn--ee under- triage

---

our goal is to solve the following minimization problem
minimize rEll.mEM [( m) subject to [x[m(x)] s
where b given parameter that limits the level of triage, i.e., the percentage of samples human experts need to provide predictions for, and
L m) gx,, [[1 - π (x)) ((m(x)) v) T(x) E(h, 3)1.
Here, one might think of replacing h with its point estimate ym (x) = argmin,,, MAn Bxix le (h, mn)l. where (-) is general loss function. However, the resulting objective would have a bias term, as formalized by the following proposition for the quadratie loss fl (h, pa) (h- =hh)
Proposition 1 Let ( y) = (ύ - y)² and assume there exist x E X for which the distribution of human predictions P(h x) is not point mass. Then, the function
I(π, m.) Ex.y [11 x(x)) El (m (2)xx)))))))))))))))). y) + Tx() El (x)((x)) v)l
a biased estimate of the true average loss defined in Eq.
The above result implies that, to find the optimal triage policy and predictive model, we need to take into account the amount of expert disagreement, or expert uncertainty, on each feature vector x rather than just an average expert prediction.
3 On the Interplay Between Prediction Accuracy and Triage
Let mz be the optimal predictive model under full automation, i.e., mi = argmin L(TO, m), where no (x) 0 for all E X. Then, the following proposition tells that, if the predictions made by mo are less accurate than those by human experts on some instances, the model will always benefit from algorithmic triage:
Proposition 2 If there is a subset V c X of positive measure under P such that
[
Eylx (((mm((), v)] dP / By (e(h, v)l dP,
XEV
X
then there exists a nontrivial triage policy क + To such that mm) < L(TO, mğ)- Moreover, if we rewrite the average loss
L(T,, m) Ex [1 T(x) Eyyx (x)m y)1 x(x) Ey. [e(h, y)]].
it becomes apparent that, for any model E M,, the optimal triage policy x argmin [(x, m) a deterministic threshold rule in which triage decisions are derived by thresholding the difference between the model and human loss per-instance level. Formally, we have the following Theorem:
Theorem Let m M be any fixed predictive model. Then, the optimal triage policy that minimize the loss [(T, m) subject constraint [x[T(x)) < b on the maximum level of triage is given by:
[1
Eyjx [f(m(x),)) 3) Et,x [e(h, f)l] > tp.bmm
(3)
otherwise,
where tp..
argmin
Ex [r b + max (Fyii [((m(xx,)) v) RAjx u)]l - r, 1.
T20
Then, if we plug Eq. into Eq. we can rewrite our minimization problem as:
minimize mEA
L( m)
(4)
where
L(T m)
Ex [Ex) w)] THRES (Eylx [e(m(xx, 8) [njh [/), 00]
(5)
"One might think that minimizing over the set of all possible trage policies not a well-posed problem. če., the optimal triage policy is extremely complex function. However, our theoretical analysis will that optimal triage policy does have simple and complexity depends on the considered hypothesis class of predictive models (refer to Theorem B..
3
All proofs can be found Appendix
3

---

x
with THRES, (I, val)
val otherwise.
Here, note that, in the unconstrained case, f,,b. = O and THRES 0) = max(x,0).
Next, building on the above expression, we prove that the optimal predictive model under full automation ma; suboptimal under algorithmic triage as long as the average gradient across the subset of samples which are assigned to the human under the corresponding optimal triage policy is not zero. Formally, our main result is the following Proposition: T "mag.
Proposition 4 Let maz be the optimal predictive model under full automation within a hypothesis class of parameterized models M(O), Tmag the optimal triage policy for maz defined in Eq. 3 for
a given maximum level of triage and {x Tmogg (x) =13. 1f
[ Bylx (ol(mo(x)) dP #P#00 #0.
(6)
then it holds that [(Tman by mo;) minsee m ma).
Finally, can also identify the circumstances under which any predictive model mer within hypothesis class parameterized predictive models M(O) is suboptimal under algorithmic triage:
Proposition 5 Let ma be predictive model within hypothesis class of parameterized models M(O), TI am b the optimal triage policy for ma defined in Eq. 3 for given maximum level of triage b, and V {x| Tma ss (x) = 03. If
Eyxx (ol(mm(()), dP + O.
XEV
then it holds that [(m
mgr) mingee L( ma).
The above results will lay the foundations for our practical gradient- algorithm for differentiable learning under triage in the next section.
How to Learn Under Triage
In this section, our goal is to find the policy ma- within hypothesis class of parameterized predictive models M(O) that minimizes the loss I( mp) defined in Eq.
To this end, we now introduce general purpose gradient-based algorithm that first approximates ma- given a desirable maximum level of triage b and then approximates the corresponding optimal triage policy Tm . To approximate ma-, the main obstacle we face is that the threshold value f, in the average loss I(TT'n,,> mo) given by Eq. 5 on the predictive model mg which we are trying to learn. To overcome this challenge, we proceed sequentially, starting from the triage policy no. with (x) 0 for all X, and build sequence of triage policies and predictive models ((Ting bb me, TT More specifically, in each step t. we find the parameters of the predictive model me, via stochastic gradient descent (SGD) 28 i.e.
8(-) - 1) a 10-1 Li mg) 0=0(1-1)
01-111 - Lali (1) V I (mmo (x) Ey,iix [e(h, v)] +(1 - n Tim,_ (x)) Eyxx v)]]
0=0 1
= 0(1111) ()--))
T Ti (x)) x Ey x (o E(ma y) -
(8)
where (1) is the learning rate at iteration j. Moreover, the following proposition shows that, under mild conditions, the performance of the triage policies and predictive models improves in each step:
Al Lest time, given predictive model my and an unsenn sample x. we cannol directly evaluate (or, more precisely, estimate using Carlo) the value of the optimal triage policy aing (x). given Eq 3 depends Eyj x+I]] and xI+].

---

Proposition 6 Assume that (mo (x), y) for all x E & and y E y and a < 1/A for all j > 0 for some constant A > If. each step t. we find the parameters of the predictive model ma. using Eq. 8 with 8( = 1.- then, it holds that L(T (Tmm, ma,) < L m
In addition, the following theorem shows that, whenever the loss function {(-) is convex with respect to 8 our algorithm enjoys global convergence guarantee:
Theorem Let E(-) be convex with respect to 9 and the output of the SGD algorithm O argming L(TTE, 1.b' me). Moreover, assume that Aminll Vă {mg (x), y) Amaxi, with
Amin > 0, and f(-) be H- H-Lipschitz, i.c., {(mo(x), y) t(mer (x),)) S H 110 0'l. Then, we have that
2
lim ( bo mo,) L .b ma-) S
(9)
A
b)2
In practice, given a set of samples D {(x,, Mis h4),, we can use the following finite sample Monte-Carlo estimator for the gradient V ( mo):
D
1
Val(f m)) Vo
(mmo(x)):)) - THREStpa.ma (E(mo(xs)) v.) l(hs. vi),
D
max [[1 (1)..))
1
Vol(mo(((a))) V1a)
D
where '[A] denotes the i-th sample in increasing value of the difference between the model and the human loss E(mo._, (xq)), m(a) ((1a) (p)) and p is the number of samples with (xpe) v1a) 2(1p), M[a) < O.
In the above, we do not have explicitly compute the threshold tpp.mme,., nor the triage policy T "mm__,ib (xx) for every sample x; in the set D, we just need to pick the max [[1 (P[1. samples with the lowest value of the model loss minus the human loss {(mo__, (x1)); y(a) - {(h1g M1) using the predictive model mo,- fitted in step 1. To understand why, note that, as long as i P.b.ma..- O. by definition, L t.p.m..,_. needs to satisfy that
d dr
Σ [Tb max({(mo,_n (x)) vs) E(hi, vi) - T, -,00]] VED
0
and this can only happen if {mmo_., (x:)).11 91) E(h y4) t...mma,-1 0 for [b|D] out of [D] samples. Here, we are implicitly estimating the optimal triage policies using the observed training labels and human predictions-ww we are not approximating them using parameterized model-and. and, due to Proposition 6 the implementation of the above procedure with Monte-Carlo estimates is guaranteed to converge to local minimum of the empirical loss. Moreover, note that we can think of the procedure as a particular instance of disciplined parameterized programming [29 30), where the differentiable convex optimization layer is given by the minimization with respect to the triage policy.
While training each of the predictive models mg. we can implicitly compute the optimal triage policy T described above, however, at test time, we cannot do the same since we would need to observe the label and human prediction of each unseen sample x. To overcome this, after training the last machine model may, we also fit model Rt (x) to approximate T . (x) using SGD, i.e.
700== - - (1-1) -(b-11 V
Σ
el (x:) a mmp.. (x:))
(10)
where ( is the learning rate at iteration j and the choice of loss (((-) depends on the model class chosen for n.. In our experiments, we have found that, using the above procedure, we can
NNte that, if the set of samples contains several predictions big by different human experts for each sample Xjsj. we would use all them estimate the (average) human loss.
5

---

Algorithm DIFFERENTIABLE TRIAGE: it returns the weights of a predictive model mh and the weights of triage policy Ky.
Require: Set of training samples maximum level of triage number of time steps T, number of epochs N, mini batches M, batch size B, learning rate a.
I: function TRAINMACHNEENNDERRRUUKCC
M, B.
2: ((0) INITIALIZETHETTO
1,..., do 4: 8, f D,
6: refurn 0r,7
7:
function TRIAGE(D, b, 0)
8:
number of instances in D with ((mo E(h,, y) <
D'
10: for I...., max D). do
11:
D' (i-th sample from 2 increasing value ((ma(x); v) E(h, 8)
12:
return D
13: function TRAINMODEL D. M, O)
14: ((0) 88'
for O...., M do
16:
D(1) + the i'th mini batch of DD
D(1) TRIAGE(D(5)
(()
18:
<-00
19:
for (x, y. h) E D do
20:
V <- V + Vo Elmoo (x), v)lo= g(s+1) < g(s)
21:
B
22:
return ((M)
23:
function APPROXIMATETRRIAGGPOLLCYO D, M, b,b,
24:
7 ((1) INITIALIZEGAMMA ()
25:
for ,,...NN do
26:
7 (0)
+ 7 (M)
27:
for i O,..., - do
28:
D(9) <- the i'th mini batch
29:
V 4-
30:
for (x, ht) E D do
31:
V V V, (m)))
32:
((41) (0) a B
33:
return (M)
approximate well the optimal triage policy Tme, b. However, we would like to note that this problem
can also be viewed as finding an estimator for the a-superlevel set ()) = {x E X' f(x) 2 a} f.p.mm, from set of
of the function (x) = E ( v) E, noisy observations. Under this view, it might be possible derive estimators with error performance bounds building upon recent work on level set estimation BT B2]. which is left as future work. Refer Algorithm [ for pseudocode implementation of the overall gradient-based algorithm, which returns fr and 7 Appendix B provides detailed scalability analysis, which suggests that our algorithm does not significantly increase the computational complexity of vanilla SGD.
[ (((((x), ), y)]] with
Experiments on Synthetic Data
In this section, our goal is to shed light the theoretical results from Section 3 To this end, we use our gradient-baan baase algorithm in a simple regression task which the optimal predictive model under full automation is suboptimal under algorithmic triage]"
All algorithms implemented Python and Nvidia 32GB of memory

---

EjeW--99 g)l - 0.0014 me(a).
Elich vil 0. 0,0026
EROM 90) 0.0094
EO0 v0) 0.0071
2(A.,) ((on())))))
(T))
1.0
1.
1.0
Bespouse
Respostas
0.3
0.5
8L5
0.5
01.01
0.0
0.00
0.01
Foature
Foature
0.08
0.08
E(ma(a)).) 1.)
0.08
E(mu(et).
0.08
E(mafa
8ues
0.02
0.02
0..00
0.03
0.02
0.10
0.00
0.01
0,02
0.00
0,000
0.01
0.03
0.500
0,02
0.02
((h.9)
2(h, 8)
((h.a))
0(h.3)
No triage
Triage policy x mmgggh
Triage policy miag
Triage policy mmy
(a) Predictive model me, trained under full automation
(b) Predictive model my trained under triage
Figure Interplay between the per-instance accuracy of predictive models and experts under different triage policies. In both panels, the first row shows the training samples (I, y) along with the predictions made by the models mxx) and the triage policy values π (I) and the second row shows the predictive model loss (z), 3) against the human expert loss (m)(), y) per-instance level. Columns correspond to the settings (I-4) from left to right. The triage policy Tmeg b is optimal for the predictive model me, and the triage policy Tm is optimal for the predictive model mg. Each point corresponds to one instance and, for each instance, the color indicates the amount of noise in the predictions by experts, as given by Eq. [1] and the tone indicates the triage policy value. In all panels, we used L(g. v) (g - y)² and the class of predictive models parameterized by sigmoid functions, i.e., ma So (z).
Experimental setup. We generate [D] = 72 samples, where we first draw the features x E R uniformly at random, i.e., 3|-3, 3, 3). and then obtain the response variables y using two different sigmoid functions (1) = 1+exp(- (app((0r) ozj More specifically, we set = (zr) if x E [-3, - 1.5) U [o, 1.5) and y Sz) if E 1-1.5, U [1.5,3] Moreover, assume human experts provide noisy predictions of the response variables, i.e., h(x)) = + c(x), where c(z) ~ N/(0, (f)) with
8 10-3
if
1-3,-1133 if x E 1.5.0) if x E 1.5) if x [15,,3]
1 x 10-3
xx
(1I)
4 10-3
2 10-3
In the above, we are using heteroscedastic noise motivated by multiple lines of evidence that suggest that human experts performance on per instance level spans wide range [ 2 27]. Then, we consider the hypothesis class of predictive models M(O) parameterized by sigmoid functions, ie. m=(x) (z). and utilize the sum of squared errors the predictions as loss function, i.e., E(D. 8) = (ū- u) to train the models and triage policies in the following four settings:
1. Predictive model trained under full automation m&o without algorithmic triage, i.e., b(x) = (x) = 0 for all x E X.
2. Predictive model trained under full automation ma, with optimal algorithmic triage T
3. Predictive model trained under algorithmic triage my, with b = with suboptimal algorithmic triage T 6. Here, we use the triage policy that is optimal for the predictive model trained under full automation.
4. Predictive model trained under algorithmic triage mg, with b = 1, with optimal algorithmic triage T "mo.b.
In all the cases, we train the predictive models ma, and ma using our method with b = 0 and i = 1, respectively. Finally, we investigate the interplay between the accuracy of the above predictive models and the human experts and the structure of the triage policies at per-instance level.
7

---

2.00
20.52
0.66
9.6
59,50
0.11
Hatespeech
Galaxy 200
Hatespeech
Galaxy 200
(a) Training losses by the predictive models mie
(b) Training losses by the triage policies #7,00)
Figure 2: Average training losses achieved by the predictive models me, and the triage policies t on the Hatespeech and Galaxy zoo datasets during training. In Panel (a), each predictive model mg, is the output of MRDEN)) at step and in Panel (b), each triage policy *,,0 is the output of TRA TRAINTRIAGE() at epoch i. Both functions are defined in Algorithm Error bars correspond to plus and minus one standard error.
Figure I shows the training samples (z, y) along with the predictions made by the predictive models mø. and my and the values the triage policies To (z),,, x Time & and T m as well as the losses achieved by the models and triage policies (1-4) on a per-instance level. The results provide several interesting insights.
Results.
Since the predictive model trained under full automation mé, seeks to generalized well across the entire feature space, the loss achieves on per-instance level is never too high, but neither too low, as shown in the left column of Panel (a). As consequence, this model under no triage achieves the highest average loss among all alternatives, [(TO, ma.) 0.0053 (setting 1). This may not come as surprise since the mapping between feature and response variables does not lie within the hypothesis class of predictive models used during training. However, since the predictions by human experts are more accurate than those provided by the above model in some regions of the feature space, we can deploy the model with the optimal triage policy T given by Theorem 3 and lower the average loss to L b. mg.) = 0.0020 (setting 2), as shown in the right column of Panel (a) and suggested by Proposition
In contrast with the predictive model trained under full automation ma, the predictive model trained under triage mg learns to predict very accurately the instances that lie in the regions of the feature space colored in green and yellow but it gives up on the regions colored red and blue, where its predictions incur very high loss, shown in Panel (b). However, these latter instances where the loss would have been the highest if the predictive model had to predict their response variables y are those that the optimal triage policy Ti hand in human experts make predictions. As result, this predictive model under the optimal triage policy does achieve the lowest average loss among all alternatives, L( b. ma) = 0.0009 (setting 4), as suggested by Propositions and
Finally, our results also show that deploying the predictive model my under suboptimal triage policy may actually lead to a higher loss L(T), m)) 0.0031 (setting 3) than the loss achieved by the predictive model trained under full automation me, with its optimal triage policy T This happens because the predictive model mg is trained to work well only on the instances æ such that Tmo & (z) 0 and not necessarily on those with n ( O.
6 Experiments on Real Data
In this section, we use our gradient- based algorithm in two binary and multi-class classification tasks in content moderation and scientific discovery, We first investigate the interplay between the accuracy of the predictive models and human experts and the structure of the optimal triage policies different steps of the training process. Then, we compare the performance of our algorithm with several competitive baselines.
Experimental setup. We use two publicly available datasets content moderation and the other for scientific discovery
33
34
one from an application in
We chose these two particular applications because the corresponding datasets among the only publicly available datasets that we found containing multiple human predictions per instance, necessary to estimate the human loss al an instance level. and relatively large publicly available under
number of instances. The Hatespoech dataset publicly available under license Galaxy

---

((mg(a))))) ((h.))
(imu(z).n)
(imz(e).0)
((h.a)
((h.u)
E(h,
Conelshaa)
2
E(h,
E(h,
E(K,
(a) Step 1
(b) Step 2
(c) Step f 3
Figure 3: Predictive model and expert losses at per-instance level randomly selected subset of 500 samples of the Hatespeech (top row) and Galaxy zoo (bottom row) datasets throughout the execution of our method during training. The maximum level of triage is to b 0.4 for Hatespeech and b = 1.0 for Galaxy zoo dataset. Each point corresponds to an individual instance and, for each instance, the color pattern indicates the triage policy value.
Hatespeech: It consists of [D] = 24,783 tweets containing lexicons used in hate speech. Each tweet is labeled by three to five human experts from Crowdflower as "hate-speech", "offensive", or "neither".
Galaxy 200: It consists of [D] = 10,000 galaxy images Each image is labeled by 30+ human experts as "early type" or "spiral".
For each tweet in the Hatespeech dataset, we first generate 100 dimensional feature vector using fasttext (35) as similarly as in De et al. (2). For each image in the Galaxy zoo dataset, we use its corresponding pixel mag") as x. Given an instance with feature value x, we estimate P(h x) = mx(h) Enrey (h)) where (h) denotes the number of human experts who predicted label h and we set its true label to y = argmax P(h| x). Moreover, at test time, for each instance that the triage policy assigns humans, we sample h ~ P(h] xx.
In all our experiments, we consider the hypothesis class of probabilistic predictive models MM()) parameterized by softmax distributions, i.e.,
mo(x) P0:x Multinomial (exp (cy. (dy.
where, for the Hatespeech dataset, 6.,0 is the convolutional neural network (CNN) by Kim 33 and, for the Galaxy zoo dataset, it is the deep residual network by He et al. [37] During training. we use a
cross entropy loss the observed labels, L.e., ((j. v) log P y x). Here, if an instance is assigned to the predictive model, we have that
exp (
P(Ù = y/x) x) = Ey exp (0y' (xx))
and. if an instance is assigned to a human expert, we have that P //x) P(h = y|x). For the function ft, (x), we use the class of logistic functions, i.e., (x) = T4expl (x))' where o, is the same CNN and deep residual network as in the predictive model, respectively. Here, we also use the cross entropy loss, i.e., 5' (x),,, Tmoe (xx)) = ग Tma.. (x) log (x) - (1 - ग mey . (x)) log(1 nt (x)). ). In each experiment, we used 60% samples for training, 20% for validation and 20% for testing. Refer to Appendix for additional details on the experimental setup.
Results. First, we look at the average loss achieved by the predictive models me, and triage policies 7,(0) throughout the execution of our method during training. Figure summarizes the results, which
Creative Commons AtrributinnNoneea Noncommercial.X No Derivative Works license. Since our algorithm general, may useful in other applications in which predictive models trained under full automation performs worse than humans in some instances (scc Proposition
The original Galaxy zoo datasel comsists of 61,577 images, however, we report results on randomly chosen subset 10,000 images dhe to scalability reasons. We found similar results in other random subsets.
The pixel maps each image available httpps: //ww::///wwwwww// kaggle con/c/ cow////alaeeeee http///aagy/oo////ee sootyy---oot--eeee cete--gexyy-cchllle

---

Differentiable Triage (ours)
Comfilenna-lwwed Triagr Surmusto-hoad Tringe
Strew-lseed Tringe
Poll. Automation Triage
0.20-
0.25
0.15
0.00
0.17
0.10
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) Hatespeech
(b) Galaxy Z0O
Figure 4: Misclassification test error P + y) against the triage level b on the Hatespeech and Galaxy 200 datasets for our algorithm, confidence-based triage [5), score-based triage M. surrogate- bbaed triage [6] and full automation triage. Appendix C contains more details on the baselines.
reveal several insights. For small values of the triage level, the models me, aim to generalize well across a large portion of the feature space. As a result, they incur a large training loss, as shown in Panel (a). In contrast, for b 2 0.4, the models ma are trained to generalize across smaller region of the feature space, which leads to a considerably smaller training loss. However, for such a high triage level, the overall performance of method is also contingent on how well Ωt approximates the optimal triage policy. Fortunately, Panel (b) shows that, as epochs increase, the average training loss of f, decreases. Appendix D validates further the trained approximate triage policies ft..
Next, we compare the predictive model and the human expert losses per training instance throughout the execution of the our method during training. Figure 3 summarizes the results. At each step I, we find that the optimal triage policies F hands in to human experts those instances (in orange) where the loss would have been the highest if the predictive model had to predict their response variables y. Moreover, at the beginning of the training process (i.e., low step values l), since the predictive model mg, seeks to generalize across a large portion of the feature space, the model loss remains similar across the feature space. However, later into the training process (i.e., high step values t), the predictive models me, focuses on predicting more accurately the samples that the triage policy hands in to the model, achieving a lower loss on those samples, and gives up on the remaining samples, where it achieves a high loss.
Finally, compare the performance of method against four baselines in terms of test mis- classification error. Refer to Appendix for more details on the baselines, which we refer to as confidence-basete triage (5), score-based triage (I], surrogate-based triage [6] and full automation triage [T0) Figure summarizes the results. We find that, in each dataset, our algorithm is able to find the predictive model and the triage policy with the lowest misclassification across the entire span of b values. One could view these values of maximum triage levels as the optimal automation levels (within the level values we experimented with).
7 Conclusions
In this paper, we have contributed towards a better understanding of supervised learning under algorithmic triage. We have first identified under which circumstances predictive models may benefit from algorithmic triage, including those trained for full automation. Then, given a predictive model and desired level of triage, we have shown that the optimal triage policy is deterministic threshold rule which triage decisions are derived deterministically from the model and human per-instance errors. Finally, we have introduced practical algorithm to train supervised learning models under triage and have shown that outperforms several competitive baselines.
Our work also opens many interesting venues for future work. For example, we have assumed that each instance is predicted by either a predictive model or a human expert. However, there may be many situations in which human experts predict all instances but their predictions are informed by a predictive model [38]. We have shown that our algorithm is guaranteed to converge to a local minimum of the empirical risk, however, it would be interesting to analyze the convergence rate and the generalization error. Finally, would be valuable to assess the performance of supervised learning models under algorithmic triage using interventional experiments on a real-worke application.
10, Among all the buselines we are aware of [1 we did not compure with De el al. 2 because they only allow lineur models and SVMs and did compare with Wilder et al. f
bocause they not provide enough details to implement their method.
10

---

Acknowledgment. Okati and Komez-Rodrigee acknowledge support from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 945719). De has been partially supported by DST Inspire Faculty Award.
References
[1] Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv: 12220, 2019.
[2] Abir De, Paramita Koley, Niloy Ganguly, and Manuel Gomez-Rodriguez. Regression under human assistance. In AAAI, 2020.
[3] Abir De, Nastaran Okati, Ali Zarezadeh, and Manuel Gomez-Rodriguez. Classification under human assistance. In AAAI, 2021.
[4] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In IJCAI, 2020.
[5] Gagan Bansal, Besmira Nushi, Eee Kamar, Eric Horvitz, and Daniel S. Weld. Optimizing AI for Teamwork. In AAAI, 2021.
[6] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to expert. In ICML, 2020.
[7] Justin Cheng, Cristian Panescu-Niculescu-Mizil and Jure Leskovec. Antisocial behavior in online discussion communities. In ICWSM, 2015.
[8] Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection.
In OOPSLA, 2018
[9] Eric Topol. high-performance medicine: the convergence of human and artificial intelligence. Nature medicine, ):44):44, 2019.
[10] David Madras, Toniann Pitassi, and Richard S. Zemel. Predict responsibly: Improving faimess and accuracy by learning defer. NeurlPS, 2018.
[11] Peter Bartlett and Marten Wegkamp. Classification with reject option using a hinge loss. JMLR, 9(ug): 1823)1808 2008.
[12] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In ALT, 2016.
[13] Yonatan Geifman, Guy Uziel, and Ran Yaniv.... Bias-reduced uncertainty estimation for deep
neural classifiers. 2019.
[14] Harish Ramaswamy, Ambuj Tewari, and Shivani Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic of Statistics, 12(1):530-554, 2018
[15] Yonatan Geifman and Ran El- Elniv. Selectivenet: A deep neural network with an integrated reject option. In ICML, 2019.
[16] Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis Chiis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. In NeurIPS, 2019.
[17] Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd-Yusof. Combating label noise in deep learning using abstention. arXiv preprint arXiv:1905. 10964. 2019.
[18] Liu Ziyin, Blair Chen, Ru Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Learning not to learn in the presence of noisy labels. arXiv preprint arXiv:2002.06541 2020.
[19] David Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. In NeurlPS, 1995.
[20] Steven Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu. Batch mode active learning and its application to medical image classification. In ICMI., 2006.
[21] Masashi Sugiyama. Active learning in approximately linear regression based on conditional expectation of generalization error. JMLR, 7(Ja))1141166 2006.
[22] Rehecca Willett, Robert Nowak, and Rui M Castro. Faster rates regression via active learning. In NewrIPS, 2006.

---

[23] Yuhong Guo and Dale Schuurmans, Discriminative batch mode active learning. In NeurIPS,
2008.
[24] Sivan Sabato and Remi Munos. Active regression by stratification. In NeurIPS, 2014.
[25] Xue Chen and Eric Price. Active regression via linear-sample sparsification. In COLT, 2017.
[26] Abolfazl Hashemi, Mahsa Ghasemi, Haris Vikalo, and Ufuk Topcu. Submodular observation selection and information gathering for quadratic models. In JCML, 2019.
[27] Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mul- lainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. In
ICML, 2019.
[28] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[29] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as layer in neural networks. In ICML, 2017.
[30] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. In NeurIPS, 2019.
[31] Rebecca M Willett and Robert D Nowak. Minimax optimal level-set estimation. IEEE Transac- tions on Image Processing, 16(12):2965-2999 29999, 2007.
[32] Aarti Singh. Nonparametric Ser Estimation Problems in Statistical Inference and Learning. PhD thesis, University of Musconsin-Madison 2008.
[33] Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. ICWSM, pages 512-515.
[34] Steven P Bamford, Robert C Nichol, Ivan K Baldry, Kate Land, Chris J Lintott, Kevin Schaw- inski, Anže Slosar, Alexander S Szalay, Daniel Thomas, Mehri Torki, et al. Galaxy zoo: the dependence of morphology and colour on environment. Monthly Notices of the Royal Astronomical Sociery, 393(4): 2009.
[35] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint 1rXiv:1612.03300 2016.
[36] Yoon Kim. Convolutional neural networks for sentence classification, 2014.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[38] Brian Lubars and Chenhao Tan. Ask not what ai can do, but what should do: Towards framework of task delegability. 2020.
[39] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[40] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
12


Reviews:
[{"review_id": "woEvf60Ns-", "reviewer": "Reviewer_Z9up", "summary": "This paper considers the setting of \"algorithmic triage\", where a predictive model can choose to route some data points to a human expert for manual review.  The authors formally define the settings under which triage is useful, and show a motivating result that training a model for non-triage settings may be sub-optimal for learning under triage.  They then derive the optimal triage policy as a deterministic threshold rule, show a gradient-based algorithm for this, and demonstrate strong empirical performance on several synthetic and real datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an approach to algorithmic triage where the goal is to learn both (i) a predictive model (ie the standard classifier) and *also* (ii) a triage policy model which determines whether the predictive model or a human expert should label a given instance, by optimizing the overall loss subject to some limit on human annotator time.  The paper:\n- (i) Formalizes the notion that expert annotator uncertainty, as a function of the data, needs to be taken into account to optimize the loss in Prop. 1\n- (ii) Establishes formally that a model worse than human experts on some instances will always benefit from an optimal triage policy (which seems like a trivial statement...?)\n- (iii) Establishes that the optimal model without triage is *not* the optimal model under triage- a very interesting result and great motivation for this line of work!\n- (iv) Proposes an iterative gradient/MC-based algorithm for estimating the model and triage policy pair, which they show in Prop. 6 is locally optimal under mild conditions\n- (v) Validates the approach on a range of synthetic and empirical experiments\n\nOverall this is a strong and thoroughly written paper that motivates the setting well, proposes a sensible algorithmic approach, and shows strong empirical results.  Weaknesses include the fact that there is only local convergence shown formally for the actual proposed algorithm; the experimental setup could be better described, especially the description of exact implementation of baselines; and more ablations exposing limitations of the proposed method would have been very helpful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "wPtDitelmdL", "reviewer": "Reviewer_E3Mu", "summary": "This paper studies the problem of algorithmic triage, where a predictive model and human experts together generate predictions. The authors introduced a theoretical analysis to model the interplay between the prediction accuracy of the model and the human experts. The authors later introduced a gradient based algorithm to find a sequence of predictive models and triage politics of increasing performance.  Extensive experiments on both synthetic dataset and two real world datasets showed improvement of proposed method over baseline methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies a very interesting problem, where human experts can work together with predictive models to generate predictions. Relevant but slightly different from active learning, which tries to generate better predictive models with humans in the loop, the algorithmic triage problem targets at minimizing the overall loss with constraint to cost on human experts.\n \nHowever, it might still be good to include more discussions on how active learning based methods would perform in this setup. \n\nThe paper is well written and easy to follow. The authors also provided theoretical analysis and support for their proposed method, which seems to be technically sound and efficient in providing policy and predictive models given cost constraints. \n\nThe author conducted experiments on synthetic datasets to show how the proposed method works and then showed the improvement on two real world datasets. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "jDoqqUfxsA", "reviewer": "Reviewer_nQQC", "summary": "This paper presents a method triaging  / deferring examples to human labellers in machine learning systems. Their approach is conceptually simple: begin with a model trained on the full dataset and then iteratively improve the triaged model by updating it on a subset of the data that excludes examples where the previous iteration was worst compared to the human labellers. The approach is justified through a series of theorems showing that, as long as there exist parts of the input space for which the human labelers outperform the model, then triaging offers the possibility of improving the (combined model & human) performance; and further, that gradient descent can improve the model on the restricted space implied by the thresholding approach.\n\nThis is not my area, but I generally found the paper to be clear and well written (aside from style issues discussed below). ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[Building a sequence of triage policies]** I understand that we can sequentially improve the combined policy of the models an humans up to our budget by retraining after excluding the subset of examples that the previous iteration performed worst on, but, assuming I know $b$, it wasn't clear whether that sequence is of length two (i.e. train and unconstrained model and then retrain with the subset implied by $\\max(\\lceil(1−b) |D|\\rceil,p)$), or whether I sequentially retrain with a sequence of $b_i \\leq b$? And if so, how do I choose that sequence? \n\n**[Evaluation]** I really would have liked to see more than two datasets evaluated. This is a generic method that seems to apply to any dataset, so aside from compute requirements, there doesn't seem to be anything preventing an evaluation that check whether the observed improvements over prior approaches hold across a statistically significant number of datasets?\n\n**[\"Mathiness\" style issues]** While the prose is generally well-written, the presentation of the propositions in section 3 (and to a lesser extent in the other sections), runs dangerously close to what Lipton & Steinhardt [2018] call 'mathiness' to \"bulldoze rather than to clarify\". I don't think that the authors intended this---and these comments haven't influenced my review score---but the paper could be a lot clearer if the propositions included a sentence or two on why each statement is true. For example, Prop 4 is prefaced with, \"we can *identify the circumstances* under which the optimal predictive model... is suboptimal under algorithmic triage. Formally, *our main result* is the following Proposition...\" (my emphasis). That sounds mysterious - when the \"circumstances\" are simple - as long as the thresholded gradients don't sum to zero, the model is suboptimal under triage. Making it clear that you're mostly just using the fact that loss and gradients on a subspace are typically not the same as those on the whole space would make the whole presentation easier to follow. \n\nTo be clear - I'm very happy for theorems to formally delineate an argument, but as far as possible, we should use them to tell the reader why something is true, not just what is true.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are relatively small issues, but I would have liked to see some discussion on them:\n\n[Interpolation regimes] Modern deep networks are often massively over-parameterized such that they can interpolate the training data (driving the loss essentially to zero on each training example); essentially memorizing training examples, while still generalizing. It seems like the method would fail in this setting because you wouldn't threshold anything? Can the approach be generalized to select which examples to triage on a validation set to avoid this issue?\n\n[Non-convexity] It is not clear how the sequential approach that is presented here interacts with non-convexity? Is there a concern that starting the optimal model on the full dataset and then thresholding can lead you to a worse solution than you would have found if you could optimize for the thresholded loss directly? I see this isn't an issue in the convex setting, but I'm curious whether you've thought about it in the non-convex setting?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XoJvN4YXiCf", "reviewer": "Reviewer_3WRj", "summary": "The paper proposes a methodology for learning models when some of the prediction tasks are deferred to human experts (i.e., triage). \nIn the paper, the authors: (i) first derive the optimal level of triage for a given a model; (ii) then, they provide conditions under which models learnt under full automation will achieve lower predictive performance in presence of triage; (iii) then, they propose an iterative procedure to learn model and optimal level of triage jointly; (iv) lastly, they empirically verify the proposed algorith via a simulation and two case studies of real-world datasets. \nThe key contributions of this paper are: (1) theorem 3, i.e., the analytical derivation of the optimal triage level; (2) the proposed methodology to learn the model under triage. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the paper to be well written, except for some typos and lack of clarity in the figures (see below). In particular, I found the introduction of the framework to be very clear (initial part of section 2). \nThe results were interesting--especially theorem 3--and the derivation of most of them was fairly simple, which is a plus. \nThe results also seem to be novel, but I am not fully aware of all works in this part of the literature and in particular I only have passing knowledge of [6]. \n\nOne minor concern: I am confused by the definition of \\mu_h(x) in line 90-91: One may be interested in the majority prediction (in case of binary classification) or in the average prediction (in case of regression), so I do not fully understand why \\mu_h is defined as the argmin of the same loss (with respect to h) that appears in (2). In addition, the proof of proposition 1 seems to rely exactly on \\mu_h(x)=E[h(x)], which I think it may not correspond to the \\mu_h as defined in lines 90-91 (e.g., it does for the squared loss but may not for other types). That said, I found the result in this proposition to be interesting. \n\nOther minor issues and suggestions:\n* line 28: reference [7] is not about models having higher predictive performance than humans\n* lines 34-36: the sentence seems slightly confusing\n* line 39: algorithm*s*\n* when the paper is the subject of the sentence, sometimes it is taken as singular and other times as a plural noun (line 40 vs. 42)\n* another perhaps useful reference for the learning to defer literature is [Madras et at., 2017]\n* line 107: it become*s*\n* line 147: the loss should decrease, not increase (the inequality in the proof seems to be in the correct direction)\n* figure 1: the caption is fairly clear (if one also carefully reads the main text), but the legend is not. For example, do the four columns correspond to models 1-4? In particular, I found confusing the fact that in this figure the color refers the noise whereas in figure 3 it refers to the triage level. There is also a similar issue for the tone\n* line 210: there is no panel (a) in figure 1\n\nMadras, David, Toniann Pitassi, and Richard Zemel. \"Predict responsibly: improving fairness and accuracy by learning to defer.\" arXiv preprint arXiv:1711.06664 (2017).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
- Sensible algorithmic approach
- Sound theoretical analysis for setup and algorithm
- Thorough experiments on both synthetic and real data

Weaknesses:
- Clarity could be improved, over-usage of math and notation
- In experiments - better describing baselines and adding ablation studies


Summary:

Reviewers are in agreement that this is a good paper and should be accepted. Most concerns regarded particular inclarities, but discussions were helpful in this respect. The authors also extended their local convergence result to a global results under some assumptions, which addressed a concern raised by one of the reviewers. 

Decision: Accept (Poster)