Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints
Authors: Jean Vieira Alves
Diogo Leitão
Sérgio Jesus
Marco O. P. Sampaio
Javier Liébana
Pedro Saleiro
Mario A. T. Figueiredo
Pedro Bizarro
Venue: Accepted by TMLR ()

Abstract: Learning to defer (L2D) aims to improve human-AI collaboration systems by learning how to defer decisions to humans when they are more likely to be correct than an ML classifier. Existing research in L2D overlooks key real-world aspects that impede its practical adoption, namely: i) neglecting cost-sensitive scenarios, where type I and type II errors have different costs; ii) requiring concurrent human predictions for every instance of the training dataset; and iii) not dealing with human work-capacity constraints. To address these issues, we propose the \textit{deferral under cost and capacity constraints framework} (DeCCaF). DeCCaF is a novel L2D approach, employing supervised learning to model the probability of human error under less restrictive data requirements (only one expert prediction per instance) and using constraint programming to globally minimize the error cost, subject to workload limitations. We test DeCCaF in a series of cost-sensitive fraud detection scenarios with different teams of 9 synthetic fraud analysts, with individual work-capacity constraints. The results demonstrate that our approach performs significantly better than the baselines in a wide array of scenarios, achieving an average $8.4\%$ reduction in the misclassification cost. The code used for the experiments is available at https://github.com/feedzai/deccaf

Published in Transactions on Machine Learning Research (07/2024)
Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints
Jean V. Alves Feedzai
jean.alee@feeddza..co
Diogo Leitão
Peedzai
Sérgio Jesus
ffniio.jesus@@edzzaa..c com
Peedzai
Marco O.P Sampaio Peedaai
Javier Liébana
hiaieriieehno@@fedaaa.coo com
Feedzai
Pedro Saleiro
pedro..aleirro@foeddaa..
Peedzai
Mário A. Figueiredo
mavio.fiuueiveaoooSttcniioo wwisseoo
Instituto Superior Técnico, UTishon Instituto de Telecomunicaçées
Pedro Bizarro
Battobbiarroofeedzaat com
Peedsai
Reviewed on OpenReview: https: //
// openreview. net/ forum? id= TAuGZm2Rqb
Abstract
Learning to defer (T.2D) aims to improve human-AT collaboration systems by learning how to defer decisions to humans when they more likely to be correct than ML classifier. Existing research in L2D overlooks key real-world aspects that impede its practical adoption, namely: i) neglecting cost-sensitive scenarios, where type I and type II errors have different costs; ii) requiring concurrent human predictions for every instance of the training datasot; and iii) not dealing with human work-capacity constraints. propose the deferral under cost and capacity construints framework (DeCCaF). DeCCaF is novel L2D approach, employing supervised learning to model the probability of human error under less restrictive data requirements (only one expert prediction per instance) and using constraint. programming to globally minimize the error cost, subject to workload limitations. We test DeCCaF in a series of cost-sensitive fraud detection scenarios with different teams 9 synthetic frand analysts, with individual work-capacity constraints. The results demonstrate that our approach performs significantly better than the baselines wide array of scenarios, achieving an average 8.4% reduction in the misclassification cost. The code used for the experiments is available at https:/ // htt....com/eedeza///ccaaaaa
address these issues, we

---

Published in Transactions on Machine Learning Research (07/2024)
DeCCaF
Class Probebliity
Human Expert Team
(1) ML Classifier
Probabilities of each
Decisioning Queue
Capacity Constraints
expert being correct
(3) Assigner
(1) ML Classifier
(2) Human Expertise Model
Figure 1: Schematic Representation of DeCCaF
1 Introduction
An increasing body of recent research has been dedicated to human-Al collaboration (HAIC), with several authors arguing that humans have complementary sets of strengths and weaknesses to those of AI (De- Arteaga al.) 2020; Dellermann et al.] 2019). Collaborative systems have demonstrated that humans are able to rectify model predictions in specific instances (De-Arteaga et al. 2020), and have shown that humans collaborating with ML model may achieve synergistic performance: a higher performance than humans or models alone (Inkpen et al., 2022). In high-stakes scenarios where MI models can outperform humans, such as healthcare (Gulshan et al.j 2016), HAIC systems can help address safety concerns (e-g., the effect of changes in the data distribution GGama et al.) 2014)), by ensuring the involvement of humans in the decision-making process.
The stale-of-the-art framework to manage assignments in HAIC is learning to defer (L22D) (Charusaie al. Mozannar et al. 2023
2022 Hemmer al. 2022 Raghu et al. 2019b Mozannar & Sontag 20206
Madras et al. 2018b Steege 2023 Verma & Nalisnick 2022a Verma et al. 2023 L2D aims to improve
upon previous approaches, such as rejection learning [Chow] 1970 Cortes et al. 2016), which defer based solely on the MT. model's confidence, by also estimating the human confidence in given prediction and
passing the instance to the decision-maker who most likely to make the correct decision.
Previous work in L2D does not address several key aspects of collaborative systems. In real-world scenarios, multiple human experts employed to carry out the classification task, the volume of instances to process cannot be handled by single human. However, only small subset of L2D research focuses the multi-expert setling, where decisions have to be distributed Uhroughout leam comprised of single ML classifier and multiple human experts (Keswani et al,, 2021 Hemmer et al.) 2022 Verma et al. 2023). To the best of our knowledge, Verma et al. (2023) propose the only two consistent multi-expert L2D formulations, with both approaches assuming the existence of every expert's predictions for all training instances.
real-world applications, we often have more limited data availability, with only a single expert (De-Arteaga) let al., 2020)) or small subset of the experts (Gulshan et al., 2016) providing predictions for each instance. This means that practitioner aiming to train L2D algorithms would have to purposefully collect the set of every expert's predictions for sample of instances, which could incur large costs or even be unfeasible. We will propose L2D) architectures that allow for assigners to be trained assuming that each instance is accompanied by the prediction of only one expert out of the team.
Current multi-expert T.2D methods also neglect human capacity limitations. Should there be an expert that consistently outperforms the model and all the other experts, the optimal assignment would be to defer all cases to that expert, which in practice is not feasible. Furthermore, current work neglects cost-sensitive scenarios, where the cost of misclassiffcation can be class even restance-dependens (e.g., in medicine, false alarms are generally considered less harmful than failing to diagnose discase).
To address the aforementioned L2D limitations, we propose the deferral under cost and capacily constraints framework (DeCCaF): a novel deferral approach to manage assignments in cost-sensitive human-Al decision- making, which respects human capacity constraints. Our method comprised of three separate components, represented schematically in Figure ! (1) an ML classifier modelling the probability of the target class given the instance features; (2) human expertise model (HEM) that models the probabilities of ness of
2

---

Published in Transactions on Machine Learning Research (07/2024)
each of the experts in the team; and (3) an assigner that computes the best possible set of assignments given misclassification costs and capacity raists
Due to the lack of sizeable datasets with multiple human predictions and to the high costs associated with producing one, we empirically validate DeCCaF in a series realistic cost-sensitive fraud detection scenarios, where a team of synthetic fraud analysts and one MI. classifier are tasked with reviewing financial frand alerts, We conclude that, across all scenarios, DeCCal' performs similarly to, or significantly better than L2D baselines, measured by average misclassification costs.
In summary, our contributions are:
DeCCal:: novel L2D method that models human behavior under limited data availability, using constraint programming to obtain the optimal set of assignments (Section 3
A novel benchmark of complex, feature-dependent synthetic expert decisions, in realistic financial fraud dctection sccnario (Section 4.1
Experimental evidence that our approach outperforms baselines in set of realistic, cost-sensitive fraud detection scenarios (Section 5/.
In Section 2 we describe the relevant related work, focusing recent developments in multi-expert L2D and examining the shortcomings of the deferral systems proposed so far. We also discuss current practices in L,2D evaluation, particularly the use of synthetically generated expert predictions, and how they can be improved. We then describe DeCCaF in Section 3 first by formulating a novel training method, compatible with limited data availability,, for the commonly used classifier-rejector framework (Mozannar & Sontay, 2020a; Verma & Nalisnick] 2022a] Verma al.) 2023 Cortes et al.) 2016), demonstrating that it produces optimal unconstrained deferral. As the traditional classifier-rejector approach fails to consider the existence of human work-capacity limitations,
propose a novel formulation for global assignment optimization under capacity constraints. In Section | we detail a realistic fraud detection experimental setup as well as the method for the synthetic expert prediction generation. We also discuss the capacity- aware baselines used in our L2D benchmark. The experimental results are reported and analyzed in Section 5
2
Related Work
2.1 Current L2D Methods
The simplest deferral approach in the literature is rejection learning (RcL), which dates back to the work of Chow (1970): Cortes el al.] (2016). In a HAIC setting, ReL defers to humans the instances that the model rejects predict (Madras et al. 20188b Raghu et al.] 2019a). A simple example (Hendrycks & Gimpel. 2017) is to obtain uncertainty estimates of the model prediction for cach instance, rejecting to predict if the uncertainty estimate is above a given threshold.
Madras et al. 2018b) criticize ReL, arguing that it does not consider the performance of the human involved in the task and propose learning to defer (L2D), where the classifier and assignment system are jointly trained, taking into account a single model and a single human and accounting for human error in the training loss. Many authors have since contributed to the single-cxpert framework: Mozannar Sontag (2020b) show that the loss proposed by Madras al. (2018b) is inconsistent, proposing consistent surrogate loss that yields better results in testing; Verrm & Nalisnick (002bb) critique the approach of Mozannar & Sontag (2020b), demonstrating that their surrogate loss has degenerate parameterization, causing miscalibration of the estimated probability of expert correctness. Keswani et al. (2021) observe that decisions can often be deferred to one or more humans out of a learn expanding L2D to the multi-expert setting. Verma el al. (2023) propose the first consistent and calibrated surrogate losses for the multi-expert setting, by adapting the work of /Verma & Nalisnick (2022b) and the softmax surrogate loss of Mozannar & Sontag (2020b). All aforementioned studies focus on deriving surrogates for the 0 loss, meaning they are not directly applicable to cost-sensitive scenarios, where the cost of erring can be class-dependent (i.e., different. costs
3

---

Published in Transactions on Machine Learning Research (07/2024)
for false positive and false negative errors), or even instance-specifie (i.e., every instance has an associated misclassification cost).
Another key facet of L2D research is the interplay between the two components of an assignment system: the rejector (which decides whether to defer and to whom) and the classifier (which produces automatic predictions if the rejector chooses not to defer). Mozannar & Sontag (2020b) argue that the main classifier should specialize on the instances that will be assigned to it, in detriment of those that will not. This is done by jointly training the classifier and the rejector, without penalizing the classifier's mistakes instances that the rejector defers to an expert. The approach of Verma et al. (2023) differs in that the rejector and the classifier are trained independently, mcaning that the classifier is encouraged to predict correctly on all instances, likely to be deferred or not. However, in the multi-expert setting, the two-stage one-vs.-all approach of Verma et al. (2023) outperforms their adaptation of the softmax loss of Mozannar & Sontag (2020a), which employs joint-learning, due to the same calibration problems observed in the single- expert setting. A more recent joint learning approach proposed by Mozannar et al.] (2023) is theoretically shown to enable the system to achieve an improvement in overall performance, as their surrogate loss differs from previous work in that it is realizable H-consistent. In more recent work, however, Mao et al. (2024) demonstrate that two-staged learning, the separate training of the classifier and rejector, with their proposed surrogate losses, can also guarantee realizable consistency. We argue that joint learning is not be suitable for real-world applications, as, by design, the instances that are most likely to be deferred are those in which the classifier will perform worse, which would make the system highly susceptible to changes in human availability, should the AI have to predict on instances that were originally meant to be deferred.
As previously mentioned, the key barrier to the adoption of current L2D methods is that they require predictions from every human in the team, for every training instance. Current L2D research tackling data-efficient learning focuses only on the single-expert setting, using active-learning principles (Charusaic et al. 2022) to construct a smaller dataset with every human's predictions for all instances, or employing semi-supervised learning (Hemmer et al. 2023 to impute the missing human predictions.
When deferring, constraints in human work-capacity are rarely considered in multi-expert L,2D, where the goal is usually to find the best decision-maker for each instance, disregarding the amount of instances that are deferred to each individual agent. To the best of our knowledge, only MMo et al. (2024) consider control of the amount of deferrals in multi-expert I.2D, by allowing for the definition of a constant deferral cost ej for each available expert j, which is included in their realizable H-consistent two stage losses. Similar approaches had been considered in the single-expert case, allowing for the inclusion of a regularization term or expert deferral cost (Mozannar & Sontag 2020b Steege 2023 Narasimhan et al. 2022). While such a formulation allows for regularization of deferral across experts, it would not be adequate for a real-world system, where the availability of experts may vary drastically over time. In a real-world system, different teams of experts may work at different schedules or have days off, which would result in experts being available during certain time intervals while absent in others.
The existence of changes in expert availability within real-world HAIC scenarios means that an L2D system must be able to accommodate expert capacity constraints which are defined at inference-time, and not during training. This restriction is similar to that considered by Zhou et al. (2022) in the context of sparsely activated mixturre-ofcexpets models, where the routing strategy (f.e., how tokens are distributed throughout experts) is akin to the task of L2D algorithms. The authors propose allowing for a hard constraint on the number of tokens selected by each expert and the number of experts each token is routed to in inference. This approach allows for dynamic limitation of the compute resources necessary, without needing to retrain their models.
In our work, we propose a multi-expert L.D algorithm that can be used in cost-sensitive scenarios, trained with restrictive data requirements, and taking into account individual human work-capacity constraints that are imposed only at inference time.
2.2 Simulation of Human Experts
Due to the lack of sizeable, public, real-world datasets with multiple experts, most authors use label noise to produce arbitrarily accurate expert predictions on top of established datasets found in the ML. literature.
4

---

Published in Transactions on Machine Learning Research (07/2024)
Mozannar & Sontag (2020b) use CIFAR-10 (Krizhevsky al. 2009) and simulate expert with perfect accuracy on fraction of the 10 classes, but random accuracy the others (see also the work by Verma & Nalisnick] (002bb) and Charusaie et al. (2022)). The main drawback of these synthetic experts is that their expertise rather simplistic, being either feature-independent or only dependent on a single feature or concept. This lype of approach has been criticised (Zhu el al. 2021 Berthon el al, 2021), and instance- dependent label noise (TDN) has been proposed as a more realistic alternative, as human errors are likely be dependent on the difficulty of given task, and, as such, should depend on its features. In this work, we propose IDN approach to simulate more complex and realistic synthetic experts.
3 Method
The main goal of our work is to develop multi-expert assignment system that able optimize assignments in cost-sensitive tasks, subject to human work-capacity constraints. For this method to be applicable in real- world scenarios, it is crucial that the assigner can be trained with limited human prediction data (only one expert prediction per instance) and that, in inference, our method be robust to variations in expert work-capacity.
To tackle these objectives, we propose DeCCaF: a novel assignment system that optimizes instance allocation team of one or more analysts, while respecting their work-capacity constraints. Given a set of instances with one associated expert prediction, we train a human expertise model (HEM) that jointly models the human team's behavior. This model predicts the probability that deferral to given expert will result in a correct decision. An ML classifier is trained over the same sample with the aim of estimating the likelihood of correctness in automatically predicting on given instance. We then employ constraint programming (CP) to maximize the global probability of obtaining correct decisions under workload constraints.
3.1 Deferral Formulation
Data Assume that, for each instance i, there is ground truth label y V, vector x; E X representing its features, specific cost of misclassification G, prediction mj,i E ) from a given expert (1,...,J). and ji, identifying which expert made the prediction. The training set can then be represented as S { Vr, m, Co. J:
Classifier Rejector Formulation We first focus on building our L2D framework using the classifier-
rejector approach (Madras et al.] 2018a; Mozannar Sontag 2020a) Mozannar et al., 2023 Verma & Nalismick 2022a Verma et al., (022). first introduced by Cortes et al. (2016). This approach focuses on learning two models: a classifier denoted h X + y and rejector r X + [0. If r(æ.) = the classifier h will make the decision instance i; if r(æ;) jj, the decision
instance will be deferred
to expert j. We will consider the 0 loss, as proposed by Verma et al. 22023 as the learning objective. According to this formulation, when the classifier makes the prediction (i.e., r(x) 0), the system incurs a loss of if the classifier is incorrect. When expert makes the prediction (i.c., r(x) = j), the system incurs a loss of 1 if the human is incorrect. Formally, the 1 expected loss defined as
Lo-1(h,r) = Em my 1/ [r(x)= = 0J [1h(x)) # vl + Σ [[((x) = 5l (x) #v]
(1)
Verma al. (0023) demonstrate that the Bayes-optimal classifier and rejector, i.c., those that minimize
To-1, satisfy
h²t (x) arg max P(ylæ) yEy
(2)
if Pf = h² Ply m=)())ar)) Vj E (I.
r (x)
arg max Pyy my(x)(=) jE(l.....)
otherwise,
(3)
5

---

Published in Transactions on Machine Learning Research (07/2024)
where P(y[z) is the probability of the label under the data generating process, and P(y mylæ) is the true probability that expert is be correct. As Tn-1 is non-convex, thus computationally hard to optimize, previous L2D work (Verma & Nalisnick] 2022a Mozannar & Sontag) 2020a) focuses on the derivation of consistent convex surrogate losses, whose minimization results in h and that approximate the Bayes optimal classifer-rejecttor pair. This approach assumees that h and r will be modelled by algorithms that fit the statistical query model (Kearns) 1998) (e.g., neural networks, decision trees), whose lcarning process involves the empirical approximation of the expected value the surrogate losses by using a training sample The minimization of the approximation the expected value (e.g., via gradient descent) produces a classifier h and a rejector f whose decisions are shown converge h" and r* the amount of data increases.
In this work, rather than deriving consistent surrogate loss for simultancously training h and т, train both components separately. In the following paragraphs, we propose training process for the classifier and the rejector under limited data availability, demonstrating their convergence to h and r². For simplicity, we consider the binary classification case, where y = {0, 1].
Classifier Training To obtain h, we can train a binary classifier using any proper binary composite loss with a well defined inverse link function wh-1, such as the logistic loss (Reid & Williamson, 2010), for which 1-1 (g) e/(1+- The empirical estimator of the expected value of the logistic loss is given by
yo log (o(xz)) (11 (1-1i)) 14) (1 ( (1-0-) (4)
(aanniiter((;;))))) Celussifier ((x;, yi, C.))))
Σ
where g(x;)) is the score of sample Ø;- Proper binary composite losses ensure that B[4] is minimized when
of (g(x)) = Pp(y = 1 (x) (Buja et al. 2005 Reid & Williamson 2010), meaning that minimization of this
empirical loss will converge to I as N As such, defining
+ 00, a scoring function producing calibrated estimates P(y = 1(x).
h(x) = -(o
(f(=)) 0.5
(5)
otherwise,
ensures that h, in the large sample limit, will agree with the Bayes-optinal classifier h².
Rejector Training In order to obtain A, we resort to a human expertise model (HEM) with scoring function 91 : X x (1, J} + R, predicting whether an expert correct (1) or incorrect (0). By conditioning the model's output on the expert index, we can train this model using the training set S, which only contains one expert prediction per instance. Although it would be possible train one scoring function per expert on the subset of the data with said expert's predictions (which would be similar to the approach of Verma et al. (2023)), we choose to use single model for the entire leam. In scenarios with limited data pertaining to each expert, it can be beneficial to model the team's is behavior jointly, as there may not be enough data to train reliable models for every single expert. By conditioning the model's output on the expert's index, we still allow the model to adapt to each individual expert's decision- making processes, should that be beneficial. Note that, should additional information e E be available the experts (e.g., MI. model score), the HEM scoring function can take the form 91 X U J} U + E, thus conditioning the expert correctness probability estimate on all the information available to the expert. We obtain the optimal scoring function ف by minimizing
N N
LHEM ((x,, Vi. my J
Mmy = y)] log (g1
(6)
Minz, # v) log - (aIL
6

---

Published in Transactions on Machine Learning Research (07/2024)
where (z) is the score associated with deferring instance æ; expert j. This allows us to obtain the estimates of Ply m), given by (0= (a. j)). As such, defining the rejector as
max (v ( v (o,,,)), vj (I,....) arg max w- (x. otherwise, jE(I....))
f(x)
(7)
implies that f will agree with the Bayes-optimal rejector r", thus proving that our approach in modeling and training P and h under limited data availability yields a classifier-reeector pair that converges to h" and r", the minimizers of Lo-1,
3.2 Definition of Capacity Constraints
To shift focus to deferral under capacity constraints, we start by formalizing the work-capacity limitations of the expert leam. Humans are limited in the number of instances they may process in any given time period (e.g., work day). In real-world systems, human capacity must be applied over batches of instances, not over the whole dataset at once (e.g., balancing the human workload over an entire month is not the same as balancing it daily). A real-world assignment system must then process instances taking into account the human limitations over given "batch" of cases, corresponding to pre-defined time period. We divide our dataset into several batches and, for each batch, define the maximum number of instances that can be processed by each expert. In any given dataset comprised of N instances, divided into Abaaches: capacity constraints can be represented by vector b, where component by denotes which batch the instance i E NY belongs to, as well as a human capacity matrix H E N NubatebessJ] where element Hos a non- negative integer denoting the number of instances batch that human expert can process.
3.3 Global Loss Minimization under Capacity Constraints
In the previous sections, we detailed our approach to train the classifier-rejector pair, showing that h and i converge to the Bayes-optimal h' and r". Note that this classifeerrreeector formulation produces optimal point-wise solutions, disregarding any work-capacity limitations. Should an expert be more likely to be correct than all their colleagues and the ML classifier over the entire feature space, the set of optimal point- wise assignments would be always defer instances to said expert, a clearly unfcasible solution in any real-world system.
A point-wise formulation of h and f is clearly inadequate to optimize the assignments over a set of instances while respecting expert capacity constraints. However, the scoring functions and 9 obtained in the training steps detailed above, will still be of use in providing calibrated estimates of the probability of correctness. To formulate the assignment problem under capacity constraints, we consider that our objective is to maximize the expected probability norrectness over all instances be deferred. Recall that y = (0, 1} and consider the assignment decision for each instance a; E {l...., J + 2}, where a; y + 1, with y E }, is automatic prediction of class y for instance i, whereas a; = + 2 denotes the decision to defer instance to the jth expert. The estimate of the probability of correctness for all possible assignments given instance is then given by
(0(x))
ifa; if 2
((correct (a,rectt/a,, ai)
1 (g(z:))
(8)
(0L (20;
if a, = +
To represent the assignment docisions over batch b comprised of nn instances, without loss of generality indexed in (1...., ng). consider the ng (2+J) matrix of assignments A, where each element A binary variable that denotes if the assignment decision aş taken for instance i. The optimal set of assignments is

---

Published in Transactions on Machine Learning Research (07 (07/2024)
given by
J+2
=
arg min
P(correct [2 ai) Ai,z:
04
s.l.
Ai.:: = Haw for ai E (3, J + 2}.
(9)
1=1
J+2
Σ
Ar.: for (I,),...,nB.
0,=1
The first constraint refers to human capacity: with an equality constraint, the number of instances assigned to each decision-maker is predefined the problem statement; this constraint may be changed to an inequality expressing the maximum number assignments per decision-maker. The second constraint states that each instance must be assigned one and only one decision-makse the constraint programming solver CP-SAT from Google Research's OR-Tools (Perron & Didier, 2023 Finding solutions to these combinatorial problems is not trivial, solution times grow exponentially with the problem's ss size (Huberman et al. 1997). CP-SAT is parallel portfolio solver, leveraging the available processor cores to run different solution search strategies each. By doing so, portfolio solvers aim to more efficiently cover the solution-space, under the assumption that different algorithms, using different houristios, may compliment each other's weaknesses (Huberman et al. 1997 Gomes & Selman 2001). This approach proves effective as selecting the optimal search algorithm for a given constraint problem is considered a non-trivial task. We selected CP-SAT in our implementation as has been repeatedly shown to be the best- performing publicly available solver in wide array of constraint programming problems in the MiniZinc challenge (Stuckey et al.] 2014), where solvers are tasked to finding the optimal within a time-limit. While this solver can find feasible solutions relatively quickly, proving solution's 's optimality can take an extremely long time, meaning that in practice it is often better to set a time limit, having the system return the best solution it found so far. On our machine (Intel(R) Xcon Gold 5120 CPU @ 2.20GHz), no significant improvements were obtained for timeout limits above seconds.
We solve the assignment problem using
near-oplimal solution
3.4 Cost-Sensitive Learning
Finally, having discussed how to optimize for the 0-1 loss under capacity constraints, we now focus on adapting our method work under an arbitrary cost structure. To do so, we follow an instance re-weighting approach (Zadrozny et al., 2003 Elkan, 2001), where each point-wise loss over the training set is multiplied by the cost e, associated with said instance. This guarantees that the minimization of the surrogate losses used to train and 91. will result in scoring functions that minimize the misclassification cost instead of the error-rate. A more detailed description of the re-weighting approach follows.
Training a classifier with score function involves approximating the expected value of its surrogate loss function l by the empirical average of the point-wise losses over the training set,
1 Σ ((:;)(())) v.). ((e))=)(((, N
(10)
where D denotes the data distribution. Note, however, that instances may have different misclassificatiin costs, in which case a surrogate for the loss not adequate. Assuming that each instance a; has an associated misclassification cost G, the goal then to learn classifier that minimizes the expected cost, Ex I(h(z) v)]. Minimizing surrogates to the (-1 loss ensures that minimize Ex [I(h(=)) # v)), which is misaligned with our objective. Zadrozny el al. (2003) show that if we have examples drawn from a different distribution,
D(a c) Eanple D(r,,c)),
(11)
8

---

Published in Transactions on Machine Learning Research (07 (07/2024)
then
1 Bo-ple Ev -.w.c--Dcllhhlal. + y)].
E a.y e. (L(((())))))))))))))))(()))))() # U)] =
(12)
shows that selecting the decision rule to minimize the error rate under is equivalent to selecting h minimize the expected misclassification cost under D. This means we can obtain our desired classifier by training it under the distribution D, using the log-loss. To train classifier under D, common approach (Zadrozny al., 2003 Elkan 2001) is re-weight the instances according to their costs. In this way, we obtain classifier that prioritizes correctness according the instances" weights, by minimizing the empirical estimate of the misclassification cost:
Equation 12
N BewDl cll(a, (())).
(13)
As such, we just have to reweight the instances in Celaesifier where is the point-wise log-loss (see Eq. when training the scoring function 9. We follow the same approach with respect to the rejector, reweighting the instances LHEM (see Eq. 6 when training gu. The re- reightt empirical losses are given by
N
Celassifier ( (ar yo, N
Σ i=1
Cj
- yi log (v (o(za)) - (1 :):) log 11- (1-x-(((aa)))
(14)
N
CHEM y1, Ci m.sne. mi
Σ 1=1
- Immy, = vl log (1= (91
c:
(15)
I[my, + vi log ( (1- w به (oL
The full algorithm is described in the following pseudo-code blocks. Note that, for the training process, the function "Minimize-Losss represents any optimization algorithm that minimizes an empirical loss (e.g.. gradient descent, gradient boosting).
Algorithm Pseudo- Code for DeCCaF- Training
Input: Training data S (xx,, yb, m 1 Initialize g, 91
Initial scoring functions (c.g., NN with random weights) see Eq.
9 Minimize-Loss ('lasssiber (fxi, Vir Ci J.N :))))
14
91
Minimize-Loss (CHEM (IHEM((2))) (x,, Ui, cs my (
Eq. 15
4 Experiments
4.1
Experimental Setup
Dataset As the base dataset, we choose use the publicly available bank-account--rand dataset. (Jesus et al.] 2022) (Version 1). This tabular datasct comprised of one million synthetically generated bank account opening applications, where the label denotes whether the instance is fraudulent (1) legitimate (0). The features of each instance contain information about the application and the applicant, and the task of a decision maker (automated or human) is to either accept (0) or reject (1) it.
These applications were generated based on anonymized real-world bank account applications, and, as such, this dataset poses challenges typical of real-world high-stakes applications. Firstly, there a high class imbalance, with 1.1% fraud prevalence over the entire datasct. Furthermore, there are changes the data distribution over time, often referred to as concept drift (Gama al. 2014), which can severely impact the predictive performance of MT. Models.
9

---

Published in Transactions on Machine Learning Research (07/2024)
Algorithm 2 Pseudo-Code for DeCCaF- Inference for Batch b
1: Input: Instances {xt (er))r Batch Capacity Constraints {Ho
H E No N XJ see Section 3.2
2: Initialize P E R [J+x)
b Stores P(correct|2; ai), see Eq.
g
3: Initialize A E {0, 1]
> Stores As,, denoting if decision de is taken for instance i, see Eq. g
for i (1,..,
do
for a, - [I, + 2) do
if a; 1 then
7:
P, f- - 10- (p(a:))
8: else if a = 2 then
9:
P, f /- 1 (i(2:))
10: else
11:
Ps.a, - (0) (2;; (2:000 0i 2))
12: end if
13: end for
14: end for
15: A t CPSolver HHb)))
Optimization Problem in (7)
Alert Review Setup There are several possible ways for models and humans cooperate. In previous L2D research, it is a common assumption that any instance be deferred to either the model the expert team. However, in weal-world settings, due to limitations in human work-capacity, it is common to use Alert Model to screen instances, raising alerts that are then reviewed by human experts (De-Arteaga et al. 2020 Han et al.] 2020). In an alert review setting, humans only predict on a fraction of the feature space, that is, the instances flagged by the Alort Model. We will train a L2D system to work in tandem with the Alert Model, by deferring the alerts in an intelligent manner. We calculate the Alert Model's alert rale ar, that is, the fraction of instances flagged for human review, by determining the FPR of the Alert Model in validation. We create distinct alert review scenarios by varying the alert rate a, (5%FPR, 15%FPR).
Alert Model Training We train the Alert Model to predict the fraud label the first three months of the dataset, validating its performance the fourth month. We use the LightGBM (Ke et al. 2017] algorithm, due to its proven high performance on tabular data (Shwartz-Ziv Ziv & Armon 2022 Borisov et al. 2022). Details on the training process of the classifier are given in Section A.1 of the Appendix.
Optimization Objective This is a cost-sensilive lask, where the cost of a false positive (incorrectly rejecting a legitimate application) must be weighed against the cost of a false negative (incorrectly accepting a fraudulent application). Due the low fraud prevalence, metrics such as accuracy are not adequate measure the performance of both ML models and deferral systems. The optimization objective used by Neyman-Pearson Jesus et al. (2022) is Neyman- Pearson criterion, in this case, maximizing recall at 5% false positive rate (FPR), which establishes an implicit relationship between the costs of false positive and false negative errors. However, for the cost-sensitive learning method described in Section 3.4 we need to have access to the explicit cost structure of the task at hand.
In a cost-sensitive task, the optimization goal is to obtain a set of predictions á that minimize the expected misclassiffcation cost E(C). Assuming that correct classifications carry no cost, the relevant parameter the ratio À= CFP/CFN, were CFp and CFN are the costs of false positive and false negative errors, respectively. The objective is thus
|l[[ = 0 A = 1| 1ly 1 d. =0].
(16)
N
Minimizing this quantity is equivalent to minimizing the average cost, as division by a constant will not affect the ranking of different assignments. As such, all that remains to be established is a relationship between the Neyman-Pearson criterion and the value of X. To do so, we follow the approach detailed in Section A.2 of the Appendix, which yields the theoretical value X = 0.057. To test performance under variable cost structures, will conduct experiments for the values À E (x+/5, >1, 5..). These alternative scenarios not.
10

---

Published in Transactions on Machine Learning Research (07/2024)
strictly comparable, as if the cost structure were À (4/5, the optimization of the Alert Model would not correspond to maximizing the recall at 5% FPR. Nevertheless, we choose to introduce these variations to test the impact on system performance of changing the cost ratio A. Combining the different alert rates with these values of X, we obtain 6 distinct text scenarios.
Synthetic Expert Decision Generation Our expert generation approach is based on instance- instance-dependent noise, in order to obtain more realistic experts, whose probability of error varies with the properties of each instance. We generate synthetic predictions by flipping each label y, with probability P(mj, # yr ( y:). In some HAIC systems, the model score for given instance may also be shown to the expert (Amarasingho et al. 2022 De-Arteaga et al. 2020 Levy et al., 2021), so an expert's decision may also depend on an ML model score m(x;). We define the expert's probabilities of error, for a given instance, as function of a pre- ree-procsssed version of its features o; and the Alert Model's score M (x,), so that the feature scale does not impact the relative importance of cach quantity. The probabilitics of error are given by
P(mj.: 1ly, = 0,, = o (86 w E; + (UMM(()))
w/M w T. + wM M M(x)) w w + w/e
VW-
(17)
P(mj.: 0lys = 1,);) = o (31 ( ta
where a denotes a sigmoid function. Each expert's probabilities of the two types of error depend on five parameters: 9o, 31, a, w, and wy. The weight vector w embodies a relation between the features and the probability of error. The feature weights are normalized that can separately control, via a, the overall magnitude of the dependence of the probability of error on the instance's features. The values of ß1 and 80 control the base probability of error. The motivation for this approach is explained further in section C.1 of the Appendix.
The expected cost resulting from an expert's decisions given by
E[C], = E, [AP(m, ^ y = 0) + P(my A y = 1)]
N
(18)
AMP(m = 1/y, (y 0) + Pmmm, = 0lys = 1)P(w = 1)]
For our setup to be realistic, we assume an expert's decisions must, on average, incur a lower cost than simply automatically rejecting all flagged transactions. Otherwise, assuming random assigument, having that expert in the human team would harm the performance of the system as whole. As the expert's average misclassification cost is dependent on the prevalence P(y = 1) and cost structure as defined by X, team of experts with the exact same parameters will have different expected misclassification costs, depending on the alert review scenario in question. For this reason, in each of the six aforementioned settings, we generate a different team of 9 synthetic experts by first fixing their feature weights w, and sampling their expected misclassification cost. Then, we calculate the values of 30, B1 that achieve the sampled misclassification cost, thus obtaining a team of complex synthetic experts with desirable properties within each scenario. Further details on the sampling and expert generation process, as well as a description of each of the expert teams' properties are available in Sections C.2 and C.3 of the Appendix.
Expert Data Availability To generate realistic expert data availability, we assume that the Alert Model is deployed in months 4-7, and that cach alert is deferred to randomly chosen expert. This generates history of expert decisions with only one expert's prediction per instance. To introduce variability to the L2D training process, the random distribution of cases throughout experts was done with 5 different random seeds per scenario. For each of the three 5% FPR alert rate scttings, this results in a set of 2.9K predictions per expert. So that all settings have the same amount of data, for the 15% FPR alert rate scenarios, we also sample 2.9K predictions from each of the 10 experts.
Expert Capacity Constraints In our experiments, we want reliably measure the ability of our method to optimize the distribution of instances throughout the human team and the classifier h. In real-world
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table Distribution of Expert Decision Outcomes
SCENARIO DECISION OUTCOME (%)
X
fp
fn
tp
tn
0.05
0.0114
30.2
0.3
11.1
58.4
0.05
0.057
25.0
1.9
9.4
63.7
0.05
0.285
16.9
5.6
5.9
71.6
0.15
0.0114
34.4
0.3
5.5
59.9
0.15
0.057
30.6
1.4
4.2
63.8
0.15
0.285
10.4
2.7
3.1
83.8
scenarios, the cost of querying h is much lower than that of querying a human expert, the work-capacily of the classifier is virtually limitless. However, in order to ensure that the assignment systems are able to reliably model the human behavior of all experts, we will force the assigner to distribute the instances throughout the expert team and the classifier h in equal amounts. As the expert teams are comprised of 9 experts, this means that 1/10 of the test set will be deferred to each expert/classifir. To introduce aopert/classifier /classifier has a different capacity, by sampling the values from N (N 10, N5). As such, for each scenario defined by the (ar, A) pair, there are a total of 25 test variations, which result from combining each of the training seeds with the different capacity constraint settings.
variability, we also create four distinct test settings where each expert
4.2 Baselines
One vs. All (OvA) For an L2D baseline, we use the multi-expert learning to defer OvA algo- rithm, proposed by |Verma al.] (2023). This method originally takes training samples of the form Dj (a,, Vi, mi,1,, -..; mj.e) and assumes the existence of a set of every expert's predictions for each training instance; however, this is not strictly necessary.
The OvA model relies on creating a classifier x + the classifier makes the decision; if r(x,) j, the decision is deferred to the jth expert. The classifier is where denotes the class index. These are related The rejector, similarly, composed of J functions: タTT X + R for j J}, which are related to the probability that expert ý will make the correct decision regarding said instance. The authors propose combining the functions 91, OK, I1,11 91.. in OvA surrogate for the 0-1 loss. The OvA multi-expert L2D surrogate is defined as:
x y and a rejector : X + 10. J}. If r(x;) = 0,
composed of functions: 9k : X + R, for h E (1...., K},
to the probability that an instance belongs to class
VorA= [[gg(x)] +
Σ WEV'''
(1)(
Σ 1=1
Φ Φ - (x)) + Σ [m, v)(0(g s())((((])]]) -1-gi
1=1
where +. (+1) x R R. is a strictly proper binary surrogate loss. (The authors also propose using the logistic loss.) Verma et al. (2023) then prove that the minimizer of the pointwise inner risk of this surrogate loss can be analyzed in terms of the pointwise minimizer the risk for each of the + underlying OvA binary classification problems, concluding that the minimizer of the pointwise inner VovA-risk, g", is compriscd of the minimizer of the inner Ø-risk for each ith binary classification problem, g. As such, in a scenario where only one expert's prediction is associated with each instance, each binary classifier g can be trained independently of the others. By training each binary classifier gi.f with the subset of the training sample containing expert j's predictions, we obtain the best possible estimates of each pointwise inner P-risk minimizer 9f given the available data. To adapt the OvA method to a cost-sensitive scenario, we can again use the rescaling approach detailed in Section 3.4 minimizing the expected misclassification cost.
The OvA method does not support assignment with capacity constraints, as such, we proceed similarly Verma et al. (2023). by considering the maximum out of the rejection classifiers' predictions. If the capacity of the selected expert is exhausted, deferral is done the second highest scoring expert, and on.
12

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Random Assignment In this baseline, which aims to represent the average performance of the ex- pert/model team under the test conditions, alert deferral is done randomly throughout the experts/model until each of their capacity constraints are met.
Only Classifier (OC)
In this baseline, all final decisions on the alerts are made by the classifier h.
Full Rejection (FR) All alerts are automatically rejected.
4.3 Deferral System Training
For the classifier and the HEM, we again use a LightGGB model, trained on the sample alerts raised over months 4 to 6, and validated on month 7. The models were selected in order to minimize the weighted log-loss, where the weight for label-positive instance G = and the weight for label-negative instance is given by c = X.
For the OvA method, we follow the process detailed in Section 4.2 first splitting the training set into the data pertaining to each expert's prediction to obtain each scoring function 91.j. In the binary case, only one classifier scoring function g is nceded for the OvA approach. As each scoring function is trained independently, the optimal classifier the same classifier h for both deferral systems. Details on the training process and hyper-paramen selection are available in Section D of the Appendix.
obtained in the same manner as our method. As such, we will use
5 Results
5.1
Classifier h Performance and Calibration
We first evaluate the predictive performance and calibration of the classifier h. We assess how well the classifier is able to rank probabilities of correctness, and then evaluate its calibration, by using the expected calibration error (Guo et al. 2017). Note that each classifier had a distinct training process, with varying sample weights 6 dependent on X. As such, these measures must be calculated under the re-weighted data distribution D (see section 3.4b. In Table 2 we show that the classifier is able to rank instances according their probability of belonging the positive class.
Table 2: Expected calibration error (ECE) and ROC-AUC for classifier h for all testing scenarios, denoted by the (alert rate (ar); cost-structure (A)} pairs classifier h is able to reliably estimate PPy; 1)
SCENARIO
CLASSIFIER h
Or
X
ROC-AUC ECE (%)
0.05
0.011
0.70
1.1
0.05
0.057
0.71
4.8
0.05
0.285
0.71
4.6
0.15
0.011
0.76
3.8
0.15
0.057
0.75
4.2
0.15
0.285
0.73
3.3
5.2 Expert Decision Modeling
Performance and Calibration
We now evaluate the probability ranking and calibration of the models that estimate expert correctness. We will again check that the scoring functions (HEM) and gı (OvA) are able to model individual expert correctness by using the ROC-AUC, then checking for calibration. For each training random seed, the ROC-AUC was calculated for every individual expert's decisions. Again, these mcasures were calculatod under the data distribution D. In the top row of Figure 2 we observe that the distribution of ROC-AUC is similar across both methods, with consistent overlap of error bars, despite superior average across almost
13

---

Published in Transactions on Machine Learning Research (07/2024)
Alert Rate 0,05
Alert Rate
0.15
Alert Rate 0.05
Alert Rate 0.15
0.65
0.65
Method
Mathod
20.0
Method
Method
Ovi HEM
OVA HEM
OvA HEM
OvA
0.60
0.60
17.5
20
HEM
ROGAUC
ROCAUC
0.55
15.0
ECE(N)
15
12.5
0.50
0.50
10.0
10
7.5
0.45
0.45
0.00 0.05 0.
15 0.25 0.30
0.00 0.05 0.10
0.15 0.150.2000000 0:25 0.30
0.00 0.000.05 0.10 0.15 0.20
0.25 0.30
0.00 0.05 0.10
0.20 0.25 0.30
Value
Figure 2: Mean ECE and ROC-AUC for estimates of P(N = my,)). Values are calculated for each expert j and averagod, with error bars representing 95(%) confidence interval. Both methods obtain similar ROC- AUC, however, the value of has significant impact on their ranking terms calibration, showing the importance of testing L2D methods under a wide variety of cost-structures.
all scenarios for the HEM method. It is notable, however, that for both alert rates, the value of the ROC- AUC consistently falls below 0.50 when À = 0.0114, indicating that these models are not able to reliably rank probabilities of correct of the expert team under these conditions. In these scenarios, however, L,2D methods can still theoretically achieve improvements in performance by choosing which instances should be deferred to the classifier h, whose probabilitics of correctness have been shown be reliably modeled. In the bottom row of Figure ? we observe that HEM achieves significantly lower ECE for X 0.057, with the reweighted OvA baseline outperforming HEM in scenarios with À 0.285. This demonstrates how the performance ranking of different L2D methods can change significantly by altering the data distribution and cost-structure of the experimental setting.
5.3 Deferral under Capacity Constraints
In this section, we evaluate the quality of deferral as measured by the average misclassification cost per 100 instances. The average experimental results are displayed in Table with 95% confidence intervals, whore we refer to method as DeCCaF. We observe that, on average, both L2D methods pulperform the non-L2D approaches, and that DeCCaF is significantly better in most scenarios. As the number of test seeds (25) is relatively low, the approximation of the 95% confidence interval may be unreliable, as the value shown is derived based on the central limit theorem.
Table 3: Expected misclassification cost per 100 instances [ECC//100, assuming OFP = X, CFN = 1) for cach {a,,A} pair. In each row, values are averaged across all test variations, and displayed with 95% confidence intervals. FR and OC represent the "Full Rejection" and "Only Classifier" baselines.
SCENARIO
DEFERRAL STRATEGY
a
A
FR
OC
Random
OvA
DeCCaF
0.05
0.0114
0.96
0.96
0.80+0.08
0.84+0.06
0.79
0.05
0.057
4.79
4.58
4.0.0.2
3. 3..37
3.4.0.04
0.05
0.285
23.95
14.08
12.2+0.2
11.13+0.04
10.2+0.2
0.15
0.0114
1.05
1.07
0.87+0.05
0.87+0.02
t.74
0.15
0.057
5.27
4.01
3.540.1
3.55.1.03
3.19
0.15
0.285
26.36
6.93
6.0.0.1
5.23.10.08
10.09
In Table 4 we report the percentage of test variations, for each scenario, in which our method outperforms other deferral basclines. These results demonstrate that, across all scenarios, DeCCaF outperforms the full rejection and the "Only Classifier baseline. Notably, the relative performance of DeCCaF when compared to random assignment. changes drastically depending on the scenario properties, performing no better than
14

---

Published in Transactions on Machine Learning Research (07 (024))
Table 4: Percentage of the 25 test variations, for each {ar,A} pair, where DeCCaF beats other methods. According to binomial statistical significance test, with a 0.05, DeCCaF is significantly better than other methods for values [0.68, 1J, while values E [0.28, 0.68] mean that we cannot conclude which method is best. Across all scenarios, DeCCaF is shown to be superior in 5, while the comparison is inconclusive in onc.
SCENARIO DEFERRAL STRATEGY
dr
A
OvA
Random
FR
OC
0.05
0.0114
0.52
0.56
0.88
0.88
0.05
0.057
0.76
1.00
1.00
1.00
0.05
0.285
0.96
1.00
1.00
1.00
0.15
0.0114
0.84
0.88
1.00
1.00
0.15
0.057
1.00
0.96
1.00
1.00
0.15
0.285
0.96
1.00
1.00
1.00
random assignment for o, = 5%FPR, 0,0114. This again demonstrates how the cost structure and distribution of expert performances has significant impact on the performance of L2D systems. Neverthe- less, we observe that DeCCal performs significantly better than the OvA baseline in 5 of scenarios, outperforming in both scenarios with À X Note that the only scenario where the comparison between DeCCaF and other baselines is inconclusive corresponds to the most imbalanced scenario, where the preva- Ience is higher, and the cost of false positives is lowest. We therefore conclude that there are benefits in jointly modeling the human decision-making processes.
Varying Data Avaliability Finally, assess the impact of the amount of training data by repeating the experiments for X = d = 0.057, but training the OvA and DeCCaF methods with less data.
Alert Rate 0.05
Alert Rate 0.15
4.2
Deferral
Deferral
OvA
OvA
DeCCaF
DeCCaF
3.8
3.4
3.2
0.25
0.50
1.00
0.25
1.00
% of Total Training Data
% of Total Training Data
Figure 3: Expected Misclassification Cost per 100 instances (E,C)/100, assuming CFP X, CFN = 1) for {0.05, 0.15]. A = 0.057, and different amounts of training data. In each point, values are averaged across all 25 test variations, and displayed with 95% confidence intervals DeCCaF remains significantly better in most scenarios.
In Figure 3 we show the variations in misclassification cost result of the undersampling of training data. As expected, both methods are significantly impacted by reducing the amount of training data. We again observe that DeCCaF performs either similarly to, outperforms the OvA bascline.
6 Conclusions and Future Work
In this work, we expand multi-expert L2D research to consider several real-world issues that limit adoption of these systems. We consider limited data availability, the existence of human expert work-capacity constraints, cost-sensitive optimization objectives, and significant class imbalance, which are key challenges posed by many high-stakes classification tasks.
15

---

Published in Transactions on Machine Learning Research (07/2024)
We propose novel architecture that aims to better model human behaviour and to globally optimize the system's performance under capacity constraints. We conducted constrained deferral under a wide variety of training and testing conditions, in a realistic cost-sensitive classification task, empirically demonstrating that variations in the cost structure, data distribution, and human behavior can have significant impact on the relative performance of deferral strategies. We retonstrate that DeCCaF performs significantly better than the basclines in wide array of testing scenarios, showing that there are benefits to jointly modcling the expert team.
For future work, we plan on testing the addition of fairness incentives to our misclassification cost opti- mization method, to study the impact this system may have in ensuring fairness. For scenarios where the misclassification costs are instance-specifie (f.e., transaction fraud), we will also study the possibility of direct cost estimation by using regression models instead of classification.
Finally, important to consider the othical implications of adopting these systems in real-world scenarios, as these may impact the livelihood of the human experts involved in the deferral process. In a system without intelligent assignment, cases are distributed randomly throughout the human team, ensuring an ii.d. data distribution for each expert. If a multi-expert L2D system were to be adopted, the subset of cases deferred to each expert would follow a separate distribution. If we consider a highly skilled expert with very high performance throughout the feature space, this system could choose to assign the hardest cases to said expert, damaging their performance. As the performance of analysts is routinely evaluated, this could create unfair disparities across the human expert team, with certain analyst's 's performance being degraded while others could be inflated. To tackle this issue, we could periodically assign randomly selected instances to each expert, in order to evaluate them in i.idd set, which would also be useful data to retrain the system, as human behaviour may change over time. We hope this work promotes further research into L2D methods in realistic settings that consider the key limitations that inhibit the adoption of previous methods in real-world applications.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next- gencration Hyperparameter Optimization Framcwork. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 2623 2631. ACM, 2019. doi: 10.1145/3292500.3333 333330000
Saar Alon-Barkat and Madalina. Busuioc. Human ai interactions in public sector decision mak- automaanini bias" and "selective adherence" to algorithmic advice. Journal of Public Administration Research and Theory, 33(1):153 169, 2023.
Kasun Amarasinghe, Kit T Rodolfa, Sérgio Jesus, Valorie Chen, Vladimir Balayan, Pedro Saleiro, Pedro Bizarro, Ameet Talwalkar, and Rayid Ghani. On the importance of application-groundee experimental design for evaluating explainable ml methods. arXiv preprint ar Xr0: 13503, 2022.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International conference on machine learning, pp. 825-836. PMLR, 2021.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Newral Networks and Learning Systems, 2022.
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3:13, 2005.
Richard I. Burden and J Douglas Faires. 2.1 the bisection algorithm. Numerical analysis, 3, 1985.
Mohammad-Amin Charusaie, Hussein Mozannar, David A. Sontag, and Samira Samadi. Sample Efficient Learning of Predictors that Complement Humans. Tn Kamalika Chaudhuri, Stefanie Jegelka, Le Song,
16

---

Published in Transactions on Machine Learning Research (07/2024)
Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, TCML, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 2972 3005. PMLR, 2022.
C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory, 16(1):41 46, 1970. doi: 10.1109/TI11 WTT.TT.997..104444.4
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with Rejection. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles (eds.), ), Algorithmic Learning Theory 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings, volume 9925 of Lecture Notes in Computer Science pp. 67-82, 2016. doi: 10.1007 /9707 3-119--3------- 163-47779--
Shai Danziger, Jonathan Levav, and Liora Awnaim-Pesse Extrancous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108((7):6889) 6892, 2011.
Maria Arteaaga,, Riccardo Fogliato, and Alexandra Chouldechova. Case for Humans-in-the-Loog De- cisions in the Presence of Erroncous Algorithmic Scores. In Proceedings of the 2020 CHI Conference Human Factors in Compuling Systems, pp. 1 12, Honolulu HI USA, April 2020. ACM. ISBN 978-1-4503- 6708-0. doi: 15545//33111133366666
Dominik Dellermann, Philipp Ebel, Matthias Soellner, and Jan Marco Leimeister. Hybrid Intelligence. Business & Information Systems Engineering, 61(5):637 643, October 2019. ISSN 2363-7005, 1867-0202.
doi: 10.1007 1.1007/s12590001--00053
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference artificial intelligence, volume 17, pp. 973-977777 978. Lawrence Erlbaum Associates Ltd, 2001.
João Gama, Indré Zliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-33) 2014.
Sharad Goel, Ravi Shroff, Jennifer Skeem, and Christopher Slobogin. The accuracy, equity, and jurisprudence of criminal risk assessment. In Research handbook on big data law, pp. 9-28. Edward Elgar Publishing, 2021.
Carla P Gomes and Bart Selman. Algorithm portfolios. Artificial Intelligence, 126(1-2):43-8 62, 2001.
Stein Grimstad and Magne Jørgensen. Inconsistency of expert judgment-beas based estimates of software devel- opment effort. Journal of Systems and Software, 30711):1700-177, 2007.
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswany, Sub- hashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama, 316(22):2402 2410, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Jingguang Han, Yuyun Huang, Sha Liu, and Kieran Towey. Artificial intelligence for anti-money laundering: a review and extension. Digital Pinance, 2(3-4):211 239, 2020.
Patrick Hemmer, Sebastian Schellhammer, Michael Vössing, Johannes Jakubik, and Gerhard Satzger. Form- ing Effective Human-Al Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts. In Lue De Raedt (ed.), Proceedings of the Thirty-Fit First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-2 29 July 2022 pp. 2478-2484. ijeai.org, 2022.
doi: 10.24963 00.24963/jjcai.2000 /ttp:/jaaaiii2//33333
Patrick Hemmer, Lukas Thode, Michael Vössing, Johannes Jakubik, and Niklas Kühh. Learning defer with limited expert predictions. arXiv preprint ar Xxx::22304.004 2023.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In 5th International Conference on Learving Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Track Proceedings. OpenRevvew.net, 2017.
Bernardo A Huberman, Rajan M Lukose, and Tad Hogg. An economics approach hard computational problems. Science, 25555296):51-54, 1997.
Kori Inkpen, Shreya Chappidi, Keri Mallari, Besmira Nushi, Divya Ramesh, Pietro Michelueci, Vani Man- dava, Libuse Hannah Vepšek, and Gabrielle Quinn. Advancing human-ai complementarity: The impact of user expertise and algorithmic tuning on joint decision making. arxi preprint 211:2208007900 2022.
Maia Jacobs, Melanie F Pradier, Thomas H McCoy Jr, Roy H Perlis, Finale Doshi-Veler, and Kraysztol Z. Gajos. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry, 11(1):108, 2021.
Sérgio Jesus, José Pombal, Duarte Alves, André F Cruz, Pedro Saleiro, Rita P Ribeiro, João Gama, and Pedro Bizarro. Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2022, 2022.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (ods.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3146 3154, 2017.
Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM), 45 (6):983 1006, 1998.
Vijay Keswani, Matthew Lease, and Krishmaram Kenthapadi. Towards Unbiased and Accurate Deferral to Multiple Experts. In Marion Fourcade, Benjamin Kuipers, Seth Lazar, and Deirdre K. Mulligan (eds.). ATES '21 AAAT/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021, pp. 154 165. ACM, 2021. doi: 11144//3666172.3625555
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ariel Lovy, Monica Agrawal, Arvind Satyanarayan, and David Sontag. Assessing the impact of automated suggestions on decision making: Domain experts mediate model errors but take less initiative. In Proceed- ings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2021.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems, 31, 2018a.
David Madras, Toni Pitassi, and Richard Zemel. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b.
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. Advances in neural information processing systems, 36, 2024.
Toby J Mitchell and John Beauchamp. Bayesian variable selection in linear regression. Journal of the american statistical association, 83(404):1023 1032, 1988.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Interna- tional Conference on Machine Learning, pp. 7076-7087. PMLR, 2020a.
Husscin Mozannar and David A. Sontag. Consistent Estimators for Learning to Defer to an Expert. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, PMLR, 2020b.
Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans. In International Conference on Arlificial Intelligence and Statistics, pp. 10520 10545. PMLR, 2023.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya K Menon, Ankit Rawat, and Sanjiv Kumar. Post-hoc ostimators for learning to defer to an expert. Advances in Newral Information Processing Systems, 35: 19992-29300 2022.
Laurent Perron and Frédéric Didier. Cp-sat, 2023. URL cp/cp_solvee solver/
Maithra Raghu, Kaly Blumer, Greg Corrado, Jon M. Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The Algorithmic Automation Problem: Prediction, Triage, and Human Effort. CoRR, 0oss/903.1122) 2019a.
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, and Jon Kleinberg, Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning, pp. 5281-5290. PMLR, 2019b.
Mark D Reid and Robert C Williamson. Composile binary losses. The Journal of Machine Learning Research. 11:2387-248 2422, 2010.
Victor S. Sheng and Charles X. Ling. Thresholding for Making Classifiers cost-sensiiive. In Proceedings, The Twendy-First National Conference on Artificial Intelligence and the Fighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pp. 476 481. AAAI Press, 2006.
Ravid Shwartz- Ziv and Amitai Armon, Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 90, 2022.
JM Steege. Leveraged calibrated loss for learning to defer. Master's thesis, University of Twente, 2023.
Peter J Stuckey, Thibaut Feydy, Andreas Schutt, Guido Tack, and Julien Fischer. The minizine challenge AT Mogazine, 35(2):55 60, 2014.
Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference Machine Learning, pp. 22184-21222 222022. PMT.R, 2022a.
Rajeev Verma and Fric T. Nalisnick. Calibrated Learning to Defer with One-vs-All Classifiers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-28 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 22184 22202. PMLR, 2022b.
Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, pp. 11415-11434 PMLR, 2023.
Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. In Third IEEE international conference on data mining, pp. 435 442. IEEE, 2003.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-ofeexperts with expert choice routing. Advances in Newral Information Processing Systems, 1477103-7114, 2022.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach learning with deseaene label noise. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 10113 10123, 2021.
19

---

Published in Transactions on Machine Learning Research (07 (07/202) 2224)
Notation
h Classifier
h Optimized Classifier
h* Bayes Optimal Classifier
9 Optimized classifier scoring function
Classifier
Optimized rejector
r* Bayes Optimal Rejector
31 Optimized rejector scoring function
b batch vector, where component b; denotes which batch instance i belongs
H Capacity matrix, where component Ha.j denotes the number of instances in batch b the human expert j can process
0 Assignment decision for instance i. a E {I...., J + 2}, where a, y +,,, with y E D, is an automatic prediction of class y for instance whereas ag f++ denotes the decision to defer instance to the jth expert.
A matrix assignments where cach element Ai,, a binary variable that denotes the assignment decision 0₃ taken for instance
A" matrix of optimal assigmments where each element Ai,, is a binary variable that denotes if the assign- ment decision a, is taken for instance i.
A Experimental Setting
A.1 Alert Model
our Alert Model is LightGBM (ee et al. 2017) classifier. The model trained the first 3 months of the BAF dataset, and validated on the fourth month. The model is trained by minimizing binary cross-entropy loss. The choice of hyperparameters is defined through Bayesian search (Akiba et al. 2019 on an extensive grid, for 100 trials, with validation done on the 4th month, where the optimization objective to maximize recall at 5% false positive rate in validation. Table 5 we present the hyperparameter search space used, as well as the parameters of the selected model.
As detailed in Scction
4.3
Table 5: Alert Model: LightGBM hyperparameter search space
HYPERPARAMETER VALUES OR INTERVAL DIST. SELECTED
boosting lype enable bundle n estimators max depth num leaves min child samples learning rate reg _s_paaph
"goss"
"goss"
False
False
[50,5000]
Log 94
[2,20]
Unif. 2
[000)
Log 145
[,500]
Log 59
[0.01, 0.5]
Log 0.3031421
[0.0001, 0.1]
Log
0.0012637
reg lambda
[0.0001, 0.1]
Log 0.0017007
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
This model yielded recall of 57.9% in validation, for threshold t 0.050969, defined to obtain 5% FPR in validation. In the meployment. split (months 4 to 8), used to train and test our assignment system, the model yields recall of MTPR = 52.1%, using the same threshold. In Figure we present the ROC curve for the Alert Model, calculated in months 4-8.
ROC-AUC Curve Alert Model - Months 4-8
0.8
0.6
TPR
0.2
0.0
0.8
FPR
Figure 4: ROC-Curve Alert Model Months 4-8
A.2 Determining À based on Neyman- Pearson criterion
Pearsan-Pearson
In our lask, the optimization goal expressed by a Neyman-Pearson criterion, aiming maximize recall subject fixed FPR of 5%. This criterion represents trade-off between the cost of false positive cFp and the cost of false negative CEN errors. In bank account fraud prevention, the consequence of committing a false positive mistake, that is, rejecting legitimate application, must be weighed against the cost of a false negative mistake, that is, accepting fraudulent application. In Section 4.1 we state that our optimization goal is to obtain set of predictions that minimize the quantity
[Tlly 0 0: =1]+ Tyy 1 d: = =]]-
(19)
However, in our task, do not have access to the values of cpp and CEN- To apply misclassification cost re-weighting approach, must then oblain the value X which is equivalent to the error cost trade-off enforced by the Neyman-Pearson criterion. This will allow us to set cpr À and Cpx = 1.
According to Elkan 2001), we can establish a relationship between the idcal threshold of a binary classifier and the misclassilical ion costs. For a given instance, the ideal classification is the one that minimizes the expected loss. As such, the optimal prediction for any given instance æ; only if the expected cost. of predicting is less than, equal to the expected cost of predicting O, which is equivalent to:
(1-p)cpp S POEN
(20)
Where p = P(y = 1[x;). that is, the probability that x; belongs to the positive class. An estimation of the value of p is given by our Alert Model, in the form of the model score output for given instance, which is an estimate of the probability that v; belongs to class 1. In the case where the inequality is in fact an equality, then predicting either class is optimal. As such, the decision threshold t for making optimal decisions leads value for A:
t
(1 f)cpp tCpN
0.057
(21)
1-1
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
As the optimal threshold t for our Alert Model was chosen such that the Neyman-Pearson criterion is met, we now may plug the value of into this equation, obtaining the theoretical value d for our optimization goal. It has been shown by Sheng & Ling (2006) that this relationship between the cost-structure and the classifier's is threshold does not always hold in practice. Secondly, the value of obtained through this method depends on the classifier trained. different classifier would yield another value for the optimal threshold according to the Neyman-Pearson criterion, which would lead to different X, despite the task being the same. However, as our aim to test different cost structures, we will use this as the default value for X, testing the cost-structures À {^r/5, Ae, 5
B Classifier h
As detailed in Section 4.1 we model the classifier h with a LightGBM (Ke et al., 2017) classifier. The model is trained on the alerts raised by the Alert Model, ranging from months four to six of the BAF dataset. The model is trained by minimizing the weighted log-loss as mentioned in section 3.1 The choice of hyperparameters is defined through Bayesian search (Akiba al.) 2019) trials, with 200 start-up trials, with validation done the seventh month. We also test several values for the initial probability estimate of the base prodictor of the boosting model. This was done in order to test if introducing a bias lowards predicting fraud can be beneficial to our model, as across all scenarios, false negatives incur a higher cost. Given instance wise feature weights C₁ the default value Ginitial of the initial estimator's prodiction
extensive grid, for 300 total
Eiy G3/
Id = logit
(22)
When training the model, run the typer-paramette search independently for Ginitial E {gd. Ja + 0.2, 9d 0.4,... Id 2}. The optimization objective is to minimize the weighted log-loss.
In a first series of experiments, we found that a low number of estimators and maximum depth resulted in the best results. As such, in our first thorough prper-paraeeter search, we use the parameter space represented in Table
Table 6: ML Model: LightGBM hyperparameter search space
HYPERPARAMETER VALUES OR INTERVAL DIST.
boosting_tyoe enable bundle bunll_bbudle
"dart"
Talso,Trud,
estimators
[50,250]
Unif.
max max_deptl depth
[2,5]
Unif.
num num_leaves
[100,1000]
Unif.
min ciin__hhld__ag Shhidd_samples
[5,200]
Unif.
learning rate reg alpha
[0.001, 1)
Unif.
[0.001,
Unif.
reg lambda.
00.001,
Unif.
In this set of experiments, across all scenarios, LGBM classifiers with a maximum tree depth of 2 achieved the best performance. As such, we conducted a second experiment, consisting of a total of 1700 trials, with 1500 startup trials, with the same hyperparameter space detailed in Table but fixing the maximum depth parameter at. 2.
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
C Synthetic Experts
C.1 Expert Desiderata
Feature and AI assistant dependence When a decision is made by an expert, it is assumed that they will base themselves on information related to the instance in question. Therefore, we expect experts to be dependent on an instance's features in order to make an informed decision.
However, in some real world deferral systems (Amarasinghe et al. 2022 De-Arteaga et al. 2020), the in- stance's s features are accompanied by an Al model's score, representing the model's estimate of the probability that said instance belongs to the positive class. The aim of presenting the model score to expert is to provide them with extra information, well as possibly expediting the decision process. It has been shown that, in this scenario, expert's performance can be impacted by presenting the model's score when deferring a case to an expert (Amarasinghe el al. 2022 Dr-Arlaa el. al., 2020: Levy el al. 2021 Note that, in this setting, a classifier trained for our task exists independently of the assignment system implemented. This approach is applicable to deferral systems such as the one proposed by Raghu et al. (0199a), who argue for the training of separate human behaviour prediction model (Raghu al.] 2019a b). In the L2D framework, however, the main classifier is trained jointly with the deferral system. As such, this framework allows users to generate experts with without dependence on separate ML classifier's score.
AI assistance and algorithmic bias Should the generated experts use Al assistant, expect experts not to be in perfect agreement with the model, due to the assumption that humans and models have complementary strengths and weaknesses (De et al. 2020 Dellermann et al.) 2019]. As such, we would assume humans and AI perform better than one another separate regions of the feature space, enabling an assignment system obtain better performance than either the expert team or the model on their own. The degree of "model dependence", or "algorithmic bias' (Alon-Barkat & Busuioc 2023 varies
between humans, as measured by the model's impact on a human's performance (Jacobs al. 2021 Inkpen
et al. 2022 As such, our synthetic expert team may also exhibit varying levels of dependence the model score.
Varied Expert Performance In order for our team of experts to be realistic, it is important that these exhibit varying levels of overall performance. Experts within a field have been shown to have varying degrees
of expertise, with some being outperformed by ML. models (Goel et al. 2021 Gulshan et al. 2016). As such human decision processes can be expected to be varied even amongst a team experts.
Predictability and Consistency It a common assumption that, when making decision, experts follow an internal process based on the available information. However, it is also known that highly educated and trained individuals still subject to flaws that inherent to human decision making processes, of these being inconsistency. When presented with similar cases, at different times, experts may perform drastically different decisions (Danziger et al. 2011 Grimstad & Jergensen, 2007). Therefore we can expect human's decision making process not to be entirely deterministic.
Human Bias and Unfairness It is also important to consider the role that the assignment system can play in mitigating unfairness. If an expert can be determined to be particularly unfair with respect to a given protected attribute, the assignment system can learn not defer certain cases to that expert. In order to test the fairness of the system as whole, is may be useful create team comprised of individuals with varying propensity for unfair decisions.
23

---

Published in Transactions on Machine Learning Research (07 (07/2024)
C.2 Expert Parameter Sampling
As stated in section we define the expert's probabilities of error, for a given instance, as function of pre-processed version of its features ã, and the Alert Model's score (a:),, given by
P(mza = 1/y: = 0, x;)= = o 30 a w F (mn,)
v Vw-++wk
(23)
P(mya 0/u: 1; = o [1 w w. BeBBttwa (w).)
Where a denotes sigmoid function. Each expert's probabilities of the two types of error are parameterized by five parameters: S0 $1, a, w and WM. The sampling process of each parameter is done as follows:
Feature Dependence Weights Generation To define w for a given expert, we sample each component from a 'Spike and Slab' prior (Mitchell & Beauchampl 1988). A spike and slab prior is a generat ive model in which a random variable u either attains some fixed value v, called the spike, or is drawn from another prior Pslah, called the slab. In our case, we set v = 0. That is, u is either zero, or drawn from the slab density N(0, 1). To sample the values of w we first sample a Bernoulli latent variable Z ~ Ber(0.3) to select if w, is sampled from the spike the slab. If Z 0, uy attains the fixed value 0, if Z = 1, wy is drawn from the slab density Pslab- As such, the spike and slab prior induces sparsity unless 0 = 1, allowing for the generation of experts whose probabilities of error are swayed by a varying nunber of features. The distribution of WM (1-2, 0.5) is defined separately to ensure all experts have some degree of model dependency. We also scparately define Up ~ (.-,0..1), with representing the protected attribute, that is, the customer's age; so that all experts have a degree of unfairness in their simulated predictions, however, an exploration of fairness was not conducted in this work.
Controlling Variability and Expert's consistency While the weight vector controls the relative influ- ence that each feature has on the probability of error, parameter a, in turn, controls the global magnitude of this influence. For 0, the probability of error would be identical for all instances. In turn, for very large a, the probability would saturate at the extremes of the codomain of the sigmoid function, 0 or 1, resulting in deterministic decision-making process. We chose a ~ N44 0.2) so that a wide variety of probability of errors exist throughout the feature space.
Controlling Expert Performance In any cost-sensitive binary classification task, the metric used to evaluate the performance of a decision-maker is the expected misclassification cost E[C].
For our setup to be realistic, we assume an expert's decisions must, on average, incur a lower cost than simply automatically rejecting all flagged transactions. Otherwise, assuming random assignment, having that expert in the human Leam would harm the performance of the system as whole. We assume that E[C], must be, at most, 70% of the cost of rejecting all applications. We also desire to have balanced distribution of expert performances. It is important that the humans in the system are not much worse on average than the classifier h, or clse there could be no benefit to having humans in the decision-systen as the cost of having the model predict on every instance is much lower than employing a team of human experts. The classifier l can be trained before we even generate our human predictions, thus allowing us to know its average performance E(CT,, ahead of time. As such, we sample the target value of E[C] for each expert from Tejoi (E([C)n, O.2))C),), to ensure that the average expert performance is evenly distributed around the average performance of the classifier h. Should Te(C), be larger than 70% of the cost of rejecting all alerts, the value is set to said value.
In Section
4.1 we demonstrate that the expected cost resulting from an expert j's decisions is approximated
by
1 N E(C,, ~ N
[AP 1 = O)P(y: = 0)
(24)
1=1
+ +P(mz 0/y 1)P(w; 1)),
24

---

Published in Transactions on Machine Learning Research (07/2024)
Assuming that the values of w, a and WM are set, the value of B[C], will only be function of 90 and B1 We must then be able to calculate values of 5o and 81 so that we obtain E[C],- = T Let us simplify the above equation
where P(my, = 1ly, 0) and P((m,,, 0lys 1) are given by Equation 23
N P(my, = 11y =0]
1
TE = XP(1) = 0)
N 1m1
(25)
+ P(y =) 1
Σ
P(my, 0ly: 1)]
N
X(1 - Ply = 1)) FPR; + P(: = 1) FNR,,
where, assuming all other parameters have been sampled, FPR; (86) and FNR, ( There is a degree of freedom in this equation: if we choose the value of either 30 B1, the other variable must attain a specific value so that the target E[C] is achieved. Inverting the expression, we obtain:
Tyjo
P(y, = FNR,,
FPR, = X(1 - (11 = 1)) X(1 P(y 1))
(26)
Mcaning that any pair of target values for FPR, and FNR, that obey the above equation will yield Te[c), Having sampled TE(C,, we sample a random value of TFNR, such, and calculate TFPR, according to the above expression, ensuring both values belong to the interval jo, 1[- We then calculate the value of 30 and
31 to obtain the desired FPR and ENR. From Equations
23 we know that
FPR,(81) N Eo
W.m: ||w||
(27)
Note that, for notational simplicity we set wMs = but the result is similar when was * 0. is then possible to show that the function (3) is monotonically increasing,
0FPR,
1 051 N
Σ o (8
w. 2 (1-0)(11))0 + ||>|| ) > 0 for TR.
(1-
(28)
Since the function is monotonically increasing,and in a bounded to the interval J0, 1[, then, for any target false positive rate T FPR, then the following equation has an unique solution:
(P) TFPR for TEPr 10,
(29)
A similar reasoning applies for Bn and T FNR. Finally,
control an expert's FPR and FNR by solving
these equations for 31 and fo. To solve these equations, partition of the datasct (month 7) is utilized to
calculate the empirical value for each rate, meaning that deviations from the target performance metrics may
occur due to temporal distribution shifts. Due to the monotonous nature of the function, and the uniqueness of the solution, we solve it through a bisection method (Burden & Faires 1985).
Feature Preprocessing For our simulation of experts, the feature space is transformed as follows. Nu- meric features in X are transformed to quantile values, and shifted by -0.5, resulting in features with values in the |-0.5, 0.5] interval. This ensures that the features impact the probability of error independently of their original scale. Categorical features are target-encoded, that is, encoded into non-negative ive integers by ascending order of target prevalence. These values are divided by the number of categories, so that they belong to the [0, 1] interval, and shifted so that they have zero mean.
C.3 Expert Properties
In this section we display and discuss some key propertics of the expert teams generated for our experiments. As mentioned in Section 4.1 one team was generated per {a,, A} pair, resulting in 6 distinct teams of 9 synthetic fraud analysts.
25

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Normalized Weight Vector Heatmap
1.00
standard@0
standard#1
0.75
0.50
standard#3
standarda4
0.000
standarda5
-0.25
standardd6
-0.50
0.75
standardag
prev address months count current, antrnes, months, count
name simil similarity
income
custontr age
days sirce, request
Intendeed balcon smouunt
payment rype zip. count, daw velecity 5n velocity 24h valocity Aw bank branch count the date birth distictt email dw employment sratug creait, relf, score email is, frae
housing status
phone home, vald
phtore moblle valid
bank months court
has other Carde
proposed creait jime
foreign request
asssion langth in minutes
source
device as keep alive susson device distect, emolls Sw devite froud count
model score
Figure Weight Vector Heatmap for each Expert Experts maintain feature weights across all testing scenarios.
Feature Dependence To ensure that the deviations between testing scenarios are mainly due to the differences in alert rate a, and cost-structure, we generate all expert teams with the same feature weights so that their probabilities are similarly swayed by each instance. The normalized weight feature vector is represented in the heatmap in Figure demonstrating the variety of decision processes within our team.
Expert Performance In the previous section, we derive
Plu 1) FPR, X(1 P(y = 1)) A(1 - Ply = 1))
B[C],
FNR,,
(30)
which allows to establish relationship between the expected misclassification cost for given expert's decisions E[[];; and their respective FPR, and FNR.. Let consider the "Full Rejection" approach, where we reject all the alerts. If we substitute the expected cost of rejecting all the alerts into Equation 30 we obtain the set of FPR and FNR R values that result in the same cost. in the lest split. As such, in a plot where the x axis is the FNR and the y axis is the FPR, all possible combinations FPR and FNR leading to the same misclassification cost as rejecting all the alerts are represented by a single line. We exemplify this in figure G where the green area represents the combinations of FPR and FNR corresponding to lower expected misclassification cost, while the red area represents higher misclassification cost.
Note that [[[C], only influences the intercept, meaning that lines parallel to the one represented in Figure G represent set of (FPR,FNR) combinations with the same value of E[C), In Figure 7 we represent the distribution of FPR and FNR within the test split. for all the generated expert teams. We also plot the performance of classifier h (assuming 100% of training data availability) within the test sct, demonstrating the relative performance of each decision-maker.
Decision-Making Complementarity In this work, we assume that we stand to gain from combining the decision-making capabilities of various experts and a ML classifier. For this assumption be true, we need to ensure that there are regions of the feature space where the probability of correctness of given expert surpasses that of other experts, and vice versa. In Figure & we represent, on heatmap, the fraction of total cases where the
in the row is correct, while the expert expert/classiffe /classifier in the column is incorrect. There is a significant number of instances, for all scenarios, where a given expert/classifee classifier is a better choice than another decision- moeii,,n-makkr, meaning that this testbed appropriate to test L2D methods.
26

---

Published in Transactions on Machine Learning Research (07 (07/2024)
FNR vs FPR Test
1.0
Cost of Full Alert Rejection S 0,048
0.8
0.6
FPR
0.4
0.2
0.0
0.0
0.2
0,4
0.6
0.8
1.0
FNR
Figure FPR FNR Full rejection performance within the alerts. Red line represents combinations of (FPR,FNR) that result in the same cost as rejecting all instances
0.-0.05, A 0.0114
a, 0.05, A- 0.057
a, - 0.05, 105.A-0.200 -02285
Experts
Coot of Full Aeett Revver Sion 0,0
Experts
Cost Full Rejection 0.240
Experts
0100 Jetpniingg uay IF3
0.6
ERR
0.0
0,0114
0.057
0.285
1.0
Cassitier
Cust at Full Alrrt Rejection 0.053
Casssierr
Cost of Fub Alert Sejuctio 0.265
Classifeer
FI And Reg Pndfectiio 011
0.4
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.8
1.0
0.0
0.2
0.4
1.0
FNR
Figure 7: Expert and Classifier performance plots for each {ar, A} pair
27

---

Published in Transactions on Machine Learning Research (07/2022 /224)
a, 0.05. 0.0114
Class. h-
Experte1
Experte2
Expertee
Experes
1.0
Experte
Experee7
0.8
Expemei
0.15,
0.6
Class
Expemed
0.4
Expemel
Expert>2
0.2
Experee
Experted
0.0
Expertet
Expertat? 7
Experees
Class
Experteo
Experte
Exparte2
Experte3
Export t4
Export es Emertes el Experte?
Experaa
Class. Experte0
Emeree1
Experta
Experted
Exportes
Expert ee Emerte? Expettea
Class
Experteo
Experte1
Emerte2
E
Exporte4
Expertes
eeportes
Export
Emprtes
Figure 8: Fraction of instances in which ROW is correct and COLUMN is incorrect.
28

---

Published in Transactions on Machine Learning Research (07 (07/2024)
D OvA Classifiers and HEM
We model both the OvA Classifiers and the HEM with LightGBM (Ke et al. 2017 classifiers. These models are trained on the alerts raised by the Alert Model, and the corresponding expert predictions, ranging from months four to six of the BAF dataset. Both methods are trained by minimizing the weighted log-loss. The choice of hyperparameters is defined through Bayesian search (Akiba et al., 2019) on an extensive grid, for 120 total trials, with 100 start-up trials, with validation done on the seventh month. The hyperparameter search space is detailed in Table 7
Table 7: LightGBM hyperparameter search space: OvA Classifiers and HEM
HYPERPARAMETER VALUES OR INTERVAL DIST.
boosting lype
"dart"
enable bundle estimators
[False, True]
[50,250]
Unif.
max depth
[2,20]
Unif.
num leaves
[100,1000]
Log.
min child samples
[5,100]
Log.
learning rate
[0.005, 0.5]
Log.
reg alpha
[0.0001, 0.1]
Log.
reg lambda.
[0.0001, 0.1]
Log.
29


Reviews:
[{"review_id": "wipLqtoMeC", "reviewer": "Reviewer_UTzZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper presents DeCCaF, a novel framework designed to defer decisions to humans within the cost and capacity constraints, addressing practical challenges in the field of Learning to Defer (L2D). DeCCaF utilizes supervised learning to model human error probabilities using one expert prediction per instance. It employs constraint programming to minimize error costs while considering workload limitations. Empirical evaluation with nine simulated fraud analysts showcases DeCCaF's superiority over existing L2D methods, resulting in reduced misclassification costs across diverse scenarios.\n\n\n### Weaknesses/Questions\n\nThe method requires data labels $m_{j,i}$ for each data point $i$ and each expert $j$. It may be impractical to get labels for all data points from each expert. The method has multiple steps and it is not clear how they are executed in sequence. My current understanding is that first $h^*$ is learned then $r^*$ and then the allocation matrix $A$. Shouldn’t the allocation be taken care of while learning $r^*$ itself? Do the allocation, accuracies, and costs get revised after each instance, e.g. the available capacity of an expert will be reduced after making a prediction?  What are the trade-offs between the overall performance and the cost, capacity, and accuracies of the experts, and classifiers?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is based on Learning to Defer (L2D), which aims to enhance human-AI collaboration systems by deferring decisions to humans when they are more likely to be correct than a machine learning (ML) classifier. Existing L2D research overlooks practical challenges such as cost-sensitive scenarios, concurrent human predictions, and human work capacity constraints.\n\nTo address these challenges, the paper proposes the deferral under the cost and capacity constraints framework (DeCCaF). DeCCaF employs supervised learning to model human error probabilities with less restrictive data requirements and uses constraint programming to minimize error costs while considering workload limitations. The authors test DeCCaF in cost-sensitive fraud detection scenarios with teams of synthetic fraud analysts and an ML classifier.\n\nThe paper provides a new benchmark of complex, feature-dependent synthetic expert decisions in a realistic financial fraud detection scenario. Through empirical testing in various cost-sensitive fraud detection scenarios, the paper demonstrates that DeCCaF outperforms existing L2D baselines in terms of average misclassification costs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes. \n\n1. Please add an algorithm block, detailing the step-by-step procedure. \n\n2. Clear definitions of capacity constraints, $H$, $b$, $A$, etc. with some examples would be helpful. Please have a summary of the notations in the paper.\n\n\n3. Minor notation nitpick, commonly the optimal solution ( Bayes optimal solutions) are denoted as $h^*, g^*,$ etc. and the solutions of the empirical problem are denoted as $\\hat{h}$, $\\hat{g}$. The paper uses $\\hat{P}$ but it is defined on $h^*, g^*$ which are the empirical solutions in the paper. This leads to confusion. \n\n\n4. The rejector function $r: \\mathcal{X} \\to \\\\{0,1,\\ldots, J\\\\}$, is rather an assignment/deferral function. Could you please clarify or update it.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CroFmMXwxs", "reviewer": "Reviewer_N8kV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The problem of learning to defer has huge practical relevance, and deferral to multiple experts with budget constraints is a natural and realistic setting to look at.\n- Unlike the setting with a single expert (Mozannar & Sontag, 2020), its not immediate how one may modify existing approaches to incorporate budget constraints on multiple experts.\n- The idea of solving a constraint program on estimates of classifier and expert accuracies is a simple and effective idea; in fact such post-hoc methods have proven to be effective in other learning to reject/defer settings.\n\n**Weaknesses:**\n- The main contribution is setting up a post-hoc constraint program to assign experts to samples. I do not consider the inclusion of asymmetric misclassification costs or allowing for limited expert annotations to be significant contributions. Given this, the paper's novelty seems limited.\n- The experimental evaluation is also limited, in that all the comparisons are done on a *single* binary classification dataset. \n- The formulation of the constraint program in (7) is confusing to parse through. The objective in (7) simplifies to:\n$$\n\\sum_{i=1}^n \\sum_{j=1}^J \\mathbb{P}(correct | x_i) \\cdot A_{ij} = \\sum_{i=1}^n \\mathbb{P}(correct | x_i) \\cdot \\sum_{j=1}^J  A_{ij} = \\sum_{i=1}^n \\mathbb{P}(correct | x_i), \n$$\nwith the optimization variables $A_{ij}$ summing up to 1 and vanishing from the objective. Moreover, the definition of $\\mathbb{P}(correct | x_i)$ is confusing as it uses an assignment function $a(x)$ (see (6)) which is never explicitly included in the optimization problem in (7).\n- The experiments are all on LightGBM models, due to which the authors are not able to jointly train the classifier $h$ and human expert model with shared parameters. In contrast, Verma et al. model both $h$ and the defer logits using the same neural network backbone, but different prediction heads. This allows for joint training with shared parameters; consequently, one could expect the learned logits to generalize better. \n\n**Typos / Presentation**\n- Page 5: $g_\\perp: \\mathcal{X} \\cup \\\\\\{1,\\ldots, J\\\\\\} \\rightarrow \\mathbb{R}$. Change to: $g_\\perp: \\mathcal{X} \\times \\\\\\{1,\\ldots, J\\\\\\} \\rightarrow \\mathbb{R}$\n- Page 5: $\\psi^{-1}(gg(x))$. Change to $g(x)$\n- Sec 3.4: This section was difficult to follow. Specifically, it wasn't clear how it relates to the content in the previous sections. For example, how would the inclusion of misclassification costs change the Bayes-optimal solution in (2). I think the authors are trying to say that the inclusion of costs is equivalent to computing the solution in (2) with an altered distribution. However, this wasn't entirely clear from the discussion in Sec 3.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the learning to defer setting, a classifier is allowed to defer to an expert on samples it deems to be difficult. Recently, Verma et al. (2023) provide a learning algorithm for a variant of this setting, where the classifier has multiple experts it can defer to. The present paper builds on this prior work and considers applications where there is a capacity constraint on each expert. \n\nThe proposed solution assumes access to a dataset labeled with expert predictions, and requires learning a classifier and a separate model for predicting the accuracy of each expert. During inference on a batch of sample, the authors propose solving a *constraint program* to assign experts to instances where the classifier ought to defer. This approach additionally accommodates asymmetric costs for misclassification errors, and does not require each training sample to be annotated with predictions from all the experts.\n\nOn a bank fraud detection, the authors demonstrate the efficacy of their approach over that of Verma et al. (which they alter using a heuristic to accommodate capacity constraints).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some adjustments the authors could do to improve the quality of the paper include:\n- Present experimental comparisons with more datasets (preferably with those used in prior papers on L2D), and with different models (e.g. neural networks). Explore multi-class datasets too.\n- Evaluate the efficacy of their approach across a range of capacity constraints. In settings with a single expert, one could vary the capacity for the single expert, and plot deferral curves that trace accuracy as a function of capacity (see e.g. Verma & Nalisnick). With multiple experts, such a clean evaluation may be difficult. The authors could find a single parameter to vary, such as the sum of capacities or the skewness in capacity amongst experts, and trace out the resulting accuracies as a function of this parameter. Deferral curves like this would allow us to visually compare different methods across a range of capacity constraints.\n- Fix the notational issues with the constraint program formulation in (7)\n\nHaving said this, given that my main complaint is about novelty, I am not sure if making the experimental evaluation more comprehensive alone will suffice for acceptance.\n\nHere's a related direction that would make the paper more compelling. I do not expect the authors to pursue this idea for this submission cycle:\n- It would be interesting to incorporate the capacity constraints into the training objective instead of at inference time. You could e.g. consider the OvA objective of Verma et al. (equation in Section 4.2). Using a Lagrange formulation, the capacity constraints could then be translated into \"weights\" on the individual expert loss terms. The challenge would then be to choose these weights in a efficient manner, so as to satisfy the capacity constraints.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SJvNQNBHkl", "reviewer": "Reviewer_zmkA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Idea is intriguing.\n- Experimental result are promising.\nWeaknesses\n- it is not clear the theoretical background. How this positive results can be supported with a theoretical justification?\n- it is not clear the challenges of the work especially for (i) and (ii) where quite standard approaches can be leveraged", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper proposes the deferral under cost and capacity constraints framework extending the current works on learning to defer by\n(i) dealing cost-sensitive scenarios, where type 1 and type 2 errors have different costs;\n(ii) not requiring concurrent human predictions for every instance of the training\n(iii) dealing with human work capacity constraints", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discuss\n- the theoretical background. How this positive results can be supported with a theoretical justification?\n- the challenges of the work especially for (i) and (ii) where quite standard approaches can be leveraged", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Suggested changes for final version:
- Figure 1, consider moving to top of page 2 rather than having inline in Abstract
- notation, is there a reason for $y$ to not be italicised?
- pg 4, typo: "depend on ithe features"
- Equation 1, make the LHS $L_{01}(h, r)$ to specify the variables of interest
- Equation 2 and elsewhere, align "if" and "otherwise"
- Equation 7, split the constraints onto two separate lines
- Equation 7, please comment on the computational complexity of the suggested CP-SAT solver.
- pg 7, is there a need to introduce a new symbol $k$ in defining the cost-sensitive loss?
- Algorithm 1, suggested to split the Training and Inference algorithms into two
- Algorithm 1, not ideal to overload "Train" for fitting of both $\hat{g}$ and $\hat{g}_{\perp}$. More precise equation references in Section 3.1 are advisable.
- Algorithm 1, specify the size of $\mathbf{H}$
- Algorithm 1, give a comment specifying the meaning of $\mathbf{A}$
- Algorithm 1, line 9 and following, $i \leftarrow \\{ 1, \ldots, J + 2 \\}$
- citations to the MoE literature are potentially relevant, as these also consider expert load, albeit for slightly different reasons. e.g., Zhou et al., Mixture-of-Experts with Expert Choice Routing, NeurIPS 2022.
- there are more relevant citations for multi-expert deferral, e.g., Mao et al., Two-Stage Learning to Defer with Multiple Experts, NeurIPS 2023.
- use $c_{\rm FP}$ instead of $c_{FP}$
- Equation 12, use of braces is not necessary

- if possible to obtain results on even a single task that was used in prior work on learning to defer to multiple experts, it would strengthen the paper.
Decision: Accept (poster)