Probabilistic Learning to Defer: Handling Missing Expert Annotations and Controlling Workload Distribution
Authors: Cuong C. Nguyen
Thanh-Toan Do
Gustavo Carneiro
Venue: ICLR 2025 Oral (2025)
Keywords: learning to defer
expectation - maximisation
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Recent progress in machine learning research is gradually shifting its focus towards *human-AI cooperation* due to the advantages of exploiting the reliability of human experts and the efficiency of AI models. One of the promising approaches in human-AI cooperation is *learning to defer* (L2D), where the system analyses the input data and decides to make its own decision or defer to human experts. Although L2D has demonstrated state-of-the-art performance, in its standard setting, L2D entails a severe limitation: all human experts must annotate the whole training dataset of interest, resulting in a time-consuming and expensive annotation process that can subsequently influence the size and diversity of the training set. Moreover, the current L2D does not have a principled way to control workload distribution among human experts and the AI classifier, which is critical to optimise resource allocation.  We, therefore, propose a new probabilistic modelling approach inspired by the mixture-of-experts, where the Expectation - Maximisation algorithm is leverage to address the issue of missing expert's annotations. Furthermore, we introduce a constraint, which can be solved efficiently during the E-step, to control the workload distribution among human experts and the AI classifier. Empirical evaluation on synthetic and real-world datasets shows that our proposed probabilistic approach performs competitively, or surpasses previously proposed methods assessed on the same benchmarks.

Published as a conference paper at ICLR 2025
PROBABILISTIC LEARNING TO DEFER: HANDLING MISSING EXPERT'S ANNOTATIONS AND CONTROL- LING WORKLOAD DISTRIBUTION
Cuong Nguyen
Thanh- Toan Do Department of Data Science and AI Monash University, Australia
Centre for Vision, Speech and Signal Processing University of Surrey, United Kingdom
Gustavo Carneiro
Centre for Vision, Speech and Signal Processing University of Surrey, United Kingdom
ABSTRACT
Recent progress in machine learning research is gradually shifting its focus to- wards human-Al cooperation due to the advantages of exploiting the reliability human experts and the efficiency of AI models. One of the promising approaches in human-Al cooperation is learning to defer (L2D), where the system analyses the input data and decides make its own decision defer to human experts. Al- though L2D has demonstrated state-of-the-an performance, in its standard setting, L2D entails severe limitation: all human experts must annotate the whole train- ing dataset of interest, resulting in time-consuming and expensive annotation process that can subsequently influence the size and diversity of the training set. Moreover, the current L2D does not have principled way to control workload distribution among human experts and the Al classifier, which is critical opti- mise resource allocation. We. therefore, propose new probabilistic modelling where the Expectation Maximisa- address the issue of missing expert's annotations.
approach inspired by the mixture-of-experre tion algorithm is leverage Furthermore, we introduce a constraint, which can be solved efficiently during the E-step, to control the workload distribution among human experts and the Al classifier. Empirical evaluation on synthetic and real-world datasets shows that our proposed probabilistic approach performs competitively, or surpasses previously proposed methods assessed on the same benchmarks.
INTRODUCTION
Human experts have the remarkable ability learn concepts and make accurate predictions. However, in many applications where the volume of data scales up exponentially, like radiology (The Royal College of Radiologists, 2023), relying solely on human experts becomes economically im- practical or even infeasible due to significant increases in hiring costs and processing time. In contrast, machine learning or Al models excel processing large amounts of information but may be prone to biases. Thus, an ideal system would therefore consist of neither human experts nor AI models in isolation but rather a judicious combination of both, leveraging the reliability of humans and the efficiency of Al models.
Such an ideal system must be capable identifying the strengths and weaknesses each human expert and Al model, so that samples can be assigned to decision-makers with high predictive ac- curacy at low processing cost. Learning to defer (L2D) (Madras et al., 2018) is one of the learning paradigms that provide such capability. L2D extends learning to reject (Chow, 1957; Cortes al., 2016) by deferring samples to human experts when the AI model is not confident. Although L2D surpass the performance of human experts or the Al model alone, the learning paradigm has notable limitation: every human expert must annotate every sample within the training dataset. This requirement places substantial burden the human experts involved, resulting in resource-

---

Published as a conference paper at ICLR 2025
intensive and cimecconu annotation process. Consequently, this may reduce the size of the training set, in addition to decreasing the diversity of labelling and potentially increasing biases due to the probable involvement of fewer annotators.
Another problem in L2D is the lack of workload control, where some experts (either human or AI model) more frequently queried to make decisions, while the remaining experts entirely ig- nored. Although solely relying on a few human experts is desirable in some applications where rewards are proportional the amount of work (e.g., crowd-soureing), it becomes unfair in fields like radiology, where payment is fixed. In such cases, the more accurate the radiologist is, the more work they are burdened with without corresponding compensation. Moreover, in those high-stakes applications, overloading one expert can easily lead to fatigue, increasing the risk of errors, and resulting in potentially poor life-hnd-death decisions. One typical example is reported in (Berlin, 2000), where a radiologist misdiagnosed because he interpreted 162 cases in one day while the average interpretation was only 50 cases. One workaround solution for the imbalanced workload distribution is to integrate a specific cost for each expert (e.g., proportional to salary) to balance out the workload (Narasimhan et al., 2022; Zhang et al., 2024). Such a solution is impractical because human experts will most likely be paid similar salaries, making the quantification of cost per pert infeasible. Furthermore, our preliminary experiments have shown that such a naive solution may need many iterations to fine-tune the cost to converge to the desirable workload distribution or may not converge at all in several cases. Furthermore, as L.2D trades off the reliability of human experts for the efficiency of AI classifiers, it is crucial to have the ability to control the workload distribution between AI and human experts to study that trade-off. Despite such importance, con- trolling workload is, however, overlooked in the L2D literature. The current practice is employ a post-hoc method by sorting the deferral score of a trained L2D method evaluated on the test set and varying the threshold to obtain different deferral rates for the AI model (Mozannar al., 2023). Such post-hoe approach is impractical because it requires access to all test samples before making a prediction, and it does not allow for workload control to be set during training and tested during evaluation. Hence, this prompts further studies to develop a principled mechanism to control the workload distribution in L2D.
We, therefore, propose a probabilistic framework inspired by the mixture-of-experts with constraints to address both problems: handling missing expert's 's annotation and controlling workload distribu- tion. Treating the missing expert's s annotation latent variable enables to formulate the pro- posed framework with the Expectation Maximisation (EM) algorithm. In addition, the constraints introduced control the workload distribution among experts and between AI and human experts yield an efficient solution, and hence, can be integrated seamlessly into the EM algorithm without compromising computational efficiency. Our contributions can be summarised follows:
We propose a new probabilistic modelling approach for L2D with missing expert's 's annota- tions and leverage the EM algorithm to optimise it. Such solution extends the capacity of L2D to practical settings where each expert annotates only portion of the training dataset.
We also introduce a constraint in the optimisation of the E step. which can be solved effi- ciently, to control the workload distributed to each human expert and the Al model.
The empirical evaluation on both synthetic and real- world datasets at different missing rates demon- strates the capability of our proposed method when dealing with missing expert's 's annotation settings, while being able to control the desired workload for human experts and the AI model.
2 BACKGROUND
x
Learning to defer (Madras et al., 2018) is mo- tivated from the observation that in certain see- narios or datasets, a machine learning model trained on ground truth cannot perform on- par with human experts. Hence, L2D is pro- posed by integrating deferral or gating mech- anism to query either a human expert or an AI classifier to make decisions. In other words, L2D is similar to a mixture of M human ex- perts and an Al classifier. The AI classifier is
N
Figure 1: L2D is modelled as mixture of M human experts and an Al classifier. The human expert's 's an- notation is either observed (standard L2D) hidden (our setting).
2

---

Published as a conference paper at ICLR 2025
learnt from ground truth data support the classification for the easy examples, reducing the cost of querying human experts.
A C-way classification task in LDA consists of dataset D = (f(x;, Vr. (t. (m) ) 1) where mm
X, E X S IR is an input sample, E E = {l..... . C} is the corresponding ground truth label, and t (m) E y represents the annotation of sample x; made by an expert indexed by m. In addition, an AI classifier, parameterised by fM--1 is introduced as learnable expert.
We follow the mixture of experts to model the L2D problem. The training data of L2D can be modelled as follows:
1. draw a sample from data distribution: x~ Pr(x).
2. draw an annotation from each expert: ((m) ~ Pr(t/xx, 0m) = Categorical(((/((())))
3. draw categorical variable to select expert: Pr(z|x, 7) Categorical(zl(((;>>)
4. draw a label from the belief of that expert: y Pr(y|z, t) Categorically
where: ((i)) X + {l,..., M + 1} is the gating model, and ((:0mm) X + Ac-1 {v [] v 1 13 classification model representing an expert (including the Al classifier) indexed by m E fl..... M + 1}. Note that in this setting, every sample in the training set is annotated by all experts, where the function representing each human expert f(:: mm) simply mapping table that returns an annotation t (m) given an input x,, with (m)) fx, m. Vm E {1..... 3.. The data modelling can also be illustrated through the graphical model in Fig. 1.
The objective of L2D is to learn the gating model's s and AI classifier's parameters by maximising data-likelihood on observed data, which can be written as follows:
m EN In Pr
M+1 (m)) Yos II m= t/
Xx, Y, (0) M+1
mmm))
maxx.004 EN In Pr Ye Xi, M t. ,7)
(mm))) + In Pr Pn
((M+))) x2, 0x.11.+)
The first term on the right hand side of Eq. (1) is to train the gating model, parameterised by y. while the second term is used to train the classifier. The annotation of the Al classifier, therefore, has to be modified accordingly: in the first term, t is the prediction of the Al classi- fier: (M+1) f (x,; 011+) so that the gating model can assess the performance of each expert (including human and the AI classifier) to defer, while in the second term, it is the ground truth label: e! (M+1) y; in order to train the AI classifier. Such a modification is required because L2D has both un-learnable (i.e., human) and learnable experts (i.e., the classifier). Equivalently, one can remove the second term from Eq. (1) by training the classifier on ground truth labels separately, then considering that trained classifier as un-leamable human expert. In that case, Eq. (1) is simplified consist of only the first term, where the annotation t! ( does need be modified.
In this modelling, L2D is a latent variable model, where the latent variable is the index z of the expert being selected. Hence, the objective in Eq. (1) can be optimised by the EM algorithm as shown in Appendix A.1.
3 MISSING EXPERT'S ANNOTATIONS
In this section, we relax the assumption that every sample in the training set is annotated by all human experts, presented in Section 2. In particular, for each training sample, annotations made by some human experts are assumed to be observed, while the ones made by the remaining experts are unobserved (or missing).
For each sample, we denote Dobs, as the set of expert indices that annotate sample x,, and Dumobs, as the set of expert indices who do not annotate sample x,. The objective function originally presented in Eq. (1) that maximises log- log-liieldhood on observed data can be rewritten as follows:
max. x,
M+1 mm1
EN In Pr yis I Imeepaa. (m) x;, 7, {0)
m=1
M
(2)
3

---

Published as a conference paper at ICLR 2025
Posterior of expert selection
pr(z)x:m)
q(z)
input
gating model
g(:in)
E-step
gating model
annotations & ground truth
6(::01)
g
g
posterior of missing annotations
f(.:01a)
((M +
pseudo-complete annotations
f(::0M:))
expert's models
loss expert's 's models
M-step
Visualisation of the proposed probabilistic L.2D where the E-step infers the posteriors of expert selection and missing annotations, while the M-step maximises the "completed" data likelihood to estimate the gating function and Al model parameters and 0
Figure 2:
The objective function in Eq. (2) learns both the gating model parameter and the parameter of M experts plus the AI classifier, denoted by (0m) ) while the standard (or "complete-annotation I.2D Eq. (1) learns only the parameters of gating model and classifier.
Due to the latent variables and t(5), for Dunobs. it difficult to calculate the log-likelihood in Eq. (2). Thus, the EM algorithm is employed to optimise lower-bound of the objective function
Eq. (2). The "complete" data likelihodood in Eq. (2) for one sample can be written as follows:
Q: = g(x,, DeDonb (())
In Pr y1, ImEEDbb. t (m)) II t.
X1, (0m)
)
M+1
m=1
In Pr (y: 2s Immeppla M JEDumooss. {!!0
= g(x., IT DuDanbb
+ +Σ, DEDDnnob In Pr t: x 0,) - In Pr + Empppt In Pr t {(m))
X,, x2, 8m), (3)
where qz; II Dunubb.. denotes the posterior of the latent variables.
E step The posterior of both and (()) can be obtained through mean-field variational inference as follows:
q Zi II IEppnoto. (1)) ((z;) II JED 9 (I.0)) argmin, KL ((z,) II JEDDubbs. &
(t.
(1)
Pr Zi IIjeppaabr. ((00 X.. Vis meDpbe. ((m) (x),, (a!l) M++1 p
argmin KL (g(2))) 2x
+E IjDDmobb KL (((())) |Pr (1) Xi, o(x))
E (xx;)
((()))
In Pr yr Imeppre t
IISED
(4)
where (k) and ((k) denote the gating and expert model parameters at the k-th iteration.
The optimisation in Eq. (4) can be expanded further and solved by the fixed-point iteration method.
Please refer to Appendix B for the detailed derivation of the solution. Note that the log-likelihood
of the last term in Eq. (4) may lead numerical instability when t is hard label (i.e., when observed). our implementation, we smooth the observed annotations when calculating that log-likelihood to avoid computational instability issue.
4

---

perfect balanced work- Published as a conference paper at ICLR 2025
M step The posteriors of latent variables obtained in the E step allows the calculation of the com- plete data log-likelihooo in Eq. (3). In the M step, that completed data log-likelihood is maximised with respect to the parameters of interest y and follows:
((tt1) ( Jm=1 argmax, 18 1
Σ Giml Qi(7,0, (01)) j
N
Ex Ejeppooba E vice(i) In Pr (66) (/00xxx/ Xi, xx:0:)]
argmax, 7,10m 1
Emepy In Pr t; (m))
x; xm.mm) + Bn((s)) [In Pr(z(((,
(5)
Note that the term related to Pr(y in Eq. (3) is omitted in the M step because it is considered constant w.r.t. y and {0m m=l The whole training procedure can be visualised in Fig. 2.
Remark 1 The first and the second terms in Eq. (5) resemble the minimum entropy principle in semi-supervised learning for expert models parameterised by 1M (Grandvalet & Bengio, 2004). In particular, the first term is equivalent to minimising the entropy of the missing expert's s annotations, while the second term trains the expert models on observed annotations.
CONSTRAINING THE NUMBER OF ASSIGNMENTS IN THE E-STEP
A naive implementation of the EM algorithm often results in workload-imballnce, where one or a few experts are queried more frequently, while the remaining experts remaining idle. Due this nature, a naively-trained gating model would become biased toward the high-pprforming experts, resulting in a sub-optimal workload distribution among: (i) human experts themselves and (ii) human experts and the AI model, as mentioned in Section 1.
Ideally, in many applications where human experts are load among experts and the Al model can be expressed follows:
N EN Bi(,.) [z] 1 Σ )>))(())) M
(6)
where q is used to denote a constrained posterior to distinguish from the unconstrained posterior presented in Section 3. Note that this step happens between the E and M steps.
The workload balancing constraint in Eq. (6) is a special case, where all experts (including human and the Al classifier) are treated equally. To further generalise the control of the workload distributed to each expert, we propose the following workload constraint:
# Σ (2:) I Eus
where E., El E [0, M are hyper- pyper-parameters, and = is the element wise operator.
The constrained optimisation is integrated into the E step when all annotations are observed (stan- dard L2D) or considered as "second E step" when annotations are partially observed as presented Section Nevertheless, the constrained optimisation can be written as follows:
. argming KL (7(z,) [g" (24)], Vi E [I, s.t.: El 인 Ei Ō(24) 1 Eus
where (z,)) denotes the unconstrained posterior of z, and can either the true posterior M+1
Pr z; yi, (t. (m))) Jm= (0m))) in case of full-observed annotations (standard L2D) or
the approximate posterior qzz) obtained in Eq. (4).
The constrained optimisation Eq. (8) is also known as the I-projection in information geometry and can be solved efficiently through its duality (Graça et al., 2007). Readers are referred to Ap- pendix C for the detailed derivation of the solution. In general, the Lagrange multipliers obtained through the duality can be written as follows:
A A argmin A. 20 ATE, AT + EN" En, q² (x.))xpp( exp - -A) z, 1). (9)
This can be solved efficiently using the projected gradient solver. The constrained posterior of the latent variable then be obtained as follows:
q. (2:) x q² (z:) exp (- (A - -7) 2, -1).
(10)
5

---

Published as a conference paper at ICLR 2025
Please refer to Appendix D for detailed training procedure of our proposed method.
The controllable workload constraint proposed in this section not only allows to distribute the work- load more flexibly, but also facilitate the analysis of L2D, especially the accuracy coverage curve (see Subsection 5.2). Current practice when analysing accuracy coverage is to either:
vary the cost of selection for human experts and classifier (Narasimhan et al., 2022),
M-sepp + (1 - c)², where add another loss term in the M step:
the desired coverage value and Ç is weighting hyper-parammter,
use the prediction values from the gating model (Mozannar et al., 2023) to rank the test samples, and then plot the accuracy-coverage curve by adjusting the prediction threshold in a post-hoc fashion. (see Algorithm 2 in Appendix E).
The first two approaches above often require multiple attempts to fine-tune the cost per human expert or the hyper-parameter associated with the new loss, resulting in a time-consuming and resource-wastirr wasting training procedure. In experiments, we were unable to successfully train for some coverage values. In fact, we empirically observed that both of the approaches to constrain workload distribution are highly sensitive to pre-defined coverage values c. resulting in notably brittle performance. For example, when running on Cifar- 100 as described in Section 5 using the second method to constrain the coverage coverage, while setting Ç = 50 leads to a coverage of 1. Applying strategy similar to the bisection method to gradually narrow the range of Ç did not lead to successful convergence during training.
0.2, we observed that setting Ç 1 leads
The third approach of threshold adjusting for post-hoc plotting (Mozannar et al., 2023) requires human intervention after each test sample has been processed by the system. Hence, it is inconsistent with the L2D training, resulting in an unreliable evaluation. Furthermore, it needs to know the gating model estimates for all of the testing samples before performing deferral, which is impractical, especially for model deployment.
In contrast, our proposed workload constrained optimisation in the step effective in terms of distributing the workload both among experts and between experts and the Al model without fine- tuning any hyper-parameter. In addition, results in a train-test consistent L2D model, where the gating model makes decision at test time without post-hoc human intervention.
EXPERIMENTS
We evaluate our proposed method and the state-of-the-art methods on both synthetic (i.e., Cifar-100 (Krizhevsky, 2009)) and real-world (i.e., WhH-Chestrray (Majkowska et al., 2020), Chaoyang (Zhu et al., 2022) and MiceBone) datasets to benchmark L2D approaches.
For Cifar-100, we follow a similar setting as in (Hemmer al., 2023), which simulates two experts, each following an asymmetric label noise. In particular, each expert performs correctly on 10 super- classes, while making 50 percent of labelling mistakes on the remaining 10 super-classes.
For real-world datasets, we use the annotations made by the human experts associated with the datasets. For each dataset, we randomly remove the annotations made by each human expert at predefined missing rate to form the dataset with missing expert's 's annotations. In terms of human ex- perts, NIH-AO dataset has one expert who performs at 99% prediction accuracy; Chaoyang dataset has three experts, with one performing 91% accuracy. the other at 87%. and the last one 99%; while MiceBone has eight experts, each performing at approximately 80% to 85% prediction accu- racy. Note that for Chaoyang, we evaluate with the first two experts (excluding the best expert with 99% accuracy).
settings: one with all three experts and the other
All results are reported from the checkpoint obtained at the last iteration in each training. Please refer to Appendix F for further details on the datasets and hyper-parameters used. The full implementation can be found at https: //gi thub. com/ c/tpyennn//ppp//
5.1 BASELINES
Three baselines are used in the L2D evaluation with missing expert's annotations: "naive" (Madras al., 2018) and two variants of semi-supervised learning (SSL.) (Hemmer et al., 2023).

---

Published as a conference paper at ICLR 2025
expert
Classifier
Lix
FixMatch-1
Probabilistic
0.85
0.85
0.85
0.8
0.8
Accuracy
0.75
Accuracy
0.75
Accuracy
0.75
0.7
0.7
0.7
0.65
0.65
0.65
0.6
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Coverage
Coverage
Coverage
(a) Cifar-100 experts miss.
(b) Cifar- 100 experts 0.5 miss.
(c) Cifar-100 100m--000--00 experts 0.7 miss.
0.94
0.94
0.94
0.92
0.92
0.92
Accuracy
Accuracy
0.88
0.88
0.88
0.86
0.86
0.86
0.82
0.82
0.82
0.2
0.4
0.6
0.8
0.2
0.8
0.2
0.4
0.6
0.8
Coverage
Coverage
Coverage
(d) Chaoyang experts 0.3 miss.
(e) Chaoyang experts 0.5 miss. (f) Chaoyang 2 experts 0.7 miss.
Figure 3: Comparison of coverage accuracy curves between different L2D methods on a variety datasets, each at a different missing rate.
The naive baseline built from the original L2D method in (Madras al., 2018), but with the gating model being modified to repeatedly sample either a human or Al classifier until the selected one is able provide an annotation for that imput data, Such selection equivalent to setting the probability Pr(z|x, ?) of the missing experts to zero in our EM framework (see Appendix A.2).
For the SSL baselines (Hemmer et al., 2023), each expert is modelled as an independent SSL clas- sifier, where labelled data is represented by samples with observed annotations, while un-labelled data is the one with missing annotations. Although different SSL methods can be used to train each classifier to model each expert, for simplicity, we use MixMatch (Berthelot et al., 2019) and FixMatch (Sohn et al., 2020) because of their well-documented effectiveness dealing with SSL problems. Missing annotations are then replaced by labels predicted by the SSL classifier. Finally, the standard L2D method (Madras et al., 2018) 8) with the integration of workload constraint presented in Section 4 is trained on the "pscudo-complete annotation dataset.
Note that although other L2D methods that handle multiple experts can be used (e.g., multi- L2D (Verma al., 2023)), they do not provide mechanism control the workload distribution to study the trade-off in the ewverag-accuurayy
and hence, are omitted from our evaluation.
5.2 RESULTS
Because L2D trades off the reliability of human experts for the efficiency of Al classifier, such tradeoff is, therefore, analysed through the coverage accuracy curve. Coverage computes the percentage of test samples assigned and predicted by the Al classifier, while accuracy measures the final prediction of the L.2D system.
Fig. 3 shows the anverage-accuruc curves of the baselines and our proposed method on various datasets, each at different missing rates. Cifar- 100, our proposed probabilistic L2D outperforms the baselines at small coverage, while being on par larger coverage values. The SSL baseline per- forms poorly because the synthetic annotations (i.e., randomly flipping labels between two classes) do not always satisfy the smooth manifold assumption in which similar samples have similar annota- tions. In Chaoyang and NIH-AO (shown in Appendix G. datasets, the baselines perform similarly while our method slightly surpasses them. In MiceBone, all methods perform worse than the best

---

Published as a conference paper at ICLR 2025
Table cowpnnddr-covrrgg---cuuree
(x 100) of different learning defer methods when dealing with missing annotations on different datasets (larger is better). Note that at no missing annotation (or zero missing rate), all methods are simplified to the same learning algorithm, and hence, share the same result.
Dataset
N experts
Missing rate
Naive L2D
MixMatch
FixMatch
Probabilistic (ours)
75.38
75.38
75.38
75.38
0.3
72.80
66.95
67.39
73.33
Cifar-100
0.5
70.83
66.80
67.27
72.13
0.7
67.68
65.52
66.52
71.19
NIH-AO
0.2
91.79
92.26
91.41
93.40
89.49
89.49
89.49
89.49
0.3
87.48
86.90
86.86
90.50
Chaoyang
0.5
87.46
88.12
86.66
90.03
0.7
87.09
87.11
86.92
88.41
0.5
89.94
88.55
90.33
91.07
0
79.58
79.58
79.58
79.58
0.3
79.67
81.02
81.87
83.48
MiceBone
8
0.5
80.30
82.19
82.12
83.69
0.7
81.31
80.60
82.55
83.10
human expert at small coverage. This is due to the slightly inconsistent performance of human ex- perts in training and testing sets (see Table Appendix F). Nevertheless, our approach shows relatively better results, particularly at low coverage.
(AUC-CAC) of each coverage accuracy curve in Fig. and the additional results in Appendix G.I, and report in Table 1. In general, our proposed probabilistic L2D outperforms the baselines across all datasets at various missing rates.
For a further comparison, we calculate the preeuumeerooveen--cwwweeyaaaaa
In terms of missing expert's annotation rates (see Fig. 6 in Appendix G.2 for further results of everrag-accuraay curves), the results Cifar- offar-1001a in Table 1 agrees with our intuition, in which the higher the missing rate, the lower the AUC-CAC. This is slightly different in Chaoyang and MiceBone datasets, especially MiceBone at lower missing expert's annotation rates. Such differ- ence is mainly attributed to the inconsistent performance in training and testing sets (see Table 2 in Appendix coupled with missing data. For example, without any regularisation, training a naive L2D baseline on the MiceBone dataset would prefer the human expert id 534 when there is no missing data, while selecting expert id 290 when 70 percent of data missing due their highest performance in each setting (see the rows highlighted in Table 2). Such selections, coupled with their inconsistent performance between training and testing, lead to the observation: the system performs better at higher missing rate.
5.3 ABLATION STUDY CONTROLLABLE WORKLOAD
To study the controllable workload further, we perform additional experiments using the proposed probabilistic L2D under settings: 1) imbalanced in which the workload of each human ex- pert is not controlled, and 2) balanced in which the workload is divided evenly for all human experts. equivalent setting El 0 and E, 1 in the former configuration, while setting R = (1 coverage) / M for each human expert in the latter one.
Figs. 4a and shows the results on Chaoyang dataset with and 3 human experts, respectively. At the same coverage, the imbalanced -bballnceddworkloadd approach performs slightly better than the balanced one. However, is clear that the imbalanced one is biased toward the human expert with highest per- formance note expert in Fig. 4a and expert 3 in Fig. 4b. This bias is more severe in Fig. 4b where almost half of all testing samples are deferred to the expert 3, who has 99% prediction accuracy.
Fig. 4c shows another disadvantage of the imbalanced approach MiceBone dataset with inconsistent human performance in the training and testing sets (see Table in Appendix F). Without balancing the workload, L2D tends to prioritise the human expert with the highest performance in the training set, resulting in an overfitting. In contrast, the balanced approach regularises the training, and hence, leads to higher performance overall.
Nevertheless, the gain overloading to one expert (or imbalanced approach) is small (less than percent prediction accuracy in the Chaoyang dataset) negative (in MiceBone), and hence,
8

---

Published as a conference paper at ICLR 2025
53 74
46 27
0.88
Best human Classlier Imbalanced
Imbolanced (accuracy 97.12%)
Deferral rate (%)
40
Balannee (accuracy 96. 95.35%)
0.86
Iimbalancedd (accuracy = 90. 13%)
Balanced
BiBalanced (accuracy (9.6
Deferral rate
Accuracy
0.84
30
30
31
23,09
25.334
0.82
24.89
20
20
12.00
0.78
10
Expert 91%
Expert 87%
Classifier 82%
Expert
Expert
Expert 99%
Classfier
0
0.2
0.1
0.5
0.8
91%
87%
82%
Coverage
(a) Chaoyang coverage of 0.5.
(b) Chaoyang coverage of 0.25. (c) MiceBone missing rate of 0.3
Figure 4: (a) and (b) show comparisons of two different workload constraints on Chaoyang dataset with 50% missing annotations per expert, where in the imbalanced setting, Et = 0 and Eu = 1 for each human expert, while in the balanced setting, ss (1 coverage) /M for each human expert, and (c) coverage accuracy curve on MiceBone at 30% missing rate.
may not compensate for overworking the best expert. In fact, by overworking that expert, it is hard to guarantee that the expert's 's accuracy will not be affected. In practice, it is, therefore, preferred to distribute the workload evenly across all human experts.
RELATED WORK
Learning to defer extends the capability of learning reject (Chow, 1957) by integrating human experts into the decision making process (Madras et al., 2018). In its simplest form, L2D consists of a classifier and a gating model that analyses an input sample and queries an expert (either a human or the classifier) to predict the label of that sample. By complementing the strengths of human experts and the classifier, L.2D can perform better than the human experts or the classifier in isolation at a lower cost compared with if all samples need to be analysed by a human expert. Most studies in the L2D literature focuses on statistical learning that develops surrogate losses for the 0-1 loss to make the learning consistent with the optimal Bayes classifier and gating model (Mozannar & Sontag, 2020) or well-calibrated (Verma & Nalisnick, 2022). In addition, most studies consider only the setting of single human expert, while in practice, there may be several human experts available. There are a few studies that considers the practical setting of multiple experts by either revising the gating model to output weight for each expert (Keswani al., 2021), using mixture of experts to produce a weighted average over the predictions of human experts and Al classifier (Hemmer al., 2022), or extending further the surrogate loss (Verma al., 2023). In contrast, our proposed method follows a probabilistic approach by modelling L2D as a variant of mixture of experts, and addition, given that based on mixture
can be optimised by employing the EM algorithm. of experts, our method can be straightforwardly extended to multiple human experts. The main difference between (Keswani et al., 2021; Verma et al., 2023) and our method, beside the modelling. is the codomain of the gating model. In our method, the output space of the gating model is the probability simplex AM := {y deferring one expert is influenced by the probability of deferring to other experts. In (Keswani et al., 2021), the codomain the gating model is the hypercube [0, 1] meaning that the deferral probability of each expert is independent. In (Verma et al., 2023), the codomain is RC+M+1 due to the unification of the classifier and gating model into a single model.
E [0, 1] y1 =1}. meaning that the probability of
Missing experts' annotations one of the challenges in L2D (Leitão al., 2022) because the standard L2D setting requires that every expert must annotate the whole training dataset. This is impractical because that increases the annotation cost significantly, or even infeasible for large-scale datasets. To our best knowledge, there is one study in the literature addressing such an issue (Hem- mer et al., 2023). That study proposes a two-phase method: (i) train a semi-supervised learning model for each expert where the observed annotations are represented by the labels and the missing ones are the unlabelled samples, then (ii) use the synthesised pseudo-labed labels to replace the missing annotations and obtain a "complete" annotation dataset to train an L2D system. Due to the nature of two phase training, an eтror in the semi cemi-supervised learning phase may be amplified in the training of L.2D systems, potentially deteriorating performance. In contrast, our method models the missing

---

Published as a conference paper at ICLR 2025
annotations train L2D system in end-to- manner. Such modelling avoids enror due to separate training, resulting in perf-onrn and efficient training.
7 LIMITATIONS AND DISCUSSION
Scalability The proposed method, however, has a limited scalability with the number of human experts. As the number of human experts grows, the number of models required to represent them also increases, leading to a substantial rise in GPU memory and computational time needed for training (e.g, see Table 3). Although such an issue can be worked around by distributing each model of human expert future work will focus on addressing the scalability of the proposed method, such as clustering M experts into K groups (M > K); where experts in the same group perform similarly. In that case, we only need K models to represent the experts in each group, making it feasible to work with datasets consisting of hundreds
different GPU, it is expensive due to the need of multiple GPUs. Thus,
thousands of human experts. We also apply further dimensional reduction techniques, such as hierarchical clustering, if the labelling pattern of human experts is diverse. Alternatively, one can design conditional model, denoted as h(x, () with being the embedding of human expert, to model each human expert. The embedding Ç can be obtained by extending from sample-wist wise to set-wise representation learning through the usage of deep set (Zaheer et al., 2017). Such conditional modelling can adapt to any number of human experts, and hence, addresses the scalability issue.
Deferral at low coverage As we mentioned in Section 1, L2D aims to maximise reliability, while keeping the cost within the acceptable limits. The choice of the ideal operating point that balances reliability and costs typically depends on specific priorities. high-stakes applications (e.g.. health- care), deferring uncertain cases offers significant advantages, such as improving decision accuracy, enhancing trust by acknowledging the limitations of the elassifier, and leveraging human oversight to ensure accountability in high-stakes scenarios. Deferred cases can also provide valuable learning opportunities to improve the AI model. However, deferring cases can be costly due to increased annotation budgets, potential expert burnout, and possible delays in decision-making. On the other hand, minimising deferrals maximises the efficiency of the L,D system, allowing for lower costs and quicker decision-making. This approach also reduces reliance on human experts and demon- strates the capability of the classifier to handle easier cases independently. However, not deferring uncertain cases poses risks, such potential misdiagnoses and diminished oversight. These risks can have severe consequences for patient safety and undermine trust in the system, especially when high-stakes decisions are involved.
Dynamic expert performance can be accommodated by incorporating temporal dynamics and performance variability through sequential modelling techniques or adaptive mechanisms. For fast- changing performance, such as fatigue, the model could include real-time performance tracking using metrics such as dynamic accuracy, allowing the adjustment of the deferral strategy based on current conditions. Slow-changing performance, like expertise improvement through learning, can be modelled using techniques such as Bayesian updating to adapt expert-specific parameters over time. Temporal modelling (e.g., recurrent neural networks) can also capture patterns in performance changes, enabling the system to make adjustments. By integrating such adaptive mechanisms, the model would better reflect the evolving nature of human expertise, improving its decision-making capabilities in dynamic scenarios.
CONCLUSION
This paper proposes probabilistic modelling for I.2D that extends the capability of the standard L2D to a practical setting where each human expert does not need to annotate the whole dataset of interest, but only a portion of the dataset. By modelling the expert selection and missing human experts' annotations as latent variables, the proposed method is optimised through the EM algorithm. We also propose to integrate a workload control mechanism as a constrained optimisation in the E step to enable the capability of distributing the workload to human experts and the classifier efficiently. Empirical evaluation demonstrates that the proposed probabilistic L2D outperforms prior methods in both synthetic and real-world datasets with different missing rates of annotations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research was supported by the Engineering and Physical Sciences Research Council (EPSRC) UK Research and Innovation through the grant EP/Y018036// We also thank Dr David Rosewarne from Royal Wolverhampton Hospitals NHS Trust for providing useful discussion on the point of view of radiologists to further motivate this study,
REFERENCES
Bjöm Barz and Joachim Denzler. Do we train on data? Purging CIFAR of near-duplicates. Journal of Imaging, 6(6):41, 2020.
Leonard Berlin. Liability of interpreting too many radiographs. American Journal of Roentgenology, 175(1):17-22. 2000.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. MixMatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Olivier Cappé and Eric Moulines. Online Expectation-Maxxiiiaan algorithm for latent data mod- cls. Journal of the Royal Statistical Society Series B: Statistical Methodology, 71(3))5933613, 02 2009.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Trans- actions Electronic Computers, pp. 247-254, 1957.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory, pp. 67-82. Springer, 2016.
João Graça, Kuzman Ganchev, and Ben Taskar. Expectation Maximization and posterior constraints.
In Advances Neural Information Processing Systems, volume 20, 2007.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in Neural Information Processing Systems, volume 17, 2004.
Patrick Hemmer, Sebastian Schellhammer, Michael Vössing, Johannes Jakubik, and Gerhard Satzger. Forming effective human-Al teams: Building machine learning models that complement the capabilities of multiple experts. In International Joint Conference Artificial Intelligence, 2022.
Patrick Hemmer, Lukas Thede, Michael Vössing, Johannes Jakubik, and Niklas Kühl. Learning to defer with limited expert predictions. In AAAI Conference on Artificial Intelligence, volume 37, pp. 6002-6011. 6011, 2023.
Vijay Keswani, Matthew Lease, and Krishnaram Kenthapadi. Towards unbiased and accurate de- ferral to multiple experts. In AAAI/ACM Conference on AI, Ethics, and Society, pp. 154-165, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, repoor, University of Toronto, 2009.
Diogo Leitão, Pedro Saleiro, Mário AT Figueiredo, and Pedro Bizarro. Human-Al collaboration in decision-making: Beyond learning to defer. ICML 2022 Workshop on Human-Machine Collabo- ration and Teaming, 2022.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and aceu- racy by learning to defer. In Advances in Neurul Information Processing Systems, 2018.
Anna Majkowska, Sid Mittal, David F Steiner, Joshua J Reicher. Scott Mayer McKinney, Gavin E Duggan, Krish Eswaran, Po-Hsuan Cameron Chen, Yun Liu, Sreenivasa Raju Kalidindi, Alexan- der Ding. Greg S. Corrado, Daniel Tse, and Shravya Shetty. Chest radiograph interpretation with deep learning models: Assessment with population-adjusted evaluation. Radiology, 294(2):421 -411, 2020.
reference standards and

---

Published as a conference paper at ICLR 2025
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to expert. In International Conference on Machine Learning, pp. 7076-7087. PMLR, 2020.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? Exact algorithms for learning to defer to humans. In International Confer- ence on Artificial Intelligence and Statisties, pp. 10520-10545. PMLR, 2023.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya K Menon, Ankit Rawat, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. In Advances in Neural Information Pro- cessing Systems, volume 35, pp. 29292-29304, 2022.
Lars Schmarje, Claudius Zelenka, Ulf Geisen, Claus-C Glüer, and Reinhard Koch. 2D and 3D segmentation of uncertain local collagen fiber orientations in SHG microscopy. In German Con- ference Pullern Recognition, pp. 374-386. Springer, 2019.
Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann, and Reinhard Koch. Is one anno- tation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation. In Conference on Neural Information Processing Systems - Datasets and Benchmarks Track, 2022a. URL https: httptppss///ppneeeeee.... http:/iwwweeeeeeeeeee wwwp://wwwww...eeooouuouuuaa wwtp:/eettetoonoooeeeeeeaaaa
Lars Schmarje, Monty Santarossa, Simon-Martin Schröder, Claudius Zelenka, Rainer Kiko, Jenny Stracke, Nina Volkmann, and Reinhard Koch. A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clustering. In European Conference on Computer Vision, pp. 363-380. Springer, 2022b.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-super learning with consistency and confidence. In Advances in Neural Information Processing Systems, volume 33, pp. 596-608, 2020.
The Royal College of Radiologists. Clinical radiology UK workforce census 2023. Technical report, The Royal College of Radiologists, 2023.
Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In Inter- national Conference Machine Learning, pp. 22122222..2. PMLR, 2022.
Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. Learning to defer to multiple experts: Consis- tent surrogate losses, confidence calibration, and conformal ensembles. In International Confer- ence on Artificial Intelligence and Statistics, pp. 11415-11434. PMLR, 2023.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. ChestX-ray8: Hospital-scall chest X-ray database and benchmarks weakly classification and localization of common thorax diseases. In JEEE Conference on Computer Vision and Paltern Recognition, pp. 2097-2106, 2017.
Mou-Cheng Xu, Yu-Kun Zhou, Chen Jin, Stefano B Blumberg, Frederick J Wilson, Marius deGroot, Daniel C Alexander, Neil P Oxtoby, and Joseph Jacob. Learning morphological feature pertur- bations for calibrated semi-supervised segmentation. In International Conference on Medical Imaging with Deep Learning, pp. 1413-1429. PMLR, 2022.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, vol- ume 30, 2017.
Maximilian Zenk, David Zimmerer, Fabian Isensee, Paul F Jäger, Jakob Wasserthal, and Klaus Maier-Hein. Realistic evaluation of FixMatcc on imbalanced medical image classification tasks. In German Workshop Medical Image Computing, pp. 291-296 Springer, 2022.
Zheng Zhang, Wenjie Ai, Kevin Wells, David Rosewarne, Thanh- Toan Do, and Gustavo Careeir. Learning to complement and to defer to multiple users. European Conference on Computer Vision, pp. 162. Springer Nature Switzerland, 2024. doi: 10. (0077/978-33031-7299992
Chuang Zhu, Wenkai Chen, Ting Peng, Ying Wang, and Mulan Jin. Hard sample aware noise robust learning for histopathology image classification. IEEE Transactions on Medical Imaging, 2022.
12

---

Published as a conference paper at ICLR 2025
A BASELINE OF LEARNING TO DEFER
A.1 FULL DATA ANNOTATIONS
This follows the standard setting of learning to defer, in which every sample is annotated by all experts, In this subsection, we present the EM algorithm to optimise the objective function in Eq. (2).
The "complete" data oogllikeldhod in this can be written follows:
M+1
(mm) ) + (rIn(((2(x))))
(,00+)) (r (im)) ,00)))
Pn
Yi
Z4 II
)
+ 1mnPr(t; ya/x, 0m1+1).
(11)
step
calculates the posterior of the expert index as follows:
M+1
Pr
2:
X v
II
(m) Pr
Ye
Z4. II
(m)
Pr(zi(x,,)) 7)-
(12)
m=1
M step maximises the complete-data log-likelihood as follows:
(1+11) 0M+1 <- argmax
2,,
(13)
7,0x4+
As mentioned in Section 4, can apply the constrained E step right after the E step obtain constrained posterior on z before performing the maximisation in the M step.
A.2 MISSING EXPERT'S S ANNOTATIONS
In this subsection, every sample is no longer annotated by all experts, but least one expert. The objective function can be written as follows:
max
Pr
Yi.
II mAD
(m)
0MS+1
N
max N.04+1
Σ
Pn
Yi
Xis
II mED
(m))
+ In Pr (M+1) xi, 0M,0)
(14)
For the naive learning to defer baseline, we sample an expert from Pr(z|x, 7) until the sampled expert has an annotation available for the sample of interest. This equivalent to:
Pr(z: ,|x,, 7) 0, Vj Dinobs.
(15)
We then follow the exact EM algorithm specified in Appendix A.I.
B UNCONSTRAINED POSTERIOR OF LATENT VARIABLES IN THE E STEP
We employ the variational EM to optimise the objective function in Eq. (2). The variational inference applied in the E step approximate the posterior of the latent variables z and The variational distribution can be written as follows:
glz,
II
((
gZz p)
II
((())
(16)
jE Dunhh...
Categorical(x;p) jEDunnhe
Categorical((()):00))
where the variational parameters p and g(0) are probability vector and obtained through the optimi-
sation in Eq. (4).
13

---

Published as a conference paper at ICLR 2025
The optimisation in the "unconstrained" E step is restated here to ease the derivation:
argmin KI. [(2z: p:) Pill
(:/2:/x (1:)
+
E
KL
(
(
Pr |
xs, xx, )
E (( II Jempuobe. (o)) In Pr Yi
Zi Impppt. (m) NEDDpuon t )
The objective function in Eq. (4) (also known as variational-free energy) can be expanded as follows:
M+1
Lise =
Σ
Pin
In
Pin
In
Ingn
(x:; +
Σ
Σ
( ic
In d - In fc(xx; f())
n=1
JED c=1
M+1
Σ
Σ
Pin
II
of ey In Siys:
(17)
jEDinobr.
where: e2 is the w-th unit vector (often known as one-hot vector), and G3 is the j-th element of (n) defined as follows:
((-)
(1)m) (x);))))
if Dobs
(18)
if Dunobs.
Variational parameter We minimise the variational energy in Eq. (17) w.r.. p. which is the probability to select which expert (either human the classifier) among total of M + experts. Note that because p probability, it is constrained by ΣM+1 Pin
We form the Lagrangian by isolating the terms which contain and adding the appropriate Lagrange multiplier, 77 0, as follows:
M+1
M+1
Lvtelpt] Σ
Pin In Pén In gn (xx11
II
d ey, In In
Pin
jEDunobe
M+1
1 )
+ T
Σ Pin
P
(19)
Taking the derivative with respect to Pin gives:
dlvtelps]
In Pin In (x;: (8)) +
Σ
II
Le In clo + 7h-
(20)
dpm
c=1 JeDucb...
Setting the derivative zero yields the optimal value of the variational paramter:
Pin In (x(37(00) exp Σ II
Dl ey. In c/as) -
(21)
JEDunobe.
can also be rewritten as follows:
((2)) x Pr zi xix (k)
In InPr(y; 2i. -lle
Dobn t (m) II Dunanab t(r))] -13.
x cxp
Denobe a (1))
(1)
(22)
14

---

Published as a conference paper at ICLR 2025
Variational parameter & Similarly, can obtain the variational parameter & and write the solu- tion the following form:
qt) ((t)))
x Pr
(5) ti x 091))
E ((x)) II yED 2 (1)57) [in [inPr('s (2 ImeDpia (/m) IIEEDpor (.0
exp
(23)
Eqs. (22) and (23) result in system of (M + + C (unbb))) non-linear equations in the form == f(x) which can be solved by fixed-point iteration method or the Newton Raphson method.
For simplicity, we use the fixed-point method with fixed number of iterations 10 to solve the above system of non-linear equations.
C CONSTRAINED POSTERIOR OF LATENT VARIABLES IN THE E STEP
This section provides the detailed derivation of the constrained optimisation in Eq. (8), which can be rewritten in the form of standard optimisation as follows:
N
argmin
Eqiz.) [ln - (2;)]
N
(2-
1=1
1
1
s.L.:
Σ [(z,)
Eu
Σ q(2,) -1i
(24)
N
N
The Lagrangian of the constrained optimisation can be written as follows:
N
(
N
1
1 N
]
B(s)) [In (2,)) In (2.)) + AoT
Σ
q(z,)
- Eu + +
El -
Σ q(zi)
N
N
N
i=1
i=1
N
B7(()) In [lz)(z) - In (2:)) + (Au - A1) g(z:)) - A,Euu Eu + + E
N
(25)
where An A E AR+++ are the Lagrange multipliers.
Taking the functional derivative (similar to the derivation of variational inference explained in David Blei's lecture: https: cs princeton. edu/ sourses/archive/aalll
-iteaaanceeeeeeeeeeeeeeeeeeeeeee i. .dff) with respect to g gives:
dL 1
dą. [In)( - Inq (z.) +1+ >1l
(26)
Setting the KKT condition gives:
dL dæ
= + ln] - Ing" + Au - -1 + +1= =0
(27)
Solving for ((z)) gives:
In ((1)))) x (2;" - Au + A - 1.
(28)
Or:
g'
((z)) Z(Au, A1) exp (An - A1) + 1)
(29)
where: 7(An; Ai) is the normalisation constant defined as follows:
(z())
Z(>,, A1) Σ
(30)
exp (An A1) 2. +1)
15

---

Published as a conference paper at ICLR 2025
Algorithm The EM algorithm for learning to defer with missing expert's annotations
1:
procedure TRAINING(D,EN, E1, K)
2: Di: training dalaset defined Section
3: EiEi, the lower and upper workload constrained vectors
4: n: mini-batch size
5: K: mumber of iterations
6: initialise parameter of gating model 7
7:
initialise parameters of expert models (0mh )
)
8:
for E {1....,K } do draw mini-mateh of n samples for i (1.....,)) do
9:
10:
D Iterate through every sample in the mini-batch
11:
9¹ (z.). (9 (t, M) < min VARIATIONAL-FRES ENERGY E Step in Eq. (4) i't (Zi) & CONSTRAINED POSTERIOR( (2)))
12:
t -Section
13:
calculate the cotalcodpleted log- log-likelihood Qi(y, 0, 0,)) ((1))
see Eq. (3)
(l+11), ((k+1) argmax,,, Ei=1 Qi(7: 0, 7(),, o(k)) + In Pr(?) EM=1 In Pr(0m)
14:
return 7, 0,0
Parameters of gating model and classifier
Substituting ((2)) in Eq. (29) into the Lagrangian in Eq. (25) gives:
L 1 E Bg(e,) [- N
d1
In Z(Au: X1) + (2. A1) ((2:)
AT + ATT
N
-TTTen + A/T8 11-1- A()
i=1
N
(2(2i)
= -AT +
m
(31)
exp
(2u
-1) +
1=1
According to the duality, An and \ can be obtained through the following optimisation:
('(z,)
X A = argmax L = argmin AT. ATE +
In
(32)
Am,A.20 A,
N 1=1
exp (Au A1) 2: + +1
D TRAINING PROCEDURE
The training of the proposed probabilistic L2D with workload constraints is described in Algo- rithm
E POST-HOC METHOD FOR ACCURACY COVERAGE CURVES
In the standard learning to defer, the coverage constraint is imposed as a post-hoc during test- ing. The post-hoc procedure is shown in Algorithm (refer helpers/metrics.py in https: ///ittt ggithbb. com/ h/n//cl celinicall calmmammmmmmmmmmmmmmmmm httu/hma__aaa _ii__aeeeea_allleeeeeee_e__ as the original code to implement the post-hoe algorithm). In particular, the confidence values produced by the gating model to select the Al classifier are used to sort the test samples, with the top-k samples being selected to defer to the Al classifier. Such post-hoc testing procedure is inconsistent with the training procedure in learning to defer due the human intervention. Furthermore, the procedure requires to access all of the testing samples ahead of time before performing deferral. This is, however, impractical, especially in model deployment.
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 Current practice in calculating coverage and accuracy at test time
procedure COVERAGE--ACCURCCY((( (t, (m)) (t/" )
1:
2:
Xy. imput sample
(m)) annotation made by expert indexed by
3:
4:
D Yı ground truth
A
5:
parameter of gating (deferral) model
6:
parameter of ML classifier
7:
coverage value of interest
8:
initialise an emptied ordered set of confidence selecting the classifier: P P+0
9:
for (I...., N} do
10:
D probability of routing/selection > store confidence of selecting the classifier [TN Indices of samples classified by classifier Texperts + NB....,N]JA. Indices of samples classified by human experts classifier
P: + g(x;7) P + P MI.M1
12:
Zelr & ARG-TOO- PrroP-K(ppulattonn222
14:
return total accuracy
F EXPERIMENT SETTING
F.1 DATASETS
Cifar-100 consists 50,000 training and 10,000 testing images which are categorised into 20 superclasses. These superclasses are then divided further into 100 classes. We follow the approach of (Hemmer et al., 2023) to simulate two experts, each exhibiting asymmetric label noise. In particular, each expert classifies correctly 10 designated superclasses, while making errors 50% of the time on the remaining 10 superclasses. The annotations made by each expert is then randomly sampled missing rate of interest. Moreover, given that the Cifar- 100 dataset has about 10 percent of testing images that are duplicated or almost identical to the ones in the training set, we decide to use ciFAIR- 100 (Barz& & Denzler, 2020), which replaces those duplicated images by ones that belong to the same classes.
NIH-ChestXray consists of 4,374 chest X-ray images taken from the ChestXray-8 dataset (Wang al., 2017). Each X-ray image initially reviewed by three radiologists over four aadi--graphic findings. If there is disagreement between the three radiologists, the image is returned for an additional review. The labels and notes from the previous rounds are made available during each iterative review. Adjudication continues until consensus is reached or for a maximum of five rounds to determine the ground truth label. Although there are a total of 22 radiologists (including board- certified and resident radiologists), each radiologist annotates only subset of the whole dataset with small mutual overlapping, increasing the difficulty to form a test set for learning to defer. We follow similar approach in (Hemmer al., 2023) select the radiologist with id 4295342357 who has the largest number of annotations 2,350 images - for the evaluation. To form a train test split, we randomly select 20% of patients be the test subset, while the remaining patients used in the training subset. We further randomly remove 20% of the annotations from the training subset to simulate the missing annotations. addition, the original image size of 1,024-by-1,00 is resized to 356-by-256 before randomly cropping to 224-by-22- during training.
Chaoyang consists of 4,021 training and 2,139 testing images of colon patches obtained from the Chaoyang hospital in China (Zhu et al., 2022). Each image annotated by three professional pathologists over four categories: normal, serrated, adenocarcinoma, and adenoma. The ground truth labels are obtained via the majority vote the three annotations. The performance of the three pathologists are 91%, 87% and 99% accurate according this majority vote. In our experiments, we show results with all three experts, and with two experts, where we exclude the third expert.
MiceBone consists of 4,736 haronnic generation microscopy images from 35 3D-scans of 6 mice where mice had the disease osteogenesis imperfecta also known brittle bone disease, while the others do not (Schmarje et al., 2019). We follow the same preprocessing procedure in (Schmarje al., 2022b) by using the given segmentation masks to cut the original 2D image slices
17

---

Published as a conference paper at ICLR 2025
into many patches, each consisting of one class, resulting in total of 7,240 2D-images. Each image then annotated by one to five professional annotators to classify into three classes (Schmarje et al., 2022a). Among the 79 annotators, only 8 annotators annotate the whole dataset. Hence, annotations those 8 annotators to represent experts our experiment, while using the majority vote the ground truth. The performance of those 8 experts shown in Table 2. For the train test split, we use the first four folds as the training set, and the remaining fold as the test set.
the
Table 2: The performance of experts from the MiceBone dataset.
Prediction accuracy (%)
Expert id
Train
(with missing rate)
Test
0.3
0.5
0.7
047
86.34
64.45
49.15
34.68
85.74
290
85.82
65.63
49.82
36.81
85.29
533
86.29
64.37
49.61
35.30
84.96
534
86.59
64.47
50.11
35.00
84.45
580
80.76
60.40
47.90
33.79
79.07
85.83
64.12
50.43
36.37
84.32
966
86.54
64.47
49.41
35.62
87.56
745
85.01
65.61
50.71
35.81
84.38
F.2 TRAINING HYPER- -AA--PPRAAA PARAMETERS
The model used throughout the evaluation is mainly Resnet-18 for datasets that have large size images (e.g., NIH, Chaoyang and MiceBone), while Pre-Act-Resnet-18 is used in datasets with smaller image size (e.g., ciFAIR-100). Each method is trained for 300 epochs (1,000 epochs for ciFAIR-100) using stochastic gradient descent with momentum of 0.9 and learning rate of 0.01. The learning rate is decayed through cosine decaying scheduler, and the gradient norm is clipped the maximal of 10 for numerical stability. In addition, mixed precision using bfloat is applied over all methods and datasets to speed up the training. The proposed method is implemented in Jax a Python library that accelerates array computation and program transformation achieve high-performance numerical computing for large-scale machine learning.
For the baseline relying on semi-supervised learning, we also use the same backbone. The trained model is then used to produce pseudo-labels for the missing annotations. A standard L,D is then trained on the observed labels and pscudo-labels to produce the final results.
G ADDITIONAL RESULTS
G.1 COVERAGE ACCURACY CURVES ON NIH-AO, CHAOYANG AND MICEBONE
This subsection provides more results in addition the ones presented in Subsection 5.2. In partic- ular, Fig. 5 show the coverage accuracy curves for NIH-AO, Chaoyang with 3 human experts and MiceBone.
G.2 COMPARISON FOR DIFFERENT MISSING RATES
This subsection provides further results of our proposed method evaluated Cif and Chaoyang at different missing rates.
FixMatch L2D Although FixMatch often surpasses MixMatch in several computer vision bench- marks, in the L2D setting shown in Fig. 3 and Fig. 5, FixMatch is par with MixMatch and inferior in some settings. This due to the strong data augmentation applied in real-world medical image datasets. The strong data augmentation in FixMatch is designed for natural images, and hence, when applicd to medical images, it creates undesired artifacts or transformations, creating more difficul- ties for the learning algorithm to utilise unlabelled data (Zenk al., 2022). In practice, the shearing
18

---

Published as a conference paper at ICLR 2025
Best human expert
Clossifier
Naive L2D
MixMatch-L20
FixMatch-L2
Probabilistic L2D
0.95
0.95
Accuracy
Accuracy
0.9
0.85
0.85
0.8
0.2
0.6
0.8
0.2
0.4
0.8
Coverage
Coverage
Chaoyang experts 0.5 miss.
(b) NIH-AO expert 0.2 miss.
0.88
0.88
0.88
0.86
0.86
0.86
0.84
0.84
0.84
Accuracy
Accuracy
0.82
0.82
0.82
0.8
0.8
0.8
0.78
0.78
0.78
0.76
0.76
0.76
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Coverage
Coverage
Coverage
(c) MiceBone- experts 0.3 miss.
MiceBone 8 experts 0.5 miss.
(e) MiceBone 88 experts 0.7 miss.
Figure 5: Additional coverage accuracy curve results on: Chaoyang, NIH-AO and MiceBone.
Best human expert
Classifier
no missing
missing 0.3
missing 0.5
missing 0.7
0.88
0.94
0.85
0.92
0.86
0.8
Accuracy
Accuracy
0.9
Accuracy
0.84
0.75
0.88
0.82
0.7
0.86
0.8
0.84
0.65
0.78
0.82
0.6
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Coverage
Coverage
Coverage
(a) Cifar-100
(b) Chaoyang
(c) MiceBone
Figure 6: The coverage accuracy curves of our method different missing rates, including the setting with complete expert's annotation.
transformation in FixMatch is often omitted when applying on medical image datasets (Xu et al., 2022). In the paper, we use the vanilla FixMatch as a baseline in Section keep the consistency with previous studies (Hemmer et al., 2023).
H RUNNING TIME
We also report the running time to train the naive, semi-supervised baselines and our method in Table 3. These running times are reported for whole training using the same number of epochs (1,000 for Cifar-100 and 300 for others). For the semi-supervised learning baseline, the running time includes the semi-supervised training for all experts (the first number) and the standard L2D (the second number).
19

---

Published as a conference paper at ICLR 2025
Table 3: Running time of different L2D approaches (in GPU-hour): for semi-super learning baselines, the reported one includes semi-supervised training for all experts (the first number) and the standard L2D training (the second number).
Running time (GPU-hour)
Dataset
Ns experts
Naive
MixMatch
FixMatch
Probabilistic L2D
Cifar- 100
2
5.1
9.8
10,9 + 5.1
5.0
Chaoyang
2
1.8
5.4 1.8
5.8 + 1.8
3.8
MiceBone
8
4.4
22.4 + 4.4
25.3 + 4.4
12.8
I ABLATION STUDY ON THE MINI-BATCH SIZE FOR THE CONSTRAINED E STEP
We conduct another ablation study understand the influence of mini-batch sizes to the constrained optimisation of the workload distribution presented in Section 4. Recall that the workload distribu- tion global constraint, so larger batch sizes may lead to more accurate learning. We evaluate of sizes 20, 50 and 100 at different missing annotation rates Chaoyang dataset and plot the results Figs. 7a and 7b. The area under curve the evaluations presented in Fig. agree with the intuition that the larger the batch size, the higher the performance.
Best human expert
Classifier
Batch size 20
Batch size 50
Batch size 100
0.93
0.93
Area under curve [x 100)
90.13
90-
89 45
Accuracy
0.9
Accuracy
0.9
89
IaBatch size
0.87
0.87
BaRatch sime 50
BaBatch 100
88
0.84
0.84
50
87.16
0.81
0.81
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.3
0.5
Coverage
Coverage
Missing annotation rate
(a) Chaoyang 2 experts 0.3 miss.
(b) Chaoyang 2 experts 0.5 miss.
(c) Area under curve
Figure 7: Coverage accuracy curves of probabilistic I.2D trained on different mini-batch sizes and their corresponding area under curve.
J ONLINE EM
Conventionally, the EM algorithm requires to load and process the whole dataset at each iteration. This is, however, impractical, especially for large datasets that cannot fit into memory. Given this limitation, we employ an online version of the EM algorithm (Cappé & Moulines, 2009), which could be performed in a mini-batch setting. The online EM algorithm maintains an exponential moving average as approximation of the full average Q defined in Eq. (3). In particular, the stochastic EM algorithm can be presented as follows:
E-step calculates the complete- data log- likeliioii as the one in the conventional EM algorithm.
M-step optimises the exponential weighted average w.r.t. the parameters of interest:
(k+1), g(++1) + argmax(1 - Tx)Q (0, 7, (k-1), 0( + T2 (0.7, 000,,,00,,00 ,00),
(33)
0,7
where Tę is the step size satisfying the decaying conditions: Σte Th +00 and _k=1 Th 400.
20

---

Published as a conference paper at ICLR 2025
Such moving average update in the M step is similar to the momentum used in stochastic gradient descent. To this end, our implementation uses the SGD with a momentum of 0.9 as a solution that integrates the online EM into our method.
21


Reviews:
[{"review_id": "xLsc2jUKut", "reviewer": "Reviewer_CxS1", "summary": "This paper extends the concept of *learning to defer* (L2D) to scenarios with missing expert annotations and balanced expert workloads. The authors propose a formulation that relies on a clever application of the expectation-maximization algorithm, which naturally handles missing data. Additionally, they introduce a constraint within the expectation stage of the algorithm to manage expert workloads. The proposed L2D is tested on both synthetic and real-world datasets, resulting in a higher area under the coverage-accuracy curve compared to the evaluated baselines.", "questions": "+ Q1. I found the data training process and the L2D objective on page 3 difficult to understand, possibly due to ambiguous notation. Does $\\mathbf{y}$ represent the ground truth label or output prediction? If $\\mathbf{y}$ refers to the ground truth, why does it depend on the expert annotations $\\mathbf{t}$? Should not $\\mathbf{y}$ be dependent on $\\mathbf{x}$ but independent of expert selection and annotation? The notation in Eq. 1, where $\\mathbf{t}_i$ becomes $\\mathbf{y}_i$, is particularly confusing. Revising the notation could improve clarity and help readers understand the logic.\n+ Q2. Regarding Eq. 1, if $\\mathbf{t}_i$ are deterministic expert annotations from a look-up table and $\\mathbf{y}_i$ represents the ground truth label, is it still valid to compute the log-likelihood with *hard labels*?\n+ Q3. Could you provide a more detailed explanation of why the accuracy of probabilistic L2D surpasses that of the best human expert (even with a 70% missing rate and especially for low coverages)? This was not explained in the discussion.\n+ Q4. Why does the area under the coverage-accuracy curve increase with higher missing rates on the MiceBone dataset? Why does having fewer annotations lead to better performance? It is mentioned that this is due to inconsistent human expert behavior between the training and test sets. How do missing annotations help in this case?\n+ Q5. The discussion on balanced workload and inconsistent human performance between training and test sets could benefit from further elaboration (Page 8, Line 423). Would assigning less weight to a particular expert act as a form of regularization?\n+ Q6. (Minor suggestion) It would be helpful to present the results in Figure 3 with equal Y-axis scales for subplots corresponding to the same dataset.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper addresses a research question relevant to real-world applications by providing a solution for settings where expert annotations are incomplete. \n+ The results show that reducing the workload of highly accurate (and typically overloaded) human experts only slightly decreases overall accuracy and can lead to higher accuracy in scenarios with inconsistent expert performance between the training and test sets. \n+ The proposed controllable workload formulation simplifies the evaluation of accuracy-coverage ratios compared to existing methods, which often require assumptions or post-hoc adjustments to balance learnable models and human experts.", "weaknesses": "+ As acknowledged by the authors, the proposed formulation does not scale well with the number of human (or learnable) experts. While grouping experts into clusters is suggested as potential future research direction, this introduces the number of clusters as a hyperparameter, necessitating additional tuning and potentially hindering scalability.\n+ Although the paper is concise and generally well-written, the notation is ambiguous in some places (see Q1 and Q2), and the discussion of the results is very brief and could benefit from additional explanations (see Q3 and Q4). \n+ (Minor comment) I recommend the authors release the source code to reproduce results. While not mandatory, providing the code would help readers understand how to implement the algorithm proposed on page 14, especially the implementation steps required to solve the optimization equation formulated in Eq. 4 on page 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gKgKl4bxq0", "reviewer": "Reviewer_6swQ", "summary": "The paper studies the problem of learning to defer for a realistic setting where expert annotations can be missing and where workload balancing among experts is crucial. The authors propose a probabilistic modeling approach where EM is used to address the missing annotations, In particular, a constrained optimization during the E step regulates workload balancing among human experts and the AI classifier. The proposed method is evaluated on synthetic and real-world datasets and is shown to perform on par or better than the considered baselines.", "questions": "I am unsure about the computational cost of the method compared to the considered baseline. Could the authors elaborate more on this? Ideally, by comparing and reporting the runtimes of the proposed method and the baselines across the different datasets.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The proposed probabilistic modeling techniques and the use of EM in this setting seem novel and an interesting contribution. Experimental results show the performance gain of the method compared to the baselines.", "weaknesses": "A key weakness is highlighted by the authors in the paper: Bad dependency on the number of human experts. Although, they discuss potential remedies, e.g., clustering. However, this probably wouldn't work for a setting with diverse human experts (where the number of clusters is large). Are there other dimensionality reduction approaches (e.g., hierarchical clustering) that one could consider for this setting and how would they affect computational cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGpK0BRgM4", "reviewer": "Reviewer_1P43", "summary": "This paper introduces a probabilistic framework for “Learning to Defer” (L2D) that enables an AI system to either make decisions independently or defer them to human experts based on confidence levels. The approach addresses the limitations of existing L2D methods by handling cases where not all experts can annotate every data sample. This approach is designed to reduce the annotation burden. The approach is based on the Expectation-Maximization (EM) algorithm to optimize workload distribution between AI and human experts. The proposed method shows promising performance across synthetic and real-world datasets.", "questions": "Could the authors answer points/questions 2 and 3 in the list above \"weaknesses\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an interesting issue in L2D and proposes a sound solution based on a probabilistic approach. \n- The workload management is particularly promising in many areas where AI is supporting expert decision such as in medicine. \n- This is also relevant in addressing ethical and practical constraints, and possibly even regulations and laws. \n- The ablation study offers an insight on the mechanism that lead to prioritise highest performing humans with the imbalanced approach, with possible overfitting.\n-  It is interesting that the study allows for the conclusion that in practice it may be desirable to distribute workload evenly across all human experts.", "weaknesses": "1. Overall, the approach has some limitations, which I acknowledge are also partially discussed. However, it's unclear how well the system can scale given that each expert requires a probabilistic model. It's unclear to me how well the clustering of expert would work and what are the risks associated with that. \n\n2. I would be interested in reading more about the trade-off between the case for fewer deferring cases or deferring cases with the highest uncertainty, which is not much discussed. Clearly, there will be cases, e.g., healthcare, where deferring on uncertain cases would be quite important. \n\n3.  How could the model be adapted to take into consideration fast and slow changing expertise performance? The model assume static performance, however, experts could have fast performance changes, e.g. due to fatigue, or slow performance changes, e.g. due to learning through a period of time. It would be nice to understand how the model could accommodate for such dynamic scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhHTM9Il9G", "reviewer": "Reviewer_edcU", "summary": "The paper deals with human - AI cooperation. It presents a modification of the learning to defer technique which can handle incomplete expert annotations and balance the workload of the experts.", "questions": "* Would it make sense to repeat the learning experiments several times in order to be able to estimate the uncertainty of the results?\n* Does it make sense to give the results in Table 1 with four valid digits? Are the results really that accurate?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It seems to me that the topic has been addressed very comprehensively\n* The comparisons include all the mentioned relevant predecessor methods", "weaknesses": "I can't see any significant weaknesses. However, this may also be because the topic is new to me.\n\nFurther comments:\n\n* In the case of “In contrast, machine learning or AI models excel at processing large amounts of information but may be prone to biases (Meehl, 1954)”, the reference chosen cannot be used as evidence for the statement because “machine learning or AI models ... processing large amounts of information” were not available until long after 1954.\n* I find statement “Ideally, a perfect balanced workload among experts and the AI model can be expressed as follows” a little strange. After all, you will only strive for an equal distribution if all experts are equally competent.\n* I wonder about “slightly-similar”, how can something be slightly similar?\n* I find it a bit irritating that there is no section called “Conclusion”.\n* “50 %” -> “50%”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)