Probabilistic Learning to Defer: Handling Missing Expert Annotations and Controlling Workload Distribution

Authors: Cuong C. Nguyen, Thanh-Toan Do, Gustavo Carneiro
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: learning to defer, expectation - maximisation

Abstract:
Recent progress in machine learning research is gradually shifting its focus towards *human-AI cooperation* due to the advantages of exploiting the reliability of human experts and the efficiency of AI models. One of the promising approaches in human-AI cooperation is *learning to defer* (L2D), where the system analyses the input data and decides to make its own decision or defer to human experts. Although L2D has demonstrated state-of-the-art performance, in its standard setting, L2D entails a severe limitation: all human experts must annotate the whole training dataset of interest, resulting in a time-consuming and expensive annotation process that can subsequently influence the size and diversity of the training set. Moreover, the current L2D does not have a principled way to control workload distribution among human experts and the AI classifier, which is critical to optimise resource allocation.  We, therefore, propose a new probabilistic modelling approach inspired by the mixture-of-experts, where the Expectation - Maximisation algorithm is leverage to address the issue of missing expert's annotations. Furthermore, we introduce a constraint, which can be solved efficiently during the E-step, to control the workload distribution among human experts and the AI classifier. Empirical evaluation on synthetic and real-world datasets shows that our proposed probabilistic approach performs competitively, or surpasses previously proposed methods assessed on the same benchmarks.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729628984154

thread_type:
official_review

<thread_object>
creator: reviewer_edcu

<content>
summary:
The paper deals with human - AI cooperation. It presents a modification of the learning to defer technique which can handle incomplete expert annotations and balance the workload of the experts.

strengths:
* It seems to me that the topic has been addressed very comprehensively
* The comparisons include all the mentioned relevant predecessor methods

weaknesses:
I can't see any significant weaknesses. However, this may also be because the topic is new to me.

Further comments:

* In the case of “In contrast, machine learning or AI models excel at processing large amounts of information but may be prone to biases (Meehl, 1954)”, the reference chosen cannot be used as evidence for the statement because “machine learning or AI models ... processing large amounts of information” were not available until long after 1954.
* I find statement “Ideally, a perfect balanced workload among experts and the AI model can be expressed as follows” a little strange. After all, you will only strive for an equal distribution if all experts are equally competent.
* I wonder about “slightly-similar”, how can something be slightly similar?
* I find it a bit irritating that there is no section called “Conclusion”.
* “50 %” -> “50%”

questions:
* Would it make sense to repeat the learning experiments several times in order to be able to estimate the uncertainty of the results?
* Does it make sense to give the results in Table 1 with four valid digits? Are the results really that accurate?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730394615416

thread_type:
official_review

<thread_object>
creator: reviewer_1p43

<content>
summary:
This paper introduces a probabilistic framework for “Learning to Defer” (L2D) that enables an AI system to either make decisions independently or defer them to human experts based on confidence levels. The approach addresses the limitations of existing L2D methods by handling cases where not all experts can annotate every data sample. This approach is designed to reduce the annotation burden. The approach is based on the Expectation-Maximization (EM) algorithm to optimize workload distribution between AI and human experts. The proposed method shows promising performance across synthetic and real-world datasets.

strengths:
- The paper addresses an interesting issue in L2D and proposes a sound solution based on a probabilistic approach. 
- The workload management is particularly promising in many areas where AI is supporting expert decision such as in medicine. 
- This is also relevant in addressing ethical and practical constraints, and possibly even regulations and laws. 
- The ablation study offers an insight on the mechanism that lead to prioritise highest performing humans with the imbalanced approach, with possible overfitting.
-  It is interesting that the study allows for the conclusion that in practice it may be desirable to distribute workload evenly across all human experts.

weaknesses:
1. Overall, the approach has some limitations, which I acknowledge are also partially discussed. However, it's unclear how well the system can scale given that each expert requires a probabilistic model. It's unclear to me how well the clustering of expert would work and what are the risks associated with that. 

2. I would be interested in reading more about the trade-off between the case for fewer deferring cases or deferring cases with the highest uncertainty, which is not much discussed. Clearly, there will be cases, e.g., healthcare, where deferring on uncertain cases would be quite important. 

3.  How could the model be adapted to take into consideration fast and slow changing expertise performance? The model assume static performance, however, experts could have fast performance changes, e.g. due to fatigue, or slow performance changes, e.g. due to learning through a period of time. It would be nice to understand how the model could accommodate for such dynamic scenarios.

questions:
Could the authors answer points/questions 2 and 3 in the list above "weaknesses"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609344172

thread_type:
official_review

<thread_object>
creator: reviewer_6swq

<content>
summary:
The paper studies the problem of learning to defer for a realistic setting where expert annotations can be missing and where workload balancing among experts is crucial. The authors propose a probabilistic modeling approach where EM is used to address the missing annotations, In particular, a constrained optimization during the E step regulates workload balancing among human experts and the AI classifier. The proposed method is evaluated on synthetic and real-world datasets and is shown to perform on par or better than the considered baselines.

strengths:
The paper is well-written and easy to follow. The proposed probabilistic modeling techniques and the use of EM in this setting seem novel and an interesting contribution. Experimental results show the performance gain of the method compared to the baselines.

weaknesses:
A key weakness is highlighted by the authors in the paper: Bad dependency on the number of human experts. Although, they discuss potential remedies, e.g., clustering. However, this probably wouldn't work for a setting with diverse human experts (where the number of clusters is large). Are there other dimensionality reduction approaches (e.g., hierarchical clustering) that one could consider for this setting and how would they affect computational cost?

questions:
I am unsure about the computational cost of the method compared to the considered baseline. Could the authors elaborate more on this? Ideally, by comparing and reporting the runtimes of the proposed method and the baselines across the different datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648937116

thread_type:
official_review

<thread_object>
creator: reviewer_cxs1

<content>
summary:
This paper extends the concept of *learning to defer* (L2D) to scenarios with missing expert annotations and balanced expert workloads. The authors propose a formulation that relies on a clever application of the expectation-maximization algorithm, which naturally handles missing data. Additionally, they introduce a constraint within the expectation stage of the algorithm to manage expert workloads. The proposed L2D is tested on both synthetic and real-world datasets, resulting in a higher area under the coverage-accuracy curve compared to the evaluated baselines.

strengths:
+ The paper addresses a research question relevant to real-world applications by providing a solution for settings where expert annotations are incomplete. 
+ The results show that reducing the workload of highly accurate (and typically overloaded) human experts only slightly decreases overall accuracy and can lead to higher accuracy in scenarios with inconsistent expert performance between the training and test sets. 
+ The proposed controllable workload formulation simplifies the evaluation of accuracy-coverage ratios compared to existing methods, which often require assumptions or post-hoc adjustments to balance learnable models and human experts.

weaknesses:
+ As acknowledged by the authors, the proposed formulation does not scale well with the number of human (or learnable) experts. While grouping experts into clusters is suggested as potential future research direction, this introduces the number of clusters as a hyperparameter, necessitating additional tuning and potentially hindering scalability.
+ Although the paper is concise and generally well-written, the notation is ambiguous in some places (see Q1 and Q2), and the discussion of the results is very brief and could benefit from additional explanations (see Q3 and Q4). 
+ (Minor comment) I recommend the authors release the source code to reproduce results. While not mandatory, providing the code would help readers understand how to implement the algorithm proposed on page 14, especially the implementation steps required to solve the optimization equation formulated in Eq. 4 on page 4.

questions:
+ Q1. I found the data training process and the L2D objective on page 3 difficult to understand, possibly due to ambiguous notation. Does $\mathbf{y}$ represent the ground truth label or output prediction? If $\mathbf{y}$ refers to the ground truth, why does it depend on the expert annotations $\mathbf{t}$? Should not $\mathbf{y}$ be dependent on $\mathbf{x}$ but independent of expert selection and annotation? The notation in Eq. 1, where $\mathbf{t}_i$ becomes $\mathbf{y}_i$, is particularly confusing. Revising the notation could improve clarity and help readers understand the logic.
+ Q2. Regarding Eq. 1, if $\mathbf{t}_i$ are deterministic expert annotations from a look-up table and $\mathbf{y}_i$ represents the ground truth label, is it still valid to compute the log-likelihood with *hard labels*?
+ Q3. Could you provide a more detailed explanation of why the accuracy of probabilistic L2D surpasses that of the best human expert (even with a 70% missing rate and especially for low coverages)? This was not explained in the discussion.
+ Q4. Why does the area under the coverage-accuracy curve increase with higher missing rates on the MiceBone dataset? Why does having fewer annotations lead to better performance? It is mentioned that this is due to inconsistent human expert behavior between the training and test sets. How do missing annotations help in this case?
+ Q5. The discussion on balanced workload and inconsistent human performance between training and test sets could benefit from further elaboration (Page 8, Line 423). Would assigning less weight to a particular expert act as a form of regularization?
+ Q6. (Minor suggestion) It would be helpful to present the results in Figure 3 with equal Y-axis scales for subplots corresponding to the same dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732116504584

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes made in the revision

comment:
We thank all reviewers for their insightful and constructive feedback. We have carefully considered the comments from each reviewer and have incorporated these suggestions into a revised version of our paper, which has been uploaded to Open Review. All modifications are highlighted in blue for easy reference. Specifically, we have:
- enhanced the clarity of the paper by providing more detailed explanations of the modeling techniques and experimental results,
- separated the discussion from conclusion to address the limitations of the proposed method and the current setting of learning-to-defer, and
- corrected some typos and visualisation (scaling y-axes of subfigures).

We have also addressed each reviewer's concerns individually, which can be found directly below their respective reviews.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733312743155

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for constructive feedback and positive review

comment:
We thank all the reviewers for their constructive feedback to improve our paper. In particular:
 - we appreciate the initial review of reviewer 1P43 for praising our paper and suggesting insightful discussion on many practical aspects of the setting considered in the paper,
 - we thank the reviewer 6swQ for suggesting and discussing further potential solutions to tackle the limtiation of the proposed method,
 - we thank the reviewer CxS1 for helping to improve the clarity of the methodology of the proposed method, as well as suggesting to add  intuitive explanation on our empricial evaluation, and
 - we also thank the reviewer edcU for helping to make our writing precise and easier to understand.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the problem of learning to defer in scenarios with incomplete expert annotations and the need for balanced expert workloads. The reviewers generally agree that the paper is well-written, addresses an interesting and relevant problem, and presents a novel and sound solution. The workload management aspect is seen as particularly promising for applications where AI supports expert decision-making, such as in healthcare. The experimental results show the performance gains of the proposed method compared to baselines. Meanwhile, some weaknesses and limitations are noted. The method's scalability with respect to the number of human experts is a concern, and while the authors discuss potential remedies like clustering, further investigation is needed to assess their effectiveness, especially in settings with diverse experts. The trade-off between deferring fewer cases versus deferring the most uncertain cases could be discussed in more detail. Additionally, the current model assumes static expert performance, and adapting it to handle fast and slow-changing expertise performance would be valuable. Overall, the reviewers considered it a good contribution to the field.