Bayesian Inference for Correlated Human Experts and Classifiers
Authors: Markelle Kelly
Alex James Boyd
Sam Showalter
Mark Steyvers
Padhraic Smyth
Venue: ICML 2025 poster (2025)
Keywords: human-ai
consensus
bayesian
Primary area: probabilistic_methods->everything_else

Abstract: Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.

Bayesian Inference for Correlated Human Experts and Classifiers
Markelle Kelly' Alex Boyd? 2
Sam Showalter
Mark Steyvers* Padhraic Smyth"
Abstract
and expert predictions when both available (Steyvers et al., 2022: Choudhary et al., 2023).
Applications of machine learning often involve making predictions based both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class prob- ability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling Limulation-based inference about the utility of additional expert queries, as well as inference of posterior distri- butions over unobserved expert labels. We apply our approach to two real-world medical classifi- cation problems, as well as to CIFAR-10H 10H and ImageNet-166., demonstrating substantial reduc- tions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.
In this paper we focus on the problem of predicting the class labels that a group of human experts will produce, leveraging the output of pre-trained classifiers. Consider the following example: a radiology department at hos- pital has five expert radiologists, who are responsible for classifying X-ray images of patients into one of K classes. The experts do not necessarily agree each image, given that the images are noisy and that the experts have different backgrounds and levels of expertise. Ideally, for each X-ray, the hospital would elicit class labels from all five radiolo- gists and aggregate their predictions (e.g.. via majority vote) (Owen & Grofman, 1986; Surowiecki, 2005). From the hos- pital's perspective this aggregate acts as ground truth, but it is too expensive obtain on routine basis. Now suppose that the hospital gains access to a pre-trained machine learn- ing classifier with minimal query cost per image. A natural question in this context is: how can the classifier be used to learn a policy that can minimize the average querying cost while accurately predicting the final majority vote?
1. Introduction
We address this problem in a general setting for K-way clas- sification with H experts, where the experts produce hard labels-i.e., "votte." Examples æ are generated in an IID manner from some underlying unknown distribution p(z). Also available per example x are K-ary class probabilities from one or more pre-trained classifiers. Conditioned on model predictions and observed expert votes, we wish to predict the remaining unobserved expert votes, and further, to predict aggregate function of these votes, e.g., corre- sponding expert consensus or their unanimous agreement (or not). Throughout the paper we will primarily focus on the consensus function, and will use the term "consensus" to refer to the class label with the most votes (with ties broken randomly), which can include cases where the consensus vote is not a strict majority (e.g., with three labels and ten experts the consensus could be label that gets four votes, with the other two getting three votes each).
Machine learning systems are now commonly deployed across a variety of real-world applications, including medi- cal diagnosis, autonomous driving, and scientific discovery. In many of these applications there is significant interest in developing remi-autonomous workflows, where the predic- tive abilities of both human experts and machine learning models are harnessed to perform more effectively in combi- nation than either experts or models on their own (Beck et al., 2018; Bien et al., 2018; De-Arteaga et al., 2020; Jarrett et al., 2022: Gal et al., 2022; Straitouri et al., 2023; Corvelo Benz & Rodriguez, 2024). A number of different problems have been explored in this context, including building models that "leam to defer" to human experts (Madras et al., 2018: Verma & Nalisnick, 2022) and optimally combining model
'Deparmment of Computer Science, University of California, Irvine, USA 'GE HealthCare, USA "Strpe USA Department of Cognitive Sciences, University of California, Irvine, USA. Corre- spondence to: Markelle Kelly <kmarko@@ci.cduu>
The classifiers do not need to be trained specifically to pre- dict the votes of the H experts. In fact, our setup is par- ticularly well-suited to situations where the classifier was trained on some labeled dataset that was generated indepen- dently from the /l experts, with the potential for significant
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
distribution shift between the classifier predictions and the human labels and where, result, the relationship be- tween the classifier probabilities and the H human votes can be quite noisy.
Another relevant line of work focuses on the problem of assigning delegating tasks between humans and AI. For example, classifiers can learn to defer or delegate to human (Madras et al., 2018; Mozannar et al., 2023; Gao et al., 2023) and this can be customized for individual experts (Raman & Yee, 2021; Hemmer et al., 2023; Keswani et al., 2021; Verma et al., 2023). Other approaches model the behavior or abilities of humans and/or classifiers for the purpose of delegation (Ma et al., 2023; Fügener et al., 2021; Fuchs et al., 2022; Tudor Ionescu et al., 2016). In these settings the goal is to choose single agent to make a decision or perform task. In contrast, our work involves flexibly querying multiple (human and AI) agents in a sequential task-by-tatt -tsk manner and learning to leverage subsets of observed labels to infer unobserved labels.
In particular, our approach models a latent correlation struc- ture among human experts and pre-trained classifiers. We develop Bayesian model that jointly infers dependence among these experts and classifiers in an online manner given sequence of examples. This allows for updating the latent model, querying the appropriate experts on a per- example basis, and inferring the predicted label of each expert. Our primary contributions are as follows.'
We propose a Bayesian framework for online querying and prediction in the context of predieting the beliefs of a set of correlated human experts (Sections to 6).
2.2. Prediction Aggregation
We demonstrate that the proposed Bayesian approach outperforms alternative methods for this problem- using fewer expert queries on average to perfectly pre- dict aggregate functions of expert scresscroos mul- tiple experiments with real-world image classification tasks (Section 7).
Our work shares some similarities with supra-Bayesian methods, which aim to aggregate expert predictions via posterior distribution over the true beliefs of decision mak- ers (e.g., Winkler, 1981; Jouini & Clemen, 1996; Lindley, 1986). However, the frameworks underlying these meth- ods differ from our setting in that they do not allow for the combination of partially observed "hard" votes (e.g., expert labels) with related probabilistic inputs (e.g., a classifier's predicted distribution over labels). Further, many of these methods assume experts are independent from one another, which can be quite restrictive in practice (Wilson, 2017). Ex- ceptions in this context are the methods of Trick & Rothkopf (2022) and Pirš & Štrumbelj (2019), which explicitly model correlations between classifiers (but still do not allow for the incorporation of human votes).
2. Related Work
2.1. Human-AI Collaboration Workflows
A wide variety of workflows for human-Al collaboration in classification tasks have been established (Green & Chen, 2019; Rastogi et al., 2023; Donahue et al., 2022), including techniques that (implicitly or explicitly) learn about specific human experts.
More closely related to our work are Bayesian methods for combining human and classifier predictions. Kim & Ghahramani (2012) propose a method for combining class label votes from classifiers (or humans), modeling the re- lationship between the classifiers' votes and the ground truth label, including dependence between classifiers. In contrast, Steyvers et al. (2022) first query categorical con- fidence scores from humans, rather than a hard vote, and then integrate them with predicted classifier distributions. Finally, similar to setting, Kerrigan al. (2021) com- bine the probabilistic output of a single classifier with the single hard vote of human expert; we will use modified version of this method as a baseline in our experiments.
However, these prior approaches generally consider human- AI settings that are quite different from the problem ad- dressed in this paper. For example, there is an extensive body of work on customizing Al for its human teammates. One family of methods involves training classifiers that are complementary to human abilities or expectations (Bansal et al., 202 Hemmer et al., 2022). Al-advised decision- making (Bansal et al., 2021b; Zhang et al., 2020; Liu et al., 2021: Schemmer et al., 2023). in which AI support is pro- vided to aid human decision-maker, can also be tailored to specific individuals (Noti & Chen, 2023; Bhatt et al., 2025). These types of workflows differ significantly from our setting, in which the goal is not to provide support to human experts, but rather to avoid querying them as much as possible by predicting their labeling decisions. Further, our approach allows for the use of pre-trained black-box classifier, avoiding the requirement for any data about the human experts to be collected ahead of time.
Importantly, all of these prior methods prioritize the perfor- mance of humans and/or classifiers relative to some inde- pendent ground truth. This stands apart from setting wherein the human votes are considered to be the ground by the practical situation where we wish to use model to infer the votes of a set of experts (c.g.. radiologists) without the need to query all of them. In other
All code and datasets used in this paper are available at
2

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
words, unlike the combination methods discussed above, our method does not combine the model and human pre- dictions to predict some ground truth y, but instead uses classifier outputs and partially observed human votes to pre- diet the remaining unobserved human votes. The goal is not to increase overall accuracy (e.g., of humans alone or classifiers alone), given that in our setting perfect accuracy achievable by simply querying all human experts. Instead, the goal is reduce the number of human experts that need to be queried for their predictions, while still accurately inferring what the group as a whole predicts.
Ω
Σ
0H
PM
2
Yj jCU
yi ico
T
The closest work that we are aware of, which predicts hu- man expert labels in a Au setting, is that of Showalter et al. (2024). Their method models the consensus of human experts, incorporating the predictions of classifier. How- ever, their methodology differs significantly from ours in that human experts are treated as exchangeable and non- identifiable. The focus of their approach is to decide how many experts to query-not which experts to query. Like- wise, their approach does not model correlations among experts, leverage information about specific experts, or pre- diet the votes of individual experts. Our work can be seen as an extension of that of Showalter et al. (2024), in the sense that can handle the realistic case where experts are not exchangeable, e.g., when experts are highly corre- lated, or vary substantially in accuracy (either overall or on a class-wise basis).
y+
T
Figure 1. Graphical representation of the assumed generative model. Grey symbols are observed and assumed known, white are random and unobserved. Circles are random variables and rectan- gles are deterministic transformations of other values. Notation for hyperparameters and time-dependence / is omitted for simplicity.
AK i = 1..,=,.,,MM Thus, we can collect predictions from H + MM total agents (we will use "agents" to refer to both human experts and classifiers). We index these agents as fol- lows: .M = {1, M) is the set of indices corresponding to classifier models and H = (M + +1, ..., H + +M} the set of indices corresponding human experts. Further, for any specific input ar(t), we will use O C 7((1) to denote the set of human experts whose predictions have already been observed and (t(+) = H({) (++) to represent experts who have not (yet) been queried, where these sets are updated as we sequentially issue queries.
3. Problem Setting and Notation
Consider stream of imputs (e.g., images) (x(1) E X indexed sequentially by ( = 1,... T. generated IID from some un- derlying (typically unknown) distribution p(z). The task is to associate each (=(1) with a class label y E {1..... K)- To make this prediction, H identifiable human experts are available; for any input 2:(() each expert i can be queried for their class label vote yi"
For simplicity we assume that all models have zero cost and that every human expert has the same fixed cost per query. The goal is to achieve the optimal cost-accuracy trade-off minimizing the number of expert queries given a desired level of accuracy in predicting y²²), in an online sequential fashion over classification inputs (1)
(t) E K}.
The target we wish to predict is some function of these expert votes y. f(v" (...))))) ). To illustrate our method- ology, in the remainder of the paper we will primarily focus on consensus (as defined in Section 1) for our definition of " but we note that our proposed methodology can sup- port any function defined over expert votes. For examplc, decision-makers might be interested in being able to predict any expert (one or more) out of the H experts predicts a partieular class label, e.g., a rare deadly disease in a medical setting. In Section 7.4.3 we explore the use of two such functions of (other than consensus) over expert votes.
(t),
4. A Bayesian Model for Predicting Expert Labels
We seek model that (1) captures the relationships between the agents' predictions and (2) leverages this information to predict human expert votes (and thus consensus), condi- tional on any set of observed model and human predictions. To facilitate this, we model this voting process in a hicrar- chical Bayesian fashion using the graphical model shown in Figure 1. The outer plate (indicated by T) refers to the sequence of T examples; nodes within this plate implicitly depend on (or, more specifically, on z! (1), e.g., yi is really
In addition to the human experts, suppose we have access to M pretrained machine learning classifiers fi X' +
In our experiments focus the case of single classifier (i.c., M = 1), but the framework is general.
3

---

Bayesian Inference for Correlated Human Experts and Classifiers
y/
where is an index over human experts. For brevity, in the remainder of the paper we suppress when it is clear from context.
While the assumed model allows for correlations within and between agent predictions, we found in our experiments that the resulting predictive distributions over human votes have the potential to be miscalibrated. To account for this, we allow for a global temperature parameter T ~ [V[00, a,) to temper each latent human probability vector 0,. This results in the following distribution over votes:
Given an example each expert E H can produce label or vote yi E fI...., } drawn from a Kaary categorical distribution with latent probabilities 0 conditioned on a particular æ. These latent expert probabilities 02 can be in- terpreted as human confidence scores, similar to a predicted distribution from classifier. Importantly, these expert prob- abilities are not directly observed; only associated "hard decision" votes Y/ E O C H can be observed by query- ing. In addition, because we assume classifier queries have zero cost, predicted model probabilities 0, for E M are assumed to be available (observed) for each example.
yi 03, T Categorical(TS((() +)) for i E H
(3)
with TS(O,, -) := softmax(0:/7) x exp(0:/T) e A where 1/T can be viewed an exploration parameter (e.g.. Daw et al. (2006)).
The posterior distribution for μ4, Σ, and (learned from the set D of observed agent predictions for previous examples xx11) ..., ...,,.(-- ((-1)) is intractable. In practice, we sample from this posterior distribution via MCMC sampling. In particu- lar, at each time point I, we draw a set S of samples μ², E and 7 (where the superscript denotes value sampled from the posterior). Further details about the sampling pro- cedure can be found in Appendix D.
We transform the probability vectors to span the real domain, allowing us to leverage multivariate normal distribution as a mrior-similas to Blei & Lafferty (2006) and Pirš & Strumbelj (2019) and building on the logistic normal model (Atchison & Shen, 1980). (An alternative approach would be an extension of the Dirichlet distribution that allows for correlations between classes and pair of agents, e.g.. Trick& & Rothkopf (2022)). Formally, let the additive logistic transform y A RRK K- be defined as
5. Conditional Simulation of Expert Labels
Given |S] posterior samples for our underlying model pa- rameters (based on observations up to time f - 1), we can then perform inference about unobserved expert votes for a new input (,((), In particular, conditioned on the observed model predictions (logits) zM and any observed human votes yo, we want to characterize our uncertainty about the final expert agreement y!"² := f(x) = f (()))))) (where votes you have not yet been observed). 3
011 (0) log [[K]
[
0[K 1]] [[K]
log
(1)
where 0[k] indicates the 1thh entry in vector 8. For a given probability vector 6,, latent or realized, we denote the as- sociated transformed values as their corresponding logits: 2, := (0,) for i E M U H. With this, we can capture correlations between classes and agents by jointly model- ing both the observed model logits 2M= == [Ri,.... 2M and latent human logits zн := [IM-I:...,,MM =M--H" with a multivariate normal distribution
To characterize our belief about y!", we estimate p(y. k yo, RM, D) for each class A. It can be shown that
ply, k yo: ZM; D)
=
~ N(,,E
(2)
2M
E ED LRM.
E Levd
[[y. = k))(Uo
=00]]
with mean μ E Id and covariance Σ E Raxd for d (K' 1)(M+ + H). Note that while each time step will have specific set of logits z(0) associated with the in- put ( we assume they are all distributed with the same mean μ and covariance Σ. Priors are placed on these global parameters Jr, E, with the mean having a normal distribution, μ ~ (O op), and the covariance having half-normal distribution on the variances, a E R& [N](0,, ),), and tribution (Lewandowski al., 2009) the correlations, Ω E Rdxd ~ (T)()), where Σ == di. The val- ues To and n > 0 are all hyperparameters, which help control the model's 's exploration/exploitaiii behavior choosing higher oい and o, or >1 will result in higher uncertainty and thus more queries issued early-on.
(4)
(see Appendix A). Note that in the case where no votes have
been observed yet, with O = 0. this is simply:
p(v. = k I.M D)
E A.E.T
E LUN
[[y, = k)]
(5)
E.
(LKJ) dis-
Based on Equation 4, we can sample from the posterior distribution over St via Algorithm 1. (An annotated version of Algorithm with additional explanation about each step
IIn our description of inference and in our experiments we will generally focus on consensus, i.c., f(x) = m)de()xx)

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
can be found in Appendix B.) From these posterior samples for ye we can (for example) make a point prediction for the consensus y. argmaxy, (p(y. k)..
Algorithm 2 Compute the expected entropy of y, after
observing yj
exp entropy +- 0
P(U. yo, 2M) + apply Algorithm 1 with U, O.
Algorithm 1 Estimate p(y.) given observed predictions yo, 2M AM and the set S of samples from the posterior for N. Σ, and r
for k E 1.... K do
O' + O + {i} u' + u \\{ b/y k k
p-y-samples + ] {stores |S] samples for II( k)p fo),i.e. the inner expression in Equation 4}
p(y- I Iyo, [y0,3/]: zM) + apply Algorithm with U' 0',(yo,y/ v/
for sample E do
entropy H (p(y- [vo, 1/y1, =M))
n,, , f- sample
exp entropy += plys = V/ 1o, 2M) entropy end for
if [ 21 then
z 2 + draw from p(#H 2M) [N(w", 23), con- ditioned on 2M)
vu +- draws from Categorica((TS(((())..)) for i E U. concatenated
yo: ZM, D). Obtained similarly to ply. = k!...) in the previous section, have
v. + f(yo,
plyo + IIEo TS()(=f), TB)l
p(u; = k yo, RM, D) xx
else
z + draw from p(au zM) (M('", 2"), condi- tioned ZM}
E Er
E
[T = k)p(vo 20)1 M5l2,
E
vi t draws from ( +)) for
i E U, concatenated
After querying the selected expert, we update O U, and yo to reflect the corresponding observed vote and the process can be repeated, deciding which (if any) expert to query next. In our experiments, we halt querying for an example z=() when the probability of making an error (in terms of predicting the consensus, according the Bayesian model) falls below some error threshold If the model is well- calibrated this allows us to directly control the number of expert queries. on per ver-example basis, to achieve this error we will return to this point in Section 7.4.
vr - f((t)
plyo =8) + (no observed votes)
end if
p-y w.y.samples.ee appppleeeappp + OneHot [y(])
end for ply+1
No, 2M) t mean of p y samples, normalized
6. Choosing Experts to Query
Now suppose, for some input (*)), that have observed some set of model predictions 2M and O human expert predictions yo. To determine which remaining expert j E l to query next, we want to estimate how much in- formation about the consensus ye each candidate's vote will provide. More specifically, we would like to estimate the information gain associated with observing the vote of each expert j. To do so, for each potential prediction v// E {1...., K} of expert j. we act as if this hypotheti- cal prediction was actually observed, i.e., as if y = 3/z- and again draw posterior samples of y. (following Algo- rithm 1), including this hypothetical prediction with yo. We can then compute the expected entropy of the result- ing p(y. [yo vy], 2M, D). To choose the next expert to query, we repeat this process for each candidate pert and select the expert that maximizes the information gain (or equivalently, minimizes the expected entropy), i.e., argmin jeu (E.,, (H*/ZM, [yo, yil, D)). This process is shown in detail in Algorithm
7. Experiments
We evaluate approach four real-world classification tasks with corresponding classifier and human predictions, described in detail in Section 7.1. Compared to two baseline methods (described in Section 7.2), our approach consis- tently achieves 0% error using fewer queries on average than the baselines, as shown in Section 7.3. In Section 7.4, we highlight additional results regarding calibration, the explore-exploit trade-off, and alternative aggregation functions. Note that throughout this section "accuracy" and "error" are always defined with respect to the expert consen- sus, rather than some separate ground truth.
7.1. Datasets
Our experiments use two datasets of medical images, an- notated by identifiable experts: ChestX-Rap (Nabulsi et al., 2021) and Chaoyang (Zhu et al., 2021). In addition, we include results for two datasets with simulated experts- CIFAR-10H (Peterson al., 2019) and ImageNet- 16H
To compute the expectation in the last line, we need p(ys
5

---

Bayesian Inference for Correlated Human Experts and Classifiers
(Steyvers et al., 2022). These datasets do not include identi- fiable experts, so we create synthetic experts with different labeling characteristies by combining the predictions of mul- tiple non- -odeenetifiable human annotators. These datasets vary substantially in terms of the differences in performance both between individual experts and between the experts and cor- responding classifier, in terms of the numbers of experts and classes, and the difficulty of the task itself- alg us to assess our method across variety of scenarios. We provide brief summaries of the datasets below; further details are provided in Appendix C.
expert has two classes of relative expertise (about 95% ac- curacy), and a class of relative weakness (between 80% and 90% accuracy). Across classes, the overall accuracies of these experts are between 92 and 93%. The classifier used for this dataset is an AlexNet model with 86.0% accuracy.
7.2. Baseline Methods
To provide a point of comparison, we include results for two alternative approaches. Given that our approach is (to our knowledge) the first to-date to address our consensus prediction task, we modify two existing related methods (introduced in Section 2) to create appropriate baselines.
ChestX-Ray (Wang et al., 2017) is real-world radiology dataset released by the NIH, consisting of chest X-ray im- ages, labeled as normal or abnormal. Nabulsi et al. (2021) collected labels from five American Board of Radiology certified radiologists for 810 of the X-ray images. These radiologists range in accuracy from 83 to 94%. We test our method using these five sets of human expert labels and the DenseNet-basee classifier proposed in Tang et al. (2020), which achieves 85.7% accuracy on the test set.
INFEXP + c-greedy querying: The original INFEXP method (Showalter et al., 2024) provides Bayesian frame- work for querying experts given classifier predictions. It can be used to decide how many human experts to query to predict consensus, but has no mechanism for choosing which specific expert to query next (as experts are treated as non-identifiable). To apply this method to our setting, we exchange the random querying component of the original INFEXP method with e-greedy querying. To do so, we keep track of the observed accuracy (relative to consensus) of each expert. With probability a random expert is queried; otherwise, experts will be queried in order of their observed accuracy. Note that this accuracy can only be observed when enough experts are queried to determine the true consen- sus, e.g., if the querying stops after observing only single expert vote, we do not update these accuracies.
Chaoyang (Zhu et al., 2021) is a dataset of colon images, where the task is to classify each image as either normal, serrated, adenocarcinoma, or adenoma. Three pathologists labeled each image in the 2139-image test set, with accura- cies between 82 and 99%. We use the ResNet-based model described in Zhu al. (2021) as the corresponding classifier, which attains 80.5% accuracy on the test set.
CIFAR-1OH (Peterson et al., 2019) is expanded ver- sion of 10, dataset for 10-class image classifica- tion (Krizhevsky& & Hinton, 2009), which includes 50 labels from human annotators for each of the 10,000 images in the CIFAR-10 test set. CIFAR-1OH annotators are not identified individually, so we modify the dataset to create synthetic experts with varying wass expertise. First, we combine classes to create three meta-classes (e.g., classes 1-3 become the new class 1). Then, we combine human annotations to create three experts: each expert has very high accuracy (approximately 99%) for two classes and lower accuracy (approximately 70-80%) for the remaining class, such that each expert has a different class (A: E {1, of relative weakness. Across classes, the overall accuracies of these experts are between 90 and 94%. For this dataset, we use as our classifier ResNet model with 91.5% accuracy.
The original confu- sion + calibration method (Kerrigan et al., 2021) combines model and human predictions, leveraging confusion matri- ces between a human expert's predictions and some ground truth. To create a baseline that can handle multiple experts and consensus prediction, we extend this approach by leam- ing confusion matrix for each human expert, skipping con- fusion matrix updates for experts whose predictions were not observed, and defining confusion with respect to the consensus rather than some separate ground truth. (As with the INFEXP + e-greedy baseline, this means that the model is only updated when the expert consensus observed.) In this manner we can use this method to obtain a distribution over labels for the consensus, allowing to predict the final majority vote and estimate the corresponding uncertainty. In experiments with this baseline, to use this method to choose which expert to query next, we apply Algorithm 2, obtaining p(y, [yo, v5], 2M) via this modified confusion calibration method rather than via our model.
Confusion matrices + calibration:
ImageNet-166 (Steyvers et al., 2022) is similar to CIFAR- 10H, comprised of human annotations for the ImageNet dataset (Deng et al., 2009), with six (non-identifiable) hu- man labels per image. We use a modified subset of this dataset which includes three classes and three experts. Again we combine classes and human predictions to simu- late experts with class- class-wise strengths and weaknesses; each
7.3. Error vs. Cost
We generate error-cost curves by systematically varying the error threshold e at which the model stops querying experts
6

---

Bayesian Inference for Correlated Human Experts and Classifiers
ChystX-Ray (II=))
for each example. For any specific threshold value, our model sequentially processes each example 2(1) For each z(9, the model decides whether to query an expert (and, if so, which expert to query), makes that query, and repeats the process, stopping when the estimated error is below the threshold e, At that point, the model makes a final prediction. In addition, the set of posterior samples updated based on the observed predictions from the model and experts, before moving to the next example. All inferences based classifier predictions and the partially-obseree votes of any experts queried on the current or earlier examples, as described in Sections 5 and 6. Details about MCMC-based inference can be found in Section 4 and Appendix D.
Confusion calibration
INFEXP C-greddy
Bayesian correlation model
Classificr-only
Chaoyang (H=3)
0.20
We use sets of 250 examples to compute the error rate and the average querying cost for all three methods. We focus this relatively small number of examples per run the goal of our method is to operate in an online manner, with relatively few data points to learn from. We run each ex- periment with 12 different sets of examples (for datasets with less than 3000 instances, we create different sets of 250 via shuffling, since each of the methods evaluated is sensitive the order in which data points are seen). The results reported are averaged across these 12 runs.
0.10
0.00
Figure 2 shows error-cost plots for our Bayesian method, in comparison to our baselines, for all four of our experimental datasets. These plots include the actual values observed as well as lowess-smoothed curves for each method. Note that because the curves are generated by varying the error threshold e systematically (i.e.. the actual error rate and number of queries are not directly controlled), the locations of specific points and distances between points vary.
1OF (I=3)
0.08
0.05
Error
0.04
0.02
For both of the real-world medical datasets (ChestX-ray and Chaoyang), our method (in green) reaches 0% error with fewer queries on average than either baseline method. In particular, while INFEXP (in orange) requires 3.16 and 1.82 queries on average to perfectly predict consensus (i.e., achieve zero error) for ChestX- rry and Chaoyang, respec- tively, our method requires only 2.55 and 1.58 queries. Al- though the confusion + calibration method (in blue) can achieve very low error rates, it never reaches 0% (i.e., per- feet accuracy), even when all experts are queried. An intu- ition for this that the method does not "know" it is trying to predict the consensus even if two out of three experts predict the same class (i.e., forming consensus), it is pos- sible that this method will predict a different class (e.g., in agreement with the third, dissenting expert).
0.00
ImageNet- 16T (II=3)
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
We demonstrate similar results in our experiments with simulated experts. As shown in the bottom half of Figure 2, our method achieves competitive (often the lowest) error rate across error thresholds for both CIFAR-IOH and ImageNet- 16H. Again, the confusion calibration method never reaches 0% error; here, neither does INFEXP, even
Average Number of Queries
Figure Average number of expert queries versus classification error rate for each of our four datasets.
7

---

Bayesian Inference for Correlated Human Experts and Classifiers
with error threshold of O. Overall, in comparison to the baseline methods, our method consistently minimizes the error rate (i.e., reaches 0% error), and does so using the fewest queries on average.
0.25
CheatX Ray
0.20
Chaoyaug
CIFAR-LOfT
Actual Error
InngeNet-1of1
0.15
7.4. Additional Results
0.10
7.4.1. CALIBRATION AND ERROR BOUNDING
0.05
With our approach, the stopping rule for querying is based on whether or not the model's estimated error (in predicting consensus), for each example, is below an error threshold e or not; not, the querying continues with another ex- pert. While the classifier's own estimate of its error may be optimistically biased, we conjecture that the uncertainty estimates from our Bayesian framework will be reasonably well-calibrated. To assess model calibration, we compute expected calibration error (ECE), which compares the actual accuracy of a model with its confidence estimates (Kumar et al., 2019) (in our case, the confidence estimates are the confidence of the Bayesian model in its consensus predic- tion for each example 201), after it has finished querying). Indeed, in our experiments, we find that our model's error estimates align well with its actual error rate, achieving low ECEs of generally less than one percent (see Table 1). Lower error thresholds tend to correspond to better ECE scores, which is what we would expect, as the model must make more queries to achieve a lower error rate, and can leverage this additional data to improve its uncertainty estimates.
0.00
0.00
0.05
0.10
0.15
0.20
Error Threshold
Figure 3. The error threshold vs. actual error rate of our model.
each set. The contrast between the two demonstrates this explore/exploit trade-off: more querying in the first 50 exam- ples as the model learns parameters (and makes predictions), and less querying the last 50 as it focuses on prediction.
X-Ray
Chaoyang
CIFAR
ImageNet
First 50
2.56
2.10
1.98
2.07
Last 50
1.57
1.61
1.10
1.32
Table 2. Average number of experts queried by our model for the first and last examples (out of 250 examples in total) using an error threshold of e 0.01.
Further, because our method operates in an online fash- ion, it is especially important that it can handle distribution shifts in the input data. In the context of our motivating radiology example, for instance, the distribution of X-ray image data could suddenly shift if the X-ray machine is replaced with newer model its operating parameters are changed. To investigate this scenario, we create version of the ImageNet- 16H dataset with such a shift. ImageNet- 1 66H includes images with varying levels of noise (Steyvers et al., 2022); we create 48 test sets of 250 images where the first 125 are low-noise and the next 125 are high-noise. To adapt our model to this setting, we use "sliding window" of size 50, i.e., using only the 50 most recent examples in making predictions (see further details in Appendix D).
e
X-Ray Chaoyang
CIFAR
ImageNet
0.05
0.009
0.008
0.007
0.009
0.025
0.004
0.003
<0,001
0.003
0.01
<0.001
<0.001
<0.001
<0.001
Table ECC our method (in terms predicting consensus) for three different error thresholds on each dataset.
As these uncertainty estimates are generally well-calibrated, setting the error threshold is practical strategy for bound- ing the model's error rate. Figure shows how our model's actual error rate relates to the specified error threshold. Be- cause the model continues querying experts until its uncer- tainty falls below the threshold, many model predictions have an estimated error well below e (e.g.. if a model's un- certainty at a certain point is 0.051 but e = 0.05, the model will issue another query, likely reducing the uncertainty well below 0.05). Thus, as reflected in Figure 3, the model's error rate is consistently much lower than its error threshold.
Figure 4 shows the number of queries made by our model over time. The first half of this plot demonstrates the ex- ploration/explloiation strategy we expect: the model makes more queries on average when it has seen very few data points, using fewer and fewer queries as time progresses. However, the halfway point, when classifier and expert accuracies drop (due to the shift to noisier images), the aver- age number of queries immediately increases. Again, the model adapts to the new distribution, the number of queries falls falls- although it remains higher than in the easier, low- noise first half. By increasing exploration after this major shift in the input data, our Bayesian model maintains a 0%
7.4.2. EXPLORATION AND EXPLOITATION
Another advantage of our Bayesian approach is that it nat- urally balances exploration and exploitation in querying over time. Table shows the average number of experts queried by our model in the first and last 50 examples of
8

---

Bayesian Inference for Correlated Human Experts and Classifiers
aggregation functions, are shown in Figure 5. Compared to the original curve, the error-cost curve for Juny starts with a higher error rate for the same number of queries (on average), but the curve drops off more steeply, reflecting that a single query can be much more informative in this case than in the majority vote case. In contrast, when using fall- the model makes more queries on serrage-demonstratee that, on average, more expert votes are needed in this setting,
Average Number of Queries
2.0
Shift (t= 125)
1.8
1.4
8. Discussion and Conclusion
1.2
50
100
150
200
250
Data Index
Limitations There are a number of potential limitations to our overall approach and experiments. First, our evalu- ation limited to image classification tasks; our findings need not necessarily generalize to other settings, e.g.. using text or tabular data. Further, for computational and estima- tion efficiency, our experiments focused on settings with relatively small numbers of class labels K and numbers of agents (M + H), given that the underlying covariance matrices Σ grow quickly with K, M, and H. For larger values of K, a more structured model for Σ (c.g., low-rank approximations, informative priors) may be useful and an interesting avenue for further research.
Figure 4. The average number of queries issued by our model over time (averaged over 48 experimental runs) for modified version of ImageNet- 16H with a distribution shift t=125.
error rate despite the experts' drops in accuracy.
7.4.3. ALTERNATIVE AGGREGATION FUNCTIONS
Although we have focused on predicting majority vote, our method models the belief of each expert individually, en- abling the prediction of any function over the expert votes (y). (F). In this section we investigate the use of our model with two practical alternative aggregation functions.
Future Work As our model is relatively flexible, a num- ber of interesting extensions are worth exploring. One such extension would be to allow human experts to provide verbal levels of confidence for their predictions (e.g., in Steyvers et al. (2022)), which then could be associated with latent confidence scores. Another potentially useful direction for future work, particularly in real-world settings, is the incor- poration of different query costs for different agents.
In the case of predicting the presence of rare disease, even one positive prediction from single radiologist may be suf- ficient reason to pursue further testing (Adachi et al., 2025). Formally, in the case where y/ E {0, 1}, we can use the aggregation function fany (y1...., yн)
Alternatively, we may want to give positive diagnosis only if the experts are unanimously positive, i.e., aggregating with (y) yn1) = I1 Vi-
Summary In this paper we develop systematic Bayesian approach to address the under-explored problem of making joint inferences about expert labels in the context of K- ary classification,, conditioned on partial information. The method is intuitive and flexible and it can be readily adapted for various settings (for example, using the sliding- window approach from the previous section handle distribution shifts, or by simulating the consensus of only a subset of experts in the case that some experts become unavailable). Further, as shown in our experiments, this method consis- tently achieves error rates below a specified error thresh- old, and minimizes error with lower average querying costs than those baseline methods. Altogether, these findings demonstrate the potential of our methodology for streamlin- ing prediction in real-world human- Al classification tasks.
0.10
Majority vete
0.08
At least one (fany)
Unanimous (fan)
0.06
Error
0.04
0.02
0.00
Averuge Number of Queries
Figure 5. Error vs. cost for the ChestX-ray dataset in predicting majority vote (green), in comparison predicting whether any expert will vote (red) and whether the vote will be unanimous (purple).
A Acknowledgements
We would like to acknowledge the specific and construc- tive feedback from the reviewers and Area Chair, which improved the quality of the final paper. This research was
Error-cost curves for the ChestX-ray dataset, using these

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
supported in part by the National Science Foundation un- der awards RI-1900644 and RI-1927245 and in part by the Hasso Plattner Institute (HPI) Research Center in Machine Learning and Data Science at the University of California, Irvine.
cation tasks. Monthly Notices of the Royal Astronomical Society, 476(4):5516-5534 2018.
Bhatt, U., Chen, V., Collins, K. M., Kamalaruban, P., Kal- lina, E., Weller, A., and Talwalkar, A. Learning person- alized decision support policies. In Proceedings of the AAAI Conference Artificial Intelligncee volume 39, pp. 14203-14211, 2025.
Impact Statement
Our method leverages the outputs of classifier to predict the class labels provided by human experts, while reducing the costs of consulting human experts. This has the potential for positive societal impact, by improving accuracy in terms of the expert consensus when it is not practical to query many human experts, or by saving the time and money that would have been required to query each individual expert. However, while our method is designed to reflect expert agreement, it can make a prediction for particular input without querying any experts. Thus, while this methodology may be useful for scenarios in which human expertise is critical, it could also result in misplaced confidence that human experts were involved in a particular class prediction, and is not directly applicable to high-stakes settings where it is necessary to have human in-the-loop. Ultimately, responsible deployment with human oversight is crucial for this methodology to be used safely and effectively.
Bien, N., Rajpurkar, P., Ball, R. I.., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Ycom, K. W., Shpanskaya, K., Halabi, S., Zucker, E., Fanton, G., Amanatullah, D. F,, Beaulieu, C. F., Riley. G. M., Stewart, R. J.. Blankenberg, F. G., Larson, D. B., Jones, R. H., Langlotz, C. P., Ng, A. Y., and Lungren, M. P. geep-learning-asiiseee sis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLOS Medicine, 15(11):1-11 11 2018.
diagno-
Blei, D. and Lafferty, Correlated topic models. In Ad- vances in Neural Information Processing Systems, vol- ume 18, pp. 147-154. MIT Press, 2006.
Choudhary, V., Marchetti, A., Shrestha, Y. R., and Puranam, P. Human-AI ensembles: When can they work? Journal of Management, pp. 01492063231 1994228,,998 2023.
Corvelo Benz, N. and Rodriguez, M. Human -ainand- cali- bration for Al-assisted decision making. In Advances in Neural Information Processing Systems, volume 36, pp. 14609-14636, 2024.
References
Adachi, M., Chau, S. L., Xu, W., Singh, A., Osborne, M. A., and Muandet, K. Bayesian optimization for building socief-inf consensus. arXiv preprint arXiv:2502.07 166, 2025.
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. Cortical substrates for exploratory decisions 2006.
in humans. Nature, 441(7095):876-88
Atchison, J. and Shen, S. M. Logistic-nornal distributions: Some properties and uses. Biometrika, (7(2)2261- 1980.
De- Arteaga, M., Fogliato, R., and Chouldechova, A. A case for humans-in -nttu------th--oop- Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20. ACM, April 2020.
Bansal, G., Nushi, B., Kamar, E., Horvitz, E., and Weld, D. S. Is the most accurate Al the best teammate? optimiz- ing Al for teamwork. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 11405- 11414, 2021a.
Deng, J., Dong, W., Socher, R., Li, L.-J, Li. K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pat- tern Recognition, pp. 248-255 2009.
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. Does the whole exceed complemen-
Donahue, K., Chouldechova, A., and Kenthapadi, K. Human-algorithm collaboration: Achieving complemen- tarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, pp. 1639-1656, New York, NY, USA, 2022. ACM Press.
its parts? the effect of AI explanations tary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 21, New York, NY, USA, 2021b. Association for Computing Machinery.
Beck, M. R., Scarlata, C., Fortson, L. F., Lintott, J., Sim- mons, B., Galloway, M. A., Willett, K. W., Dickinson, H., Masters, K. L., Marshall, P. J., et al. Integrating human and machine intelligence in galaxy morphology classifi-
Fuchs, A., Passarella, A., and Conti, M. A cognitive frame- work for delegation between error-pront AI and human agents. In 2022 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 317-322, 2022.
10

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
Fügener, A., Grahl, J., Gupta, A., Ketter, W., and Taudien, A. Exploring user heterogeneity in human delegation behavior towards AI. In International Conference Information Systems (ICIS), 2021.
Kumar, A., Liang, P. S., and Ma, T. Verified uncertainty cal- ibration. In Advances in Neural Information Processing Systems, pp. 3787-3798, 2019.
Lewandowski, D., Kurowicka, D., and Joe, H. Generating random correlation matrices based on vines and extended onion method. Journal of Mullivariate Analysis, 100(9); 1989-2001, 2009.
Gal, Y. K. Segal, A., Kamar, E.,. Horvitz, E., Lintott, C., and Walmsley, M. new workflow for human-Al col- laboration in citizen science. In Proceedings of the 2022 ACM Conference on Information Technology for Social Good, GoodIT '22, pp. 89-95, New York, NY, USA, 2022. ACM Press.
Lindley, D. Making Decisions. Wiley, 1986.
Liu, H., Lai, V., and Tan, C. Understanding the effect of out- of-distribution examples and interactive explanations on human-AI decision making. Proc. ACM Hum. comp-uC Interact., 5(CSCW2), 2021.
Gao, R., Teechasschnn,, M., Ar-Artaaga, M., Han, L., Sun, W., Lee, M. K., and Lease, M. Learning comple- mentary policies for human-Al teams. arXiv preprint arXiv:2302.02944 2023.
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., and Ma. X. Who should I trust: AI or myself? lever- aging human and Al correctness likelihood to promote appropriate trust in Al-assisted decision maaiiig.. In Pro- ceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2023.
Green, B. and Chen, Y. The principles and limits of Angorithm-in-the-baee decision making. In Proc. ACM Hum.-Comput Interact., volume 3, New York, NY, USA, 2019. ACM Press.
Hemmer, P., Schellhammer, S., Vossing, M., Jakubik, J., and Satzger, G. Forming effective human-Al AI teams: Building machine learning models that complement the capabilities of multiple experts. In International Joint Conference on Artificial Intelligence, pp. 2478-2484, 2484, 2022.
Madras, D., Pitassi, T., and Zemel, R. Predict responsibly: Improving faireess and accuracy by learning to defer. Advances in Neural Information Processing Systems, volume 31, pp. 6150-6160, 2018
Hemmer, P., Thede, L., Vössing, M., Jakubik, J., and Kühl,
Mozannar, H., Lang, H.,, Wei, D., Sattigeri, P., Das, S., and Sontag, D. Who should predict? exact algorithms for learning to defer to humans. In International Conference on Al and Statistics, pp. 10520- 10545. PMLR, 2023.
N. Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 6002-6011, 2023.
Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C., Santos, E., Kiraly, A. P., Ye, W., Yang. J., Pilgrim, R., Kazemzadeh, S., et al. Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Scientific Reports, 11(1):15523, 2021.
Jarrett, D., Hüyük, A., and van der Schaar, M. Online decision mediation. Advances in Neural Information Processing Systems, 35: 1790-1805, 2022.
Jouini, M. N. and Clemen, R. T. Copula models for ag- gregating expert opinions. Operations Research, 44(3): 444-457, 1996.
Noti, G. and Chen, Y. Learning when to advise human decision makers. Proceedings of the Thirty-Seoond Internatiooal Joint Conference on Artificial Intelligence, IJCAI '23, 2023.
Kerrigan, G., Smyth, P., and Steyvers, M. Combining hu- man predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems, 34:4421-4433 2021.
Owen, G. and Grofman, B. Information Pooling and Group Decision Making. JAI Press, 1986.
Keswani, V., Lease, M., and Kenthapadi, Towards unbi- ased and accurate deferral to multiple experts. In Proceed- ings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 154-165, 2021.
Peterson, J. C.,, BBttleddy, R. M., Griffiths, T. L., and Rus- sakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9617-9626, 2019.
Kim, H.-C. and Ghahramani, Z. Bayesian classifier combi- nation. In International Conference on Artificial Intelli- gence and Statistics, pp. 619-627. PMLR, 2012.
Pirš, G. and Štrumbelj, E. Bayesian combination of prob- abilistic classifiers using multivariate normal mixtures. Journal of Machine Learning Research, 20(51):1-18, 2019.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, ON, Canada, 2009.

---

Bayesian Inference for Correlated Human Experts and Classifiers
Raman, N. and Yee, M. Improving learning-to-defer algo- rithms through fine-tuning. arXiv:21 2021.
International Conference on Artificial Intelligence and Statistics, pp. 11415-11434, PMLR, 2023.
Rastogi, C., Leqi, L., Holstein, K., and Heidari, H. tax- onomy of human and ML strengths in decision-making to investigate human-Mi complementarity. Proceed- ings of the AAAI Conference on Human Computation and Crowdsourcing, volume 11, pp. 127-139, 2023.
Wang. X., Peng. Y., Lu, L., Lu, Z., Bagheri, M., and Sum- mers, R. M. Chestx-ray8: Hospital-scaal chest x-ray database and benchmarks
weakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2097-2106, 2017.
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., and Satzger, G. Appropriate reliance on Al advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 410 422, 2023.
Wilson, K.
An investigation of dependence in expert judgement studies with multiple experts. International Journal of Forecasting, 33(1):325--33, 2017.
Winkler, R. Combining probability distributions from dependent information sources. Management Science, 27 (4):479-488, 1981.
Showalter, S., Boyd, A. J., Smyth, P., and Steyvers, M. Bayesian online learning for consensus prediction. In International Conference Artificial Intelligence and Statistics, pp.
Zhang. Y., Liao, Q. V., and Bellamy, R. K. E. Effect of confidence and explanation on accuracy and trust cali- bration in Al-assisted decision making, In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT '20. ACM, January 2020.
PMLR, 2024.
Steyvers, M., Tejeda, H., Kerrigan, G., and Smyth, Bayesian modeling human-AI complementariyy Pro- ceedings of the National Academy of Sciences, 119(11): e21 22111547119 2022.
Zhu, C., Chen, W., Peng, T., Wang, Y., and Jin, M. Hard sample aware noise robust learning for histopathology image classification. IEEE transactions on Medical Imag- ing, 41 894, 2021.
Straitouri, E., Wang, L.,, Okati, N., and Rodriguez, M. G. Improving expert predictions with conformal prediction. In International Conference Machine Learning, pp- 32633-32653, PMLR, 2023.
Surowiecki, The Wisdom of Crowds. Vintage, 2005.
Tang, Y.-X., Tang, Y.BB., Peng, Y., Yan, K., Bagheri, M., Redd, B. A., Brandon, C. J.. Lu, Z., Han, M., Xiao, J., et al. Automated abnormality classification of chest radiographs using deep convolutional neural networks. NPJ Digital Medicine, 3(1):70, 2020.
Trick, S. and Rothkopf, C. Bayesian classifier fusion with an explicit model of correlation. In Valps,Valls, G., Ruiz, F. J. R., and Valera, (cds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 2210-2310, 2022.
Tudor lonescu, R., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, P., and Ferrari, V. How hard can it be? estimating the difficulty of visual search an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2157-2166, 2016.
Verma, R. and Nalisnick, F. Calibrated learning to defer with one-vs-all classifiers. International Conference on Machine Learning, pp. 22184-22202. PMLR, 2022.
Verma, R., Barrejón, D., and Nalisnick, E. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles.
12

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
A. Derivations of Inference Equations
Below we assume that expressions are implicitly conditioned on (and reintroduce it notationally at the end of this section). For each class we wish compute
= - p(u- ki, yo; =M duy
plu. = k 10,, =M)
yr
/ p(u- = yuu yo: ZM) I yo, ZM )d)du
Since y, = mode( (i.e., is a deterministic function of yuu and yo) we have:
f I[(-- x))(um yo, AM ddyu
(6)
Focusing on the latter probability we can obtain:
p(yu/
yo. =m)
p(yu zu No, =M)
mu
p(uti zu Mo. =M)p(=u vo PM day
p(yu zu)p(2u yo, 2M )az
(7)
Narrowing our focus again, we can manipulate the second probability in 7:
p(zu yo, zM)
p(zu: zo yo, =M)do
20
p(yO, 21, 20 2M) dzo
plvo (2
20
p(yo 2u, (M)(0 =m)dzo
20
p(vo aM)
p(vo =0)p(<z; zo a d=o
20
(8)
p(vol zo)p(zo aM)d20
Reincorporating 8 into 7 we obtain:
p N0, 2M)
p(vo 20)p(zu 20 2m)d20
/ su
p(yra =x)
d
pluo 20)p(=0 | =
p(ye 1u)pyo =0))p()) zol zM Sm)dzoday
(9)
plvo =0)p(=0 =m)d20
13

---

Bayesian Inference for Correlated Human Experts and Classifiers
Reincorporating 9 into 6 have:
ply = No, zM)
p(uu/ zu)p(uo | (o) zm)dzodeu
T(ye = k) 3M 2o
dyju
plvo zo)p(zo =M)deo
Ily+ =k) / p(yu zu)p(uo =0)p(zu so zm)dzodzzdyu
x
10
Ty = ki)p(uu zu)((vo zo)p(zu. =o =x)dgndzzddz
E
E
[[y- k)p(vo) =0)]
2u.30 3MM
Libe RM
Ultimately we have:
ply- =k 20,
[[ k)pluo =0)]]
(10)
1r, S.TS
14

---

Bayesian Inference for Correlated Human Experts and Classifiers
B. Algorithm 1 with Annotations
Algorithm 3 Estimate p(y.) given observed predictions yo, RM and the set of samples from the posterior for μ4, E, and
p-y-samples <-1 |
for sample E S do
"","" + sample {These correspond to single sample from the posterior distribution. This loop will draw a single, corresponding sample for zy and zo. followed by a sample for yuu giving us a complete, joint sample from our model over agent predictions.)
if |O| 2 then
=q. = draw from p(zH =M) {If any human predictions have been observed, we will need re-weigh by p(yo =0), which compute in this block. The first step is to sample 24, which is drawn from N( ),), conditioned on ZM (using the closed-form conditional multivariate normal distribution).}
vu + draws from Categorical(TS(((E3))) r*)) for i E U, concatenated {In this step, we are drawing a sample for each unobserved expert vote ys from the corresponding categorical distribution for expert i.}
V. - f'yo: wi) {Here, we compute the final output of the aggregation function f on all expert votes: both the observed expert votes yo and the sampled, unobserved expert votes Vis This sampled result y² is needed to compute the corresponding entry of p-y_samples (see below)-}
plyo =)) lioo TS()(==),,+)) [M]] {This step computes p(yo zo) for the sample zi. Since the yes are conditionally independent given this is the product of each p(uilz)). for i E O. The inside of the product is the v element of the corresponding categorical distribution, i.e., the probability of observing the class label y; under our sampled categorical distribution (S(()(2)),7*))
else
zu draw from p(zu 1M {N(He, Σ²), conditioned on 2M3 {In this if-block, we cover the case in which we have not yet observed any experts. Similarly above, we sample 2H (which is equal zu), drawing from N(ue, E*), conditioned 2M-}
vis + draws from Categorical(TS((())),))) r")) for i E U, concatenated (We draw a sample for each (unobserved) expert vote ys. from the corresponding categorical distribution for expert i.}
y + +-(v() { We compute the final output of the aggregation function f on these sampled expert votes. This sampled result y is needed to compute the corresponding entry of p-y-py-saa p-y-samples (see below).}
p(yo =0)= < {Since we have not observed any expert votes, the sampled result y is not re-weighted.)
end if'
P saaplees.apenndpp(e =0)== OneHot K (y])) The array P y samples contains samples of the innermost part of the expectation in Equation 4, i.e., for each sample, we add K-dimensional vector to p-y-samples, where the element corresponding to the sampled expert agreement y is p(yo z8) (in the case where no expert votes have been observed, this is 1) and the rest of the elements are O. This corresponds to [S] samples of the expert agreement, re-weighted based on the expert votes that have already been observed.} end for
ply= No, zM) < mean of p samples, normalized {In this final step, we take the normalized mean of tty-ssamples to obtain probubility distribution over the agreement y.-}
15

---

Bayesian Inferenee for Correlated Human Experts and Classifiers
C. Additional Details on Datasets
ChestX-ray Here we report the accuracies of five experts and the classifier, a DenseNet-121 convolutional neural network trained for this task (see (Tang et al., 2020) for model details). Note that we treat this as binary classification task (normal vs, abnormal).
Overall
Classifier
86.2%
Expert
90.5%
Expert 2
92.3%
Expert 3
83.7%
Expert 4
93.2%
Expert 88.3%
Table 3. Overall accuracies for ChestX-ray classifier and experts
Chaoyang Here we report the accuracies of the three experts and the classifier, a ResNet model trained for this task (see (Zhu et al., 2021) for model details).
Class
Class 2
Class 3
Class 4
Overall
Classifier
91.2%
55.3%
96.8%
67.2%
80.5%
Expert I
67.5%
78.8%
99.2%
96.3%
85.9%
Expert 2
92.3%
62.9%
98.4%
59.9%
81.7%
Expert 3
100.0%
96.5%
100.0%
100.0%
99.2%
Table 4. Class-wise and overall accuracies for Chaoyang classifier and experts
CIFAR-10H We combine classes so that the new class 1 corresponds to the original classes 1, 2, and 3; class 2 corresponds to classes 4, 5, and 6; and class 3 corresponds to classes 7, 8, 9, and 10. We then create the combined experts as follows. The predictions of expert 1 are taken from the first 15 columns of annotator labels; for classes and their prediction the consensus of the first 10 of these labels; for class their prediction is a "minority opinion" of all 15 labels (a random prediction that is not part of the consensus, if one exists). Expert is formed similarly from the next 16 labels; the majority vote of the first 10 for classes and 2 and minority opinion from all 15 for class 3. Expert is formed in the same way using the following 15 columns, with the majority vote for classes 2 and 3, and the minority opinion for class 1. The class-wise and overall performance of these experts and the classifier are shown in Table 5.
Class
Class
Class 3
Overall
Classifier
82.0%
95.6%
95.6%
91.5%
Expert
100.0%
78.9%
99.8%
93.7%
Expert 2
99.7%
99.7%
75.5%
89.8%
Expert 3
69.5%
99.6%
99.9%
99.6%
Table 5. Class-wis and overall accuracies for CIFAR- 10H classitier (ResNet) and experts
ImageNeet-16H We combine classes as follows. The new class corresponds to the original classes for "clock," "knife," "knffe" "oven," " "chair," "bottle," "keyboard" (roughly: "objects")). The new class combines the original classes "cat," "elephant," "dog." "bird," "bear" ("animals"). The new class 3 combines "airplane," "boat," "car," "truck," " "bicycle" ("transportation") Experts are then combined following the same strategy as described the first 67 annotation columns, expert 2, the next 66, and expert 3, the last 67. As there are multiple predictions per image depending the noise level, choose annotations based noise level to further diversify the experts: experts 1 and 3 use predictions for Ω=95 and 10, while expert 2 uses predictions for Q== 10 and 125. The final class-wise and overall accuracies are shown in 8.
CIFAR-10H, with expert using predictions from
16

---

Bayesian Inference for Correlated Human Experts and Classifiers
Class
Class 2
Class 3
Overall
Classifier
81. 1%
95.9%
82.4%
86.0%
Expert 96.6%
82.4%
97.9
92.6%
Expert 95.5%
98.1%
81.2%
91.7%
Expert 86.9%
95.9%
96.0%
92.5%
Table 6. Class-wise and overall accuracies for ImageNec-166 classifier (AlexNet) and experts
D. Experimental Details
Modeling
Experiments on an NVIDIA GeForce 2080ti GPU over the course of several days. The following hyperparameter values were used for all experiments:
or 0.4
ON 0.1 1 = 0.75
0, = 1
Parameters were updated after every example for the first 20 examples, then every 10 examples until 100 total examples were reached, at which point the update rate was further reduced every 50 examples.
For all inference tasks (for example, estimate the parameters each time step (), used three independent Markov chains, each comprising 1,500 warm-up iterations followed by 2,000 post-warm-up (posterior) samples. The resulting 6,000 samples were combined into our final sampled approximate posterior distribution. Chain convergence plots generally indicated that chains mixed; R-hat values across parameters and experiments were consistently within +/-,001 of 1.
Distribution Shift Experiment
We include here the performance of the classifier and each expert before and after the distribution shift. Both overall per-agent performance and inter-agent, inter-class differences change substantially with the distribution shift.
Class 1
Class 2
Class
Overall
Classifier 85.3%
100.0%
88.8%
87.0%
Expert 100.0%
100.0%
100.0%
100.0%
Expert 100.0%
100.0%
86.0%
93.3%
Expert 80.7%
100.0%
100.0%
89.9%
Table 7. Class-wiss and overall accuracies for ImageNet-166t 16H classifier and experts, "low noise" (Q=80) examples
Class 1
Class 2
Class 3
Overall
Classifier
78.8%
44.4%
82,4%
81.6%
Expert 100.0%
44.4%
100.0%
98.7%
Expert 99.5%
100.0%
57.7%
81.3%
Expert 35.0%
88.8%
100.0%
64.5%
Table 8. Class-wiss and overall accuracies for ImageNet- 16H classifier and experts, "high noise"
(Ω=125) examples
These experiments were run with a "sliding window," done by re-training the model after each new data point and passing in only the most recent 50 observed points. The model was given an error threshold of = 0.01.
17


Reviews:
[{"review_id": "FLORRpyJSK", "reviewer": "Reviewer_DV8P", "summary": "This paper addresses an interesting problem of predicting human consensus among correlated experts and machine learning models. The end goal is an active learning task, where the consensus label must be predicted using as few experts as possible. To this end, a hierarchical Bayesian model is presented to model \"agent\" correlations (i.e., correlations of both human experts and machine learning models), and an information gain criteria is presented and numerically estimated for active learning. Several experiments are presented which show the proposed method is able to recover the consensus label with fewer expert queries than other baselines.\n\n## Update after rebuttal\n\nI am glad that the authors will incorporate further prior work and more complete exposition in the final paper. While a theoretically simple modification, performing \"online\" inference would require non-trivial reimplemtation, in my opinion. Overall, my evaluation of the paper has improved, but I maintain my score of 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The main claim made by this submission is that the proposed Bayesian model is appropriate and effective at tackling the proposed problem. I believe this claim to be well-founded, both from the modelling perspective (with similarities to many similar approaches and well-motivated modifications) and the empirical perspective (with its experiments).", "methods_and_evaluation_criteria": "Yes, I think the proposed methods and evaluation criteria are appropriate. It is difficult to derive good baselines, since the problem formulation itself is novel, but the choices taken seem reasonable.", "theoretical_claims": "The main theoretical claims are regarding the derivations of expectation quantities and algorithms. I checked the expectation expressions, which are relatively simple and seem correct. Algorithm 1 seems correct, though is potentially lacking some details (e.g., in line 3: draw from what posterior?). Algorithm 2 is a straightforward way to approximate the expected entropy using Monte Carlo samples.", "experimental_designs_or_analyses": "The experiments are well-designed, and I reviewed all of the experiments in Section 7 (i.e., Cost vs. Error, Calibration and Error Bounding, Exploration vs. Exploitation, and the \"online\" (or continual) learning experiment).", "relation_to_broader_scientific_literature": "This paper proposes an interesting setting where correlated experts should be queried in a cost-aware way. I believe the approach is solid, (though perhaps not particularly novel technically), and forms a valuable contribution to the literature of Bayesian combination of expert knowledge.", "essential_references_not_discussed": "Following the line of Kim \\& Ghahramani (which was discussed in the paper), Trick \\& Rothkopf present a solution to correlated experts [1]. This is directly relevant to discussions about the use of Dirichlet priors, as their solution is a version of the Dirichlet distribution with explicit correlation structure. Perhaps closer to the current work is that of Pirš \\& Štrumbelj, who use an inverse additive logistic transformation [2].", "other_strengths_and_weaknesses": "### Strengths\n\n**Regarding the Setting** I think the proposed setting is very interesting, and to my knowledge novel.\n\n### Weaknesses\n\n**Regarding Presentation of Equations** I think the presentation of some equations (e.g. Eq. (4) and Eq. (5), and throughout Appendix A) may be improved by including (in words) a brief description. For example, Eq. (4) may be read as \"the predictive distribution of $y_*$ can be obtained as a nested expectation, where first correlations between experts are considered\"\n\n**Regarding Novelty** There have been several methods in the Bayesian combination literature that attempt the modelling of correlated experts (e.g., [1], [2]), and the entropy-based active learning approach is classical. That said, I think this is balanced by the interesting problem setting.", "other_comments_or_suggestions": "I have a few editorial remarks:\n1. There is no punctuation after Eq. (6).\n2. The marker for footnote 3 should be placed after the period. \n\nAs a more stylistic remark,\n1.  The content in Appendix A could be made significantly more readable by including some discussion of each step.", "questions_for_authors": "1. I would appreciate if the authors could comment more about the computational demand of the proposed method; for example, what is meant by \"over the course of several days.\" This would help clarify the expense of the proposed method, which can be practically relevant.\n2. It is stressed several times that the method is an online one, but to my knowledge, the entire posterior is re-sampled from scratch using NUTS for every value of $t$; is this understanding correct?\n\n### References\n\n[1] Trick, Susanne, and Constantin Rothkopf. \"Bayesian classifier fusion with an explicit model of correlation.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n[2] Pirš, Gregor, and Erik Štrumbelj. \"Bayesian combination of probabilistic classifiers using multivariate normal mixtures.\" Journal of Machine Learning Research 20.51 (2019): 1-18.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQjsqvsFPG", "reviewer": "Reviewer_g7pw", "summary": "This paper introduces a Bayesian framework for predicting the consensus of human experts in K-ary classification tasks, leveraging pre-trained classifiers to minimize the cost of querying experts while maintaining high accuracy. The correlation between human experts and classifiers was modelled by a joint latent representation to infer posterior distribution over unobserved expert labels. Results on medical datasets and image benchmark datasets CIFAR, ImageNet were reported.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes, the claim was basically supported by results in Section 7.3 and 7.4. But it seems the authors make a specific niche setting under this topic, and no other existing algorithms can be directly used to be compared with their method. They modify two existing methods as benchmarks, but it is hard to know whether the benchmarks are suitable.", "methods_and_evaluation_criteria": "The proposed method tries to predict expert consensus while minimising the query number. The zero error point is used as an important condition for evaluation, which seems reasonable. \n\nThe proposed method adopts MCMC for sampling and posterior computing. It is common that MCMC might take a long time to burn-in or fail to estimate the correct posterior region in the defined search space. But I cannot see any ablation studies or results related to how MCMC hyper-parameters may affect overall performance, as well as the failure rate (if any) of MCMC in this problem etc.", "theoretical_claims": "Problem setup and Algorithm 1 seem fine.", "experimental_designs_or_analyses": "Seems lacking experiments about sensitivity of MCMC hyperparameters to the final performance. \n\nSeems also lacking evaluation of how different pretrained ML classifiers affect the final results.", "relation_to_broader_scientific_literature": "This work seems contribute to the consensus prediction in a human-AI using a Bayesian method, highly related to the broader literature in this field, but with a niche problem setting.", "essential_references_not_discussed": "Not sure, I am not an expert in this specific field.", "other_strengths_and_weaknesses": "Strengths:\n\n* Well structured paper, easy to follow\n* The proposed algorithm seems novel in the field, adding new knowledge to the community\n\nWeaknesses:\n* The reported results need to be further enhanced, e.g. MCMC and ML classifier design and influence to the final performance were not reported.", "other_comments_or_suggestions": "* Might be good to further compress the introduction of datasets, - a bit too long in the main content of a conference paper, leaving less space for more important parts.", "questions_for_authors": "1. Does the conclusion still hold with different types of classifiers? \n\n2. Did MCMC fail to estimate the correct posterior region in some of the repeated experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHQ7r4myOt", "reviewer": "Reviewer_qbrk", "summary": "The paper proposes a hierarchical Bayesian model for aggregating predictions from  pretrained classifiers and  human experts, aiming to estimate the majority vote outcome. It assumes the ground truth corresponds to the majority vote of human experts, with each expert's latent probability correlated and modeled using a multivariate normal distribution. This distribution employs hyperpriors: the mean follows a normal distribution, and the covariance is modeled using the Lewandowski-Kurowicka-Joe distribution. The proposed model not only predicts the estimated majority vote results but also strategically determines which expert to query next by maximizing information gain, equivalently minimizing the expected entropy. Empirical results demonstrate that the Bayesian model proposed outperforms existing aggregation methods.\n\n## Update after rebuttal\nI found the authors’ rebuttal satisfactory. Their proposed revisions addressed my main concerns—namely, the general applicability of the social welfare function for consensus and the motivating example for the new setting. Accordingly, I have increased my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The proposed method is based on a Bayesian framework, and its effectiveness is empirically validated on four datasets against two baseline methods. The results clearly demonstrate that the proposed method consistently outperforms both baselines across all datasets in terms of the trade-off between the average number of expert queries and classification error. Additionally, the authors claim that their Bayesian model is well-calibrated, as evidenced by low expected calibration error (ECE) metrics in Table 1 and Figure 4.", "methods_and_evaluation_criteria": "Overall, the adoption of a hierarchical Bayesian approach is well-motivated for the problem at hand, especially given the inherent uncertainty and correlation among human experts. The evaluation approach, focusing on the trade-off between classification accuracy and the number of expert queries, is appropriate and clearly demonstrates the cost-effectiveness of the proposed method. Additionally, the emphasis on model calibration through expected calibration error (ECE) metrics further strengthens the validity of the approach.", "theoretical_claims": "Not applicable; The paper does not claim theoretical results.", "experimental_designs_or_analyses": "I did not verify the implementation details explicitly; however, the choice of datasets and the use of the NUTS sampler appear reasonable.", "relation_to_broader_scientific_literature": "Human-AI collaboration has become increasingly important, especially in light of the rapid emergence of large language models (LLMs). Aligning AI models with multiple human preferences is an essential topic in the broader scientific literature. However, this paper assumes a fixed and identifiable set of human experts. A more general and practically relevant setting would relax this assumption, considering scenarios where expert identities are neither fixed nor necessarily known.", "essential_references_not_discussed": "The paper appears to cover essential references adequately; however, it remains possible that there are important related works not mentioned, particularly those outside my familiarity.", "other_strengths_and_weaknesses": "Strength\n- The manuscript is well-written.\n- Figure 1 helps significantly in understanding the overall pipeline.\n\nWeakness\n- The proposed approach appears rather straightforward and incremental. Indeed, the authors themselves acknowledge in Section 2.2 that this paper extends the work of Showalter et al. (2024) by incorporating correlations among experts' predictions. Consequently, the primary novelty lies in modeling the latent probability $z$ using a hyperprior. However, I find this hierarchical Bayesian approach to be quite standard practice within the community. While there is certainly nothing problematic about utilizing established methods, for a submission to a top-tier conference like ICML, I would expect a more innovative methodological contribution. Therefore, I suggest the authors consider submitting their paper to a journal such as Knowledge-Based Systems, where incremental methodological advancements combined with practical significance are well-received. For Bayesian experimental design part, a similar setting has been investigated in [1], albeit with slight differences.\n[1] Bayesian Optimization for Building Social-Influence-Free Consensus, https://doi.org/10.48550/arXiv.2502.07166\n- This paper appears to conflate the concept of consensus with majority voting. It is important to clarify that Bayesian consensus is not inherently restricted to a utilitarian (majority-based) approach. I am not entirely convinced that utilitarian aggregation is universally appropriate, especially in high-stakes contexts requiring careful deliberation, such as X-ray classification tasks. In the context of medical diagnostics like X-ray analysis, human expert labels should be treated as informed advice rather than definitive ground truths. Definitive ground truths in such scenarios are typically only attainable via invasive procedures, such as surgery, which are prohibitively costly. Hence, expert labels serve as practical proxies. The primary motivation behind aggregating multiple expert opinions is to enhance the reliability and robustness of these proxy judgments. Adopting a majority-vote strategy aligns with the \"wisdom of crowds\" principle. However, an egalitarian approach may be more suitable in situations where avoiding the risk of overlooking critical indicators is paramount. Given that individual doctors possess varied experiences and specialized expertise, aggregating multiple expert assessments serves primarily to mitigate the risk of missing rare but significant findings. In this egalitarian scenario, if any expert identifies an abnormality, the consensus decision should categorize the image as abnormal, independent of the majority opinion. Fortunately, the approach described in this paper seems sufficiently flexible to accommodate both utilitarian and egalitarian aggregation schemes. I encourage the authors to further develop their methodology to explicitly support a general framework, allowing practitioners to select the appropriate social-welfare functional based on their specific decision-making needs.", "other_comments_or_suggestions": "The paper does not provide an explanation of what the Expected Calibration Error (ECE) represents.", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tSD4EeyqCD", "reviewer": "Reviewer_YjoC", "summary": "This paper addresses the challenge of predicting human expert consensus in $K$-ary classification by developing a Bayesian framework that leverages model outputs and expert queries. Given a limited budget for expert input, the method efficiently maximizes prediction accuracy by modeling expert correlation and inferring unobserved labels. Experiments on medical and image datasets demonstrate significant cost reductions while maintaining accuracy.\n\n------post rebuttal\n\nThe rebuttal clarified a few points for me, and I now have a better appreciation of the motivation. The authors claim that the method can be extended to incorporate varying expert costs, but since they do not provide detailed derivations, it is difficult to verify this during the rebuttal phase. I also find that the Bayesian method adopted in the paper is fairly standard, and I would have liked to see more advanced Bayesian techniques. Therefore, I maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no proofs provided.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The discussion of related work in Section 2 is properly written.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "I have concerns regarding the motivation for Consensus Prediction. Specifically, given the assumption that pre-trained classifiers are provided, it is unclear why expert consultation is necessary for prediction. Furthermore, it raises the question of why pre-trained classifiers cannot themselves be considered experts.\n\nFurthermore, after framing this problem, the Bayesian framework is rather natural, and I do not consider the proposed methodology innovative and significant.\n\nAlso, they assume all experts cost the same, which isn't very practical.", "other_comments_or_suggestions": "There is no conclusion provided.", "questions_for_authors": "1. Could the authors elaborate on the motivation for the problem, and the innovation and significance of the proposed methodology?\n2. Could the methodology be extended to incorporate varying expert costs, rather than assuming uniform costs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduced a rather interesting method to make predictions with the input of human annotators and trained classifiers by explicitly modeling the correlation between the human experts and the classifier. In general, the reviewers were positive about the paper and the authors provided a suitable rebuttal to the critiques. Therefore I vote to accept the paper.
Decision: Accept (poster)