Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration
Authors: Gavin Kerrigan
Padhraic Smyth
Mark Steyvers
Venue: NeurIPS 2021 Poster (2021)
Keywords: human-machine
human-AI
human-in-the-loop
calibration
classification

Abstract: An increasingly common use case for machine learning models is augmenting the abilities of human decision makers. For classification tasks where neither the human nor model are perfectly accurate, a key step in obtaining high performance is combining their individual predictions in a manner that leverages their relative strengths. In this work, we develop a set of algorithms that combine the probabilistic output of a model with the class-level output of a human. We show theoretically that the accuracy of our combination model is driven not only by the individual human and model accuracies, but also by the model's confidence.  Empirical results on image classification with CIFAR-10 and a subset of ImageNet demonstrate that such human-model combinations consistently have higher accuracies than the model or human alone, and that the parameters of the combination method can be estimated effectively with as few as ten labeled datapoints.

Combining Human Predictions with Model
Probabilities via Confusion Matrices and Calibration
Gavin Kerrigan'
Padhraic Smyth'
Mark Steyvers
Department of Computer Science
DDpartment of Cognitive Sciences
University of California, Irvine smyth@ics.
gavin.kOuch edu
mark steyvers@uci.edu
Abstract
An increasingly common use case for machine learning models is augmenting the abilities of human decision makers. For classification tasks where neither the human model perfectly accurate, key step in obtaining high performance is combining their individual predictions in a manner that leverages their relative strengths. In this work, develop set of algorithms that combine the probabilistic output of model with the class-level output of human. We show theoretically that the accuracy of our combination model is driven not only by the individual human and model accuracies, but also by the model's confidence. Empirical results image classification with CIFAR-10 and subset of ImageNet demonstrate that such human-model combinations consistently have higher accuracies than the model human alone, and that the parameters of the combination method can be estimated effectively with as few as ten labeled datapoints.
Introduction
One of the main goals of machine learning is to develop algorithms that can operate robustly in autonomous fashion without human supervision. However, there are many applications where hybrid likely to be preferred mode of operation for variety of different reasons, such as improving trust between humans and machines, and allowing for human or model take over in situations where the one the other lacks expertise [Kamar, 2016, Vaughan, 2018, Kleinberg et al., 2017, Riedl, 2019, Trouille et al., 2019, Johnson and Vera, 2019, Zahedi and Kambhampati, 2021].
human-machine approaches
The performance benefits of combining multiple predictors, rather than relying on a single predictor, have been clearly demonstrated in past work in several fields. For example, in machine learning there a rich vein of research over the past few decades on combining models using a variety of different estimation and algorithmic approaches [Kittler al., 1998, Dietterich, 2000, Kuncheva, 2014, Sagi and Rokach, 2018]. This existing line work emphasizes that combinations of models that have diversity in how they make predictions can systematically outperform a single model. In parallel, in the behavioral science literature, there has been extensive prior work studying combinations of human opinions where, again, diverse combinations tend outperform any single individual [Hong and Page, 2004, Lamberson and Page, 2012].
This naturally leads to questions about hybrid combinations of human and machine predictions, rather than just combining one type or the other. For example, one motivation for hybrid combinations empirical evidence that human and machine classifiers do not make the same types of errors for problems such as image classification [Geirhos et al., 2020, Rosenfeld et al., 2018, Serre, 2019], i.e., they diverse in their predictions. These ideas have begun to have impact in real-world applications, where hybrid human-machine teams have been found to be effective in areas such as crowdsourcing [Kamar al., 2012], citizen science [Beck al., 2018], speech transcription [Gaur al., 2015],
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

12
Dog
8
6
Piñear) 0.76 P(Dag) 0.21
Error
Classifeer
Classifier
FiBear 0.24 PiDog) 0.25
2
Human
Comb.
Human
Comb.
0.86
P(Dog) 0.13
CIFAR-10H ImageNet-16H
Figure 1: Left: Combining a human's 's label and a classifier's probabilities for an ImageNet- 16H (Steyvers et al., 2022] image (true label: bear). Right: lower error rates on average than the human or classifier alone (ResNet- 164 on CIFAR-IOH, VGG-19 on ImageNet-16HD).
combinations (purple) achieve
face identification [Phillips et al., 2018], and clinical radiology [Bien et al., 2018, Patel et al., 2019, Rajpurkar et al., 2020].
In this paper, we focus on a simple yet realistic instantiation of the general problem of hybrid combinations of human and machine predictions. In particular, we consider a K-way classification task such as image classification, with a single human making categorical classification decisions (i.e. predicting only label) and single classification model predicting a distribution over labels. While humans can provide confidence estimates with their label predictions, calibrated self-assessment of confidence can be difficult [Keren, 1991, Kahneman and Tversky, 1996, Klayman et al., 1999] as well as consumnn..
A key question in this context is whether the non-probabilistic information from human predictor can be effectively combined with the probabilistic information from machine learning model. We answer this question in the affirmative and show both theoretically and empirically how relatively simple probabilistic combination techniques can robustly outperform each of a human and machine on their own. In particular we show that human can augment their predictions with those of classification model, improving classification accuracy and producing calibrated predictions, even when the model is less accurate than the human. Similarly, from the model's 's perspective, accuracy can often be significantly improved by augmenting it's class probabilities with human's 's labels, while improving calibration performance, even when the human is less accurate than the model.
Figure shows example using our proposed methodology for an image from the ImageNet-161 dataset [Steyvers et al., 2022]. The human incorrectly predicts the label dog. The classification model deep network) predicts the correct label bear with probability of 0.86 (uncalibrated) and 0.78 (calibrated). The combined prediction is for bear with confidence of 0.74, with dog having higher confidence given the human's 's prediction. The histograms on the right show the overall average reduction in error rate on test images: even though the classifiers are on average less accurate (6%, 11.1% % error) than the human (4.62% 9.99% error), the resulting combinations have lower error rates than either (2.85% 6.62% error).
The primary contributions of work as follows:
We propose and investigate general framework for combining predictions using instance-level confidence from model and class-level information from human. The methods we propose are straightforward to implement in practice and -abel-efficient.
We empirically validate our approach the CIFAR- [Peterson al., 2019] and ImageNet- 16H [Steyvers et al., 2022] image classification datasets and show that human- machhiee combinations in this context systematically more accurate and better calibrated than either alone.
We develop a theoretical understanding, for this framework, of the key tradeoffs related to cali- bration and accuracy for both the individual human and model, introducing the notion of model and human confidence ratios. We illustrate how these factors affect the combination, showing for
'Code for our estimation methods and experiments is available at: attps://githu: com htt///ninnneiiannneneeaaa www.raattii..i and caritiiioo..

---

example how two models with the same accuracy but with different calibration properties can have different performance when combined with human predictions.
The paper begins in Section 2 by introducing notation and deriving our combination method. Section 3 discusses related work and in Section 4 we propose number of different estimation methods. Section 5 describes our experimental results with two image classification datasets, with individual human labelers, individual models, and combinations. Complementing the experimental results in Section we develop theoretical results that characterize combination performance. Section 7 discusses the potential societal impact and limitations of the work, and Section 8 concludes the paper.
2 Combining Human Labels and Model Probabilities
Notation. We consider K-ary classification problem, where the goal is to predict a label y E (1,... K} from features x E X. The random variable (x, y) follows an unknown dis- tribution with support X' x V. We assume access to individual human labeler represented by the function : X + D. where h(x) E y is the label predicted by the human. addition, we have access to trained machine classifier m X RK where m(x) R is the normalized probability vector output by the classifier. Both the human and classifier are assumed to be noisy labelers relative to the ground truth y. The true labels could be determined, for example, by expert labelers or additional information not contained in x.
Combining Predictions. Given imput x E X, our goal is to predict true label conditioned on the predictions hxz) and m(x). The key challenge in combining human and classifier predictions is simultaneously leveraging both the class-level outputs from the human and the predictive distributions output by the classifier. Although we focus on the particular case where is a human and m is a classifier, our setup could be applied more generally combinations of a non- propprobabilistic labeler (whose output is categorical) and a probabilistic labeler (whose output is a distribution over classes).
There are variety of functional forms that could be used to combine the predictions. We pursue probabilistic approach, where the conditional distribution over labels that we seek can be factored via Bayes' rule as
(1) In this work, we pursue a conditional independence (CT) approach where the human labels h(x) and the probabilistic predictions m(xr) are assumed to be conditionally independent given y. Under this assumption, the quantity of interest factors as
p(ulh(x) m(z)) x p(h(z))
plylh(x), m(x)) x (((h())))))()M)g
In the above, the term p(h(x)ly) can be interpreted as calibrated probabilitics at the class level. We parameterize the term p(h(x)) (y) by the confusion matrix for the labeler h, which we denote by yo with entries PPg = p(h(x))= = ily =). The second term (m))m())) can be interpreted as calibrated probabilities at the instance level. However, the probabilistic output of the classifier m(x:) may differ from p(y mx)).. For example, modern neural networks tend to be overconfident in their predictions [Guo et al., 2017]. To remedy this, post-hoc calibration maps m(x) to well-calibrated probabilities via a learned calibration map with parameters 0. We note that there are many possible choices for such a post-hoc calibration map [Guo al., 2017, Zhang et al., 2020, Kull al., 2019, Patel et al., 2021], but our method is agnostic to this choice. this work, we use m² (x) to denote the output of the classifier after applying such calibration map. The second term in Equation (2) is then parameterized by the calibrated classifier probabilities m² (x).
Altogether, method expresses the predicted probability for class j. given that the human predicts class i and the model produces a class probability vector m(x) as:
(x)
puu 51h(2) i, mx)))
Ak= parm (x)
The CI assumption above is common (both implicitly and explicitly) in prior work on combining predictions, such as additive classifier ensembles [Kuncheva, 2014, Sagi and Rokach, 2018] and (log))linea opinion pools [Genest and Zidek, 1986, Jacobs, 1995]. As our primary motivation is to develop a relatively simple and robust methodology for combining human and model predictions, the
3

---

additional functional or parametric assumptions (and parameters) required to specify a joint model for p(h(z), m(x)ly)) are beyond the scope of the work in this paper. In addition, although the CI assumption is unlikely to hold exactly, prior work [Kuncheva, 2006] notes that a CI model can be an optimal discriminant even when the CI assumption violated. As further motivation, for the two datasets we use in this paper, CIFAR-10H and ImageNet- 16H, the conditional dependence of the predictions h((x)) by the human and m(x) by the model appears to be relatively weak (see Appendix A for details).
3 Related Work
We summarize below relevant aspects of related literature. While there is significant amount of prior work in machine learning and related fields on combining predictors, this work has in general not addressed the specific problem of combining hard label predictions from a human with probabilistic label predictions from a model.
Ensembles and Opinion Pools. There is rich literature in machine learning studying pre- dictions based on ensembles of classification models. For non-probabilistic classifiers, the most common aggregation methods are variants of (weighted) majority voting [Dietterich, 2000, Sagi and Rokach, 2018]. However, in our case of only two predictors, a weighted majority vote ensemble can never improve accuracy over the best of its two components. Beyond majority voting, naive Bayes aggregation [Xu et al., 1992. Kuncheva, 2014] fits a class-level confusion matrix to each predictor. Kim and Ghahramani [2012] develop fully Bayesian extension of this, which relaxes the independence assumption by explicitly modeling correlations between predictors. However, because these confusion-matri aggregation methods are at the class level they are unable to take full advantage of the instance-level uncertainties produced by the probabilistic labeler.
In the context of aggregating predictions from multiple humans, there has been considerable amount of prior work in the behavioral sciences and forecasting literature. Approaches include additive linear and log-linear opinion pools for subjective distributions [Genest and Zidek, 1986, Jacobs, 1995], techniques for weighting linear combinations of real-valued human predictions [Lamberson and Page, 2012, Davis-Stober et al., 2015], and voting methods for combining label predictions from more than two human predictors [Lee and Lee, 2017]. A key difference between these methods and our work is that they do not address the problem of how to combine probabilistic and non non-probale -on-pobbabilistic predictions in a coman-machine context.
Leveraging Human and Model Predictions. Combining human predictions with model predic- tions to solve classification problems has also been a topic of recent interest in a number of different areas. For example, in [Wright et al., 2017] simple averaging is used to combine the labels of multiple human annotators with the output of a classifier for astronomical image classification, achieving better performance than with either the humans or the classifier. In crowd-sourcing, classification models have been used to automatically filter examples to improve human annotation efficiency [Kamar et al., 2012, Russakovsky et al., 2015, Vaughan, 2018, Trouille et al., 2019]. A similar line of research focuses on algorithmic deferral techniques where a model defers to human predictions based on the model's confidence [Madras et al., 2018, Raghu et al., 2019], as well as work adapting prediction models to the human decision maker [Branson et al., 2017, Mozannar and Sontag, 2020, Wilder al., 2020, Bansal et al., 2021, Okati et al., 2021] The results in [Mozannar and Sontag, 2020] in particular describe experiments with the same CIFAR-I01 dataset that we use in this paper. However, in addition being different our work in terms of its focus on deferral (rather than combining) we also note that in [Mozannar and Sontag. 2020] the improvements in performance are demonstrated using relatively large numbers of human labels. In contrast, as we demonstrate in Section 5 on the CIFAR-10F 10H and ImageNet-1661 datasets, our methods require only small number of human labels to yield combined predictions that are more accurate than either human or model alone. In general, existing work on filtering and deferral strategies complements the combining methods that we develop in this paper. All of these approaches are potentially useful in a broad range of human AI applications, but in different contexts.

---

4 Estimation Methods and Algorithms
Combining human and machine predictions via Equation (3) requires learning two sets of parameters: confusion matrix parameters for the human and calibration parameters for the classifier. The choice of procedure used to infer these parameters impacts the label efficiency and quality of the resulting combination. In this section, we describe in detail several inference procedures and empirically evaluate them in the context of numan-maching combinations.
To estimate our combination model, we assume access to a combination dataset Dc {(h(zre), m((r)) 3, we) with human labels, machine probabilities, and ground truth labels. We assume the machine classifier is already pre-trained using true labels on a separate training set Dr.
Confusion Matrix Estimation. Recall that yp denotes a K x K confusion matrix for the human, where SPy is p(h(2;) = ily = j). The most straightforward estimate for this quantity is the maximum likelihood estimate, where Pij is estimated by the number of datapoints in De where the human labeler predicts h(x)) = i when the ground truth is y = j. normalized by the number of points in De where y j.. j.
However, the size of the confusion matrix is quadratic in K, this estimate will have high variance for small amounts of labeled data and collecting enough human labels to overcome this variance could be prohibitively expensive. Moreover, such matrix depends on the particular labeler, and would potentially need to be re-estimated for different individual human labelers. We can instead take Bayesian approach and incorporate informative prior information. Given the true label y j. the human label fuly = j ~ Discrete(yPa)) ( is assumed to be drawn from a discrete distribution with parameters corresponding to the jth row in the confusion matrix. We place conjugate Dirichlet prior Paj ~ Dirichlet (ay) over each column with parameters 0g E R The prior parameters ag are chosen such that
3 i ly i
(az):
44)
That is, the prior matrix is Y E Ry along the diagonal and B E R on the off-diagonal. We choose B and such that the resulting Dirichlet distribution has mode equal to the train-set accuracy of the classifier, which can be obtained from the training data without needing additional human labels. This choice of prior reflects our belief that the confusion matrix will have diagonally dominant structure, and moreover corresponds to a prior belief that the human is equally accurate across all classes. Posterior estimates of the confusion matrix can then be obtained straightforwardly by conjugacy.
Calibration Parameter Estimation. Scaling-based calibration maps are typically fit by optimizing the log- likellieioood on held-out calibration set [Guo et al., 2017, Zhang et al., 2020, Kull et al., 2019]. In this section, we describe Bayesian version of temperature scaling [Guo et al., 2017], allowing us to incorporate informative prior information. A temperature T E (0, 1) indicates underconfidence, and T E (1, oc) indicates overconfidence. To account for this difference in scale, we place Gaussian prior on the log-temperature log T a posteriori (MAP) temperature is estimated via gradient-based optimization. In our experiments, we choose a 0.5 for the CIFAR-10 models and 0.75 for the ImageNet models, and we use = 0.5 throughout. These parameters were chosen to reflect our belief that deep models tend to be overconfident and concentrate the prior on reasonable temperature values. We choose larger value of a (i.e. a wider prior) for the ImageNet models, as this task is more difficult than CIFAR- 10 and hence could potentially lead to a stronger degree of miscalibration. In Appendix B, we derive a fully Bayesian approach where we marginalize over the posterior distribution over temperature (e.g. using Monte Carlo methods [Hoffman and Gelman, 2014]). However, we find empirically that the simpler MAP approach is more effective and, as a result, focus on MAP estimation in this paper.
N(,,, d²). As this is non-conjugate prior, the maximum
Learning without Ground Truth. Requiring both human and ground truth labels De can be potentially limiting assumption in domains where ground truth labels are unavailable or ex- pensive to obtain. To avoid this, we propose an unsupervised approach that is able to learn both calibration parameters and confusion matrix parameters from a combination dataset of the form De = ((h(ze), m(xx))), consisting only of human labels and machine probabilities. We treat the ground truth labels as latent and fit the required parameters using Expectation- Expeetation-Maximizatioo (EM)
5

---

Table 1: Summary of combination methods studied in this work. Except for logistic regression, parameter counts correspond to calibration using MAP temperature scaling (one parameter), and confusion matrices are fit with MAP inference. The human output is always label.
Method Name
Acronym
Parameters
Model Ourput
Groan Truth?
Label Efficient?
Logistic Regression
LR
+ 2h
Probubilities (P)
Calibrated Machine Probs & Single-Parameter Confusion
SP
2
Probabilitics (P)
Machine Labels & Human Labels
L+L
21
Labels (L)
Calibrated Machine Probs. Human Labels
A
Probabilitics
Calibrated Machine Probs, Humon Labels (EM)
P+L-EM
41
Probabilities (P)
[Dempster et al., 1977] (details Appendix C). This approach can be seen as an extension of the Dawid-Skene model [Dawid and Skene, 1979]. In particular, the Dawid-Skene model fits confusion matrix for the classifier, whereas contrast our work fits calibration parameters for the classifier. We perform both maximum likelihood and MAP estimation with this method, using the same priors as above.
For clarity of exposition, we focus on three types of combinations in our results:
Machine Labels & Human Labels (L+L), baseline where the instance-level probabilities from the model are discarded and instead a confusion matrix is fit for the model, as well as a confusion matrix for the human. The confusion matrices are estimated via supervised MAP inference. This can be viewed as a naive Bayes' combination [Kuncheva, 2014, Chapter 4] for non-probabilistic predictors.
Calibrated Machine Probabilities & Human Labels (P+L. combined via Equation (3) using supervised MAP estimates for both the calibration parameters and confusion matrix parameters. Calibrated Machine Probabilities & Human Labels (P+L-EM) combined via Equation (3) using unsupervised MAP estimates fit with our EM algorithm, i.e. using human labels but no ground truth in correspondence with these labels.
In terms of complexity, these methods sit between simple model with only one parameter for the human confusion matrix (SP) (where the diagonal entry corresponds to the human's 's marginal accuracy), and a full multinomial logistic regression model (LR).. We find that LR obtain slightly lower error rates in some cases, but requires significantly more labeled data than the other methods to do so. At the other extreme, SP is highly data efficient, but underfits compared to preferred methods. We provide additional discussion of these methods to Appendix E. The various estimation methods discussed in this section are summarized in Table 1.
5 Experiments
Datasets and Models. We evaluate various combination strategies two pre-existing image classification datasets that include human annotations: CIFAR- [Peterson et al., 2019] and 166 [Steyvers et al., 2022] 3 CIFAR-1OH contains 10-way human classifications for 10,000 images from the standard CIFAR10 test set [Krizhevsky et al., 2009]. ImageNet-16H contains 16-way human classifications for noisy images from the ImageNet test set [Deng et al., 2009], distorted by phase noise at each spatial frequency based on four levels of phase noise (80, 95, 110, and 125). The human classifications for CIFAR-IOH and ImageNet-I66 come from the Amazon Mechanical Turk platform.
We experiment with four CNN models CIFAR-10H (ResNet-110, 110, Resnet- 164 [He al., 201 6a]. PreResNet- 164 [He et al., 2016b], DenseNet [Huang et al., 2017]), and eight models (four VGG- [Simonyan and Zisserman, 2015]. four GoogLeNet [Szegedy et al., 2015]) of varying accuracy on ImageNet-166F Our models are chosen to span a range of performance, from below human accuracy exceeding human accuracy. See Appendix F for details regarding our model architectures and training procedures.
In fact, combination with temperature scaling can seen as a special case of multinomial logistic regression taking m(:)) and h(z) as inputs (see Appendix D). 'The ImageNet- 16H dataset is available on the Open Science Foundation at https://osfio/nntr/.

---

Human
Model
L+L
P+L (EM) (Ours)
P+L (Ours)
ResNet-110
ResNet-164
NreResNet-164
DenseNet-BC
Error Rate (%)
4
4
4
2
10
102
103
10
102
10a
10
102
10a
102
10a
Noise Level 80
Noise Level 95
Noise Level 110
Noise Level 125
12
21
Error Rate (%)
14
29
10
12
18
26
15
23
10
6
12
20
10
102
10a
10
102
10a
10
102
103
10
102
10a
Number of Labels
Figure 2: Learning curves for various models on CIFAR-IOH (top) and VGG-19 on ImageNet-166 (bottom). For the supervised methods (L+L, P+L), the x-axis corresponds the number of human labels, machine predictions, and ground truth labels. For the unsupervised method (P+L (EM)), the x-axis corresponds the number of human labels and machine predictions (but no ground truth). In all cases, the y-axis is the eттоr rate, with shading indicating one standard deviation.
Both datasets are partitioned into three disjoint subsets: (i) a model training set Dr, (ii) a combination training set Dc, and (iii) an evaluation set Dg. The model training set Dr the same the suggested training set for the original CIFAR- 10 and ImageNet datasets, and is used to fit the classification models. The combination training set De used to estimate any calibration parameters and confusion matrices, and the held-out evaluation set DE is used solely for testing. The combination training set and evaluation set are subsets of the original test sets, where 70% of the data used for fitting the combinations and 30% is used for evaluation. The true labels (ground truth) for both CIFAR- 10 and ImageNet correspond to the originally originally-provided labels for each of these datasets. our experiments, (i) is fixed and we average over randomly selected splits for (ii) and (iii).
Importantly, note that each individual human only labels small subset of the images in each dataset (approximately 200 images per individual on CIFAR- and 50 images per noise level and individual on ImageNet-1616
To overcome this limitation in our experiments, we select a single human- geraaantde label for each image by random sampling. This sampled label is fixed across all trials. In Appendix G. we demonstrate that our results are robust to this sampling procedure by fitting Bayesian P+L combination to each individual human (i.e. without the aforementioned sampling procedure). In short, this results in numan-moder combinations with performance on par with that obtained by sampling.
Calibration Methods. In our experiments, model calibration is carried out by MAP temperature scaling (TS) [Guo et al., 2017]. In Appendix H, we experiment with two additional calibration methods: ensemble temperature scaling [Zhang et al., 2020] and I-Max binning (Patel et al., 2021]. We find that the combination performance is robust to the choice of calibration map, and hence restrict our focus to TS in this section. In addition, find that combinations using uncalibrated model probabilities produce less accurate combinations than those using calibrated model probabilities.
Learning Curves. Given that it is highly desirable to learn numan-machinn combinations from small amounts of data, we empirically study the data efficiency the various inference methods previously described. In Figure 2, we plot the combination enror rate on the evaluation data function of dataset size for the four CIFAR-10 models (first row) and the VGG-19 model on ImageNet (second row). For supervised methods, the dataset size corresponds to the number of (h(z), m((x)))))))) v)
7

---

No Calibration
10 Datapoints
5000 Dutapoints
Metric
Model Name
Model
Comb.
Model
Comb.
Model
Comb.
ResNet-110 10
5.23 = 0.35
2.08 + 0.25
3.03 + 0.58
1.30 = 0.23
2.99 + 0.36
1.76 + 0.18
ECE (10-3)
ResNet-164
2.98 + 0.34
1.63 + 0.23
1.95 + 0.33
1.25 + 0.18
1.89 + 0.32
1.39 + 0.18
Pre-esNet-164
3.03
1.87 0.22
2.31 +: 0.33
0.26
2.27 d: 0.31
1.43 0.21
DenseNet-BC
2.18 0.27
1.53 + 0.20
1.76 + 0.28
1.34 = 0.14
1.73
+ 0.28
1.27 + 0.13
ResNet-110
0.81 0.07
0.23 +: 0.05
0.58 +: 0.07
0.24 + 0.05
0.58 + 0.06
0.19 0.06
ResNet-164
0.39 1 0.06
0,15 0.03
0
0.15 0.04
0.31 1: 0.05
0.13 1 0.03
c=BCE (10
PreResNet-164
0.29 0.04
0.13 0.03
0.28
0.13 0.03
0,28 + 0.04
0.13 +
DenseNet-BC
0.23 0.03
0.11 0.02
0.24
0.12 0.02
0.24
0.02 0.02
0.11 = 0.02
ResNet-110
0.40 0.02
0.16 + 0.01
0.35 0.02
0.13
0.35 + 0.02
0.14 0.01
ResNet-164
0.24
0.11 0.01
0.00
0.10 0.01
0.20
0.10 0.01
NLL
164
0.23 0.02
0.13 0.02
0.09
0.02
0.10 0.01
DenseNet-BC
0.17 0.01
0.10 0.01
0.14 0.01
0.09 + 0.01
0.01
0.08 + 0.01
Table Calibration metrics for various CIFAR-10H models with P+L combinations. Even a small amount of labeled data (I0 labels) reduces the calibration error of both the classifier and combination. For all metrics, lower is better.
triples used for learning, whereas for unsupervised methods the dataset size corresponds to the number of (h(x)), m(x)) pairs without ground truth y.
As Figure 2 demonstrates, the P+L method able to learn a human-model combination that out- performs both the human and model alone, with very few datapoints. While the P+L-EM method requires more human labels than the P+L method, it is able to learn such a combination without any ground-truth labels. The baseline L+L method fails to learn an effective combination on the CIFAR-10 dataset, and only does on the VGG-19 ImageNet dataset with a large number of ground truth labeled datapoints. This demonstrates that the instance level probabilities from the model are a key component in efficiently learning human- model combinations with high accuracy. In Appendix E, we provide similar plots for GoogLeNet and for maximum likelihood based inference procedures. In general, we find that maximum likelihood estimation requires more data than MAP estimation, and hence we focus our presentation of results on MAP
Calibration Properties of Combinations. In addition to the error rate, we study the calibration properties of P+L combinations. In Table 2, we report the ECE [Guo et al., 2017], classwise-ECI (cwECE) [Kull et al., 2019, Patel et al., 2021], and negative log-likelihood (NLL) for our various CIFAR-1O models (Model) and the resulting human-machine combinations (Comb.) on the held-out evaluation set. The ECE and ewECE are evaluated using 15 bins containing equal number of data points. In addition to P+L combinations with 10 or 5000 labeled datapoints, we evaluate a combination consisting of the uncalibrated classifier probabilities with the MAP human confusion matrix estimated with 5000 labeled datapoints (No Calibration).
Combining classifier probabilities with human labels generally results in combination that is better calibrated than the model alone. Moreover, MAP TS can be fit using a very small number of labeled datapoints. Our results show that the calibration properties (of both the classifier alone and the resulting human-machine combination) significantly improve with only ten labeled examples. However, increasing the number of labeled examples to 5000 does not result in further calibration gains. We provide similar results for our ImageNet-16f models in Appendix I.
Theoretical Analysis
The key quantity in our analysis is the confidence ratio of predictor, which is a random variable representing a predictor's 's confidence for the correct class relative to the predictor's confidence for other classes. This quantity can be thought of the predictor's instance-level odds for making a correct prediction. More specifically, the confidence ratios I' and Th for machine and human labelers are defined as
Confidence Ratios.
mx
(hh())
(=) 1 1-mg(e)
ra(2:) 1 --PM())
(5)
8

---

Classifier
Error (9)
Human
ECE
3.35
0.91
15.99
0
2
20
40
Temperature
log(r.(x))
Error Rate (%)
12.5
Classifier
10.0
Human
ECE
8.94
7.5
1.65
17.93
5.0
10
40
Temperature
log(rm(x))
Figure 3: Left: Single human labeler combined with various classifiers of equal accuracy, resulting in combinations with varied accuracies. The orange point corresponds to the P+L combination. First row: ResNet-164 on CIFAR-IOH. Second row: VGG-19 on ImageNet-16H noise level 80.
We note that unlike the machine classifier, the human does not directly output such confidences rather, this quantity is estimated empirically through the human's confusion matrix.
If the model has confidence ratio of (x:) 1 (indicating that the model has confidence greater than 0.5 for the correct class), then the model is guaranteed to correctly label x. On the other hand, correctly label instead, the model must be
I'm (ar) >> is not sufficient for the combination sufficiently confident in its prediction as well. The following theorem formalizes this notion by lower bound the accuracy of the combination in terms of the confidence ratios of the individual predictors. For binary classification tasks, this lower bound is achieved, i.c. we have equality.
Theorem (Combination Model Accuracy.). The accuracy of the P+L combination c(x) is least the probability that the confidence ratio for exceeds the inverse confidence ratio for h.
E 11 (c(x) = v)1 2 (() 2 (rx(x))-+)
(6)
A detailed proof is provided in Appendix J. An analogous result holds for the combination of two probabilistic classifiers or two non-probabilistic classifiers. As our focus is on combining human predictions with model probabilities, discuss these cases in Appendix J.
Theorem (1) is further illustrated Figure 3 for ResNet-164 classifier on CIFAR-IOH (first row) and a VGG-19 classifier on ImageNet- 16H (second row). For each row, we create family of classification models where each model makes the same class-level predictions (and hence has the same error rate) but with different confidence ratio distributions. This is achieved by tempering the output probabilities of base classifier via the map m(X)
(m² 1/T ..... ... m& ( E my 1/) with temperature T' > [Guo et al., 2017, Zhang et al., 2020]. In the first column of each row, the solid curve plots the error rate of the combination of fixed human with the various classification
models. Despite each classification model having the same accuracy, the accuracy of the resulting human-model combination varies.
This behavior can be explained by Theorem (1), which tells us that the combination accuracy driven not only by the human and machine accuracies but by their confidence ratios as well. At large temperatures (purple), the classifier becomes underconfident in its predictions, and the combination error rate approaches the human error rate. At small temperatures (green), the model becomes overconfident its predictions, and the combination error rate approaches the model error rate. When the combination is fit by our P+L method (orange), the classifier is well-calibrated (reflected by its low ECE), and the resulting combination obtains a lower enror rate than each of the under- and over-confident classifier combinations.
Relationship between combination and calibration/confusion error. We additionally quantify the estimation error our PtL method, incurred by empirically estimating the human confusion

---

matrix and calibration parameters. The result below shows we can upper bound our estimation error by the estimation error for the confusion matrix and the E marginal calibration error (MCE) [Kumar et al., 2019].
Theorem (Estimation Error Upper Bound.). Let nfa,, v) (p(h(=)))) (o(((())ll))((ss(((()))s)))))))) - m (3)) Fh(x)yl be the estimation error (up to normalizing constants), where Piz represents an estimate of p( = i j). Under the CI assumption, in expectation over random (x, y) pairs,
E Exx,Y [n(X, & 11 - G11 +ME((m²)
(Proof in Appendix J). With sufficient amount of labeled data, the confusion matrix error || د11 can be made arbitrarily small (i.e. the MAP confusion matrix estimate is unbiased). This is not necessarily the case for MCE(m²). However, if an asymptotically unbiased calibration method is used, this result guarantees that our posterior estimation error will converge to zero.
7 Limitations and Potential Societal Impacts
Limitations. One limitation of our work is that our experiments only involve two datasets and both involve image classification. Thus, there is guarantee that similar results (in terms of combined improvements) are achievable for other tasks, such as ceesstioon-nswwringg from text data more general problem-solving tasks. Another potential limitation is reliance conditional independence in our approach. A reverse view of this is that both our theoretical and experimental results demonstrate that there is ample room for improving human and machine performance by combining their predictions, even without taking dependence into account.
In Appendix K, find that combining a probabilistic model and non probabilistic model (rather than human) often does not improve over the better of the two. We hypothesize that this is due to a lack of diversity in the predictions of two models. Precisely characterizing the dependence of our method on the predictor diversity is an interesting direction for future work, as well as developing methods for combining probabilistic and non-probabilistic predictions that are robust to this lack of diversity.
Potential Societal Impacts. Combining human and machine predictions to improve overall classi- fication accuracy has the potential for positive societal benefit, particularly for example in high-stakes applications such medical image diagnosis and autonomous driving. However, there are also potential negative societal impacts. For example, if there is a lack of transparency in terms of how the system operates (e.g., how predictions are being combined arrive at final result), augmenting an individual's ' predictions with machine's could have negative psychological consequences for the individual, such as decreasing trust, reducing individual autonomy, and eventual disengagement.
8 Conclusions
We investigate methods for combining predictions using instance-level confidence from model and class-level information from a human. Across a variety of image classification experiments our proposed combination framework leads to systematic increases in accuracy over both the model and human alone while requiring few human labels. Supporting theory illustrates how combined human-model performance is affected by calibration properties of the model.
Acknowledgements and Funding Disclosure
We thank Heliodoro Tejeda for contributing trained models to this project. This material is based upon work supported in part by the HPI Research Center in Machine Learning and Data Science at UC Irvine, by the National Science Foundation under grants number 1900644 and 1927245, by the Center for Statistics and Applications in Forensic Evidence (CSAFE) through Cooperative Agreements 70NANB 151176 76 and 70NANB20H019 between NIST and Iowa State University, which includes activities carried out at the University of California Irvine, and by a Qualcomm Faculty Award. Additional revenues potentially related to this work include: research funding from NIH, PCORI, DARPA, and SAP: consulting income from Amazon.com.
10

---

References
Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S. Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13): 1405411414, May 2021.
Melanie Beck, Claudia Scarlata, Lucy F Fortson, Chris J Lintott, BD Simmons, Melanic A Galloway, Kyle W Willett, Hugh Dickinson, Karen L Masters, Philip J Marshall, et al. Integrating human and machine intelligence in galaxy morphology classification tasks. Monthly Notices of the Royal Astronomical Society, 476(4):5516-5534 2018.
Nicholas Bien, Pranav Rajpurkar, Robyn L.. Ball, Jeremy Irvin, Allison Park, Erik Jones, Michael Bereket, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Safwan Halabi, Evan Zucker, Gary Fanton, Derek F. Amanatullah, Christopher F. Beaulieu, Geoffrey M. Riley, Russell J. Stewart, Francis G. Blankenberg, David B. Larson, Ricky H. Jones, Curtis P. Langlotz, Andrew Y. Ng, and Matthew P. Lungren. Deep-learning-asiste diagnosis for knee magnetic resonance imaging: Development and retrospective validation of mrnet. PLOS Medicine, 15(11):1-19, 11 2018.
Steve Branson, Grant Van Horn, and Pietro Perona. Lean crowdsourcing: Combining humans and machines in an online system. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6109-61 18, 2017.
Clintin P Davis-Stober David V Budescu, Stephen B Broomell, and Jason Dana. The composition of optimally wise crowds. Decision Analysis, 1203)113--143, 2015.
Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society; Series C (Applied Statistics), 28 (1):20-28, 1979.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Computer Society Conference Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pages 248-255, Los Alamitos, CA, USA, jun 2009. IEEE Computer Society.
Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1-15, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg.
Yashesh Gaur, Florian Metze, Yajie Miao, and Jeffrey P Bigham. Using keyword spotting to help humans correct captioning faster. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18900119022 Curran Associates, Inc., 2020.
Christian Genest and James V. Zidek. Combining Probability Distributions: A Critique and an Annotated Bibliography. Statistical Science, 1(1):114 135, 1986.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 630-645. Springer, 2016b.
11

---

Matthew D Hoffman and Andrew Gelman. The no- no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1):1993-1623 2014.
Lu Hong and Scott E Page. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101 (016::66355-16688 2004.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
Robert A. Jacobs. Methods For Combining Experts Probability Assessments, Neural Computation, 7(5)8867-888, 09 1995.
Matthew Johnson and Alonso Vera. No ai is an island: The case for teaming intelligence. Al Magazine, 40-1))16-28 Mar. 2019.
Daniel Kahneman and Amos Tversky. On the reality of cognitive illusions. Psychological Review, pages 582-591, 1996.
Ece Kamar. Directions in hybrid intelligence: Complementing Al systems with human intelligence. In Proceedings of IJCAI, pages 4070-4073, 2016.
Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, volume 12, pages 467-474, 2012.
Gideon Keren. Calibration and probability judgements: Conceptual and methodological issues. Acta
Psychologica, 77(3):217-273, 1991.
Hyun- Chan-Chll Kim and Zoubin Ghahramani. Bayesian classifier combination. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 617, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.
Kittler, M. Hatef, R.P.W. Duin, and J. Matas. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3):226-239) 1998.
Joshua Klayman, Jack B Soll, Claudia Gonzalez- Vallejo, and Sema Barlas. Overconfidence: It Gonzalez-Vlleejo, depends on how, what, and whom you ask. Organizational Behavior and Human Decision Processex, 79(3):216-247, 1999.
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1):237-293, 08 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto., 2009.
Meelis Kull, Miquel Perello Nieto. Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ludmila Kuncheva, On the optimality of naïve bayes with dependent binary features. Pattern Recogn. Leut. 27(7):30--37,, May 2006.
Ludmila Kuncheva. Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons, 2014.
PJ I Lamberson and Scott Page. Optimal forecasting groups. Management Science, 8854))805-810 2012.
12

---

Michael D Lee and Megan N Lee. The relationship between crowd majority and accuracy for binary decisions. Judgment and Decision Making. 12(4):328, 2017.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving faimess and accuracy by learning to defer. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bssa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Leurning Research, pages 7076-7087. PMLR, 13-18 Jul 2020.
Nastaran Okati, Abir De, and Manuel GodezzRodriguez. Differentiable learning under triage. In Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2021.
Bhavik N Patel, Louis Rosenberg, Gregg Willcox, David Baltaxe, Mimi Lyons, Jeremy Irvin, Pranav Rajpurkar, Timothy Amrhein, Rajan Gupta, Safwan Halabi, et al. Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digital Medicine, 2(1):1-10, 2019.
Kanil Patel, William H. Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty calibration via mutual information maximization-based binning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617-9626, 2019.
P Jonathon Phillips, Amy N Yates, Ying Hu, Carina A Hahn, Eilidh Noyes, Kelsey Jackson, Jacque- line G Cavazos, Géraldine Jeckeln, Rajeev Ranjan, Swami Sankaranarayanan, al. Face recogni- tion accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proceed- ings of the National Academy of Sciences, (15(24))6171-6176 2018
Maithra Raghu, Katy Blumer, Greg Corrado, Jon M. Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. CoRR, abs/1903. 12220, 2019.
Pranav Rajpurkar, Chloe O'Connell, Amit Schechter, Nishit Asnani, Jason Li, Amirhossein Kiani, Robyn Ball, Mare Mendelson, Gary Maartens, Daniël J van Hoving, et al. Chexaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. NPJ Digital Medicine, 3(1):1-8, 2020.
Mark O. Riedl. Human-centered artificial intelligence and machine learning. Human Behavior and Emerging Technologies, 1(1):33-366 2019.
Amir Rosenfeld, Markus D Solbach, and John K Tsotsos. Totally looks like how humans compare, compared to machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1961-1964 2018
Olga Russakovsky, Li-Jia Li, and Li Fei-Fei. Best of both worlds: human-machine collaboration for object annotation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2121-2131, 2015.
Omer Sagi and Lior Rokach. Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e 1249, 2018.
Thomas Serre. Deep learning: The good, the bad, and the ugly. Annual Review of Vision Science, (1):399-426, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
13

---

Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-Al complementarity Proceedings of the National Academy of Sciences of the United States of America, to appear, 2022.
Christian Szegedy, Wei Liu, Yangqing Jía, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1-9, 2015.
Laura Trouille, Chris J. Lintott, and Lucy F. Fortson. Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human-machine systems. Proceedings of the National Academy of Sciences, 16(()): 1902-1999,, 2019.
Jennifer Wortman Vaughan. Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research, 18(193):1 46, 2018.
Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI-20, pages 1526-1533. International Joint Conferences on Artificial Intelligence Organization, 7 2020.
Darryl E Wright, Chris J Lintott, Stephen J Smartt, Ken W Smith, Lucy Fortson, Laura Trouille, Campbell R Allen, Melanie Beck, Mark C Bouslog, Amy Boyer, et al. A transient search using combined human and machine classifications. Monthly Notices of the Royal Astronomical Society, 17222):1155-1323, 2017.
Lei Xu, Adam Krzyzak, and Ching Y Suen. Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man, and Cybernetics, 22 (3):418 4355, 1992.
Zahra Zahedi and Subbarao Kambhampati. Human-Al symbiosis: A survey of current approaches. arXiv preprint 2eX.x:22103009990 2021.
Jize Zhang, Bhavya Kailkhura, and Yong-Jin Han. Mix-n-match : Ensemble and compositional methods for uncertainty calibration in deep learning. In Hal Daumé III and Aarti Singh, edi- tors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11117-11128 PMLR, 13-18 Jul 2020.
14


Reviews:
[{"review_id": "pG0qZnQ4M0t", "reviewer": "Reviewer_YY6P", "summary": "This paper presents and investigates methods for combining human and machine predictions. In the proposed setup, such a combination relies on instance-level probability outputs from a machine learning model and class-level information from human annotators. In a series of extensive experiments, the authors show that their method, which relies on model recalibration and confusion matrices, outperforms both humans and ML models. To support the strong empirical results, the authors demonstrate theoretically how this combination gains from the recalibration process.\n\nThe main contributions of this paper are its simple and sound methodology, its strong experimental analysis and its useful theoretical grounding. It is also clearly written and insightful.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a strong, clearly written paper that improves our understanding of an important subject. While it is not totally original in its approach (there is a plethora of existing work on combining humans and machines for better decision making), it is extremely clear in its approach and it makes several important contributions to this literature. \n\nMainly, the proposed framework generalizes existing approaches to human-machine combinations and can be used in the future for other applications and with other algorithmic solutions. Second, the empirical analysis sheds light on the sensitivity of such approaches to model calibration, and specifically to overconfidence. Deep neural networks are notoriously over confident, and understanding how this affects their usefulness as decision support systems is crucial. Finally, the theoretical analysis is insightful and relevant, shedding light on the sensitivity to model overconfidence.\n\nHowever, while I appreciate the authors’ acknowledgement of the paper’s limitations, I do think that more can be made to address them. First, the reliance on conditional independence is indeed problematic and might result in the approach being less useful when estimating conditional independence is infeasible. Second, it would be interesting to see the performance of the approach on datasets which do not involve image classification. Specifically, extending to NLP, where overconfidence is even more of a problem, would also be interesting. Third, it might be interesting to explore more advanced recalibration methods, or optimize for calibration in the first place (for example see Kumar et al 2018).\n\nFinally, it is unclear whether the results are robust to other human selection methods. The authors choose humans randomly, but it is unclear if their method is sensitive to cases where each human annotates many examples and is biased towards specific types of decisions. I might have missed that in the paper, so perhaps better explaining the sensitivity to the human selection process in the CIFAR and ImageNet datasets might be helpful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper can potentially have a substantial positive societal impact, as successfully combining human and machine predictions can improve decisions made in high-stakes settings. Moreover, better-understanding the effect of model calibration on downstream decisions supported by machine learning models can surely improve their safe dissemination in the wild. I believe that the authors have properly explained the benefits in the context of high-stakes applications, though more emphasis can be made on the importance of understanding model calibration. \n\nI also think that the authors have adequately addressed the limitations of their work, which are the focus on computer vision applications and their reliance on the conditional independence assumption. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WcFJ9br9a33", "reviewer": "Reviewer_X6gK", "summary": "The paper \"Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration\" proposes a method to obtain a well-performing ensemble model of both an ML classifier (which outputs a probability for every class) and a human observer (who outputs a categorical decision). This is a setting that traditional ensemble methods do not account for; e.g. majority vote would not be possible here since there are only two decision makers in total (1 model, 1 human). The problem may not be common in standard ML but since ML applications are increasingly being used in real-world settings, there are areas where this approach can be applied (such as high-stakes medical decisions). Combining model and human predictions is shown to lead to lower error rates than either of the two alone.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nAccording to the authors (lines 177ff) , this is the first method that combines a hard label prediction with a probabilistic label prediction. The related work section (Section 3) references relevant prior work, and clearly delineates how the paper differs from existing publications.\n\nMinor comment: The authors write that \"This work emphasizes the point that combinations of models that have diversity in how they make predictions can systematically outperform a single model.\" -> I'm not convinced that this point really needs emphasis, it's rather well-known that correlated predictors are bad, right? (e.g. multicollinearity in regression).\n\n\n## Quality\n\nThe paper uses a nice combination of experimental and theoretical methods, and the results are presented clearly in text, figures and tables. Disclaimer: I am not an expert in this area. That being said, the results (e.g. lines 55f) are convincing despite the simplicity of the approach - for instance, if the achieved accuracy gain were the result of an architectural modification, it would be considered a significant improvement. Of course, this comes at the cost of obtaining human decisions, which is not feasible or practical in many cases, but for specific use cases described by the authors (e.g. medical diagnostics) it may well be useful. \n\nEquation 2 assumes conditional independence, which is an assumption that typically does not hold in practice, at least not completely (see Table 3 in the appendix) but the numerical results are good in spite of a potential violation of this assumption, so I don't consider this to be a big problem.\n\nWhile the authors focus on the setting where human and machine decisions are combined, their approach is generally applicable to any two decision makers whenever there is a combination of two models, one with confidence scores for all classes and one without (i.e. only outputting a single class). In order to increase the potential appeal and applicability of the paper, it might be helpful to additionally showcase one example where instead of combining human and machine predictions, two machine predictions are combined in this way. This might be an interesting setting e.g. for cases where there is a proprietary model for which only the output decision (but not the confidence) is known, and it would additionally be a nice example for settings where there might be more than a handful (16) of classes.\n\n\n## Clarity\n\nThe introduction is well written and clearly motivates the problem. The paper is well organized and structured.\n\nWhile the authors perform multiple experiments testing calibration properties, it would be helpful to explicitly state the limitations of the approach: e.g. are there any limitations on the accuracy difference between the two decision makers that need to be met? At which point of poor calibration does the combined model perform worse than e.g. a human observer alone? How \"different\" / uncorrelated / ... do model and human predictions need to be?\n\nTwo minor comments regarding clarity:\n\n- Figure 1 right: it would be helpful to mention the datasets below the two bar plot groups (I'm assuming the left one refers to CIFAR-H and the right one to ImageNet-16H?).\n\n- I was confused about the sentence in line 98: why does a single observer have a confusion matrix? Is the observer shown the image multiple times and then decides differently? Or is this an aggregated confusion matrix over multiple images? Generally, how can you parameterise a probability with an entire confusion matrix? This became a bit more clear when looking at equation (3) but it would be helpful to make the entire paragraph starting with line 98 more precise / self-contained. I'm still not entirely sure about how the values for the confusion matrix are obtained.\n\n- In equation (3), \"i\" is defined twice: once on the left-hand side of the equation and also again on the right-hand side as the sum counter; this is confusing\n\n#### Code clarity\n\nI appreciate that code was provided. A README would go a long way in making the code more accessible; currently it's mostly a list of files and scripts. The 20+ TODOs don't really help in providing confidence in the code quality: \"Do these edge cases really need to be separated out??\", \"This isn't the right file for this\", ... plus, as a suggestion for future code submissions, I would make sure that hard-coded paths are properly anonymized.\n\n## Significance\n\nAs explained above, the paper addresses a relevant topic, more precisely, how human and model decisions may be combined. This is probably less relevant for tasks like the investigated one (such as standard ImageNet object recognition), but might become very relevant in the context of medical imaging (where peak performance truly matters, and spending the time of obtaining a human classification decision is a fair price to pay for improved performance).\n\n## MISC\n\n- it would be helpful to use a citation style etc. where one can click on a number in the paper and directly end up at the reference (currently this is not possible, so looking up a reference involves a lot of scrolling down)\n- typo line 39: remove \"being\"\n- humans are refered to as \"non-probabilistic labelers\"; perhaps a footnote could be added to explain that this refers to the fact that humans output a category instead of a probability vector. The term \"non-probabilistic labeler\" could lead to confusion if understood in the sense that humans would be deterministic and always choose the same label, i.e. that humans wouldn't have internal noise, which is not the case.\n- typo line 146: remove \"that\"\n- Table 2: for readers not familiar with the metrics, indicate whether lower=better or higher=better (and perhaps make the leading entries bold?)\n- the paper sometimes refers to \"human labelers\", which may be confusing since usually \"labelers\" are those who generate labels for the dataset (rather than classify images). Perhaps rephrase accordingly?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In Section 7 of their paper, the authors discuss limitations and potential societal impact. In particular, I appreciated that the authors took the time to think about potential _negative_ societal consequences as well, which can often be difficult at first (especially for standard \"pure research\" papers). Well done!\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h 40min"}, "extra_text": {}}, {"review_id": "CulHYVHD6MV", "reviewer": "Reviewer_NHxx", "summary": "The current work focuses on the interesting prospect and problem of combining human and machine classification information per input, especially in the small-data regime, for the sake of ultimately improving accuracy. The authors take a probabilistic approach to motivating their strategy for combining human-machine information, leveraging a carefully chosen prior to enable inference of a reliable human confusion matrix when the number of annotators is low. They ultimately find that the resulting combination (evaluated in the context of popular image classification tasks) improves accuracy over both humans and models alone, and with drastically less data than previous work. They show more formally that this success is in part driven by both the accuracy of humans/models as well as the confidence values of each.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To my knowledge, the methods employed, formal arguments, and conclusions are generally valid. The paper is atypically clear in explaining its aims, proposal, and evaluation. While I expect that these results will be robust, I would have preferred that the authors discuss them in the context of past work as well as different interpretation of the human behavioral datasets.\n\nFor example, the CIFAR-10H dataset contains over 500,000 judgements spanning thousands of annotators having viewed different subsets of images. Under the assumption that human annotators are noisy labelers with similar classification behavior, each labeler provides a sample from a \"mental class-conditional distribution\". However, if classification behavior is diverse, the resulting labels are more like argmax predictions from an ensemble of humans. In the former case, we can think of this sort of data as sort of representing a model, whereas in the latter, we might instead think of it as representing an ensemble of models in itself. The implications for the proposed method and its applicability given these two interpretations is not clear. Since the authors sample from single annotators, I think this is worth discussing.\n\nMoreover, the original CIFAR-10H work performed a sort of validation of the soft labels by evaluating models trained on them on out-of-distribution datasets and against adversarial attacks. This was effectively a kind of argument that the confidence information conveyed in this sort of data is useful for classification. Given these results, why shouldn't we expect this confidence information to be useful in the current work? Further, why wouldn't the authors evaluate their hybrid method using OOD datasets, where accuracy on validation sets can be misleading as a target of combination?\n\nNonetheless, I think the work provides a unique contribution, and provides a good reason to be optimistic about leveraging these sorts of human datasets without having to scale dataset collection by a factor of n_annotators_per_image in a way that is prohibitively expensive.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss plausible possible negative impacts that I did not foresee myself.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9pnsPGNMHXJ", "reviewer": "Reviewer_nC5z", "summary": "The authors introduce a simple method for combining predictions from a human and a model (or more generally a non-probabilistic and probabilistic labeler).  The resulting combination model is more accurate than either models or humans alone, and better calibrated than models alone.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Note: I am not familiar with this area of work.  \n\nOverall, I think the problem statement is well-motivated and seems to be under-researched.  However, I would've liked to see more thorough experiments on the methods.\n\nI think it would be natural to have baselines where: the model m(x) is finetuned on the labels h(x).  another variant is having m(x) re-trained on ground truth but where it can condition on h(x) for the portion of inputs h(x) is available.  I expect this should be worse in the low data regime but it's less clear in the 5000 datapoint regime.  (It would also be interesting to have humans that get to condition on model predictions, although that is understandably much more difficult to try.)\n\nThe combining method at its core is simple, but I would've liked to see ablations on the functional forms for confusion matrix estimation and calibration parameter estimation.  For example, learning K^2 parameters doesn't seem that bad in the 5000 data point regime.  It seems natural to parametrize with a low rank factorization as well.\n\n\nOther Qs:\n\n- In Figure 2, it's not clear to me why the original CIFAR models aren't already well calibrated on NLL, given that there is no distortion.  \n\n- I might have misunderstood, but did Theorem 2 require the conditional independence assumption?  that seems pretty important to state, even if it's kind of obvious after thinking about it.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "seems good", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
- Simple, well-executed approach
- Clever combination of machine and human strengths (probabilistic vs categorical predictions)
- Clear idea, clear writing

Weaknesses:
- Some unclarity regarding robustness of empirical results
- Needs discussion regarding individual vs. collective human-labeling behavior
- Authors should be more precise about limitations, and address them

Summary:

After some helpful discussions between the reviewers and the authors (including detailed responses by the authors, which the reviewers appreciated), reviewers are mostly in agreement that this is a strong submission which should be of interest to the community. Concerns mostly regarded the experimental setup. Here, reviewers were concerned about whether the empirical results would prove to be results under reasonable variations of the setup. Relatedly, reviewers also pointed out that results could be specific to how the authors chose to select labelers (at random), and asked the authors to clarify their reasoning behind this and they way in which they present their results in light of this experimental design decision. Authors are encouraged to clarify the issues regarding diversity in individual user behavior vs. collective user behavior, and how it relates to their experiments.
Finally, two reviewers were glad to see that the authors acknowledged some of the limitations of their approach, but at the same time, urged the authors to discuss them with more precision, and – more importantly – take steps to address them in the final version of the paper.

Decision: Accept (Poster)