Multi-annotator Deep Learning: A Probabilistic Framework for Classification
Authors: Marek Herde
Denis Huseljic
Bernhard Sick
Venue: Accepted by TMLR ()

Abstract: Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.

Published in Transactions on Machine Learning Research (09/2023)
Multi-annotaton Deep Learning: A Probabilistic Framework for Classification
Marek Herde
morek.hendeeuuit-asse....
Intelligent Embedded Systems University of Kassel Kassel, Hesse, Germany
Denis Huseljic
dbuseljjcChuniikkssscl
Intelligent Embedded Systems
University of Kassel Kassel, Hesse, Germany
Bernhard Sick
bsick&humi-ken weans-hunnsasell...
Intelligent Embedded Systems University of Kassel Kassel, Hesse, Germany
Reviewed on OpenReview: https: // openreview. net/ net/forum? forum? id= Ngdowaimix
Abstract
Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e-g,, crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We ad- dress this issue by presenting probabilistic training framework named mulli-annolator deep learning (MaDL). A downstream ground truth and annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers proba- bilistic estimates annotators' performances. modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' den- sities within latent space proxies of their potentially correlated annotations. Together with weighted loss function, improve the learning from correlated annotation patterns. In a comprchensive evaluation, we examine three rescarch questions about multi-annotator supervised learning. Our findings show MaDL's state-of the--ar performance and robustness against many correlated, spamming annotators.
1 Introduction
Supervised deep neural networks (DNNs) have achieved great success in many classification tasks Pouyanfar et al., 2018) In general, these DNNs require vast amount of annotated data for their successful employ- ment Algan & Ulusoy 2021). However, acquiring top-quality class labels annotations is time-intensive and/or /or/or financially expensive (Herde et al. 2021). Morcover, the overall annotation load may excced single annotator's workforce (Uma el al. 2021). For these reasons, multiple non-expert. annotators, e.g,, crowd- workers, are often tasked with data annotation (Zhang) 2022 Gilyazev & Turdakov 2018). Annotators' missing domain expertise can lead erroncous annotations, known as noisy labels. Further, even expert an- notators cannot be assumed to be omniscient because additional factors, such as missing motivation, fatigue, annotation task's ambiguity (Vaughan 2018), may decrease their performances. A popular annotation

---

Published in Transactions on Machine Learning Research (09/2023)
quality assurance option is the acquisition of multiple annotations per data instance with subsequent aggre- gation (Zhang et al. 2016), e.g,, via majority rule. The aggregated annotations are proxies of the ground truth (GT) labels to train DNNs. Aggregation techniques operate exclusively on the basis of annotations. In contrast, model-based tochniques use feature or annotator information and thus work well in low-redundancy settings, e.g,, with just one annotation per instance (Khetan et al. 2018 Through predictive models, these techniques jointly estimate instances' GT labels and anmotators performances (APs) by learning and in- ferring interdependencies between instances, annotators, and their annotations. As result, model-based techniques cannot only predict GT labels and APs for training instances but also for test instances, i... they can be applied in transductive and inductive learning settings (Vapnik) 1995).
Despite ongoing research, several challenges still need to be addressed in multi-annotatorr supervised learn- ing. To introduce these challenges, we exemplarily look at the task of animal classification in Fig. [ Eight annotators have been queried to provide annotations for the image of jaguar. Such query is difficult because jaguars have remarkable similaritics to other predatory cats, c.g., lcopards. Accordingly, the ob- lained annotations indicate strong disagreement between the leopard and jaguar class. Simply taking the majority vote of these annotations results in leopard as wrongly estimated GT label. Therefore, advanced multi-annotator supervised learning techniques leverage annotation information from other (similar) anno- tated images to estimate APs. However, producing accurate AP estimates is difficult because one needs to learn many annotation patterns. Otherwise, the estimated GT labels will be biased, e.g., when APs are exclusively modeled as a function of annotators. In this case, we cannot identify annotators who are only knowledgeable about certain classes or regions in the feature space. Another challenge in multi-annotator supervised learning concerns potential (latent)) correlations between annotators. Tn animal annotation task, illustrate this issuc by visualizing three latent groups of similarly behaving annotators. Although assume that the annotators work independently of each other, they can still share common stalistiially correlated error patterns (Chu et al. 2021). This is particularly problematic if a group of ordinary persons strongly outvotes a much smaller group of professionals. Considering prior information about the annota- tors, Lee, annotator features or metadata (Zhang et al. 2023), can help to identify these groups. Moreover, prior information enables model inductively learn performances for annotators who have provided few or no annotations. In our example, zoological interest could be a good indicator for this purpose. While the inductive learning of APs for annotators poses an additional challenge the already complex task, its use may be beneficial for further applications, e.g., optimizing the annolator selection in aclive learning setting (Herde et al. 2021 or training annotators to improve their own knowledge (Daniel et al. 2018
A Ausolators
Instance
ampotare
Latent Annotator Groups:
Professional Zoologists
Hobihy Zaologists
Ordinary Persons
Ground Truth: Jaguar
Annotations:
Jugune
Leopard
Jugune
Jagune
Leopord
Lespard
Leogurd
Leopard
Age
38
46
23
44
53
32
19
Zoological Interenl:
Very High
High
High
Very High
Very Low
Medium
Law
Medium
Aunolator Features
Annotation Time:
46
Figure 1: Animal classification an illustration of multi-annotalor supervised learning problem.
In this article, we address the above challenges by making the following contributions:
We propose mulli-unnolalor deep learning (MaDL) a probabilistic and modular classifi- cation framework. In an end-to-end training via a weighted maximun-likelihod approach, it learns embeddings of annotators to account for possible correlations among them.
We specify six properties concerning the estimation of APs and application scenarios for categorizing related multi-annotator supervised learning techniques.
Associated with these properties, we formulate three research questions (RQs), which we experimentally investigate, including comparisons of MaDL related techniques.
2

---

Published in Transactions on Machine Learning Research (09/2023)
The remainder of this article is structured as follows: In Section 2 we formally introduce the problem setting of supervised learning from multiple annotators. Subsequently, we identify central properties of multi-annotator supervised learning techniques as a basis for categorizing related works and pointing out their differences to MaDL in Section Section !| explains the details our MaDL framework. Experimental evaluations of MaDL and related lechniques are presented regarding RQs associated with the aforementioned properties in Section S
Finally, we conclude and give an outlook regarding future research work in Section
2 Problem Setting
In this section, formalize the assumptions and objectives of multi-annotator supervised learning for classification tasks.
Without loss of generality, we represent data instance as a vector x= (x x (D)), D D N in a D-dimensional real-valued input or feature space Ωx == D. The N E N>o instances jointly form a matrix X = (X1,..,,xN) and originate from an unknown probability density function Pr(x). For each observed instance x, ~ Pr(x), there is a GT class label yn E fl == [I....: C). Each GT label yn is assumed to be drawn from an unknown conditional distribution: Yn ~ Pr(y x,). We denote the GT labels as the vector y = (y1,..., VN) yN) These GT labels are unobserved since there is omniscient annotator. Instead, consider multiple error-prone annotators. For the sake of simplicity, individual features as a vector am E U4 = RO, O E N If no prior annotator information is available, the annotators' features are defined through one-hot encoded vectors, i.e., ΩA == fei....,ee,, ,eM] with am == em to identify each annotator uniquely. Otherwise, annotator features may provide information specific to the general annotation task, e-g-; the zoological interest when annotating animal images or the years of experience in clinical practice when annotating medical data. Together, the M E N>o annotators form a matrix A- = (ai,...,a,))" We denote the annotation assigned by annotator an to instance X,, through 2nm f2 [80), where 2mm = indicates that annotation is unobserved, i.e., not provided. An observed annotation is assumed to be drawn from an unknown conditional distribution: 2um Pr(z Xn, Am, y). Multiple annotations for an instance X, can be summarized as a vector 2,, == (Zal,..., 2n M) Thereby, the set A. = [m | m E [I.....M] A 2am E f0y) } represents the indices of the annotators who assigned an annotation to an instance x,. Together, the annotations of all observed instances form the matrix == (Z),..., "NN) ZN) We further assume there is a subset of annotators whose annotated instances are sufficient to approximate the GT label distribution, i.e., together, these annotated instances allow correctly differentiate ween all classes. Otherwise, supervised learning is hardly possible without explicit prior knowledge about the distributions of GT labels and/or APs. Morcover, we expect that the annotators independently decide on instances' annotations and that their APs are time-invariant.
Prerequisites:
represent an annotator through
Objectives: Given these prerequisites, the first objective is to train a downstream GT model, which ap- proximates the optimal GT decision function UGT Ωx + Gy by minimizing the expected loss across all classes:
MGT(() = arg min (Egi [LT s/)l)
(1)
y'Ely
Thereby, we define the loss function LGT: B2 x fly {0,1) 1} through the zero-one loss:
0, if y =- v',
Lct(u.V)= 5(y v') 1, if y F v/
(2)
As result, optimal GT model for classification tasks accurately predict the GT labels of instances.
Proposition 1. Assuming LGT to be the zero-one loss in Eq. g the Bayes optimal prediction for Eq. is given by:
(xT = arg max (Pr(1/ x)).
(3)
yelly
When learning from multiple annotators, the APs are further quantities of interest. Therefore, the second objective is to train an AP model, which approximates the optimal AP decision function
3

---

Published in Transactions on Machine Learning Research (09/2023)
MAP :2x x 22a {0,1} } by minimizing the following expected loss:
yAP(x, a) = arg min (Fyi [Beix.ay [TAP (v', LoT (y, =))]))
(4)
('E(0.1)
Defining LAP and LGT as zero-one loss, an optimal AP model for classification tasks can accurately predict the zero-one loss of annotator's class labels, ie., whether an annotator a provides a false, i.., VAP(X, a) = 1, or correct, i.c., (xr(x, a) = 0, class label for an instance x.
Proposition 2. Assuming both LAP and Lor to be the zero-one loss, as defined in Eq. the Bayes optimal prediction for Eq. 4 is given by:
8AP a) =
Σ
Pr(y x) Pr(y x, < 0.5
(5)
Denr
We refer to Appendix A for the proofs of Proposition and Proposition
2
3 Related Work
This section discusses existing multi-annotator supervised learning techniques targeting our problem setting of Section 2 Since we focus on the AP next to the GT estimation, restrict our discussion to techniques capable of estimating both target types. In this context, we analyze related research regarding three aspects, i.e., GT models, AP models, and algorithms for training these models.
Ground truth model: The first multi-annotator supervised learning techniques employed logistic regres- sion models [aykar et al. 2010
Kajino et al. 2012 Rodrigues el al. 2013 Yan et al., 2014) for classification. Later, different kernel- bernellbased based variants of GT models, e.g., Gaussian processes, were developed (Rodrigues)
et al. 2014 Long et al. 2016 Gil-Gonzalez et al. 2021 Rodrigues et al. (2017) focused on documents
and extended topic models to the mulli-annotator setting. More recently, several techniques were proposed
to train DNNs for large-scale and especially image classification tasks with noisy annotations (Albarqouni
let al.
2016 Guan et al.
2018 Khetan et al.
2018 Rodrigues & Pereira 2018 Yang et al.) 2018 Tanno et al.
2019
Cao ct al. 2019
Platanios et al. 2020
Zhang et al.] 2020 Gil-González et al. 2021 Rithing Cachay
et al.
2021: Chu et al.
2021 Li el. al., 2022
Wei et al. 2022 Gao el. al. 2022 MaDL follows this line of
work and also employs (D)NN as the GT model.
Annotator performance model: An AP model is typically seen an auxiliary part of the GT model since it provides AP estimates for increasing the GT model's performance. In this article, we reframe an AP model's use in more general context because accurately assessing APs can be crucial in improving several applications, e.g,, human-in-the-loop processes (Herde et al. 2021) or knowledge tracing (Piech et al. 2015) For this reason, we analyze existing AP models regarding six properties, which we identified as relevant while reviewing literature about multi-annotator supervised learning.
(P1) Cleesedependent annotator performance: The simplest AP representation is an overall ac- curacy value per annotator. On the one hand, AP models estimating such accuracy values have low complexity and thus do not overfit (Rodrigues et al., 2013 Long et al. 2016). On the other hand, they may be overly general and cannot assess APs on more granular levels. Therefore, many other AP models assume dependency between APs and instances' GT labels. Class-dependent
AP models typically estimate confusion matrices (Raykar et al. 2010 Rodrigues al. 2014 2017
Khetan et al. 2018 Tanno et al., 2019 Platanios et al. 2020 Gao et al. 2022 Li et al. 2022
which indicate annotator-speeffc probabilitics of mistaking one class for another, c.g., recognizing
a jaguar as a leopard. Alternatively, weights of annotation aggregation functions CCao el. al. 2019
Rühling Cachay et al. 2021] or noise-adaption layers (Rodrigues & Pereira 2018 Chu et al. 2021
Wei et al.) (022) can be interpreted as non-probabilittic versions of confusion matricos. MaDL es-
timates probabilistic confusion matrices or less complex approximations, e.g., the elements on their
diagonals.
4

---

Published in Transactions on Machine Learning Research (09/2023)
(P2)
Instance-dependent annotator performance: In many real-world applications, APs are addi- lionally instance-dependen (aan et al. 2014) because instances of the same class can strongly vary in their feature values. For example, recognizing animals in blurry images is more difficult than in high-rosolution images. Hence, several AP models cstimate the probability of obtaining a correct annotation as a function of instances and annotators (Kajino et al. 2012 Yan al., 2014 Guan ct al. 2018 Yang et al., 2018: Gil-Gonzalez ct al.] 2021 Gil-González et al.) 2021). Combining
instance- and class-dependentt APs results in the most complex AP models, which estimate confu-
sion matrix per instance-annotator pair (Platanios et al.
2020
Zhang et al. 2020 Rühling Cachay
et al.
2021 Chu et al., 2021 Gao et al. 2022 Li et al.
2022 MaDL. also employs an AP model
of this type. However, it optionally allows dropping the instance and class dependency, which can benefit classification lasks where each annotator provides only a few annotations.
(P3) Annotator correlations: Although most techniques assume that annotators do not collaborate, they can still have correlations regarding their annotation patterns, e.g., by sharing statistically correlated error patterns (Chu et al.] 2021]. Gil-Gonzalez al. (2021) proposed bernel-based approach where malrix quantifies such correlations for all pairs of annolators. Inspired by weak supervision, Cao et al. (2019) and [Rühling Cachay et al. (2021) employ aggregation function that takes all annotations per instance as input to model annotator correlations. Gil-Gonzáloz et al. (2021) introduce a regularized chained DNN whose weights encode correlations. Wei et al. [2022) jointly model the annotations of all annotators as outputs and thus take account of potential correlated mistakes. Chu et al. (2021) consider common annotation noise through a noise adaptation layer shared across annotators. Moreover, similar to our MaDL framework, they learn embeddings of annotators. Going beyond, MaDI. exploits these embeddings to determine annotator correlations.
(P4) Robustness to spamming annotators: Especially on crowdsourcing platforms, there have been several reports of workers spamming annotations (Vuurens et al. 2011 e.g., by randomly guessing or permanently providing the same annotation. Such spamming annotators can strongly harm the learning process. As a result, multi-annotator supervised learning techniques are ideally robust against these types of annotation noise. Cao et al. (2019) employ an information- thhorettc approach to separate expert annotators from possibly correlated spamming annotators. Rühhing Cachay et al.) (2021) empirically demonstrated that their weak-supervised learning technique is robust large mumbers of randomly guessing annotators. MaDL ensures this robustness by training via a weighted likelihood function, assigning high weights to independent annotators whose annotation patterns have no or only slight statistical correlations to the patterns of other annotators.
(P5) Prior annotator information: On crowdsourcing platforms, requesters may acquire prior infor- mation about annotators (Daniel et al. 2018), c.g., through surveys, annotation quality tests, or publicly available profiles. Several existing AP models leverage such information to improve learn- ing. Thereby, conjugate prior probability distributions, e.g., Dirichlet distributions, represent a straightforward way of including prior estimates of class-dependent accuracies (Raykar et al.) 2010:
Albarqouni al. 2016 Rodrigues et al. 2017). Other techniques (Platanios et al. 2020 Chu al. 2021), including our MaDL, framework, do not directly expect prior accuracy estimates but work with all types of prior information that can be represented as vectors of annotator features.
(P6) Inductive learning of annotator performance: Accurate AP estimates can be beneficial in varions applications, e.g., guiding an active learning strategy to select accurate annotators (Yang (tt al. (2018). For this purpose, it is necessary that a multi-annotator supervised learning technique can inductively infer APs for non-annotated instances. Moreover, an annotation process often dynamic system where annotators leave and enter. Hence, it is highly interesting to inductively estimate the performances of newly entered annotators, e.g., through annotator features as used by Platanios et al. 2020 and MaDL.
Training: Several multi-annotator supervised learning tochniques employ the iopectation-mazimiaade (EM) algorithm for training (Raykar et al. 2010 Rodrigues
2013 Yan al. 2014 Long et al. 2016
Albarqouni et al. 2016 Guan et al.] 2018 Khetan et al. 2018 Yang et al. 2018
Platanios et al. 2020
5

---

Published in Transactions on Machine Learning Research (09/2023)
GT labels are modeled as latent variables and estimated during the E step, while the GT and AP models' parameters are optimized during the M step. The exact optimization in the M step depends on the underlying models. Typically, a variant of gradient descent (GD), e.g., quasi-Newton methods, is employed, or a closed- form solution exists, c.g., for AP models with instance-independent AP estimates. Other techniques take a Bayesian view of the models' parameters and therefore resort to expectation propagation (EP) (Rodrigues et al. 2014 Long al. 2016) or variational inference (VT) (Rodrigues al.) 2017). As approximate inference methods are computaatinnllyy expensive and may lead to suboptimal results, several end-to-end training algorithms have been proposed. Gil-Gonzalez et al. (2021) introduced a localized kernel alignment-basse relevance analysis that optimizes via GD. Through regularization term, penalizing differences between GT and AP model parameters. Kajino et al. formulated a convex loss function for logistic regression models. Rodrigues & Pereira (2018) Gil-González el. al. (2021) and Wei et al. (2022) jointly train the GT and AP models by combining them into a single DNN with noise adaption layers. Chu et al. (2021) follow a similar approach with two types of noise adaption layers: onc shared across annotators and onc individual for cach annotator. Gil-González
al. (202T) employ regularized chained DNN est.imate GT labels and AP performances jointly. In favor of probabilistic AP estimates, Tanno et al. (2019), Zhang et al. (2020) Li et al. (2022). and MaDL avoid noise adaption layers but employ loss functions suited for end-to-end learning, Cao et al. (2019) and Rühling Cachay et al. (021) jointly learn an aggregation function in combination with the AP and GT models.
Table D summarizes and completes the aforementioned discussion by categorizing multi-annotator supervised learning techniques according to their GT model, AP model, and training algorithm. Thereby, the AP model is characterized by the six previously discussed properties (P1-P6). We assign / if property is supported, x if not supported, and if partially supported. More preciscly, is assigned to property P5 if the technique can include prior annotator information but needs few adjustments and to property P6 if the technique requires some architectural changes to learn the performances of new annotators inductively. For property P4, a / indicates that the authors have shown that their proposed technique learns in the presence of many spamming annotators.
Table 1: Literature categorization of multi-annotator supervised learning techniques.
A Annotator Per corformance Model
Reference
Ground Truth Model
Training
P1
P2
P3
P4
P5
P6
Kajino et al.
2012
GD
x
/
X
x
x
X
Haykar et al.
2010
Logistic Regression Model
/
X
X
X
/
X
Rodrigues et al.
2013)
EM & CD
X
x
x
x
X
Yan et al. (0014)
1
X
Rodrigues al.
2017)
Topic Model
VI & CD
/
x
X
X
Rodrigues al. (2014)
EP
/
x
X
x
Long et all (2016)
Kernel-baser Model
EM EP & GD
x
x
x
8
X
x
Cil--onzaalez et al.
2021
GD
x
8
x
Albarqouni et al.
2016)
x
x
x
Yang ct al. (2018)
EM & GD
Khetan al. (2018)
/
8
Platanios et al. 2020
/
/
x
x
Rodrigues K Percira 2018
1
x
x
x
Guan et al. (2018)
1
x
Tanno et al. (2019)
Cao (2019)
(Deep) Neural Network
Zhang al. (2020)
1
1
x
Gil-Gonzéle et
2021
GD
8
/
x
Rüthling Cachay et al.
2021
/
x
Chu et al. (2021)
1
/
/
x
Li ct al. (2022)
/
Wei et (202
1
x
x
x
x
Gao et al.
2022
/
/
x
x
+
MaDL (2023)
/
/
/
/
/
6

---

Published in Transactions on Machine Learning Research (09/2023)
4 Multi-annotttor Deep Learning
In this section, we present our modular probabilistic MaDL framework. We start with description of its underlying probabilistic model. Subsequently, we introduce its GT and AP models' architectures. Finally, we explain our end-to-end training algorithm.
4.1 Probabilistic Model
The four nodes in Fig. depict the random variables of an instance GT label y, an annotator a, and an annotation 2. Thereby, arrows indicate probabilistic dependencies among each other. The random variable of an instance x and an annotator a have no incoming arrows and thus no cansal dependencies on other random variables. In contrast, the distribution of a latent GT label y depends on its associated instance x. For classification problems, the probability of obscrving y = c as GT label of an instance x can be modcled through categorical distribution:
II (x)) 5(kmm)
Pr(y c c|x)= = Catly = c p(x))
(x).
(6)
where p. 02x - Ape 10, E p 1} denotes the function outputting an instance's true class- membership probabilities. The outcome of an annotation process may depend the annotator's features, instance's fealures, and the latent GT label. A function P : Ωxx x Ωu + [0, 1jcxC' outputting a row-wise normalized confusion matrix per restance-annotate pair can capture these dependencies. The probability that an annotator a annotates an instance x of class y = e with the annotation = k can then be modeled through a categorical distribution:
C Pr(z = l c) = Cat (3 (== (x, =
(x
6((=h) = plaa)(x(a))
(7)
1=1
where the column vector ( (x, A corresponds the c-th row of the confusion matrix P(x, a).
Observed
Observed Annotator Features
Age:
Zookugiesl Interest: High
Mean Anpotation Time: St
Latent Ground Truth
Observed Anuotation Leopard
Jaguar
Ground Truth Model Parameters
W Annotator Performance Model Parameters
Figure 2: Probabilistic graphical model of MaDL.
4.2 Model Architectures
Now, we introduce how MaDL's GT and AP models are designed to approximate the functions of true clas-mmmbershhpp probabilities p and true confusion matrices P for the respective instances and annotators. Fig. 3 illustrates the architecture of the GT (purple) and AP (green) models within our MaDL. framework. Solid lines indicate mandatory components, while dashed lines express optional ones.
The GT model with parameters is a (D)NN (cf. 4 in Fig. 3) which takes an instance as input to approximate its true class-membership probabilities p(x) via (x)... We define its decision function in analogy to the Bayes optimal prediction in Eq. through
(x) == arg max (1(m) (x)
yefly

---

Published in Transactions on Machine Learning Research (09/2023)
Ground Truth Model
with Parameters 0
Px
Pe
203
Profucilitio
bost-meeeeellll bepplip
2000 Roatuoss
(Deep) Neural
Netwock
x007
Leurred Instance Repreentation
[C)
Pxa
PT
Probabiltiés
Ansosation
Pics
(Derp)
Anuotator Performance Model with Parameters
ORR
Nestral
P : P
Pica
Network
2043
(Deep) Neural
Netwock
(Deep)
Neural Network
(
RGRg
Figure 3: Architectures of MaDL's GT and AP models.
The architecture of the AP model with parameters w comprises mandatory and optional components. We start by describing its most general form, which con- sists of three (D)NNs and estimates annotator-, class-, and instance-dependent APs. Annotator features a are propagated through first (D)NN (cf. ! in Fig. 3 to learn an annotator embedding à E R², R E N21. During training, we will use such embeddings for quantifying correlations between annotators. Analogously, we propagate raw instance features x or a representation learned by the GT model's hidden layers through a second (D)NN (cf. ② in Fig. 3) for learning an instance embedding X E R Q e Nz1- Subsequently, instance and annotator embeddings X and ă are combined through third and final (D)NN (cf. 3 in Fig. 3) for approximating the true confusion matrix P(x, a) via (x a). Various architectures for combining embeddings have already been proposed in the literature (Fiedler) 2021). We adopt solution from recommender systems where often latent factors of users and items are com- bined Zhang et al.) 2019). Concretely, in DNN 3 we use an outer product- based layer outputting E R E N>1 to model the interactions between instance and annotator embeddings (Qu et al.) 2016). The concatenation of a, and Ō propagated through residual block (He et al. 2016), whose architecture is visualized in Fig. There, add only the annotator embedding à to the learned mapping h(ā, X, ö) E R", The motivation behind this modification is that the annotator embeddings, informing about an annotator's in- dividuality, are likely to be the most influential inputs for estimating confusion matrices as APs. Empirical investigations showed that R = Q = = 16 as the embedding size is a robust default. Finally, we define the AP model's decision function in analogy to the Bayes optimal prediction in Eq. through
Weight Layer
ReLU
Weight Layer
h@
h(a, x) + +
a Identity
ReI.U
Figure 4: MaDL's residual block combining annotator and instance embedding.
C
(x,, = Σ
(c) (x) p(a.e) (x,a)
< <.. (x,,) = á
< 0.5
(9)
prodicted correctness probability
An AP model estimating a confusion matrix per instance-annolator pair can be overly complex if there are only a few annotations per annotator or the number of classes is high (Rodrigues et al. 2013). In such settings, ignoring the instance features as input of the AP model may be beneficial. Alternatively, we can constrain a confusion matrix's degrees of freedom by reducing the number of output neurons of the AP model. For example, might estimate only the diagonal elements of the confusion matrix and assume
8

---

Published in Transactions on Machine Learning Research (09/2023)
that the remaining probability mass per row is uniformly distributed. Further, we either estimate each diagonal element individually (corresponding to C output neurons) or approximate them via a single scalar (corresponding to one output neuron). Appendix G illustrates such confusion matrices with varying degrees of freedom.
4.3 End-to-en Training
Given the probabilistic model and accompanying architectures of the GT and AP models, we propose an algorithm for jointly learning their parameters. A widespread method for training probabilistic models is to maximize the likelihood of the observed data with respect to the model parameters. Assuming that the joint distributions of annotations are conditionally independent for given instances X, we can specify the likelihood function follows:
N II
Pr A,
Pr(zn Xn, A,
(10)
We further expect that the distributions of annotations 2n for given instance x,, conditionally inde- pendent. Thus, simplify the likelihood function:
N II n=1 MEA.
Pr(Z x
II Pr(Enm Xn, am: 0 w).
(11)
Leveraging our probabilistic model in Fig. we can express the probability of obtaining certain annotation as an expectation with respect to an instance's (unknown) GT class label:
N
Pr(Z X,A;
II ==1 mEA.
II
Ex. [Pr(Enm Xn, am, Bn w)]
(12)
NN
II n=1 m.A. BB==1
II
Σ Pr(yn xni 0) Pr(Zum Xns ams Uni w)
(13)
N
II
II mEA.
IT (X,, am (x(()))
(x
(14)
Ramm
probabilities
where ea... denotes the one-hot encoded vector annotation 2nm Taking the logarithm of this likelihood function and converting the maximization into minimization problem, we get
(x,A,
Σ
Σ
In (e P am Do (xx)
(15)
mEAA.
as cross-entropp loss function for learning annotation probabilities by combining the outputs of the GT and
AP models (ef. blue components in Fig. 3 Yet, directly employing this loss function for learning may result in poor results for two reasons.
Initialization: Reason number one has been noted by Tanno et al. 2009 who showed that such a loss function cannot ensure the separation of the AP and GT label distributions. This is because infinite many combinations of membersibb probabilities and confusion matrices perfectly comply with the true an- notation probabilities, c.g., by swapping the rows of the confusion matrix as the following example shows:
T (xn, am )p(xn)
(
9 1)
:)
0
(
PT PT (Xn: am ) (xx).
(16)
probabilitios
predicted prohabilities
Possible approaches aim resolving this issue by favoring certain combinations, e.g., diagonally dominant confusion matrices. Typically, one can achieve this via regularization (Tanno et. al. 2019 Zhang et al. 2020:

---

Published in Transactions on Machine Learning Research (09/2023)
Li al 2022 and/or suitable initialization of the AP model's parameters (Rodrigues & Pereira, 2018
Wei et al. 2022). We rely the latter approach because it permits encoding prior knowledge about APs. Concretely, we approximate an initial confusion matrix for any pestace-aanntaate pair (Xn; am) through
coftmax(vv")) Am )m) + B)(..)
(1-n)
(w am) =
C-1 (1c Ic),
(17)
softmax (Xn, am) W + ()().))
where Ic E RRCx denotes an identity matrix, le E RCxC an all-one matrix, and E (0,1) the prior probability of obtaining a correct annotation. For example, in binary classification problem, the initial confusion matrix would approximately take the following values:
Pw (x am) R (1 11-1 " )
(18)
The outputs of the sof tmax functions represent the confusion matrix's rows. Provided that the initial AP modol's last layer's weights W E # H E Nso satisfy v ("(x, am ) = Oc E RCxC for the hidden
representation v(x,: am) E R of each instance-annown annotator pair, we approximate Eq. 17 by initializing the biases RCxC of our AP model's output layer
1-1 (2 Ic.
B B= In
(19)
By default, set 17 0.8 to assume trustworthy annotators priori. Accordingly, initial class- membership probability estimates are close to the annotation probability estimates.
7
Annotator Probability Densities:
Pr(a1 A) X 1
Pr(a2 A) R Pr(as A) R Pr(aq
A) R Pr(as
a1
a
a4
Pr(as A) Pr(ar
A) Pr(as
A) X
Annotator Weights:
w(a1) g
2g
w(a2) x was)) so w(as) so w(as) ~
T3
2,7
aG
(1) w(as) w(a7) = w(as)
Figure 5: Visualization of annotator embeddings (left) accompanied by an exemplary calculation of annotator probability densities and annotator weights (right).
Annotator weights: Reason number two has been noted by Cao al. (2019), who proved that maximum- likelihood solutions fail when there are strong annotator correlations, i.e., annotators with significant statis- tical correlations in their annotation patterns. To address this issue, we explore the annotator correlations in the latent space of the learned annotator embeddings. For this purpose, assumee that annotators with similar embeddings share correlated annotation patterns. Recalling our example in Fig. 1 this assumption implies that annotators of the same latent group are located near each other. The left plot of Fig. 5 visualizes this assumption for a two-dimensional embedding space, where the eight annotators are arranged into three clusters as proxies of the three latent annotator groups. We aim to extend our loss function so that its evaluation independent of the annotator groups' cardinalities. For our example, we view the three anno- tator groups three independent annotators of equal importance. To this purpose, extend the original
10

---

Published in Transactions on Machine Learning Research (09/2023)
likelihood function in Eq. 11 by annotator weights, such that obtain the weighted likelihood function:
N
Pr(Z A,
II II T= MEAN
Pr(2nm Nn. am 0, w) w((m))
(20)
where W
RM denotes vector of non-negative annotator weights. From a probabilistie perspective, we interpret such weight w(am) the effective number of observations (or copics) per annotation of annotator am- Interpreting the annotators A as samples from continuous latent space, we define an annolator weight w(am) to be inversely proportional to an annolator's am. probability density:
w(am) Pr(am |A) Z = M
Σ Pr(am A)-1
M
provided that Pr(a1
(A) 0.
Z
(21)
The normalization term Z 230 ensures that the number of effective annotations remains equal to the number of annotators, i.e. Em w(am) M. On the right side of our example in Fig. 5 expect that annotator's probability density is approximately proportional to the cardinality of the group to which the annotator belongs. As result, we assign high (low) weights to annotators belonging to small (large) groups. Inspecting the exemplary annotator weights and adding the weights per annotator group, we observe that each group provides the same number of effective annotations, i.e., 8/s. More generally, we support our definition of the annotator weights by the following theorem, whose proof given Appendix A
Theorem 1. Let there be G E (1.....,M) M) non-emphy, disjoint annotator groups, which we denote sets of indices such that A U = f1...., M}. Further assume, the annotators within each group E {1,...,G) share idenlical annolation pollerns for the observed instances, i.e.,
Vn (L...., Vm, E A 2nm Zn2 A Pr(Znm Nn. am) Pr(znd Xx, at),
(f)
and the annotators' probability densitics are proportional to their respective groups' cardinalities, i.e.,
Vm E (1....,,M M} Pr(am A) x Σ 70m
5m E (
Then, the true weighted log-likell likelihood function for all M annotators reduces to the log- bogllikeldhood for G onnotators:
N Σ
ΣD Σ w(am) In (Pr(Znm Xn, am))
In (Pr(Znm,
Xns am,))
where my E A(s) represents the indez of an arbitrary annotator of the g-th annotator group.
Intuitively, Theorem I confirms that each group A(9), independent of its cardinality (A()) equally contributes to the weighted log-likelihood function. This way, we avoid any bias toward a large group of highly correlated annotators during learning. Typically, the assumptions ( and of Theorem 1 do not hold in practice because there are no annotator groups with identical annotation patterns. Therefore, we estimate degrees of correlations between annotators by computing similarities between their embeddings A = (āj,..., aav)" as the basis for a nonparametric annotator probability density estimation:
Pr (amam A) Pr @mm A,
ky (no ho_grad (ā:) mo_grad (ā..)),
(22)
where by R Rx + R>0 denotes kernel function and ? Rz0 its kernel scale. The expression (o_grad(ām.) R indicates that gradient regarding the learned annotator embedding am is computed,
11

---

Published in Transactions on Machine Learning Research (09/2023)
which is necessary to decouple the learning of embeddings from computing annotator weights. Otherwise, we would learn annotator embeddings, which optimize the annotator weights instead of reflecting the anno- tation patterns. Although many kernel (or similarity) functions are conceivable, we will focus on the popular Gaussian kernel:
(no _rrad (am) lo_noogrrraa grad (āi)) & exp (-> ||o__ (ām) no no_grad (ā:) ll))
(23)
with 1/122 as Euclidean distance. Typically, the kernel scale 7 needs to fit the observed data, iee, annotator embeddings in our case. Therefore, its definition a priori is challenging, such that we define ry as a learnable parameter subject to a prior distribution. Concretely, we employ the gamma distribution for this purpose:
go
(r a, 3) = Gam (y a, 3) = I(a) o0 r(a)
-- exp (-87),
(24)
where I' is the gamma function and E Ry1, 3 E R>o are hyperparameters. Based on experiments, we set a = 1.25, B = 0.25 such that the mode is (a-1)/8 = 1 (desining the initial value of before optimization) and the variance with a/5² = 20 high in favor of flexible learning. As a weighted loss function, we finally get
Lx..A.a,,,,....
1
Σ
Px (0 In (e.... PT (xn, am )o((x)) In (Gam (7 aa, ()) (25)
Z
mEAn
N
Z =
8Zmm E [2y),
(26)
1=
where w. (am) denotes that the annotator weights wa(m)) are estimated by learning the kernel scale 7. The number of annotations [Z] is normalization factor, which accounts for potentially unevenly distributed annotations across mini-batches when using stochastic GD.
Given the loss function in Eq. 25 we present the complete end-to-end training algorithm of MaDL in Algorithm and an example in Appendix B During each training step, we recompute the annotator weights and use them as the basis for the weighted loss function to optimize the AP and GT models' paramoters. After training, the optimized model parameters (0,w) can be used to make probabilistic predictions, e.g., mlasemhemerrhhp probabilities (x(x) (cf. Fig. 3) and annotator confusion matrix Pw a) (cf. Fig. 3), or to decide on distinct labols, c.g., class label (x) (cf. Eq. gh and annotation crror go (x, a) (cf. Eq. 9
Algorithm 1: End-to-end training algorithm of MaDL.
input: instances X, annotators A, annotations Z, number of training epochs F,, mini-batch size B, initial model parameters (O,w), prior annotation accuracy 7, gamma distribution parameters (a, B); start:
initialize biases B of the AP model's output layer using 7 (ef. Eq. 19
initialize kernel scale y == (a-1)/8;
for epoch E f1,..., E} do
for sampled mini-batch X = (x....... Xip T (zi..... 2T with fir....,is! iB} c (1....,N N} do for b E (i,,...,4n) do
compute class- maase-mmmberhhp probabilities (x,) (cf. Fig. 3 M) do compute confusion matrix Pu (Xb, am) (cf. Fig. 3) end
for E (1,...,,, end
for (m, E {1,..., 1.4....MY 2 do compute similarity ke (no_ grad (mm).am)))) no grad a )) (cf. Eq.
23
end end
for m E (1,..., M) do compute annotator weight w(am) x W. (am) (cf. Eq. 21
and Eq
optimize parameters 0, w, with reference to L.. (0, w,w,7) (cf. Eq. 250
end end
output: optimized model parameters (0, w)
12

---

Published in Transactions on Machine Learning Research (09/2023)
5 Experimental Evaluation
This section investigates three RQs regarding the properties P1-P6 (cf. Section 8 of multi-annotator supervised learning. We divide the analysis of each RQ into four parts, which are (1) a takeaway summarizing the key insights, (2) a setup describing the experiments, (3) a qualitative study, and (4) a quantitative study, The qualitative studies intuitively explain our design choices about MaDL, while the quantitative studies compare MaDL's performance to related techniques. Note that we analyze cach RQ in the context of concrete evaluation scenario. Accordingly, the results provide polential indicalions for extension to related scenarios. As this section's starting point, we overview the general experimental setup, whose code base is publicly available at htttps://wwt http:/ww..ttttt.. www.gh...o/eeeeeeeeeeeeee rew.leerrreeracummuleeul
deeotator-deep-eess
5.1 Experimental Setup
We base our experimental sctup on the problem setting in Section
Accordingly, the goal is to evaluate the predictions of GT and AP models trained via mulli-annotator supervised learning techniques. For this purpose, we perform experiments on several datasets with class labels provided by error-pront annotators, with models of varying hyperparamcters, and in combination with collection of different evaluation scores.
2
Datasets: We conduct experiments for the tabular and image datasets listed by Table 2 LABELME and MUSIC are actual crowdsourcing datasets, while we simulate annotators for the other five datasets. For the LABELME dataset, RRoriiues & Pereira (2018) performed crowdsourcing study to annotate a subset of 1000 out of 2688 instances of eight different classes as training data. This dataset consists of images, but due to its small training set size, we follow the idea of Rodrigues & Pereira and transform into a tabular dataset by utilizing the features of a pretrained VGG-16 (Simonyan & Zisserman (015)) inputs. There are class labels obtained from 59 different annotators, and on average, about 2.5 class labels are assigned to instance. MUSIC is another crowdsourcing dataset, where 700 of 1000 audio files are classified into ten music genres by 44 annotators, and on average, about 2.9 class labels are assigned to file. We use the features extracted by Rodrigues et al. (2033) from the audio files for training and inference. The artificial TOY dataset with two classes and features serves to visualize our design choices about MaDL. We generate this dataset via a Gaussian mixture model. Frey & Slate (1991) published the LETTER dataset to recognize a pixel display, represented through statistical moments and edge counts, as one of the 26 capital letters in the alphabet. for Modern English. The datasets FMNIST, CIFARIO, and SVHN represent typical image benchmark classification tasks, each with ten classes but different object types to recognize. Appendix presents separate case study on CIFAR100 to investigate the outcomes on datasets with more classes.
Table 2: Overview of datasets and associated base network architectures.
Dataset
Annotators
Instances
Classes
Features
Base Network Architecture
Tabular Datasets
TOY
simulated
500
MLP
Rodrigues & Percira
2018
LETTER
Frey Slate
1991
simulated
20000
MLP
Rodrigues & Percira
2018
LABELME
Rodrigues
Percira
2018
real-world
2688
8192
MLP
Rodrigues & Percira
2018
MUSIC
Rodrigues el al.
2013)
real-world
1000
10
124
MLP
Rodrigues & Pereira
2018
Image Datasets
FMNIST
X lao et. al.
2017
simulated
70000
10
1
x 28 x 28
LeNet-5 (TeCun & Cortes
1998
Krizhevsky
2009
simulated
60000
10
3
32
x 32
ResNet- 18
He al. 2016
SVHN
Netzer et al. 2011
simulated
99289
10
3
32
x 32
ResNet- 18
He al.
2016
Network Architectures: Table 2 lists the base network architectures selected to meet the datasets' re- quirements. These architectures starting points for designing the GT and AP models, which adjust according to the respective multi-annotator supervised learning technique. For the tabular datascts, we follow Rodrigues & Pereira (2018) and train a mulltlayer perceptron (MLP) with a single fully connected layer of 128 neurons as a hidden layer. A modified LeNet-5 architecture {LeCun & Cortes) 1998), a simple convolutional neural network, serves as the basis for FMNIST as a gray-scale image datasct, while we employ a ResNet-18 (He et al.) (0166) for CIFAR10 and SVHN as RGB image datasets. We refer to our code base for Glooot remaining details, e.g., on the use of rectified linear wnils (ReI.U, Glorot et al. 2011] as activation functions.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Annotator simulation: For the datasets without real-world annotators, adopt simulation strategies
from related work (Yan et al., 2014 Cao et al. 2019; Rühling Cachay et al. 2021 Wei et al. 2022 and
simulate annotators according to the following five types:
Adversarial annotators provide false class labels on purpose. In our case, such an annotator provides a correct class label with a probability of 0.05.
Rondomly guessing annotators provide class labels drawn from a uniform categorical distribution. As a result, such an annotator provides correct class label with probability of I/c.
Cluster Specier-peciallzed annotators' performances considerably vary across the clusters found by the k-means clustering algorithm. For images, we cluster the latent representations of the ResNet-18 pretrained on ImageNet (Russakovsky et al., 2015). In total, there are 10 clusters. For each annotator, we randomly define five weak and five expert clusters. An annotator provides a correct class label with a probability of 0.95 for an expert cluster and with a probability of 0.05 for weak cluster.
Common annotators are simulated based the identical clustering employed for the cluster-specialized annotators. However, their APs vary less between the clusters. Concretely, we randomly draw a correctness probability value in the range [//,, 1] for each cluster-annotator pair.
Class-specialized annotators' performances considerably vary across classes to which instances can belong, For each amnotator, we randomly define [c/2] weak and [c/2] expert classes. An annotator provides a correct class label with a probability of 0.95 for an expert class and with a probability of 0.05 for a weak class.
We simulate annotation mistakes by randomly selecting false class labels. Table 3 lists four annotator sets (blueish rows) with varying numbers of annotators per annotator type (first five columns) and annotation ratios (last column). Each annotator set is associated with concrete RQ. A copy flag indicates that the annotators in the respective types provide identical annotations. This way, we follow Wei et al. 20222 Cao
et al. 2019), and Rühhing Cachay el al. (2021) to simulate strong correlations between annotators. For example, the entry "I + 11 copies" of the annotator set CORRELATED indicates twelve cluster-specialee annotators, of which one annotator is independent, while the remaining eleven annotators share identical annotation patterns, i.e., they copies of each other. The simulated annotator correlations are not directly observable because the copied annotators likely annotate different instances. This is because of the annotation ratios, e.g., ratio of 0.2 indicates that each annotator provides annotations for only 20 % of randomly chosen instances. The annotation ratios are well below 1.0 because, in practice (especially in crowdsourcing applications). is unrealistic for every annotator to annotate every instance. We refer to Appendix presenting the results case study with higher annotation ratios for CIFAR10.
Table 3: Simulated annotator sets for each RQ.
Adversarial
Common
Cluster-specialized
Class-specialieed
Random
Annotation Ratio
INDEPENDENT (RQI)
0.2
CORRELATED (RQ2)
11 copies
+ 11 copies
11 cupies
0.2
(RQ2)
90 copies
0.2
INDUCTIVE (RQ3)
0.02
Evaluation scores: Since we are interested in quantitatively assossing GT and AP predictions, we need corresponding evaluation scores. In this context, we interpret the prediction of APs as a binary classification problem with the AP model predicting whether an annotator provides the correct or a false class label for an instance. Next to categorical predictions, the GT and AP models typically provide probabilistic outputs, which we examine regarding their quality (Huseljic et al. 2021). We list our evaluation scores in the following, where arrows indicate which scores need to be maximized (f) minimized (4):
14

---

Published in Transactions on Machine Learning Research (09/2023)
Accuracy (ACC, 1) is probably the most popular score for assessing classification performances. For the GT estimates, it describes the fraction of correctly classified instances, whereas it is the fraction of (potential) annotations correctly identified as false or correct for the AP estimates:
N 8 (Un = (xn))
YI )
=
(27)
N
n=1
AP-ACC(X, y, y,
1
Σ
8 (6 (yh 2nm) Voo (Xns am))
(28)
Z
n= m.A.
Maximizing both scores corresponds to the Bayes optimal predictions in Eq. and Eq.
Balanced accuracy (BAL-ACC, t) variant of ACC designed for imbalanced classification prob- lems (Brodersen al. 2010). For the GT estimation, the idea is compute the ACC score for each class of instances separately and then average them. Since datasets fairly balanced in their distributtion of class labels, use this evaluation score only for assessing AP estimates. We may encounter highly imbalanced binary classification problems per annotator, where class represents either false or correct annotation. For example, an adversarial annotator provides majorly false annotations. Therefore, we extend the definition of BAL-ACC by computing the ACC scores for each annotator-class pair separately to average them.
Negative log-likelihood (NLL, 4) is not only used as a typical loss function for training (D)NNs but can also be used to assess the quality of probabilistic estimates:
GT-NLL(X, y, Pa)
Σ
P
(xx))
(29)
N
n=1
AP-NLL(X,y. Z. Do =
N
(6(m) Zmmm In (fo.w (Xx, am)) + 8 (yn + zmm) In (1 a (Xa, am))
(30)
Z
n. mEAn.
Moreover, NLL is proper scoring rule Ovadia et al. 2019 such that the best score corresponds to perfect prediction.
Brier score (BS, 4), proposed by Brier 1950), is another proper scoring rule, which measures the squared error between predicted probability vectors and one-hot encoded target vectors:
1 Σ lley. (x,)))
GT-BS(X, y, PO) =
(31)
N
1
AP. D.
(ō (yn 2nm) po. (Xn, ama))²
(32)
Z
m....
In the literature, there exist many further evaluation scores, particularly for assessing probability calibra- tion OOadiaa et al. 2019 As a comprehensive evaluation of probabilities is beyond this article's 's scope, we focus on the aforementioned proper scoring rules. Accordingly, we have omitted other evaluation scores, such the expected calibration error (Naeini al., 2015) being non-proper scoring rule.
Multi-annotator supervised learning techniques: By default, train MaDL via the weighted loss function in Eq. 25 using the hyperparameter values from Section g and the most general architecture de- picted by Fig. 3 Next to the ablations as part of analyzing the three RQs, we present an ablation study on the hyperparameters of MaDL in Appendix [ and a practitioner's guide with concrete recommendations in Appendix G We evaluate MaDL compared
subset of the related techniques presented Scction 3 This subset consists of techniques that (1) provide probabilistic GT estimates for each instance, (2) provide prob- abilistic AP estimates for each instance-ammolato pair, and (3) train (D)NN the GT model. Moreover,
15

---

Published in Transactions on Machine Learning Research (09/2023)
we focus on recent techniques with varying training algorithms and properties P1-P6 (cf. Section 3 As result, we select crowd layer (CI., Rodrigues & Pereira, 2018), regularized eslimation of annotalor confu- sion (REAC, Tanno et al. 2019), learning from imperfect annotators (LIA, Platanios et al. 2020), common noise adaption layers (CoNAL, Chu et al. 2021), J, and union net (UNION, Wei et al.) 2022 Further, we aggregate annotations through the majority rule as a lower baseline (LB) and use the GT class labels as an upper baseline (UB). We adopt the architectures of MaDL's GT and AP models for both baselines. The GT model then trains via the aggregated annotation (LB) or the GT class labels (UB). The AP model trains using the aggregated annotations (LB) the GT class labels (UB) to optimize the annotator confusion matrices. Unless explicitly stated, no multi-annotator supervised learning technique can access annotator features containing prior knowledge about the annotators.
Experiment: An experiment's run starts by splitting a dataset into train, validation, and test sets. For MUSIC and LABELME, these splits are predefined, while for the other datasets, we randomly select 75 % of the samples for training, 5 % for validation, and 20 % for testing. Following Rühling Cachay et al. (2021), small validation set with GT class labels allows fair comparison by finding suitable hyperparameter values for the optimizer of the respective multi-annotatorr supervised learning technique. We employ the AdamW (Loshchilov & Hutter 2019) optimizer, where the learning rates {0.01, 0.005, 0.001} and weight decays {0.0, 0.001, 0.0001} are tested. We decay learning rates via a cosine annealing schedule (Loshchilov & Hutter 2017) and set the optimizer's mini-batch size to 64. For the datasets MUSIC and LABELME, we additionally perform experiments with and 16 as bini-batch sizes due to their smaller number of instances and, thus, higher sensitivity to the mini-batch size. The number of training epochs is set to 100 for all techniques except for LIA, which we train for 200 epochs due to its FM algorithm. After training, we select the models with the best validation GT-ACC across the epochs. Each experiment is run five times with different parameter initializations and data splits (except for LABELME and MUSIC). We report quantitative results as means and standard deviations over the best five runs determined via the validation GT-ACC.
5.2
RQ1: Do class- and instance-dependent modeled APs improve learning? (Properties P1, P2)
Estimating class- (property P1) and instance-dependent (property P2) APs leads to superior performances of the GT and AP models. This observation is especially true for GT models trained on datasets with real-world annotators whose annotation patterns are unknown.
Takeaway:
We address RQ1 by evaluating multi-annotator supervised learning techniques with varying AP assumptions. We simulate ten annotators for the datasets without real-world annotators according to the annotator set INDEPENDENT in Table 3 Each simulated annotator provides class labels for 20 % of randomly selected training instances. Next to the related multi-annotator supervised learning techniques and the two baselines, we evaluate six variants of MaDL denoted via the scheme MaDI,(P1, P2). Property P1 refers to the estimation of potential class-dependent APs. There, we differentiate between the options class- dlass-ndeepndeen (I), partially (P) desendeep,,, and fully (F) class-dependent APs. We implement them by constraining the annotator confusion matrices' degrees of freedom. Concretely, class-independen. refers to confusion matrix approximated by estimating a single scalar, partially class-dependent refers to a confusion matrix approximated by estimating its diagonal elements, and fully class-dependent refers to estimating each matrix element individually (cf. Appendix Gb. Property P2 indicates whether the APs are estimated as a function of instances (X) or not (X). Combining the two options of the properties P1 and P2 represents one variant. For example, MaDL(X, F) is the default MaDL variant estimating instance- and fully class-dependent APs.
Setup:
Qualitative study: Fig. 8 visualizes MaDL's predictive behavior for the artificial dataset TOY. Thereby, each row represents the predictions of a different MaDL variant. Since this is binary classification problem, the variant MaDL(X, P) is identical to MaDL(X, F), and MaDL(X, P) is identical to MaDL(X,, F). The first column visualizes instances as circles colored according to their GT labels, plots the class-membershipp probabilities predicted by the respective GT model as contours across the feature space, and depicts the decision boundary for classification as a black line. The last four columns show the class labels provided by four of the ten simulated annotators. The instances' colors indicate the class labels provided by an annotator, their forms mark whether the class labels are correct (circle) or false (cross) annotations, and the contours
16

---

Published in Transactions on Machine Learning Research (09/2023)
across the feature space visualize the AP model's predicted annotation correctness probabilities. The GT models of the variants MaDL((, F), MaDI.(X, 1), and MaDI((, F) successfully separate the instances of both classes, whereas the GT model of MaDL(X, 1) fails in this task. Likely, the missing consideration of instance- and class-dependent APs explains this observation. Further, the class-membership probabilities of the successful MaDL variants reflect instances' actual class labels but exhibit the overconfident behavior typical of deterministic (D)NNs, particularly for feature space regions without observed instances (Huseljic jet al., 2021p. Investigating the estimated APs for the adversarial annotator (second columm), we see that each MaDL variant correctly predicts low APs (indicated by the white-colored contours) across the feature space. When comparing the AP estimates for the class-specialized annotator (fifth column), clear differences between MaDL(X, I) and the other three variants of MaDL are visible. Since MaDL(X, 1) ignores any class dependency regarding APs, it cannot differentiate bet ween classes of high and low APs. In contrast, the AP predictions of the other three variants reflect the class structure learned by the respective GT model and thus can separate between weak and expert classes. The performances of the cluster-spccialized and common annotator depend on the regions in the feature space. Therefore, only the variants MXDL(X, and MXDL(X, F) can separate clusters of low and high APs. For example, both variants successfully identify the two weak clusters of the cluster-specialized annotator. Analogous to the flass-membership probabilities, the AP estimates are overconfident for feature space regions without observed instances.
cless Bise
cluss
low
high
wrongly cluss bine instance wrongly assignod class
instance of class
Clase-mmembeshhip Probubility
Annotatar Performance
Data
Decision Rounday
MaDL(X, D
MaDL(X, F)
MaDL(A D)
MADDLCK 1
Adversarial Annotator
Cluster-specialized Annotator
Common Annotator
Clase-specialized Annotator
Classification
Figure 6: Visualization of MaDL's predictive behavior for the two-dimensional dataset TOY.
17

---

Published in Transactions on Machine Learning Research (09/2023)
Quantitative study: Table presents the numerical evaluation results for the two datasets with real-world annotators. There, we only report the GT models' test results since no annotations for the test instances are available to assess the AP models' test performances. Table 5 presents the GT and AP models' test results for the four datasets with simulated annotators. Both tables indicate whether a technique models class- dependent (property P1) and/or instance-dependent (property P2) APs. Generally, training with GT labels as UB achieves the best performances, while the LB with annotations aggregated according to the majority rule leads to the worst ones. The latter observation confirms that leveraging AP estimates during training is beneficial. Moreover, these AP estimates are typically meaningful, corresponding BAL-ACC values above 0.5. An exception is MaDI.(X, T) because this variant only estimates by design a. constant performance per annotator across the feature space. Comparing MaDL(X, F) as the most general variant to related techniques, we observe that it achieves competitive or superior results for all datasets and evaluation scores. Next to MX, F), CoNAL often delivers better results than the competitors. When we investigate the performances of the MaDL variants with instance-independenn APs, we find that MaDL(X, F) achieves the most robust performances across all datasets. In particular, for the datasets with real-world annotators, the ACC of the respective GT model is superior. This observation suggests that modeling deeendet APs (property P1) is beneficial. We recognize a similar trend for the MaDL variants with instance-dependent APs (property P2). Comparing each pair of MaDL variants with X and X, we observe that mustance-dependee APs ofien improve GT and, in particular, AP estimates. The advantage of class- and instance-dependenn APs is confirmed by CoNAL as a strong competitor of MaDL(X, F). LIA's inferior performance contrasts this, although LIA estimates class- and instance-dependent APs. The difference training algorithms can likely explain this observation. While MaDL(X, F) and CoNAL train via an end-to-end algorithm, LIA trains via the EM algorithm, leading to higher runtimes and introducing additional sensitive hyperparameters, e.g.: the number of EM iterations and training epochs per M step.
Table 4: Results regarding RQ1 for datasets with real-world annotators: Best and second best performances are highlighted per dataset and evaluation score while excluding the performances of the UB.
Ground Truth Model NLL $
Ground Truth Model NLL 4
Technique
P1
P2
ACC f
BS 4
ACC f
BS
MUSIC
LABELME
UB
0.785=0.020
0.710:0.037
0.314 02T
0.580:00 112
0.150=20.00 003
LB
0.646:10.045
1.09610.103
0.492:1.005
0.810:10.015
0.72411. 155
0.29411.022
CL
/
0.605=0.015
1.62+004000
0.521 021
0.001
1,77141 155
0.250= 011
REAC
V
0.20331..023
0.89322.061
0.410:10.033
0.843:10.006
0.833:10 088
0.254.:0.00 006
UNION
V
0..03
1.39610.144
0.50110.007
0.0004
1.07410.340
0.248110.011
LIA
0.658-00.03
1.158
0.408:10.020
0.813+00.010
0.976+40.234
0.295 0.00
CoNAL
V
0.0031
0.964110.881
0.423.10.035
0.806:00.004
2.740
0.247 0.003
MaDL(X, 1)
X
X
0.718
0..0110.027
0.394:00.009
0.815.10.009
0.61610.125
0.276.10.012
Ma P)
+
X
0.720:10.018
0.871 10.000
0.396:100009
0.8111:0.012
0.630 128
0.281 +00002
MXDL(X, P)
V
X
0.725:10.015
0.977:0.064
0.403=20.013
0.859-:0.007
1.008:10.275
0.240:10 014
MXDL(X.
X
0.713
0.876.10.041
0.402
0.816.10.008
0.55910.022
0.276.10.01
MX, P)
0.714=:0.014
0.909:20.036
0.398:1:0.00 .113
0.811:20.009
0.771::0 160
0.289: 016
MX,L(X, F)
0.743.10.018
0.87710.030
0.38110.012
0.0044
0.623
0.214
5.3 RQ2: Does modeling correlations between (potentially spamming) annotators improve learning? (Properties P3, P4)
Takeaway: Modeling correlations between annotators leads to better results in scenarios with many correlated spamming annotators (property P4). Capturing the corrclations of beneficial annotators does not lead lo consistently better results (property P3). However, estimating and leveraging APs during training becomes more critical in scenarios with correlated annotators.
Setup: We address RQ2 by evaluating multi-annotalor supervised learning techniques with and without modeling annotator correlations. We simulate two annotator sets for each dataset without real-world an- notators according to Table 3 The first annotator set CORRELATED consists of the same ten annotators as in RQ1. However, we extend this set by ten additional copies of the adversarial, the class-specialized, and one of the two chuster-specialized annotators, there are 40 annotators. The second annotator set.
18

---

Published in Transactions on Machine Learning Research (09/2023)
RANDOM-CORRELATEE also consists of the same ten annotators as in RQ1 but is extended by 90 identical randomly guessing annotators. Each simulated annotator provides class labels for 20 % of randomly selected training instances. Next to the related multi-annotator supervised learning techniques and the two baselines, we evaluate two variants of MaDL denoted via the schome MPDL Property P3 refers to the modcling of potential annotator correlations. There, we differentiate between the variant MaDL(W) using annotator weights via the weighted loss function (cf. Eq. (5) and the variant MaDL(WW training via the loss function without any weights (cf. Eq. 15 MWD corresponds to MaDL's default variant in this setup.
Table 5: Results regarding RQ1 for datasets with simulated annotators: Best and second best performances are highlighted per dalaset and evaluation score while excluding the performances of the UB.
Ground Truth Model
Annotator Performance Model
Technique
P1
P2
ACC1 1 NLL +
BS
ACC1
NLL
BS BAL-ACC
LETTRR (INDEPENDENT)
CB
06000
0.130:10.00
0.059+40.004
0.770=0:0001
0.488:50.003
0.316.40.000
0.709+00.00
LB
/
0.878=0.001
0.980+#0002 021
0.385=00.00 008
0.661=0.004
0.624 003
0.133+0.000 003
0.666+0.000 000
CL
/
X
0.886=0.013
1.06240.14 145
0.081
0.663-0.006
0.6254:00013
0.430
0.601
REAC
0.936=0.005
0.23820.018
0.097
0.685=0.002
0.560+00001
0.385++0.000
0.604 002
UNION
V
x
0.905=0.016
0.906+0.433
0.151 030
0.670=0.004
0.589+0.008
0.108
0.605 002
LIA
0.897=0.005
0.77810.052
0.305-10,021
0.669:0.0044
0.65410.010
0..47710.000
0.616:10.000 003
CoNAL
V
0.907=0.016
0.813+0.351
0.143++0.02 027
0. 0.723=0.018
0.565+0.021
0.372
0.663+0.00
MaDL(X. 1)
X
0.269 0.93/1=0.003
0.100= 004
0. 0.607=0.001
0.627=0.000
0.411 0000
0.500 000
MaDL(X. P)
0.235 0.035=0.0055
0.099 006
0.556:10.001 0.692:000001
0.981
0.606:00.00 003
MaDL(X. F)
V
0.933=00005
0. 2665 025
0.100 005
0.691=0.002
0. 0.556+0.000 001
0.381 001
0.606 002
(X. 1)
X
0.936=0.006
0..03
0.092:00008
0.770=0.004
0.492:00016
0.316++0.007
0.708:00004
MaDL(X, P)
0.940
0.24210.045
0.090110.004
0.770
0.496 49.91.0020
0.31610.0009
0.708 005
(X, F)
0.935=0.006
0.092
0.098 009
0.766=0.001
0.491=0.006
0.3174:0.00
0.70240.000 005
(INDEPEND
UB
/
0.909-0.007 0.002
0.246.10.068
0.131 00.003
0.756
10.001
0.321 10.00
0.704 0.001
0.883=0.001
0.90340..03
0.385540.001
0.644=0.007
0.645+00000
0.453:40.004
0.585 007
CL
0.892
0.31210.008
0.15810,004
0.674:0.002
0.58010.001
0.402:10.000
0.623:10 001
REAC
0.89:1=0.003
0.30990.001
0.155 001
0.703=0.001
0.535 0.001
0.361 0.00
0.6:11:0 001
UNION
0.893=0.002
0.305-10.006
0.155
0.67420.002
0.570:10.002
0.395
0.622:10.00
LIA
0.002
1.01710.016
0.442.10.008
0.665-10.024
0.628 1.628.10001
0.4337110.00
0.613.10.027
CoNAL
0.894=0.004
0.30440.009
0.155:40.004
0.725=0.016
0.521
0.35140.0016
0.679++0.018
MXDL(X. 1)
X
X
0.896=0.003
0.34010.006
0.161110.00
0.390:10.000
0.6388110 0..30000000
0.453:10.000
0.500 000
MXDL(X. P)
X
0.891=0.001
0.307 0000
0.155+0.000
0.705=0.001
0.53440.00
0.00
0.36340.000
0,6410+0.000 001
(X.L F)
V
X
0. 0.89120.002
0.307+0.008 0.06
0.155#00.00 003
0.705=00000
0.534+0..00
0.363440.000
0,6400 000
MX.L(X. 1)
0.895=0.003
0.20110.000
0,150::0.000
0.054
0.490:10.004
0.325:10.003
0.699:1.004
MX.L P)
0.89910.003
0.28610.006
0.147:10.003
0.751:10.003
0.489:10.00
0.32410.003
0.698:10.00
(X.L
0.896=0.002
0.288-10.006
0.148:10.008
0.750=0.005
0.4911:0.005
0.326:+0.005
0.697
CIRAR10
(INDEPENDENT))
UB
/
0.933=0.002
0.51910.026
0.108:10.004
0.710=100001
0.547
0.389-1..000
0.658:10.00
LB
/
0.789=0.004
1.08140.031
0.460::0.015
0.575=0.021
0.673
0.4811+0.000
0.547++00011
CL
/
0.803=0.008
0.536110.012
0.242:10004
0.664:10.001
0.604410.002
0.420.1:0.00
0.613:10.000
REAC
/
X
0.839=0.008
0.50140.010
0.245
0.676=0.003
0.580 000
0.397 0.00
0.625 002
UNION
0.83/100.003
0.595=0.022
0,249++0.00 005
0. 0.668=0.001
0.592 001
0.110-+ 001
0.617+00 002
LIA
/
0.803=0.003
1.102:10.035
0.469:10.016
0.622:10.024
0. 0.645:10.014
0.453:40.011
0.579:10.012
CoNAL
/
0.838=0.005
0.630=0.021
0.236+0.000 008
0.668=0.001
0.600=000001
0.4166
0.616 001
MaDL(X. 1)
X
0. 0.832=0.006
0.58310.021
0.256#0.000 000
0.376=0.010
0.616::0002
0.4611:0.000 002
0.500+00.00 000
MXDL(X.
0.804=0.001
0.529+0.014
0.23110.004
0.682=0.001
0.568120.001
0.390+:0.000
0.630+0..00
MaDL
0.840=00005
0.545+0.019
0.237 006
0.681=0.00
0.509++0.002
0.390++0.001
0.630++0.001
MaDL(X. 1)
X
0.8.3=0.0005
0.555 021
0.236: 008
0.697=0.002
0.559 005
0.380+ 003
0.6166+0.000 002
(X.
0.845=0.002
0.546+0.027
0.232440.000 000
0.697=0.001
0.68720.002
0.380-10.001
0.646:0.0m
MXDL(X. F)
0.846=0.0003
0.521#0.014
0.229=00005
0.697=0.002
0.557=0.004
0.379+0.002
0.616+0.003
SVHN (INDEPENDENT)
/
0.965 0.000
0.40310.024
0.064110.01
0.675:10.002
0.567
10.001
0.590
LB
/
/
0.900=0.002
0.0000
0.332410.015
0.581
0.680-10.008
0.487
0.540-1-0.000
CL
/
0.944=0.0011
0.2377100008
0.086:200007
002
0.646=0.001
0.69840.001
0.419
0.546 001
REAC
/
0.9-3=00.011
0.278+0.00 0418
0.096 0.020
0.6-8==00006
0.593+0.015
0.4144+0.0110
0.5433+0. 0.00
UNION
0.902=0.002
0.250:30.005
0.087:10.007
0.6-66=00001
0.594+10.00
0.416::0.00
0.5444:0.000 DU1
LIA
<
0.002
0.809110.162
0.333110.081
0.585.10.016
1..073
0.476.10.021
0.53610.00
CoNAL
V
0.944=0.002
0.246-10.012
0.056-40.009 002
0.688=0.036
0.56
1..844-0.026
0.602-10.050
(X, 1)
X
0.942=0.003
0. 0.23
0.093320.008
0.613=20.003
D. 0.63001:0.000
0.446::0.003
0.500-10.000 0000
(x. P)
+
0.9-0=00.002
0.262+00001
0.091+0.00 003
0.652=0.000
0. 0.586+0.000
0.408-+00000
0.5-44+0.000
M&DL (X. F)
/
0.9:0=00.002
0. 00007
0.092++0000 002
0.652=0.001
0.585 0.000
0. 0.-088+0.000 000
0.5-13+00.00 001
MX. 1)
X
0.941=0.003
0.24010.007
0. 0.085:10.003
0.6661
0.575 0000
0.399::0.000
0.565+t0.00
MaDL(X.
0.945_0.000
0.0841100004 10.001
0.669 0.002
0.572 0.002 0.002
0.39610.002
0.573
(X, F)
/
0.943=0.001
0.25410.003
0.087:10.002
0.008
0.572+100000
0.396-L0.003
0.570-10.006
19

---

Published in Transactions on Machine Learning Research (09/2023)
Qualitative study: Fig. visualizes MaDL(WW)ss leared learned annotator embeddings and weights for the dataset LETTER with the two annotator sets, CORRELATED and RANDOM-CORRELLATT Based on (W)'s learned kernel function, we create the two scatter plots via multi-dimensionall scal- ing (Kruskal 1964) for dimensionality reduction. This way, the annotator embeddings, originally located in an (R = dimenniiil space, are transformed into a two-dimensional space, where each circle represents one annotator embedding. circle's color indicates to which annotator group the embedding belongs. The two bar plots visualize the mean annotator weight of the different annotator groups, again indicated by their respective color. Analyzing the scatter plot of the annotator set CORRELATED, observe that the annota- tor embeddings" latent representations approximately reflect the annotator groups' correlations. Concretely, there are four clusters. The center cluster corresponds to the seven independent annotators, one cluster- specialized annotator and six common annolators. The three clusters in the ouler area represent the three groups of correlated annotators. The bar plot confirms our goal to assign lower weights to strongly correlated annotators. For example, the single independent cluster-spocialized annotator has a weight of 4.06, while the eleven correlated cluster-specialized annotators have a mean weight of 0.43. We make similar observations The scatter plot shows that the independent annotators also
after five training epochs.
for the annotator set form cluster, separated from the cluster of the large group of correlated, randomly guessing annotators. The single adversarial annotator belongs to the cluster of randomly guessing annotators since both groups of annotators make many annotation errors and thus have highly correlated annotation patterns. Again, the bar plot confirms that the correlated annotators get low weights. Moreover, these annotator weights are inversely proportional to the size group correlated annotators. For example, the 90 randomly guessing annotators have a similar weight in sum as the single class-specialitns annotator.
-lustet--pecializoii ammolater
cluse-spocialiedd annotator (oarrelated) advvrssriial umooaate (ocrrelited))
cluste--peciallidd annotator
common anmotator
class class-ppccialzee anmotator
commson asnotator adversarial annotator
-ppecaaaape amnolatee (coerelated)
O
tandomly guesing anmotator (oorrelttd))
Meaa Aunolator Weight
1.6
Mean Annotator Weight
10.10
10.24
Latent Dimensson 2
Lasent Dineession
0.43
01.00
Lateut Dimension
Annotator Group
Latent Dimenuion 1
Aunotator Group
CORRELATED
RANDOM-CORRELATEED
Figure 7: Visualization of MaDL(W)'s learned similarities between annotator embeddings and associated annotator weights.
Quantitative study: Table presents the GT and AP models' test performances for the four datasets with the annotator set CORRELATED and Table 7 for the annotator set RANDOM-CORRELLTTD. Both tables indicate whether technique models correlations between annotators (property P3) and whether the authors of a technique demonstrated its robustness against spamming annotators (property P4). Analogous to RQ1, training with GT labels achieves the best performances (UB), while annotation aggregation via the majority rule leads to the worst ones (LB). The LB's significant underperformance confirms the importance of modeling APs in scenarios with correlated annotators. MaDL(WW), as the default MaDL, variant, achieves competitive and often superior results for all datasets and evaluation scores. In particular, for the annotator set RANDOM- CORRELATED, MaDL(W) outperforms the other techniques, which are vulnerable many randomly guessing annotators. This observation is also confirmed when compare MaDL(W) to MaDL(W). In contrast, there is no consistent performance gain of MaDL(W) over MaDL(W) for the annotator set CORRELATED. While CoNAL is compelitive for the annotator set CORRELATED, its performance strongly degrades for the annotator set CONEOOM-TR....EEDD.... The initial E step in LIA's is EM algorithm estimates the GT class labels via a probabilistic variant of the majority rule. Similarly to the LB, such an estimate less accurate for correlated and/or spamming annotators. Besides MaDL(W), only CL and UNION consistently outperform the LB by large margins for the annotator set RANDOM--CORHLLAVEED.
20

---

Published in Transactions on Machine Learning Research (09/2023)
Table 6: Results regarding RQ2 for datasets with simulated annotators: Best and second best performances are highlighted per dataset and evaluation score while excluding the performances of the UB.
Ground Truth Model
Annotator Performance Model
Technique
P3
P4
ACC 1
NLL 4
BS +
ACC 1
NLL 4 BS +
BAL-ACC 1
LETTRR (CORRELATED)
UB
X
V
0.962
01199:0.004
0.058:1 D03
0.887:10.000
0.305:20003
0.173:00002
0.737=0.002
LB
X
0.762.10.007
1.302.10.005
0.482.10.004
0,68210.000
00,003
0.0062
0.60210.000
CL
X
O. 0.330000035
2.435:11228
0.318:10.05 0.7
0.800#0.000 008
0.416=00006
0.285=0.002
U.674 0000
REAC
X
X
0.922.10.000
1.006
0.115:10.007
0.815.10.001
0.005
0.24910.000
0.68411.001
UNION
V
O. 0.66-
1.668+0.32
0.224++0.032
0.795#00.00 0007
0.032=0.007
0.278=0.007
0. 0.667=0.006
LIA
X
X
0.823310.008
1.48310.018
0.569 1.369:10000 007
0.676.10.005
0.629:10.00
0.436
0.075
CoNAL
0..01
...001-0.449
0.213:10.024
0.840-10..014
0.390
0.238 0.0021
0.712
MaDL(W)
X
0.946:0.0006
0. 0.2933::000 0.82
0.063:00.00
0.883++0.002
0.314=0.001
0.178:000002
0.751=0.003
MaDL(W)
V
0..003
1.282.00008
0.080 1 1..044
0.887
0.308:00.004
0.115.10.002
0.756100001
FMNIST (CORRELATED)
UB
X
0.909==0.000
0.246::0000
0.131
0.002
0.333
0.198=0.000
0.741=0.002
LB
X
0.787=00.00 003
1.127:
0.475:+0 007
0.668#0.000 009
0.626 006
0.436 006
1.580=00.005
CL
X
X
0.868 0033
0.417::0002
0.217 010
0.799=500005
0.021=0.000
0.270=0.003
0.677=0.001
REAC
X
0.873:100000
0.415.10.012
0.196.10.000
0.828110.01
0.38210.000
0.237 1.0001
0..9710.001
UNION
V
0.859--:0.00 008
0.41 11000018
0.205 008
0.801 000009
0.420
0.269:10.011
0.678-10.009
LIA
X
0. 0.837=00.0m 006
1.277+0.000 008
0.553 004
0.685#0.000 0.00
0.633
00.011
0.569=0.002
CoNAL
V
0.897:00.002
0.299 0.00
0.152:+0.00 004
0.811#00.001
0.000
0.217=0.002
0.721#0.00
(aDL(W)
X
0.9044:0.002
0.272:0.007
0.1 0.139:00.00
0.803-10.008
0.337:100044
0.201 00.004
0.737=0.004
MaDL(W)
0.903=0.002
0.273+0.004
0.141#0.002
0.863+0.008
0.338=0.003
0.202=0.003
0.738=0.003
CIFAR.10
(CORRELATRD)
UB
X
0.933300.02
0.495:10.017
0.118 003
0.837:0.000
0.384::0.00
0.235
0.711=0.001
LB
X
X
0.652=0.014
1.309+0.001
0.540= 008
0.602 0101
0.623#0.000 003
0.136=0.003
0.54110.000
CL
X
0.490++0002 0.007
0.224:10.011
0.799:1.000
0.439:1.0004
0.282:10.003
0.074
REAC
X
O. 1.856==0000 003
0. 0.600 063
0,259+ 025
0.775=#0001 0.11
0.145= 015
0.287=00002
0.6-88=0.011
UNION
0.858200.000
0. 0.499:
0.211##00009
0.800
0,4322 0.0
0.276:10.002
0.675=0.003
LIA
X
0.776110.000
1.343.1..02
0.741 0.0002
0.617100.00
0.424110.003
0.617100.02
Co
V
V
0.862.10.00
0.473400000
0.003
0.800+10.001
0.43311.0003
0.277:10.002
0.670:10.001
MaDL(W) W
X
X
0 004
0.439:0.011 0.015
0.184::0. 005
0.824:100004
0.398
0.247=0.004
0.699=0.004
MaDL(W)
0.805
0.404
0.18810.0111
0.823100.00
0.39710.003
0.248 10.002
0.69810.002
SVHN (CORRELATED)
UB
X
/
0,966 1: 00.001
0.382.0.013
0.062.1 0.001
0.794:10.003
0.41410.002
0.26620.002
0.65710.004
LB
X
0.000
1.01210.0088
0.420-10.017
0.624110.023
0.604
0.0047
0.0071
CL
X
0.947=10000
0.314 0..44
0.116::0.00
0.789 009
0.033=20000
U.281 0.28110.000
000002
0.655=0.012
REAC
X
0.000
0.263
0.097110.005
0.767100.002
0.431
0.2831100000
0.620-10.008
UNION
V
0.947=00.00
0.250=40.025
0.080-10.010
0.767=:0.003
0.435=#00003
0.286=000002
0.621=0.000
LIA
X
X
0.929-#0.om 002
1.123#00 023
0,477#0 011
0.716 013
0. 0.623#0.010
0.131=0.010
0.391=0.0133
CoNAL
/
0.952:00000
0.231 10.003
0.076:10.001
0.805510.003
0.379-00000
0.235520.004
0.702=0.004
MaDL(W)
X
0.950==0.00
0. 0.23374:0000 0066
0.078+:0.003
0.790-20.008
0.416:200002
0.2699000022
0.652=0.002
MaDL(W)
V
0.952=0.001
0.227=000006
0.075:20.002
0.7811#0.000 003
0. 0.120=00000 002
0.273=000002
0.645=20000
5.4 RQ3: Do annotator features containing prior information about annotators improve learning and enable inductively learning annotators' performances? (Properties P5, P6)
Takeaway: Annotator features containing prior information about annotators improve the learning of GT and AP models (property P5). Furthermore, we can use these annotator features to inductively estimate the performances of annotators unavailable during training (property P6).
We address RQ3 by evaluating multi-annotator supervised learning techniques with and without us- ing annotator features containing prior information. For each dataset, we simulate 100 annotators according to the annotator set INDUCTIVE in Table 3 However, only 75 annotators provide class labels for training, Each of them provides class labels for 2 % of randomly selected training instances. The lower annotation ratio is used to study the generalization across annotators sharing similar features. The remaining 25 annotators form test set to assess AP predictions. We generate annotator features containing prior information by composing information about annotalor lype, class-wise APs, and cluster-wise APs. Fig. provides examples for two annotators based on two classes and four clusters. We evaluate two variants of LIA, CoNAL, and MaDL, denoted respectively by the schemes LIA(P5), CoNAL((55), and MaDL(P5). Property P5 refers to a technique's ability consider prior information about annotators. We differentiate between the variant with annotator features containing prior information (A) and the one using one-hot encoded features to separate
Setup:
21

---

Published in Transactions on Machine Learning Research (09/2023)
Table 7: Results regarding RQ2 for datasets with simulated annotators: Best and second best performances are highlighted per dataset and evaluation score while excluding the performances of the UB.
Ground Truth Model
Annotator Performance Model
Technique
Ps
P4
ACC
NLL
BS
ACC1 1
NLL 4
BS +
| BAL-ACC t
LETTER (LANDOM-CORRKKLAYHH))
UB
X
V
0.960:20.003 D03
0.131
0.059
0,937 002
0.21220.003
0.104:20000
0.516:20.002
LB
X
0.056
3.30710.044
0.96510.000
0.088 10.0000.000
9.95012.090
1.80610.002
0.500
CL
X
0.365:40.022 028
3.519=0.455
0.68220.052
0.925 000
0.237=0.004
0.124
0.506:20.00
REAC
X
X
0.607.10.024
1.810:00117
0.56110.034
0.00600.000
0.221
0.11610.002
0.507 100000
UNION
x
0.615=0.034
3.817=0.582
0.623=0.065
0.900
0.232=0.004
0.122+000002
+.500+000000
LIA
X
x
0.352:1:0.010
2.960:10.035
1.932200000
0,088 0.881:0.000 000
0.131
1.474410.041
0.500 0.0000
CoNAL
0.581110.015
2.325 249
0.599-0..07
0.925-1:0.000
0.006
0.124:10.001
0.507
MaDL(W)
x
0.5488:00.08
19902202215
0.673=0.001
0. 8.14:000044
0.02330.0033
0.265
0.506 006
MaDL(W)
V
0.932
1.27710.038
0.101 10000
0.940 0.000
0.204.10.003
0.10110.001
0.51910.001
FMNIST FRANDOM-CORRELAAN
UB
/
0.909 002
0.246=0.000
0:00000:003
000000
0.337=0.00
0. 00000
0.620-E0.000
LB
x
0.172+00.01 0119
2.296=0.005
0.899=0.0001
0,140 0.000
865 1669
1.703 0.000
0,500 0.000
CL
X
X
0.880: 003
0.462:20119
0.222=0.073
0.880 003
0.3047=0.00
0,200 0003
0.513+0.000
REAC
X
0.870
0.470.10.009
0.204
0.88510.000
0.342 0.000
0.19410.000
0.514:10.000
UNION
/
X
0.884:10.001 0022
0.387=0.022
0.182=0.007
0.88 18814:00000
0.345
0. 0.
0.514410.000
LIA
x
0.677: 008
2.094+0.0002
0.802=0.001
0.1 0.140==0.000
2.067
1.418+0.000 002
0.500 000
CoNAL
V
0.858:20.012
0. 0.6
0.219=000031
0. 0.882:+00000 0.002
0.311==00002
0.197
0.516:0.000
MaDL.(W)
x
0.337 046
2.131
0.853
0.229:40.075
1.038 146
0.814:1:0. 128
0.49--00.022
MaDL(W)
/
0.896=0.002
0.290=0.008
0.150=0.002
0.889#0.000
0.337=0.000
0.191+0.000
0.620+00000
GIFAR11
(RANIDOM-CORRRGLAAWV))
UB
/
0.932:10.00 002
0.519:10016
0.11910.004
0.886100.000
0.340::.000
0.192:10.00
0.515 0.00
LB
X
X
0.141 008
2.301=0.002
0.900=0.000
0.139=00.000
14.22446..60 699
1.70440.000 001
0,500 000
CL
X
0.676:100028
1.390
0.676:0.022
0.878
0.002
0.204
0.607:100000
REAC
X
0.462::0.011 010
2. 0.092200062
0.767=0.011
0. 0.875=t0.00 001
0.353=0.000 000
0.20
0,505 001
UNION
x
0.5400=100049
1.517=0.200
0.629-0.065
0.876:100000
0.335
0.205
0.506
LIA
x
0,211
2.273
1..041
0.139 0.139.10.000
2.096
1.429 0..002
0.500 000
CoNAL
/
0.555-10.020 V
1.379=0.058
0.502=0.020
0.870:10.001
0.35541.0002
0.206:10.002
0.506
MsDL(W)
X
x
0.217:400042
6.992:10.386
0.21970.087
0.872:40.00 000
0.39810.011
0.2294:0.00
0.502::0.000
MaDL(W)
/
0.822
1.99300,033
0.20210.011
1.885100000
0.339-10.000
0.10210.001
0.51410.000
SVHN
FRANDOM-CCRRELANN
UB
X
/
0.965.10.001
0.399:0.0017
0.064:200001
1.8771000000
0.349:0.000
0.201.10.00
0.509
LB
x
0.190-10.000
200002
0..000
0.13::10.000
24.01911.8002
1.700-10.001
0.500-10.000
CL
X
0.908:10.038
0.398=0.226
0.143:0.006
0.873:0.001
0.354:0.002
0.205110000
0,505
REAC
X
0.189.10.001
2.29411.000
1.89810.000
0.140.10.000
2.7724
1.38411..04
0.50010.000
UNION
X
0.881 104
0.529
0.
0.872:t0.002
0.356=#0.008
0.206-4:0.005
0.505-+:0.00
LIA
X
X
0.192:4:000 004
2.299400.00 004
898=00001
0, 1.13::00000 000
3.86/133.510
1.483440.11
0.500+00. 000
CONAL
0.201110.048
0.933
0.956=10.02
0.800:10.000
0.41410.008
0.242:10.003
0.500-1:0000
MaDL(W)
0.243:10. 102
6.055=3.173
1.11920.230 230
0.675+:0.352
0.70220.344
0.505-+0.319
0.500++00.00
MsDL(W)
V
0.910:10.002
0.241=0.011
0.091=0.003
0.877=0.000
0.3/9920.000
0.201+0.000
0.508-t0.000
between annotators' identitics (A). MA)L(A) corresponds to MaDL's default variant in this setup. We do not evaluate CL, UNION, and REAC since these techniques cannot handle annotator features.
Qualitative study: Fig. 8 visualizes AP predictions of MaDL(A) regarding two exemplary annotators for the dataset TOY. The visualization of these AP predictions is analogous to Fig. G Neither of the two annota- tors provides class labels for the training, and the plotted training instances show only potential annotations to visualize the annotation patterns. The vectors at the right list the annotator features containing prior information for both annotators, The colors reveal the meanings of the respective feature values. These meanings are unknown to MaDL(A), such that its AP predictions exclusively result from generalizing sim- ilar annotators' features and their annotations available during training. MaDI.(A) correctly identifies the left annotator as adversarial because it prodicts low (white) AP scores across the feature space regions close to training instances. For the right cluster-specialized annotator, MaDL(A) accurately separales the two weak clusters (feature space regions with predominantly crosses) with low AP estimates from the two expert clusters (feature space regions with predominantly circles) with high AP estimates.
Quantitative study: Table S presents the GT and AP models' test performances for the four datasets with the simulated annotator set INDUCTIVE. The table further indicates whether a technique processes prior information as annotator features (property P5) and whether a technique can inductively estimate the performances of annotators unavailable during the training phase (property P6). Note that the AP results refer to the aforementioned 25 test. annotators. Hence, there are no results (marked as -) for techniques
22

---

Published in Transactions on Machine Learning Research (09/2023)
Iustanace class blue instancce class green
Instance wrongly assigned
sunotater type
claster wine AP
instancc wroupy assignod class green
AP Amnotator Festuree
Anuolator Performance
Data
adversarial
0.04
0.06
a1
0.03
0.07
0.04
0.05
cluster-speciaalieed
0.51
0.49
92
0.95
0.03
0.95
0.07
Adversarial Annotator aj
Cluster-specialized Annotator a2
Figure 8: Visualization of MaDI.(A)'s inductive AP estimates for two unknown annotators.
with AP models not fulfilligg property P6. For completeness, we provide the results for the 75 annotators providing class labels for training in Appendix DJ As for RQI and RQ2, training with GT labels leads to the best performance results (UB), whereas learning from annotations aggregated via the majority rule mostly results in the worst performances (LBB.. Inspecting the results of MaDI(A)'s GT model compared to the other techniques, we observe competitive or partially superior results across all four datasets. Concerning its AP model, we further note that MaDL((A) provides meaningful AP estimates, indicated by BAL-ACC values greater than 0.5. Comparing the GT models' results of each pair of variants, performance gains for LIA and MaDL demonstrate the potential benefits of lcarning from annotator fcatures containing prior annotator information. In contrast, the GT models' results of CoNAL(A) and CoNAL(A) hardly differ.
6 Conclusion
In this article, we made three main contributions. (1) We started with a formalization of the objectives in multi-annotator supervised learning. Focusing on AP estimation, we then presented six relevant properties (cf. P1 P6 in Section 3) for categorizing related techniques in this research area. (2) Considering these six properties, we proposed our framework MaDL. A modular, probabilistic design and a weighted loss function modeling annotator correlations characterize its novelties. (3) We experimentally investigated the six propertics via three RQs. The results confirmed MaDL's robust and often superior performance to related multi-annolator supervised learning lechniques. The findings of this article, with a focus on AP estimation, provide a starting point for several aspects of future research, some examples of which are given below.
Although the annotator embeddings already contain information about the annotation patterns concerning instances and classes, MaDL, is currently limited to computing annotator correlations on global level, i.e., annotator weights are not an explicit function of instance-annotar direction may be valuable to quantify correlations in certain regions of the feature space. Leveraging AP estimates for additional applications, e.g., selecting the best crowdworkers to obtain high-quality annotations during a crowdsourcing campaign (Herde et al.] 2023). is also of great value. Another neglected aspect is the study of epistemic uncertainty (Huseljic et al. 2021). For example, the visualizations for the two- dimensional dataset in Fig. 0 show high certainty of the GT and AP models in feature space regions with observed instances. However, meaningful epistemic uncertainty estimates are essential in many (safety- critical) applications (Hüllermeier & Wacgeman (021) and would improve the characterization of annotators' knowledge. During our experiments, we showed the potential benefit of annolator features. We had no access to a dataset with prior information from real-world annotators, so we needed a suitable simulation for these features. Therefore, and also noted by /Zhang et al. (2023), future rescarch may acquire such prior information via crovdsourcing to verify their benefit. As the concentration of annotators may fluctuate or annotators may learn during the annotation process, taking time-varying APs into account is another potential avenue
pairs. For example, an extension in this
23

---

Published in Transactions on Machine Learning Research (09/2023)
Table 8: Results regarding RQ3 for datasets with simulated annotators: Best and second best performances are highlighted per dataset and evaluation score while excluding the performances of the UB. The AP models' results refer to the 25 test annotators providing no class labels for training. An entry marks a technique whose AP model cannot make predictions for such test annotators.
Technique
P5
P6
ACC f
Ground Truth Model
NLL 4
BS
ACC +
Annotator Performance Model NLL
BS
BAL-ACC f
LETTER (INDUCTIVE)
UB
/
0,962
0.129::0.00 003
0.058:00 002
0.672400000
0.107 0.715=0.017
0.612=0.005
LB
/
0..005
1. .090:40.001
0.429-10.008
0.569+00.088
0,730:100011
0.623:2.007
0.537
LIA(A)
X
1.875=00.006
0.901
0.350= 024
LIA(A)
/
/
0.876=00000
1.006+0.177
0.397 0.44
0.609+0.017
1.41740.815
0.597=0.105
0.545=0.035
CoNAL(A)
X
x
0.875
0,804 119
0.186: 010
CoNAL(A)
0.874:10.007
0.808:10.116
0.186:10.011
MaDL(A)
X
0.911#0.006
0.834
0.129:000088
MaDL(A)
V
0.914=0.004
0.303=0.009
0.124::0.00
0.668+0.007
0.813#0.115
0.47
0.600=0.010
PMNIST (INDUCTIVE)
UB
0.909-10.000
0.246.10.005
0.131
0.730.1..008
0.536110.019
0.357:10.010
0.606
LB
/
1.811400002
0.876:4-0005
0.370:+-0000
0.590=000023
0.68110.006
0.487
0.537
LIA(A)
x
0.852:+0.00
1.011 0.00
0.436:10.010
LIA(A)
/
/
0.855110.002
0.97210.002
0.417:10.000
0.674100003
0.6261100026
0.436:1.0024
0.601 0.0027
CoNAL.(A)
X
x
0.880-10.002
0.322 005
0.163:+
CoNAL(A)
V
0.000000002
0.323#0.011
0. 0.163:+0.005 005
MaDL(A)
X
0.895=0.002
0.297
0.152+0.000
MaDL(A)
V
0.893-20.004
0.297:0.0088
0.158:10.004
0.723:100004
0.538:10.003
0.362:20.003
0.649-0.005
CIFAR10 (INDUCTIVE)
UB
0.9831
0.527
0.122:40.000
0.68640.00 0006:000000 006
0.6:66:0.10 101
U. 0010000:016
0.613=00000
LB
/
0.001
1.05440.035
0.447:100016
0.583:100009
0.684
0.490:00.004
0.521=0.000
LIA(A)
X
0.798 008
1.072 014
0.455= 006
LIA(A)
/
0.80:1:00.011
1.056
0.417=#0001
0.607+0.020
0.670=0.017
0.477=0.016
0.001
CoNAL(A)
x
0..05220002
0.676::0.016
0.245:100005
CoNAL(A)
V
0.834:100.00
0.574
0.248.10.007
MaDL(A)
X
0.811110.008
0.626.10.036
0.277.10.014
MaDL(A)
V
0.887:500003
0.667:00022
0.2424000006
0.698110000
0.567=10.016
0.38320.004
0.617=0.004
(INDUCTIVE)
UB
/
0.965-10.001
0.393:10.015
0.063:10.002
0.613:10.004
0.943
0.511
0.024
/
LB
/
0.927=00002
0.80640.016
0.328 .00
0.58
0.704
0.509
0.511
LIA(A)
X
X
0.929.10.0m
0.8181101151
0.336.10 0.068
LIA(A)
/
0.902110.001
0.7541:0.
0.303:10.079
0.60310.0088
0.67110.024
0.47820.022
0.51320.008
CoNAL(A)
X
0.94110.000
0.258 0.009
0.090
CoNAL(A)
V
0.942:0.00 1.000
0.260.10.01
0.080
MaDL(A)
X
0.928 002
0.299::0000 019
0.109::: 005
MaDL(A)
0.905:1:0001
0.098:1.002
0.62410.007
0.632:10.013
0.444
0.006
for future research (Donmez et al., 2010). Furthermore, there are already crowdsourcing approaches CChang ct al. 2017) and concepts (Calma et al. 2016 supporting collaboration between annotators. Thus, developing techniques considering or recommending such collaborations is of practical value (Fang et al. 2012).
Finally, we limited ourselves to empirical performance results and classification tasks with class labels as annotations. Future investigations theoretical performance guarantees of MaDL and the learning with dif- ferent. annotation types, such as class labels with confidence scores (Berthon et al. 2021 or partial labels (Yu) et al. )022), are apparent. Furthermore, the extension to related supervised learning tasks, such as semantic segmentation, sequence classification, and regression, is of interest. The goal of semantic segmentation is to classify individual pixels (Minaee et al., 2021). potential approach to extend MaDL would be to implement its GT model through U-Net (Ronneberger et al., 2015) and foed its latent representations as input to the AP model for estimating pixel-wise confusion matrices per annotator. Likewise, we may adapt MaDL to be applied to sequence classification tasks, such as named entity recognition (Li et al.) 2020). Concretely, we could implement the GT model through BiLSTM-netwol with softmax outputs (Reimers & Gurevych 2017) and foed its latent word represcntations as inputs to the AP model for estimating word-wise confusion matrices per annotalor. Since both extensions involve higher computational costs than standard classifica- tion tasks, one may alternatively investigate the estimation of a single (pixel- or word-independent) confusion matrix per annotator. Regression tasks expect the prediction of continuous target variables. Therefore, the probabilistic model of MaDL has to be adapted. For example, the GT model could estimate the mean and variance of an instance's target variable, while the AP model learns annotators' biases and variances.
24

---

Published in Transactions on Machine Learning Research (09/2023)
Broader Impact Statement
Big data is a driving force behind the success of machine learning (Zhou et al, 2017). Reducing the effort and cost. required for annotating this data essential for its ongoing development In this context, MaDL, is a possible tool to leverage the workforce of cost-efficient but error-prone annotators. Yet, as central rosource for data annotation, crowdsourcing can negatively impact individuals even entire communitios. Some of these impacts include exploiting vulnerable individuals who participate in low-wage crowdsourcing tasks (Schlagwein et al., 2019), producing low-quality data (Daniel et al. 2018). and outsourcing jobs (Howe) 2008). On the one hand, mulli-annotaton supervised learning techniques can improve data quality and sup- port awarding well-performing crowdworkers. On the other hand, such a technique may intensify the already existing competition between crowdworkers (Schlagwein et al., 2019). Tt. also requires tight monitoring to ensure fair assessments of crowdworkers. Besides the benefits of annotator features containing prior infor- mation about annotators, there several risks. Collecting and leaking potentially sensitive personal data about the annotators is such a significant risk (Xia & McKernan 2020). Thus, the annotator features must contain only information relevant to the learning task. Further, lack of control over this or other processes can lead to discrimination and bias based on gender, origin, and other factors (Goel & Faltings 2019). For these reasons, is crucial to consider and address the potential risks via responsible policies and practices when employing multi-annotase supervised learning techniques.
Acknowledgments
We thank Lukas Rauch for the insightful discussions and comments, which greatly improved this article.
References
Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirei, and Nassir Navab. Aggnet: Deep Learning from Crowds for Mitosis Detection in Breast Cancer Histology Images. IEEE Trans. Med. Imaging, 35(5):1313 1321, 2016.
Görkem Algan and Iaka Ulusoy. Image classification with deep learning in the presence of noisy labels: A survey. Knowl. Based Syst., 215:106771, 2021.
Sercan Ô Arik and Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning. In AAAI Conf. Artif. Intell., pp. 6679-6687, 2021.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence Scores Make Tnstance-dependent Label-noise Learning Possible. In Int. Conf. Machine Learn., pp. 825-836, 836, Virtual Conf., 2021.
Glenn W Brier. Verification of Forecasts Expressed in Terms of Probability. Mon. Weather Rev., 78(1):1-3. 3, 1950.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. The Balanced Accuracy and Its Posterior Distribution. In IEEE Int. Conf. Pallern Recognil., pp. 3121 3124, Istanbul, Turkey, 2010.
Adrian Calma, Jan Marco Leimeister, Paul Lukowiez, Sarah Oeste-Reif, Tobias Reitmaier, Albrecht Schmidt, Bernhard Sick, Gerd Stumme, and Katharina Anna Zweig. From Active Learning to Dedicated Collabo- rative Interactive Learning. In Int. Conf. Archit. Comp. Syst., pp. 8, Nuremberg, Germany, 2016.
Peng Cao, Yilun Xu, Yuqing Kong, and Yizhon Wang. Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds. In Int. Conf. Learn. Represent., New Orleans, LA, 2019.
Joseph Chee Chang, Saleema Amershi, and Ece Kamar. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In CHI Conf. Hum. Factors Comp. Syst., pp. 2334-2346, Denver, CO, 2017.
Zhendong Chu, Jing Ma, and Hongning Wang. Learning from Crowds by Modeling Common Confusions. In AAAT Conf. Artif. Intell. pp. 1832-5840, Virtual Conf., 2021.
25

---

Published in Transactions on Machine Learning Research (09/2023)
Florian Danicl, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Sure., 51(1):1 40, 2018.
Jacob Dovlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pro-training of Deep Bidirec- tional Transformers for Language Understanding. (XX:::1100004800 2018
Pinar Donmez, Jaime Carbonell, and Jeff Schneider. A Probabilistic Framework to Learn from Multiple Annotators with Time- Varying Accuracy, In SIAM Int. Conf. Data Min., pp. 826 837, Columbus, OH, 2010.
Meng Fang, Xingquan Zhu, Bin Li, Wei Ding, and Xindong Wu. Self Seught Active Learning from Crowds. In IEEE Inl. Conf. Dala Mön., pp. 858 863, Brussels, Belgium, 2012.
James Fiedler. Simple Modifications to Improve Tabular Neural Networks. arXic:2108.0322 2021.
Peter W. Frey and David J. Slate. Letter recognition using Holland-style adaptive classifiers. Machine
Learn., 6(2):161-182, 1991.
Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Mare Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S. Boning. Learning from Multiple Annotator Noisy Labels via Sample-Wise Wise Label Fusion. In Fur. Conf. Comput. Vis., pp. 407-422, Tel Aviv, Israel, 2022.
J. Gil-Gonzalez, A. Orozco-Gutieree and A. Alvarez-Meza. Learning from multiple inconsistent and de- pendent annolators to support classification lasks. Neurocompuling, 423:236 247, 2021.
Julián Gil-González, Andrés Valencia- Aieencia-Dqque, Andrés Álvarez-Meza, Álvaro Orozco-Gutiéree Gutiérrez, and Andrea Garcfa-Moreno. Regularized Chained Deep Neural Network Classifier for Multiple Annotators. Appl. Sci., 11(12):5409, 2021.
Ruslan A. Gilyazev and Denis Y. Turdakov. Active Learning and Crowdsourcing: A Survey of Optimization Methods for Data Labeling. Program. Comput. Softw., 44(6):476-49) 2018.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. In Int. Conf. Artif. Intell. Stat., pp. 315-323, Fort Landerdale, FL., 2011.
Naman Goel and Boi Faltings. Crowdsourcing with Fairness, Diversity and Budget Constraints. In AAAI/ACM Conf. AI, Ethics, Soc., pp. 297-304, Honolulu, HI, 2019.
Keren Gu, Xander Masotto, Vandana Bachani, Balaji Lakshminarayanan, Jack Nikodem, and Dong Yin. An instance-dependent simulation framework for learning with label noise. Mach. Learn., 2022.
Melody Y. Guan, Varun Gulshan, Andrew M. Dai, and Geoffrey E. Hinton. Who Said What: Modeling Individual Labelers Improves Classification. In AAAI Conf. Artif. Intell., pp. 3109-3118, New Orleans, LA, 2018.
Maya R Gupta, Samy Bengio, and Jason Weston. Training Highly Multiclass Classifiers. J. Mach. Learn. Res., 15(1):1461 1492, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Conf. Comput. Vis. Pattern Recognit., pp. 770-778, Las Vegas, NV, 2016.
Xiangnan He, Xiaoyu Du, Xiang Wang, Feng Tian, Jinhui Tang, and Tat-Seng Chua. Outer Product-based Neural Collaborative Filtering. In Int. Joint Conf. Artif. Intell., pp. 2227 2233, Stockholm, Sweden, 2018
Marck Herde, Denis Husoljic, Bernhard Sick, and Adrian Calma. A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification. IEFE Access, 1166970-166966 2021.
26

---

Published in Transactions on Machine Learning Research (09/2023)
Marek Herde, Denis Huseljic, Bernhard Sick, Ulrich Bretschneider, and Sarah Oeste-ReiB. Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning. In Inl. Workshop on Interact. Adapt. Learn. @ Eur. Conf. Mach. Learn., pp. 14 18, Turin, Italy, 2023.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empir- ically. or 1712 00,00, 2017.
Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shlapentokh-Rochman. Learning Curves for Analysis of Deep Networks. In Int. Conf. Machine Learn., pp. 4287-428747 4296, Virtual Conf., 2021.
Jeff Howe, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, 2008
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learn., 110:457-506, 2021.
Denis Huseljic, Bernhard Sick, Marek Herde, and Daniel Kottke. Separation of Alcatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks. In IEEE Ind. Conf. Pattern Recognil., pp. 9172 9179, Virtual Conf., 2021.
Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima. A Convex Formulation for Learning from Crowds. In AAAI Conf. Artif. Intell, pp. 73 79, Toronto, ON, 2012.
Ashish Khetan, Zachary C. Lipton, and Animashree Anandkunar. Learning From Noisy Singly-labeled Data. In Int. Conf. Learn. Represent., Vancouver, BC, 2018.
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. Master's thesis, University of Toronto, 2009.
Joseph B. Kruskal. Multidimensional scaling by optimizing goodness of fit to nonmetric hypothesis. Psychometrika, 29(1):1 27, 1964.
Yann LeCun and Corinna Cortes. The MNIST database of handwritten digits, 1998.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE Trons. Knowl. Data Eng., 34(1):50-77) 70, 2020.
Jingzheng Li, Hailong Sun, and Jiyi Li. Beyond confusion matrix: learning from multiple annotators with
awareness of instance features. Machine Learn., pp. 1-23, 2022.
Chengjiang Long, Gang Hua, and Ashish Kapoor. A Joint Gaussian Process Model for Active Visual Recoguition with Expertise Estimation in Crowdsourcing. Int. J. Compul. Vis., 116(2):136 160, 2016.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In Inl. Conf.
Learn. Represent., Toulon, France, 2017.
llya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In Int. Conf. Learn. Represent., New Orleans, LA, 2019.
Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: survey. IEEE Trans. Pallern Anal. Mach. Intell., 44(7): 3523 3542, 2021.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conf. Artif. Intell., Austin, TX, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In Deep Learn. Unsupervised Feature Learn. Workshop @ Adu. Neural Inf. Process. Syst., Granada, Spain, 2011.
27

---

Published in Transactions on Machine Learning Research (09/2023)
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Takshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Adv. Neural Inf. Process. Syst., Vancouver, BC, 2019.
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J. Guibas, and Jascha Sohl-Dickstein. Deep Knowledge Tracing. In Adv. Neural Inf. Process. Syst., Montreal, QC, 2015.
Emmanouil A. Platanios, Maruan Al-Shedivat, Eric Xing, and Tom Mitchell. Learning from Imperfect Annotations. 088443 2020.
Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mci-Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar. A Survey on Deep Learning: Algorithms, Techniques, and Applications. ACM Comput. Surv., 51(5):1-35 36, 2018.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based Neural Networks for User Response Prediction. In IEEE Int. Conf. Data Min., pp. 1499-155, Barcelona, Spain, 2016.
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from Crowds. J. Mach. Learn. Res., 11(4):1297 1322, 2010.
Nils Reimers and Tryna Gurevych. Optimal Hyperparameters for Deep LSTM-Networks for Sequence Label- ing Tasks. (eXxi:1177.0000 2017.
Filipe Rodrigues and Francisco Pereira. Deep Learning from Crowds. In AAAI Conf. Artif. Intell., pp. 1611-1618, New Orleans, LA, 2018.
Filipe Rodrignes, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annotators: Distin- guishing good from random labelers. Pattern Recognit. Lett., 34(12):1488-1438 2013.
Filipe Rodrigues, Francisco Percira, and Bernardete Ribciro. Gaussian Process Classification and Active Learning with Multiple Annotators. In Inl. Conf. Machine Learn., pp. 433 441, Beijing, China, 2014.
Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C. Pereira. Learning Supervised Topic Models for Classification and Regression from Crowds. IEEE Trons. Pattern Anal. Mach. Intell., 39(12):2409 2422, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Med. Image Comput. Comput.-Assist. Interv., pp. 234-241, Munich, Germany, 2015.
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. Endd-o-End Weak Supervision. In Adv. Neural Inf. Process. Syst., Virtual Conf., 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. of Comp. Vis., 115:211-252, 2015.
Daniel Schlagwein, Dubravka
and Benjamin Hanckel. Ethical norms and issucs in crowd- sourcing practices: Habermasian analysis. Inf. Syst. J., 29(4):811 837, 2019.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recog- nition. In Inl. Conf. Learn. Represent., San Diego, CA, 2015.
Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan Daniel C. Alexander, and Nathan Silberman. Learning from Noisy Labels by Regularized Estimation of Annotator Confusion. In Conf. Comput. Vis. Pattern Recognit., pp. 11244-11253, Long Beach, CA, 2019.
Alexandra Nnemamaka Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from Disagroement: A Survey. J. Artif. Intell. Res., 12:1385-1470, 2021.
Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
28

---

Published in Transactions on Machine Learning Research (09/2023)
Jennifer W. Vauhana. Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research. J. Mach. Learn. Res., 18(193):1-46, 2018.
Jeroen Vuurens, Arjen P. de Vries, and Carsten Eickhoff. How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy. In ACM SIGIR Workshop Crowdsourcing Inf. Reir., Pp- 21- 21-26, Beijing, China, 2011.
Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. Deep Learning From Multiple Noisy Aunotators A Union IEEE Trans. Neural Netw. Learn. Syst., 2022.
Nicolai Wojke and Alex Bewley. Deep Cosine Metrie Learning for Person Re-leentificaion. In JEEE Winter Conf. Appl. Comput. Vis., pp. 748-756, Lake Tahoe, NV, 2018.
Huichuan Xia and Brian McKernan. Privacy in Crowdsourcing: Review of the Threats and Challenges. Comput. Support. Coop. Work, 29:263-301, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708. 077477777 2017.
Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy, Learning from multiple annotators with varying expertise. Machine Learn., 95(3):291-33) 297-377, 2014.
Jie Yang. Thomas Drake, Andreas Damianou, and Yoelle Maarek. Leveraging Crowdsourcing Data for Deep Active Learning Application: Learning Intents in Alexa. In Int. World Wide Web Conf-, pp. 23-32, Lyon, France, 2018.
Peilin Yu, Tiffany Ding, and Stephen H. Bach. Learning from Multiple Noisy Partial Labelers. In Tna. Conf.
Artif. Intell. Stat., pp. 11072-11095, Virtual Conf., 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. 1xXi:16605.0714, 2016.
Jing Zhang. Knowledge Learning With Crowdsourcing: A Bricf Review and Systematic Perspoctive.
IEEE/CAA J. Aulom. Sin., 9(5):749 762, 2022.
Jing Zhang, Xindong Wu, and Victor Sheng. Learning from Crowdsourced Labeled Data: A Survey. Artif.
Intell. Rev., 46(4):543-57 576,, 2016.
Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederík Barkhof, and Daniel Alexander. Discntangling Human Error from Ground Truth in Segmentation of Medical Images. In Adv. Newral Inf. Process. Syst., pp. 15750 15762, Virtual Conf., 2020.
Le Zhang, Ryutaro Tanno, Moucheng Xu, Yawen Huang, Kevin Bronik, Chen Jin, Joseph Jacob, Yefeng Zheng, Ling Shao, Olga Ciccarelli, et al. Learning from Multiple Annotators for Medical Image Segmen- tation. Pattern Recognit., pp. 109400, 2023.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv., 52(1):1-38, 2019.
Lina Zhou, Shimei Pan, Jianwu Wang, and Athanasios V. Vasilakos. Machine learning on big data: Oppor- tunities and challenges. Neurocomputing. 237:350-300 2017.
A Proofs
Proof of Proposition Minimizing the loss Eq.
results in the following Bayes optimal prediction:
(xT(x) = arg min (Eyix [5(y # v()i)) = arg min y'Ely
Σ
Pr(y x))(y +b))
y Eyt
BEflY
arg min y y'Efy
Σ Verr
Pr(v x)
= arg min (1 - Pr(y/ x)) = arg max (Pr(y/ [x)) y'Ely
0
e'y
29

---

Published in Transactions on Machine Learning Research (09/2023)
Proof of Proposition Minimizing the loss in Eq.
results in the following Bayes optimal prediction:
MAP(X. a) = arg min (Eyyxx [Exixay 18 (y' (d # 2))[]) y'E(0,1)
arg min ('E(0.1)
Pr(y
Σ WeEty
Pr(z x, x, # 8(y # 2))
yElly
Pr(z xr(= 1) + Pr(y xx,,,)) # #01))
arg min y'e(0.1)
Σ yU
Pr(y 1x(v1x)
Σ
Σ Pr(y xx) (1-1 -rry x,a,,))o(y * + Pr(y x,,, # #01) MEft
arg min (0,(0,13
8
Σ yEny
Pr(y x) Pr(y x)
Σ Pr(y x)) Pr(y xx
vEst
6
Σ
Pr(y x) Pr(y x) < Σ
Pr xx) - E
Pr(y x) Pr(y x, a, y)
yefty
vefly
Σ
Pr(y Pr(y x,a, <1 Σ
Pr(y x) Pr(y x.a.y)
yelly
yelly
=
Σ vERy
Pr(y x) Pr(y xx <
Proof of Theorem] am is given by:
Applying assumption
*
Theorem
to Eq. 21
the weight w(am) for an annotator
M
8(m E A(1)
am
(A()
Accordingly, the sums of the annotator weights uniformly distributed across the G groups:
M G
Σ
w(am)
Σ
w(am)
(o)
Inserting these annotator weights into the weighted ligeliidl function and making use of assumption
ft)
in Theorem [ get
N
M
N
Σ
w(am) In (Pr(2nm Xn, am))
Σ
w(am) ) In (Pr(znm Xn. am))
n=1 m=1
==1 g=1 meA4(9)
Σ wr(am)) In (Pr(Enm, Xns am,))
mE,((9)
G
O
M
In (Pr(znme Xns ammmm))))
g=1
Σ
In (Pr Xnx amm))
1=1
30

---

Published in Transactions on Machine Learning Research (09/2023)
B End-to-end Training Algorithm Example
This appendix illustrates the steps of Algorithm 1 for sampled mini-batch. Let assume classification
problem with C 2 classes, M = 2 annotators, and mini-batch size of B = 1. We further suppose there is no prior information about annotators such that we represent the two annotators via the one-hot encoded vectors
a1 = (1,0)² and a₂ (0,
(33)
In the first step, we compute the instances' class-membership probabilities. For a mini-batch size of B 1, we exemplarily assume we have a single arbitrary instance Xn, for which we obtain
(x = (0.8, 0.2)*
(34)
class-membership probabilities outputted by the GT model with parameters O. In the second step, we compute the confusion matrix for each mistanc-aanntator pair. For mini-batch size of B and M = annotators, we obtain
Pw a1) ( 9) and Pw( a2)
0.5
0.5
(35)
0.5
0.5
two exemplary confusion matrices outputted by the AP model with parameters Thus, we currently expect annotator a1 to be error-free and ag to randomly guess. In the third step, we determine the similarities between all pairs of annotator embeddings to compute their weights in the fourth step. Since the Gaussian kernel ky is symmetric and there are only two annotators, we obtain
(a1))) (a2) =
(36)
as annotator weights in our example. We refer to Fig. 7 for a more complex example of computing annotator
weights. Before evaluating the loss function, we assunee Zn = (2, 2)TH as the vector of class labels assigned by the annotators a and a2 instance x,. Moreover, we take 7 = as an example bandwidth and a = 2, 3 as parameters of the gamma distribution. Now, we have all the ingredients evaluate the loss function in
Eq. 25
(
0
T
00.8)
Lxx Sx a1 (0,,,, 1 In (0, 1)
1
0.2
(37)
prediction loss for aunotator a
0.5
0.5
0.8
2 1 In (0,
(38)
0.5
0.5
0.2
prediction loss for snnotator az
In (Gam (1 22,
(39)
regularization term for the bandwidth
In (0.2) In (0.5) +1. compute the cross-entropy loss between the estimated annotation probabilities
(40)
Eq. 37 and Eq.
38
wo (Xn, a1) = (
0 1
T
0.88 0.2
0.8))
0..5
0.5
T
00.8
(0.5) 0.5
(x,, ai)
(41)
0.5
0.5
0.2
and the two provided annotations as one-hot hot encoded targets to learn amnotation patterns. In contrast, Eq. 39 computes the logarithmic probability density the current bandwidth value in relation the gamma distribution regularize the possible bandwidth values. Finally, we can use common optimizer for DNNs to update the parameters 0,w,7-
31

---

Published in Transactions on Machine Learning Research (09/2023)
C
Ablation Study
This appendix presents the results of an ablation study regarding MaDL's hyperparameters. Table provides the results regarding the two datasets MUSIC and LABELME with real-world annolators and Table 10 presents the results for the dataset LETTER with the four annotator sets simulated according to Table 3 We design the ablation study following a one-factor-at-aa-iin approach in favor of roducing the computational cost. This means we define default MaDL variant for each experiment and change the value only oue hyperparameter at a time. For example, we study the effect of the AP prior 1j (0, by taking the default MaDL variant and changing only the value of this hyperparameter. Our default MaDL variant corresponds to the hyperparameter values described in Section E For the combination of the dataset LETTER with the annotator set INDUCTIVE, the default MaDL, variant gets annotator features containing prior information instead of one- encoded annotator features as input. The general setup of an experiment, e.g., the number of repeated runs, the splits into the training, test, and validation sets, etc., is identical to the one described in Section S In the following, we analyze the effects of the individual hyperparameters, whose default values are given in brackets:
Embedding size (Q= R 16): The embedding size controls the dimensionality of the instance and annotator embeddings lcarned by the AP model. For the two datasets with real-world annotators in Table y an embedding size of Q = R = 16 clearly works best. In contrast, the results of Table [0 indicate that an embedding size of Q = R = 8 is superior for the dataset LETTER with simulated annotator scts. Although Q = R 16 is a robust default embodding size for the tested learning tasks, is crucial consider the characteristies of each learning task individually. Thereby, the mumber of annotators and instance features are of particular importance.
AP prior (1)=088) The AP prior controls the initialization of the AP output layer's biases (cf. Eq. 19 and thus the parametrization of initial annotator confusion matrices. This hyperparameter is relevant for the identifiabiliy of the class-membership probabilities and annotator confusion matrices. The results of Table and [0 confirm this importance. A low value of r = 0.1 leads to poor performance across all testod datasets since it cannot identify the annotation noisc. Selecting high values, c.g., 77 E (0.77077 0.8,0,88 0.9), resolves such an issue and thus leads to much better performances.
Outer product (True): The outer product layer is one option, adopted from literature (Qu et al., 2016). to model the interactions between instance and annotator embeddings. Training MaDL with such a product layer leads to performance gains for MUSIC and LABELME as datascts with real-world annotators (cf. Table g). In contrast, there are no clear performance differences for the dataset LETTER with the four simulated annotator sets (cf. Table [O]]. Since this modeling of interactions resembles recommender systems, testing alternatives, c.g., computing the outer product of instance and annotator embeddings as input to convolutional layer (He et al., 2018), may be worthwhile.
Residual block (True): This hyperparameter determines whether we implement the residual connection (True) not (False) into the block shown in Fig. 4 The idea of this connection is to prioritize the annotator embeddings when computing APs. Inspecting the results, we see clear performance gains for MUSIC and LABELME as datasets with real-world annotators (cf. Table D.. In contrast, the residual connection leads for two of the simulated annotator sets to better and for the other two to worse results for the datasct LETTER. Similar to investigating alternatives to the outer product layer, optimizing the entire architecture in Fig. may be valuable in future work.
Gamma prior (a 1.25, = 0.25): This pair of hyperparameters specifies the prior gamma distribution of the bandwidth y used for computing kernel values between annotator embeddings. Table and I0 list the evaluated pairs of a and 3, including the option of no annotator weights The values for a and 3 are selected to test different combinations of modes, i.e., (a-11/9 E {0.5, 1, 1,2], and variances, ie., a/8² e {6, 20), for the gamma distribution. Table 10 indicates no large performance difference of varying value pairs of a and 3. As previously shown for RQ2 in Soction 5 the weak results of the op- tion "No Weights" for the dataset LETTER with RANDOM-CORRELLN
as annotator set demonstrate the importance of modeling annotator correlations. Table g indicates larger performance differences
32

---

Published in Transactions on Machine Learning Research (09/2023)
for the varying parametrizations. Here, mode of (a-1)/3 corresponding to (a 1.25, 3 = 0.25) and (a = 1.5, B = 0.5), leads to competitive performances across both datasets. In general, one could also replace the Gaussian kernel with other common kernels or similarity functions to estimate an- notators' densities as a basis for quantifying their correlations. A popular alternative would be the cosine similarity function. For example, Wojke & Bewley (2018) demonstrate how to learn an embedding space where this similarity function is optimized by re-paramctrizing the conventional softmax classification output.
Table 9: Ablation study on MaDL's hyperparameters for the datasets MUSIC and LABELME with real-world annotators: Best and second best performances are highlighted per annotator set and evaluation score.
Ground Truth Model
Ground Truth Model NLL 4
Parameter
Value
ACC1 1 NLL +
BS
ACC
BS
LABELME
Detault: MaDL
0.743:0.020
0.877+0.034
0.001
0.867:0.004
0.623 0.118
0.214:0.009
Embedding Size
Q =
0.742#0.011
0.907
0.378=0.010
0.852#0.010
0.728=0.18 146
0.280=0.018
Q= R R= 32
0.724+00029
0.889+0.067
0.388#0.029
0.856#0.012
0.9.0=0.250
0.242+0.0026
= 0.1
0,200 101
5.652 197
0.305
0.139:10.055 065
10.514.15,13 138
1.483:0.331
Prior
7
0.73410.027
0.905
0.377
0.851
0.882-10300
0.248
0.9
0.736++0.005
0.90740:047
0.396=000000
0.04
1.00==0,277
0.246+000012
Outer Product
False
0.72510.030
0.806100.06
0.39410.034
0.86011.003
1.01410.221
0,238 1.0006
Residual Block
False
0.731+0.015
0.911#0.065
0.387
0.852:0.006
0.121
0.249:0.022
No Weights
0.73610.614
0.85710.033
0.377
0.87011.008
0.117
0.21110.014
1.118,
0.236
0.731110.026
0.922.1::071
0.081
0.853:100011
1.077:1..218
0.253:10.018
1,226, 0.452
0.722:40.015
0.948=0.088
0.399=0.023
0.852#0.011
0.902#03316
0.246+000027
Gamma Prior
1.5, 8 = 0.5
0.748+0.009
0. 0.869:+00077
0.372=0.025
0.86-1 001
0.685=0.115
0.222 0101
1.56, 0.28
0.722:50.018
0.84710.026
0.301
0.860:10.012
0.67810.211
0.225:10.019
2.22, 8 0.61
0.716110.006
0.947.10.068
0.405
0.857
0.785
0.236.10.013
The above analyses show that the chosen defauut hyperparameters do not always give the best results but are competitive for most datasets studied. They also indicate that having a (small) validation set with GT labels is required if maximum performance is crucial. Obtaining such validation set in setting with error-prone annotators can be expensive. Therefore, future research may examine methods to design such validation sets cost-cfficicntly. In this context, it is also essential to claborate further on the thcoretical foundations of multi-annolator supervised learning techniques, e.g., by deriving theoretical guarantees for specific GT and AP distribution types.
D Extended Results regarding Research Question 3
Table 11 extends Table S by the AP models' results regarding the 75 annotators providing class labels for training. The GT models' results of both tables are identical and are given to avoid switching between the two lables. The additional AP models' results confirm the main lakeaway for properly P5. Excluding the UB, MaDL(A), which uses annotator features including prior information, makes the most accurate AP predictions. This means it also outperforms its counterpart MaDL((), which has no prior information about the annotators. LIA(A) estimates the APs more accurately than LIA(A) for three of the four datasets. Comparing the AP models of CoNAT(A) and CoNAT.(A), performance gains of using annotator features are observable for two of the four datasets, while there are no notable differences for the other two datasets.
E A Case Study on Varying Annotation Ratios
This appendix presents case study on the impact of varying annotation ratios on multi-annotator supervised learning techniques' performances. Fig. Fig.geen displays evaluation results for four different. annolation ratios, i.e., 0.2, 0.4, 0.6, and 0.8, as curves for the dataset CIFAR10 with the simulated annotator set INDEPENDENT. For better interpretation of the results, the upper right plot displays the means and standard deviations over five runs for two descriptive statistics of the training data. On the one hand, we calculate the annotation accuracy (ANNOT-ACC), and the other hand, calculate the accuracy of the observed annoaations
33

---

Published in Transactions on Machine Learning Research (09/2023)
Table 10: Ablation study on MaDL's hyperparameters for the dataset LETTER and four simulated annotated
sets: Best and second best performances are highlighted per annotator set and evaluation score.
Ground Truth Model
Annotator Performance Model
Parameter
Value
ACC 1
NLL 4 BS 4
ACC 1 NLL 4 BS < BAL-ACC t
LETTER (INDRPENDENT)
Default: MaDI
0.935=0.000
0,303:0 102 0.098100010
0.766:10.00
0.491:0.007
0.317:0.005
0.702 000
Embedding Size
Q = R
0.937=0.001
0. 0,097=00000 0.285+0.116
0.773=0.003
0.476+0.000
0.309+0.000
002
0.708+0.000
Q = Rt = 32
0.935=0.007
0.327:0.122
0.099:000011
0.76322.000
0.501 0.05
0.322 0003
0.698 0us
4 0.1
0.587=0.048
7.686. 1.26
0.766:10.095
0.636.10.018
1.194110.33
0.554.10.037
0.614410.015
AP Prior
7 0.7
0.938=0.005
0.273:10.047
0.095:1:.007
0.767
0.487:1:0.033
0.814:10:002
0.702:1-0.00 004
n 0.9
0.935=0.007
0.250+20.022
0.096+t0.00 0009
0.768:00.00
0.486:+00000
0.315:+0.00 002
0.703=0.000 008
Outer Product.
Fulse
0.933-0.010
0.365
0.103:10.018
0.765.10.003
0.499.10.0112
0.321 1.0006
0.701 0.701 1:00004
Residual Block
Fulse
0.935=0.003
0.326440. 131
0.099=00.007
0.762:0.000
0.513:+0.011
0.326=00.055
0.699-10.000 004
No Weights
0.909
0.302:10.097
0.094:0.011
0.7671100002
0.491
0.316.1
0.702.1 002
1.118, 0.236
0.285.10.005 0.937=0.005
0.096:10.008
0.768-10.000
0.4844:0.002
0.314:00.002
0.701
1.226, 8 0.452
0.936=0.003 003
0.307+00.08
0.097=00008
0.767=0.000 004
0.486:+0.000 006
0.315=#00003
0.702= 006
Gamma Prior
= 1.5, 8 0.5
0.910=0.002
0.271+
0.092:00004
0.767 003
0.189:00 006
0.317 003
0.703+0.00 004
1.56, 8 0.28
0.936=0.0000
0.266:100046
0.095:10.006
0.767:0.001
0.491:000000
0.316:10.002
0.702:10.000
a 2.22, 8
0.61
0.936
0.283:10.048
0.097:10.005
0.766-10.003
0.493.1..005
0.318:10.002
0.703.10.002
LETTER
(CORREI,TTE)
Default: MaDI,
0.947=0.003
0.262.10.077
0.050:10.004
0.887100000
0.308 1.00810.004
0.175.10.002
0.756 0.0601000000
Embedding Size
Q = R = 8
0.953=0.003
0.219:00017
0.072:0.004
0.888:00000
0.303
0.172: 002
0.757 002
Q 32
0.949=0.005
0.303:10.008
0.080:10.008
0.885:10.000
0.324:10.00
0.178:10.000
0.754:10.00 0.02
n== 0.1
0.588=0.059
9..12242 241
0.773=00160 166
0.782#0.033 031
0.8914:0
186
0.377=0.062
0.659+0 030
Prior
n 0.7
0.947=0.00m
0.292:::0.00 080
0.078:::0000
0.887 002
0.309:4:0.00
005
0.1444:0.000
003
0.756: 003
n 0.9
0.948=0.003
0.255 079
0:079::0:006
0.887=0.00
0.311
001
0.115 002 0.757+0 002
Outer Product
False
0.948=0.003
0.30544:0
109
14:000500005
0.88740.00
0.314:40.005
00.177:0.002
0.757-+40.01
Residual Block
False
0.936_0.016
0.541
0.1011:0.031
0.88110.005
00.18510.00
0.75110.000
No Weights
0.916=0.007
0.293:
092
0.083:::0100
0,88320.00 003
0.314=#0.000
0.178:#0007
0.751:+0.000
1.118, 8 = 0.236
0.950=0.004
0.260=+0.089
0.077=:0.007
0.887=0.00
0.308:+0.000
0.174440:003
0.758=00001
Gamma Prior
1.226, A 0.452
0.931=0.000 001
0,250+
0,075=00000 006
0.888=0.000
0.309+00
005
0,174 003
0.758=0.000
1.5, 8 0.5
0.951=0,005
0.271 093
0.075
0.887=0.001
0.311+:0 003
0.174:::.002
0.757:10.002
2
1.56, 8 0.28
0.947
0.286.10.055
0.082_10.008
0.885110.002
0.318.10 013
0.177.10.000
0.754110.00
2 0.61
0.947=0.000
0.259:::0.02
0.082:1:0003
0.885:10002
0.313::00000
0.77:000001
0.754:10.003
LETTER (RAND (BANDOOM-CORRRELAAAAA
Defanlt: MaD1.
0.932
0..2771:0.04)
0.101110:006
0.940:10.00
0..0041.0.00 .004|
0. 001000000001
0.619:10.01
Embedding Size
Q= = It 8
0.935=0.003
0.23210.012
0.097::0.004
0.940:0.0000
0.202
0.10110.0000
0.519:10.00 000
Q = R = 32
0.935=0.003
0.266.10.007
0.096:00.007
0.939-10.00
0.214110.003
0. 0:000000000
0.519:10.000
AP Prior
7 = 0.1
0.398=0.069
7.53774:1.90 9.22
0.744::0.13
0.931=0.000
0.251:::0 012
0. 0.120:10.001
0.512:+0.00
7 0.7
0.934_0.009
0.263.10.033
0.098.10.012
0.940.10.000
0.204110.003
0.101110.000
0.519100.000
n = 0.9
0.932=0.008
0.247::00015
0.101++00.00
0.939:10.00
0.205:40.000 002
0.102
0.519:20.000
Outer Product
False
0.932=0.003
0.292:40 038
0.
0.10:::0:006
0.939=0.000
0.207 004
0. 0.1002:0.001
0.519#0.000 000
Residual Block
Fulse
0.936-0.005
0.269 0.34
0.098-1-0006
0.939-10.000
0.207:10.001
0.000010.000
0.518110.000
No Weights
0.548=0.037
1.902:00.24
0.673::0.071
0.801 0.049
0.423 0.37
0.265=10.031
0.506:10.00
1.118, 0.236
0.935_0.001
0.270.10.05
0.09710.004
0.940
0.20410.0011
0.101110.000
0.51910.000
Gamma Prior
1.226, 8 0.452
0.934=0.002
0.255 039
0.097::0.005
0.910=0.000
0.204
0.101=0.000
0.519:100000
= 1.5, 8 0.5
0.935.0.000
0.252.10.047
0.098110.009
0.9401100000
0.204.10.00 003
0. 0.101110.001
0.51910.000
1.56, 8 0.28
0.932=0.004
0.306:10.087
0. 103
0.009=0.000
0.210::0.003
000
0.519
a= 2.22, B 0.61
0.932=0.000 005
0.279:+0.04 048
0.10+000000
0.939= 000
0.209++0.00 003
0. 001:+000000 001
0.51940.000 000
LETTRR (INDL:CTIVE)
Default: MaDL(A)
0.914=0.004
0.303:1..000
0.124::0.006
0.71810.004
0.583:.0.00 007
0.38310.004
0.665:10.005
Embedding Size
Q = it = 8
0.919-0.004
0.28710.018
0.120:0.005
0.704_10.033
0.56910.013
0.38410.012
0.645.10.012
Q m 32
0.916=0.005
0.330++00041
0.22::000006
0.71240.000
0.632:40.022
0.308-10.012
0.657-40.008
Prior
7 0.1
0.642_0.065
1.972
0.63010.117
0.65210.017
0.999110.144
0.525
0.605:10.015
n 0.7
0.912-00006
0.332:10.003
030
0.128:10.009
0.704_10.004
0.588:10.000
0.390.1:0000
0.650
n 0.9
0.913=0.009 003
0.306::0.02
0.20:000003
0.717=0.000
0.582:+0.000
000
0.383
0.666-20.000
Outer Product
False
0.914=0.008
0.1034+0.15)
0.127:00001
0.725=0.003
0.5964+0.013
0.381#0.001
0.676=0.003
Residual Block
False
0.92120.003
0.286:10.002
0.116:0.006
0.721:0.004
0.503:40.033
0.378
0.668820.004
Weights
0.915_0.004
0.301
0.12410.003
0.71910.007
0.577
0.377 0.06
0.66310.007
1.118, 8 = 0.236
0.917=0.006 0.308:10.033
1.3:0000000
0.718:10.004
0.593-10.015
0.3844:0.006
0.667-10.006
Gamma Prior
1.226, 0 0.452
0.918=0.000
0.298:+0.02
0, 0.121:0.002
0.719=0.000
0.583+0 016
0.38140.00
0.660+0.000
1.5.88 0.5
0.914=0.000
0.304110.006
0. 1.00
0.71610.007
0.581
0.083
0.663-100000
a
1.56, 8 0.28
0.910 000
0.341 059
0. 132+ 007
0.711
0.588 009
0.389 007
0.659+0 000
2.22, 9
0.917=0.007 0.331 072
0.121::00000
0.586 0.714=#0007
015
0. 0.385
0.660 005
aggregated via the majority rule (MR-ACC). Mathematically, both statistics can be expressed as follows:
N
ANNOT-ACO(y, Z) = Z
8 (1m = kmm),
(42)
n= mn m
1
MR-ACC(y,2) N
Σ (Yn Zn),
(43)
34

---

Published in Transactions on Machine Learning Research (09/2023)
Table 11: Additional results regarding RQ3 for datasets with simulated annotators: Best and second best performances are highlighted per datasct and evaluation score while excluding the performances of the UB. In contrast to Table S
the AP models' results refer to the 75 annotators providing class labels for training.
Ground Truth Model
Annotator Performance Model
Technique
P5
P6
ACC 1
NLL 4
BS 4
ACC 1
NL. $
BS +
BAT,-CC t
LETTER (INDUCTIVE)
UB LB
0.962:10.00 0002
0.129:10.003
0.058
0.729-0.0044
0.565-10.010
0.369-10.000 0.005
0.677=0.004
0.861 005
1. 1.000+000017
0.429 0.00
0.572=0.008
0.719+0.008
0.515+0.006
0.532=0.004
LIAGA)
0.875.10.000 006
0.901 100.06000
0.350
0.62210.002
0.71210.015
0.495 1.0007
0.593-10.003
LIA(A)
/
0.876:10.000
1.006:10.11 177
0.397
0.618:10.025
1.441110.922
0.58 0. 129
0.572=00038
CoNAL(A)
X
X
0.875::00.00 009
0.804 119
0.186:00.011
0. 0.646:00000 000
0. 0.609
1..555+0.000
0.595=0.001
CoNAL(A)
0.871=#0.000 007
0.808
0.186+#00011
0.616=0.000
0. 0.602+0.015
0. 0.153#0.000 0.00
0.595=0.001
MaDL(A)
X
0.911#0.008
0.331#0.026
0.129:0.008
0.685#0.011
0.403=0.009 0.601+0.005
0.626=0.012
MaDL(A)
0.914400004
0.303:0.009
0.12440.005
0.718-0.004
0.5832000006
0.38322.004
0.665=0.004
PMNIST (INDUCTIVE)
UB
0.909::0.00 002
0.216:0.005
0.131::00003
0. 0.759::0.001
0. 0.18620.00 0022
0.920
0.693=0.002
LB
/
0.881 0.000
0.876.10.005
0.370.10.005
0.6211:0002
0.662100.06
00.006
0.562:10.009
LIA(A)
0.852+00 003
1.011=0.020
0.436+0.010
0, 0.657=0.022
0.611+0.013
0. 0.003
0.589=0.015
LIA(A)
V
/
0.855:4 002
0.972440002
0.417 006
0. 0.001
0. 0.623+0.002 023
U. 0.032
0.610=0.022
CoNAL(A)
x
x
0.880# 002
0.322 005
0.163 003
0. 0.705+00009 000
0.547+0.010
0.372+0.000
0.653=0.010
CoNAL(A)
0.890.10.00 002
0.323.10.011
0.163.10.000
0.713100.011
0.54010.011
0.366100010
0.661
MaDL(A)
0.895.100000 002
0.29710.004
0.152:10002
0.753:10.003
0.496:10.004
0.32810.000
0.683=0.005
MaDL(A)
J
0.898:20.004
0.297:0.0008
0.158:+0.000
0.755#0.003
0.492:0.004
0.325:0.003
0.686=0.005
CIPAR10 (INDUCTIVE)
UB
0.931:50.00
0:527::0022
0.122:10.00
0.71210.00
0.60550.002
0:37210.002
0:635=0.001
LB
/
0.781#0.000 003
1.054+ 035
0.417=00001
0.584 0.006
0.682#0.00 0.00
0. 0.000
0.53-100.004
LIA(A)
X
0.798.10.00 0.008
1.072.10.014
0.455.1.0.006
0.598110.013
0.676110.014
00.013
0.547
LIA(A)
/
0.80411
1.056:0.022
0.447::00011
0.60244.000 0.02
0.673+0.017
0.479
0.549=0.014
Co
X
0.835 0.00
0.576110.016
0.245.1..00
0.670:10.002
0.599 1.0002
0.415:10.002
0.618=00.001
CoNAL(()
0.6341100.00
0.674
0.248:1.0007
0.670-10.001
0.500-10.001
0.4 0.415110.001
0.618=2.0022
MaDL(A)
0.811##0.00 0088
0.626::0.033 003
0.277+0.001
0.690
0.005
0.006
0.624=0.003
(ADL(A)
0.837.10.00
0.55710.028
0.242 0..42.10.000
0. 0.703400.04
0..011
0.308.00.005
0.640.0.005
SVHN (INDUCTIVE)
UB
0.965:10.000 0.000
0.393:10.015
0.063:1:0.00
0.650:200000
0.589110.000
0.41016.002
0.578=0.004
LB
/
/
0.927.10.00
0.805.10.016
0.328.10.00
0.509
0.70910.005
0.513
0.531
LIA(A)
X
x
0.929:10.00
0.03
0.818 133
0.336:1.0.0 068
0.548:10.00 0.043
0.695
0.502:10.027
0.528=0.009
LIA(A)
/
0.932:10.000
0.754::0.122
0.30311.007
0.000
0.672::0.027
0.480:10.024
0.539=20.004
CoNAL(A)
X
0.941#0.000
0.258=000009
0.090+0.008
0. 0.627#:0.m 005
0. 0.608 002
0. 0.42880.00m 0000
0.531=0.000
CoNAL(A)
0.942:10.001
0.260::0.012 .12
0.090+0.002
0.632=0.005
0.605=500003
0. 0.005
0.556=0.007
MaDL(A)
X
0.928::0.000 002
0.299=t0.019
0.109:::0000
0.63 1.631=:0.00 :00004
0.599920.006
0.418200.022
0.548=0.002
MaDL(A)
0.935#40.001
0.256#0.009
0.098+0.000 002
0.641#0.002
0.592#0.001
0.413+0.001
0.562=0.002
where z = -31....,, E S2 denotes the vector of observed annotations aggregated per instance via the majority rule. The ANNOT-ACC is almost constant across the different annotation ratios, as the selection of which annotations are observed is made randomly. In comparison, the MR-ACC increases because, on average, the amnotators are independent and better than random guessing. Therefore, the probability of correctly aggregated annotation increases with more observed annotations per instance.
The curves in the other seven plots report multi-annotator supervised learning techniques' mean evaluation scores and standard deviations over five runs. The evaluated MaDL, variant corresponds to its default hyperparameter configuration. The GT models' evaluation curves in the first row of Fig. g confirm our intuition that more annotations lead to better results. Further, we observe superior performances of the UB regarding ACC and BS, while the UB repeatedly provides inferior NLL scores. Since the NLL per instance is unbounded, the latter observation is likely caused by highly overconfident predictions. As expected, the GT model of the LB performs worst in terms of the ACC for each tested annotation ratio and thus confirms the general usefulness of multi-annotator supervised learning. However, its results for the NLL and BS are surprisingly better than LIA, possibly due to the difficulties of training via an EM algorithm (ef. Section 3 The GT model of MaDL outperforms other multi-annotator supervised learning techniques regarding the ACC and BS results, while it is approximately on par with CoNAL regarding the NLL results. As the annotation ratio increases, the ACC of MaDL approaches that of the UB. Still, MaDL cannot reach the ACC results of the UB, even for the highest tested annotation ratio of 0.8. This is because the negative impact of annotation noise is problen-dependent (Gu et al., 2022),, e.g., the impact may depend on the data distribution. The annotator simulation is an additional impactful aspect in our concrete case. For CIFAR10,
35

---

Published in Transactions on Machine Learning Research (09/2023)
the negative impact of annotation noise is more significant than for the other datasets, particularly evident in the large gap in the ACC between the LB and the UB. The AP models' evaluation curves in the second row of Fig. | show similar trends to the ones of the GT models. In particular, there are two noteworthy aspects. First, the AP estimates of MaDL are close to the UB. Second, the AP estimates of the LB are better than the ones of the other multi-annotalor supervised learning techniques (excluding MaDL, and UB) for annotation ratios of 0.6 and 0.8. The well-performing AP model architecture adopted from MaDL, likely leads to these results.
++ CB
UNION
+
LB
LIA
+
CL
CONAL
REAC
MaDL
+ MR-ACC -4- -NNONT--GG
Multi-annotator Supervised Learning Techniques
Training Set Statistics
100
0.40
000
0.3
Anactation Ratios
Annotation Ratior
Annotation Racior
Annotation Ratier
000
0-0.30
40 0.34
AP-PAA
0.9
0.34
0.0
0.3
002
0.94
Anactation Ratios
Annotation Ratica
Annctation Racioa
Annctation Ratica
Figure 9: Evaluation results for four different annotation ratios CIFAR10 (INDEPENDENT).
F A Case Study on CIFAR100
This appendix presents a case study analyzing multi-annotati supervised learning techniques' performances on classification tasks with more classes than in the main experiments. For this purpose, we use the dataset CIFAR100, which is similar to the CIFAR10 dataset (cf. Table 2 with the major difference that C' = 100 classes occur (Krizhevsky) 2009). The general design of the ex experiments follows that RQ1 (cf. Section 5. except that we adopt the wide residual network (WRN-28-10 Zagoruyko Komodakis 2016) as the basic network architecture and stochastic GD the optimizer with mini-batch size of B 128. For fairer comparison, perform small grid search with the learning rates (0.1, 0.01} and weight decays {0.0005, 0.0) for each technique. Thereby, we reduce the learning rate with cosine annealing (Toshchilov & Hutter 2017 over the 100 training epochs.
Table presents the obtained GT and AP models' test performances, where the evaluated MaDL variant corresponds to default hyperparameter configuration. As expected, training with GT labels as the UB achieves the best performance in terms of GT and AP estimates. The performance gap between the UB and other techniques is substantial, highlighting the challenges of learning the CIFAR100 dataset with partially erroneous annotations. However, the benefits of employing multi-annotator supervised learning techniques are also significant. This advantage becomes particularly apparent when comparing MaDL with the LB be- cause MaDL improves the GT-ACC by approximately 19 %. The comparison with the other multi-annolator supervised learning techniques further confirms the state-of-the- that only LIA and REAC appear to roughly learn the classification task among the related techniques. A conceivable explanation might be that the default values of the hyperparameters of CL, UNION, and CoNAL are only well-suited for datasets with fewer classes.
results MaDL for CIFAR100. We note
36

---

Published in Transactions on Machine Learning Research (09/2023)
Table 12: Results on the CIFAR100 dataset with simulated annotators: Best and second best performances are highlighted per evaluation score while excluding the performances of the UB.
Ground Truth Model
Annotator Performance Model
Technique
ACC 7
NLL +
BS 4
ACC 1
NLL 4
BS +
BAL-ACC
100 (INDRPRNDENT)
UB
0.795 3 0.001
0.818 2 0.008
0.297
0.003
0.727 1 0.002
0.530
1 0.001
0.355
0.001
0.665 1: 0.003
LB
0.430
0.010
2.593
1 0.038
0.762
0.012
0.599 1 0.015
0.665
0.006
0.472
0.005
0.577
1. 0.011
CL
0.U77 Σ 0.013
10,555
4 0.802
1.616 1 0.059
0,394 1 0.000
1.466
4 0.013
0.764 + 0.001
0.500 + 0.000
REAC
0.553 0.006
2.037
1. 0.175
0.630
0.023
0.629 1 0.029
0.657
0.071
0.465
0,060
0.543 1: 0.035
UNION
0.033 + 0.003
4.815
= 0.067
1.031 + 0.011
0.594 + 0.000
1.838 + 0.008
0.791 + 0.001
0.500 = 0.000
LIA
0.560 1 0.008
1.995 1 0.051
0.619
0.009
0.672 1 0.004
0.608 1 0.003
0.417
0.003
0,611 1 0.006
CONAT
0.168 0.008
7.762
+ 0.344
1.283
0.016
0.594 1 0.000
1.209
0.021
0.700
0.003
0.500 + 0.000
MaDL
0.621 1 0.003
1.628
+ 0.011
0.508
0.005
0.665 1 0.006
0.646 1 0.015
0.011 0.442
0.628 1: 0.003
G Practitioner's Guide
This appendix provides an overview of various aspects to consider when employing MaDL in practical classification tasks. Note that these aspects are closely related to each other and that the associated recom- mendations do not have general validity.
Data modalities: We can train MaDL with different modalities of data. Depending on the instances" data modality, we need to modify the GT model's architecture. For example, we may use TabNet (Arik & Pfister 2021) for tabular data, ResNet (He et al. 2016) for image data, or BERT (Devlin et al. 2018) for text data. These considerations apply analogously to the annotator features and the architecture of the AP model. However, annotator features are commonly tabular since they are collected via surveys, or only anonymized identifiers of the annotators are known if there is no prior annotator information. In the latter case, one-hot encoding converts these identifiers into tabular annotator features. Recommendatton: Use common architectures from the literature to fit the data modality.
Number of instances:
The number of annotated instances available for training is critical to DNNs gen- cralization performances (Hestness et al. 2017] and, consequently, to MaDL's generalization perfor- mance. Typically, the more annolated instances we have, the better MaDL can learn the underlying annotation patterns as a basis for inferring the GT class labels and APs. Learning curves allow us to study this behavior (Hoiem et al.) 2021 Fig. 10 shows such exemplary learning curves of MaDL's GT-ACC and AP-ACC compared to the LB and UB for the dataset LETTER. Each curve represents the means and standard deviations over five runs. The trend of improving performance with an increasing number of annotated instances is confirmed for both evaluation scores. Further, observe that MaDL improves upon the LB, even for smaller training sets. Our findings, coupled with the case study on varying annotation ratios in Appendix F. suggest that an increase in both the number of annotated instances and the number of annotations per instance generally contributes to improved results. Recommendation: If the anmolation budget allows, increase the number of annotated instances for better results.
UB
LB
MaDL
07
0.8
GT-ACC-1
AP-ACC
9.6
041
0.5
012 1000 2590
1024 3354 160e Number of Annotated Instances
Number of Annotated Instances
Figure 10: Exemplary learning curves on the LETTER dataset.
37

---

Published in Transactions on Machine Learning Research (09/2023)
Number of classes:
With an increasing number of classes, there is growing chance that some classes will be easy to separate and some classes will be highly confusable (Gupta et al., 2014). Detecting such pairs of confusable classes requires the accurate estimation of annotators' confusion matrices, whose concept is illustrated in Fig. [1 The three matrices represent three variants of MaDL with varying assumptions about APs (cf. property P1 in Section 3 The class-independent variant corresponds to estimating a single scalar as the only degree of freedom, i.c., v 1, to build the entire confusion matrix. In our example, this scalar corresponds to 0.80 as an annotator's correctness probability. Accordingly, this variant cannot distinguish between easy- and difficult-to degree of freedom of the partially class-dependent variant equals the number of classes, i..., V == C, because there is an individual correctness probability for each class. These probabilities build the diagonal of the confusion matrix, which consists of the values 0.60, 0.80, and 1.00 in our example. Only the fully class-dependennt variant can capture pairs of highly confusable classes. As a result, the degroe of freedom considcrably higher with v = C. (C - 1). On the one hand, our experiments show that this variant. is robust. across different. datasets (ef. Section 5), and even works for dalasets with C' = 100 classes (ef. Appendix F. On the other hand, the complexity increases quadratically with the number of classes, so this variant is computationally costly for datasets with hundreds thousands of classes. Note that MaDL can estimate these confusion matrices not only annotator- dependent but also instance-dependent. Recommendation: If no assumplions about the annolotors are known, and the computational effort is feasible, use the fully class-dppndden MaDL variant.
classes. The
Class-independent (I)
Partially Classddependent (P)
Fully Class-dependent (F) C. (C
C m 3
2
1)
3 2
6
Annotated Class
Annotated Class
Annotated Class
GT Class
a
x==
2=
GT Class
2 ==1
==1|== ==2 2
2
GT Class
am 1
2 2
2 3
==1 I
0.80
0.10
0.10
y=
0.60
0.20
V ===
0.60
0.05
y 2
0.10
0.80
0.10
y ==
0.30
0.80
0.10
y
0.20
0.80
0.00
v=3
3
0.10
0.10
0.80
y = 3
0.00
0.00
1.00
y = 3
0.00
0.00
1.00
Figure 11: Illustrative confusion matrices for C classes.
Number of annotators: As the number of annotators increases, the AP model usually needs to learn more varying annotation patterns. For this, the AP model's size must be sufficiently large, e.g., by incrcasing the annotator embeddings' dimensionality. Morcover, scttings with many annotators, prior information about the annotators can be particularly helpful for learning correlations ween them. Recommendation:: Ensure that the AP model has a sufficient size to learn onnotation patterns.
Training, validation, and testing: In principle, MaDL can be trained without validation or Lest set, demonstrating robust results with default hyperparameters across diverse datasets. Nevertheless, like most DNNs, careful selection of the optimizer's ' hyperparametors, such as the learning rate, is essential. Without a validation set, the training loss curve can serve as a rudimentary guide to assess if the loss is being minimized effectively. Still, better results typically stem from hyperparameter tuning using a validation set. Next to popular regularization techniques, e.g., dropout and weight decay, such a validation set is also essential to detect and avoid overfitting. Finally, a separate test set allows us to assess MaDL's final performance reliably. The setting of error-prone annotators makes obtaining validation or test set challenging. Ideally, we obtain the GT labels from experts. Alternatively, may assume the majority votes of numerous annotators as GT labels. Recommen- dation: For hyperporameter tuning, avoidance of overfitting, and a reliable performance assessment, consider acquiring a validation and test set before deploying MaDL.
Deployment: After training, MaDL's GT and AP models offer various deployment options: We can
directly apply the GT model as a downstream classifier to the associated learning task,
use the GT model's predictions to improve the dataset's 's label quality,
study the annotation patterns and correlations between annotators via the AP model,
or leverage the AP model to assess which annotator is best for annotating a particular instance.
38


Reviews:
[{"review_id": "YjFxYvqwrT", "reviewer": "Reviewer_HnGV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this work include:\n\n-\tThe paper is well-written, and the figures are clear and easy-to-understand \n-\tSection 3 provides a comprehensive review of the existing multi-annotator supervised learning techniques including GT and AP models.  Table 1 is impressive as it summarizes the past works’ predictability on the six annotator properties\n-\tThe authors also benchmarked the framework’s performance on public datasets against other approaches such as majority vote, ground truth, union set and etc. MaDL performance well in terms of accuracy, balanced accuracy, and negative log-likelihood\n-\tThe authors provide detailed mathematical equations for each new concept introduced ranging from metrics to loss function, which makes the paper theoretically sound \n\nThe weaknesses of this work include:\n\n-\tBoth instance and annotator features are crucial pieces in the framework. As the author stated, annotator features should contain prior knowledge. However, the authors did not elaborate on how the features are generated or obtained in the case studies. In a real-life annotation case, we could measure annotation time, and potentially collect annotator age or interest from surveys (annotator features in Figure 1). Still, some features could be hard to obtain due to privacy or resource constraints.\n-\tCertain datasets have a smaller amount of instances and a larger number of classes (for instance, LABELME). Could overfitting be an issue for those use cases? The authors should further elaborate.  \n-\tThe authors answered a few research questions on how to improve the AP or GT model. But how can their predictions be used in removing low-quality labels and training downstream classifiers? The paper barely touches on the applications of AP/GT models. It seems unclear how the framework could help boost the performance of downstream models. \n-\tThe scope of this paper is limited to classification tasks in a relatively low dimension (<30 classes). How could the framework scale to high dimensional classification problems (> 100 classes)? In addition, in real life, more tasks using crowd workers are on complex annotations such as keypoints, bounding boxes, and image segmentation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The author proposed a probabilistic training framework named multi-annotator deep learning (MaDL) which contains one model to predict the ground truth (GT) labels and one model to estimate annotator performance (AP). The framework could be applied to complex classification tasks requiring crowdsourcing, and multiple annotators label each instance. The AP model provides predictions on six properties including class-dependent, instance-dependent annotation performance, annotator correlations, robustness to spamming, etc. The training procedure is end-to-end. The authors also tested the efficacy of this framework on public classification datasets including images, audio, and tabular data. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n-\tElaborate on what features are suitable or feasible to obtain for both instances and annotators \n-\tExplain whether the overfitting could be an issue in the test cases\n-\tExtend the studies on applications of MaDT to improve labeled dataset quality and downstream classifier training \n-\tDiscuss how the framework could be extended to complex annotations such as high dimensional classification, keypoint, bounding boxes, segmentation, and so on", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SFkUJqOa3t", "reviewer": "Reviewer_YPct", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I learned a lot from the related work section. The related work in multi-annotator deep learning is split into six categories: class-dependent annotator performance, instance-dependent annotator performance, annotator correlations, robustness to spamming annotators, annotator prior information, and inductive learning of annotator performance. Table 1 is quite helpful. \n\nI really appreciate the “takeaway” blobs right under the research questions. \n\nThe paper is generally sound and detailed. The research questions and takeaways are clear. \n\n---\n\nA general comment: What’s the takeaway for general researchers or practitioners? What circumstances would the approach work well / would the approach be more useful in the real world? \n- For example, what if the number of classes is very large? \n- What if the number of annotators is very small, or very large? \n- Would the approach be more or less useful when there are a lot more examples in the dataset? What if there are only a small number of examples in the dataset? \n- Would the approach extend onto sequence generation tasks (e.g., text generation, where at each time-step, we choose one word out of the vocabulary of, say, 10k possibilities)? If not, what are the difficulties?\n\nAn unclear assumption in Section 2: “We further assume that there is a subset of annotators whose annotated instances are sufficient to approximate the ground-truth label distribution.” It’s not clear to me what the authors mean by “sufficient” – does the sufficiency depend on the class of algorithms?  \n\nPotential issue on the derivation of Proposition 2: how did the authors go from the third-to-last line [ delta ( \\sum_y p(y|x) p(y|x,a,y) < \\sum p(y|x) (1 - p(y|x,a,y)) ) ] to the second-to-last line [ delta ( \\sum_y p(y|x) p(y|x,a,y) < 1 - \\sum_y p(y|x) p(y|x,a,y) ) ]? If p(y | x) p(y | x, a, y) is between the right-hand-side (of the inequality) of the third-to-last line and the right-hand-side of the second-to-last line, would the equality still be true? Please let me know if I’m missing something. \n\nIn Equation 14 and Equation 15, what is e? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate multi-annotator deep learning (assuming noisy class labels given that the annotators could produce errors due to various reasons like expertise and carelessness) by proposing the approach where a ground-truth model (predicting the correct label of the classification problem) and an annotator performance model are jointly trained end-to-end using a weighted maximum-likelihood approach.  \n\nThe training in prior work usually uses an EM approach, where the ground-truth labels are estimated during the expectation step, and the ground-truth model and annotator performance models’ parameters are optimized during the minimization step. Variational inference and expectation propagation are also used in past approaches. The authors’ approach satisfies all six proposed criteria (for the annotator performance model), but prior approaches don’t. The authors’ approach relies on two models as well: the ground-truth model and the annotator performance model. The authors only consider classification, so the two models can be modeled as two categorical distributions as shown in Equations (6) and (7). \n\nFigure 3 (on page 8 in the submission version) illustrates the model architecture. The architecture is very clear. The ground-truth model takes in the input (of the example to be classified). The annotator performance model takes in one or optionally two things: the first thing being the annotator embedding; the second thing (optionally) being the instance embedding (of the given example – either using the original instance feature or using the learned instance representation from the ground-truth model). The output of the ground-truth model and the output of the annotator performance model are then multiplied (by dot product) into the final annotation probability. \n\nExperiments are done on seven datasets: Toy, Letter, Labelme, Msic, FMNIST, CIFAR10, SVHN. Two (LabelMe, Music) datasets are using real-world annotators. The rest of the datasets are using simulated annotators. \n\nThree research questions:\n\n(1) Do class-dependent and instance-dependent modeled annotator performances improve learning? The answer is yes. Performances of ground-truth and annotator performance models are improved.\n\n(2) Does modeling correlations between annotators (who are potentially spamming) improve learning? The answer is yes, when there are many correlated spamming annotators. But no (at least not consistently), when capturing correlations of beneficial annotators. \n\n(3) Do annotator features containing prior information improve learning and enable inductively learning annotators’ performances? The answer is yes. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Answer the above questions. \n- Include more information on the applicability of the approach (see above).", "broader_impact_concerns": "The statement is present and well-written. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YkYcDD4Iji", "reviewer": "Reviewer_Jzj1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The authors present a nice probabilistic framework for the multi-annotator deep learning problems and provide a good amount of discussion / analysis on what kind of properties people need to consider when developing MaDL algorithms (p1-p6);\n- The mathematical derivations are very detailed and helpful for the readers to understand the paper;\n- The experiment results outperform existing baselines under the same setting.\n\n**Weaknesses:**\n\n- The paper is extremely lengthy. A normal TMLR paper usually finishes in 12-15 pages but the paper is almost twice as long as a regular paper submission. I understood that the authors submit the paper as a \"long submission\", but I still believe that moving a good amount of detail to the appendix might be a better idea. \n- It's still about the writing of the paper. I feel that the authors prefer to directly jump into mathematical formulations and derivations without providing intuition first. This will make the paper very hard to follow. In fact, I will suggest the authors to try summarizing what message each formula is supposed to deliver and express them in plain language before diving into the details. It is also very important to provide concrete numerical examples on how key formulae are calculated (e.g. Eqn 25). It's even better if the authors could provide an overview figure for Algorithm 1. \n- I am curious about the necessity of the AP model in Figure 3. Is it possible for us to use a simple statistical model to measure the reliability of different annotators on different classes? For example, we can just use 1 - error_percentage as the entries for confusion matrices. I feel that the authors gave too little example on how this confusion matrix looks like in real world datasets and how it correlates with the simplest error percentage statistics.\n- Regarding evaluation, it seems to me that the authors are using very small datasets to perform proof-of-concept experiments. While it is usually OK for most theory-oriented papers, it might be problematic for this particular paper. From my understanding, one of the most important motivations for MaDL is that deriving a \"golden standard\" ground truth label might be very costly. But if you conduct experiments on very small datasets, the cost will not be a problem. Besides, it is true that the performance drop of MaDL from UB is slower than other papers in the field, but the gap is still considerably large. Take CIFAR10 as an example, the performance almost dropped from 93.3% accuracy to 85% accuracy, which is the difference between a very simple 3-5 layer CNN and a relatively strong, ResNet-like design. This also makes me question whether the proposed MaDL framework could have any practical value in the current stage.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study Multi-annotator Deep Learning (MaDL) from a probabilistic perspective. Machine learning datasets today are usually crowd-sourced and the annotators might not be reliable. So it is important to make sure that the models can still learn from these ambiguous labels. The framework proposed in this paper learns a ground truth model (GT model) and an annotator performance model (AP model), in an attempt to maximize the likelihood of real ground truth labels. Evaluated on both real world crowdsourced datasets and simulated academic datasets, the authors demonstrate that MaDL outperforms all existing multi-annotator deep learning methods and partially closes the gap between fully supervised learning on real ground truth labels and aggregated crowdsourced labels.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To get this paper accepted into TMLR, it is necessary to restructure the paper and make it more intuitive. Despite being a long submission, 24 pages is far too long and the authors need to consider what part of the main context can be moved to the appendix. Besides, the authors need to justify the importance of the MaDL task itself and the severe performance drop on very small scale datasets such as CIFAR 10.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for explicitly modeling the quality of individual labelers, alongside the training of deep networks, to deal with per-annotator noise.  The authors use this to incorporate the probability of the annotation, along in the predictive likelihood of the label, in the loss.  In the paper, the authors experiment with learning the annotator model using one-hot labels for the annotators or explicit annotator features.  This is a rather long submission for TMLR and the reviewers seemed to find it somewhat dense.  Reviewers found the paper well-written, interesting, and sound.  One reviewer commented that the related work section is quite thorough and educational.  As weaknesses, the reviewers pointed out that they had trouble understanding how the authors proposed to use annotator features, and that the scope of the paper was limited to relatively smaller classification tasks.  The reviewers noted in discussion that the author rebuttal was strong and convincing, and helped to answer their concerns.  Ultimately, all reviewers voted to accept the paper, with two leaning accept and one accept.  Given that the reviewers all agreed on acceptance, found the paper relevant and interesting, and agreed that the claims were substantiated, the recommendation is to accept the paper.
Decision: Accept (poster)