crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels
Authors: Marek Herde
Lukas Lührs
Denis Huseljic
Bernhard Sick
Venue: Accepted by TMLR ()

Abstract: Crowdworking is a cost-efficient solution for acquiring class labels. Since these labels are subject to noise, various approaches to learning from crowds have been proposed. Typically, these approaches are evaluated using default hyperparameter configurations, which often result in unfair and suboptimal performance, or using hyperparameter configurations tuned via a validation set with ground truth class labels, which represents an often unrealistic scenario. Moreover, both setups can yield different approach rankings, complicating study comparisons. Therefore, we introduce crowd-hpo as a framework for evaluating approaches to learning from crowds, together with criteria for selecting well-performing hyperparameter configurations using only noisy crowd-labeled validation data. Extensive experiments with neural networks demonstrate that these criteria select hyperparameter configurations that improve the learning from crowds approaches' generalization performances, measured on separate test sets with ground truth labels. Hence, incorporating such criteria into experimental studies is essential for enabling fairer and more realistic benchmarking.

Published in Transactions on Machine Learning Research (11/2025)
crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels
Marek Herde
Intelligent Embedded Systems University of Kassel Kassel, Hesse, Germany
Lukas Lührs
lukas.luchrs Shanirrsshuiii-ssssl de
Intelligent Embedded Systems
University of Kassel Kassel, Hesse, Germany
Denis Huseljic
dhuseliickhuiiiksssel.dd de
Intelligent Embedded Systems
University of Kassel Kassel, Hesse, Germany
Bernhard Sick
bsick Sarckikkskkkeeeeee de
Intelligent Embedded Systems University of Kassel Kassel, Hesse, Germany
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- Sakfhyl VLK
Abstract
Crowdworking is cost-cfficient solution for acquiring class labels. Since these labels subject to noise, various approaches to learning from crowds have been proposed. Typically, these approaches are evaluated using default hyperparameter configurations, which often result in unfair and suboptimal performance, or using hyperparameter configurations tuned via a validation set with ground truth class labels, which represents an oflen uurealistic scenario. Moreover, both setups can yield different approach rankings, complicating study comparisons. Therefore, we introduce crowd-hpo hpo as a framework for evaluating approaches to learning from crowds, together with criteria. for selecting well-performing hyperparameter configurations using only noisy crowd-labeled validation data. Extensive experiments with neural nerworks demonstrate that these criteria select hyperparameter configurations that improve the learning from crowds approaches' generalization performances, measured on separate test sets with ground truth labels. Hence, incorporating such criteria into experimental studies is essential for enabling fairer and more realistic benchmarking.
1 Introduction
Crowdworking represents popular and cost-efficient solution to label data instances for classification tasks (Vaughan 2018). However, the corresponding crowdworkers are error-prone for various reasons, c.g., missing domain knowledge, lack of concentration, or even adversarial behavior (Herde et al. 2021). Training deep neural networks with noisy crowd-labeled data decreases generalization performance because these networks tend memorize the false class labels (Zhang et al.] 2017]. Hence, many approaches intend improve the robustness against noisy labels. Together, they form the research area of learning from noisy labels (LNL.) with the core topies of regularization, sample selection, robust. loss functions, dedicated neural

---

Published in Transactions on Machine Learning Research (11/2025)
annot-mix-
#40
O #13
mv-
#130
#12
union-a-
#12O
O #11
coin-
#20
#10
madl-
#10
#9
da-
#80
#
conal
#70
O #7
trace
#60
O #
croud-ar
#90
O #5
union-b
#110
#4
# Rank
gea-u-
#100
O
Default (DEF)
geo-f-
#30
O #2
cl-
#50
O#1
Optimized (TRUB)
8
9 10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Zero-one Loss [%]]
Figure 1: Default (DEF) versus optimized (TRUE) HPCs for LFC approaches. The y-axis lists the LFC approaches, and the z-axis the zero-one loss evaluated test with true class labels of the reuters-full dataset (Rodrigues et al. 2017), whose training contains noisy class labels from crowdworkers. Default HPCs result in substantially worse performance than HPCs optimized via validation data with true class labels. Further, HPO alters the approaches' ranking. For example, cl (Rodrigues & Percira, 2018 performs best under default and only fifth-best under optimized HPCs, whereas madl (Herde et al. 2023 moves from the ninth place with the default HPCs to the first place after optimization.
network architectures (Song al. 2022 Within this area, learning from crowds (LFC, Raykar et al. 2010 approaches explicitly handle crowd-labeled data, where each instance receives a (potentially varying) number of noisy class labels and where we know which label originates from which crowdworker. Accordingly, these approaches estimate the crowdworkers' performances (e.g., labeling accuracies) to infer the instances' true (i.e., ground truth) class labels. Many experimental evaluation studies have demonstrated the performance gains of such approaches (Rodrigues & Percira 2018
Chu et al. 2021 Nguyen et al. 2024).
Translating these gains into practice demands effective hyperparameler oplimization (HPO) to find well- performing hyperparameter configurations (HPCs). While approaches for training standard neural networks are tuned against validation set with truc class labels, LFC approaches have no access to such set if all class labels originate from crowdworkers. As a result, HPO becomes a more difficult challenge. Potential workarounds are using data-agnostic default HPCs (Chen et al.) 2021 or explicitly requiring access to a validation set with true class labels (Herde et al.] (023).. We refer to both procedures as hyperparameter selection (HPS) criteria because each chooses HPCs for the LFC approaches. Figure | exemplifies that both HPS criteria lead to different. test. losses per LFC approach and Specifying default HPCs is realistic because it requires criterion often yields suboptimal performance results that are heavily influenced by nonobjective choices such as the experimenters' or software frameworks' presets, undermining fairness (Bagnall & Cawley 2017). By contrast, HPO on a validation set with true labels can produce superior and fairer results when every LFC approach receives the same search budget. Nonetheless, this HPS criterion is unrealistic in an LFC setting where only noisy crowdworkers provide labels. Existing literature lacks HPS criteria to perform experiments for fair (involving НPÒ) and realistic (with access only to crowd-labeled validation data) comparison of LFC approaches. Motivated by these obscrvations and related ones in areas such partial label learning (Wang analyze the following research questions (RQs):
rankings between LFC approaches. true class labels. Nevertheless, such HPS
et al.
2025
We the learning from crowds, whereas other publications research refer multiple annotators
et
2022 labelers (Rodrigues al. 2013) instead of crowdworkers.
2

---

Published in Transactions on Machine Learning Research (11/2025)
crowd-hpo: Research Questions and Contributions
RQ1: Given access only to crowd-labeled validation data, which evaluated hyperparameter selection criterion yields the highest performances for LFC approackes
RQ2: Given the best-evaluatdd hyperparameter selection criterion for crowd-labeled volidation data, how do learning from crowds approaches compore in performance?
Based on these research questions, we propose crowd-hpo for learning from crowds approaches with crowd-labeled validation data contributing:
framework of hyperparameler selection criteria based on empirical risk measures to be combined in a robust ensemble,
an extensive experimental study benchmarking 13 learning from crowds approaches across 5 real-world datascts, each with variants of noisy labels from humans,
a guideline for realistic and fair experimentation to compare learning from crowds approaches performances combination with hyperparameter optimization,
and comprehensive codebas to reproduce and perform experimental studies for learning from crowds approaches in combination with hyperparameter optimization.
htttpp://ggithub.coo//// resiaarcesrrraarr/mmmm/aaaa awwoaaaatttttaaataaooaa mehe www.chhie-aaaamaaa//eeeettoooooo
2
Related Work
This section presents a discussion on foundational and related works on LFC (Raykar et al. 2010) approaches for classification tasks, their experimental studies, and validation with noisy class labels in the broader area of LNL (Song al., )022). In short, this discussion confirms that most experimental studies LFC follow different experinentation protocols, none of which consider HPO with noisy crowd-labeled validation data. Although other works on LNL address validation issues in the presence of noisy labels across various contexts, they do not explicitly validate with crowdworkers of varying performances.
Table 1: Overview of experimental studics of LFC approaches training neural networks for classification tasks. Each row represents one study sorted by publication years, while the columns refer the characteristics of such a study. We denote counts by the # symbol. We account for multiple simulation methods for the same single-labeled and variants for the same crbwe-labeled dataset by (x...). The symbols TTL (True Labels) and /NL (Noisy Labels) denote the validation label type, whereas x indicates that the respective aspect has been ignored. If no information is available, we denote ? as a symbol.
Approaches
[+]] One-stage
Datasets [#]
Hyperparameter Optimization
Study
Venue
Early Stopping
Two-stage
Simulated
Roal
Per Datasel
Per Approach
Rodrigues & Pereira
AAAI
3
4
1
/TL
ml.
(x
Tanno et
CVPR
5
2
(x
2)
VTL
A1.
TMM
6
2
Wes
TNNLS
6
4
(x 4)
L.1 et all.
MLJ
5
4xx 2)
2
TTL
/
al.
TMLR
6
/TL
VTL
Ibrahmi ct
ICLR
8
2
(x
2
/TL
TTL
ml.
SIGIR
Herde ét 6]
ECAT
9
Zhiang el
AAAI
2 (X 4)
TPAMI
XX
Nguven at al.
NeurIPS
5
2
3
NeurIPS
9
13
C/10 ot al.
NeurIPS
7
2
(X
Herde 6t.
NeurIPS
(x 7)
/TL
croud-hpo
TMLR
11
(x 7)
/NL.
/NL
3

---

Published in Transactions on Machine Learning Research (11/2025)
Learning from Crowds Approaches Literature differs between two-stage and one-stage LFC ap- proaches [Li et al. 2022.. Two-stage approaches aggregate the noisy crowd-labeled class labels per instance in the first stage and use these aggregated labels as true class label estimates for training neural networks in the second stage. The most common aggregation algorithm is majority voting (mv), which implicitly assumes equal performances across the crowdworkers (Chen el. al. 2022: Jiang et al. 2022 In contrast, the Dawid-Skene algorithm (ds, Dawid & Skene 1979) leverages the empectationn-aaiiiiaan
(EM) algorithm, where the true label probabilities are estimated in the E-step to update the crowdworkers' confusion matrices in the M-step. Typically, such label aggregation approaches operate with the given labels as only inputs (Zhang et al.) 2016) and expect more than one class label per instance (Khetan et al. 2018). One-stage approaches aim to overcome these limitations by jointly training & neural network for estimating the true labels and a model for evaluating the crowdworkers' performances (Herde et al.
2023).
The latter model is often implemented ed as Chu et al. 2021] or probabilistic confusion
weights of noise adaptation layers (Rodrigues & Pereira 2018
matrices (Tanno ct al., 2019 Chu et al. 2021 Ibrahim ct al.
2023) to model crowdworkers class-dependent
performances. More complex models, designed as (deep) neural networks, estimate performances as a function
of instances and crowdworkers (Zhang et al. 2020 Li et al.
2022
Cao et al. 2023 Herde et al. 2024b).
Experimental Studies for Learning from Crowds For a better understanding of experimenting with LFC approaches, Table overviews and charactorizes recent experimental studics of LFC approaches. Most studies focus on presenting a new LFC approach compared to stale-of-the-art competitors. We report the number of evaluated two-stage and one-stage LFC approaches for each study. Here, we count individual approaches if they incorporate distinct methodological ideas. In addition, we report the number of datasets used in each study. We distinguish between simulated and real crowd-labeled datascts. Simulated datasets are built on top of standard single-labeled datasels, such as cifar10 (Krizhevsky) 2009), by simulating the labeling process of the crowdworkers. For the simulated data, most experimental studies consider multiple single-labeled datasets and multiple simulation methods for the noisy class labels. Analogously, multiple variants of crowd-labeled datasets can be constructed by subsampling the crowdworkers' labels, e.g., by retaining only a certain number of class labels per instance (Wei et al. 2022 Herde et al., 2024a) We take both procedures into account by denoting the product term (# datasets x # variants). Central to our analysis is the handling of the hyperparumelers (HPs) for the LFC approaches. Here, we note the distinction between HPO, which involves systematically searching for the best HPC, and early stopping, a regularization technique that halts training once validation performance deteriorates, thereby preventing overfitting. If the HPO is only performed per dataset, e.g., to select. the basic architecture and optimizer parameters, we set a check mark at "per dataset". If the HPO is only performed to determine values of specific HPs for an individual approach over multiple datasets, e.g., the best value for a regularization term, we set check mark at "per approach". If HPO is performed for each dataset and approach, we set a check mark at "per dataset" and "per approach". TI no HPO is performed, we set a cross for both columns. We also mark if noisy validation labels from crowdworkers are used for the HPO or if access to a validation set with true labels is assumed. For studics without any HPO, some experimentation relics on standard architectures with default HPCs across their study (Tanno et al. 2019 Zhang et al.) 2024 Li et al., 2024 Nguyen et al. 2024 Han et al., 2024 Guo et al.] 2024). In contrast, others specify the HPC for each dataset and approach without. further explanation (Cao et al.) 2019 Li et al. 2021: Wei al. 2023). Several studies (Tanno et al. 2019
Herde et al., 2023 2024b Zhang et al. 2024 Nguyen et al. 2024
Guo et al.
2024 provide an extra ablation
study for the HPs of their own LFC approaches.
Validation with Noisy Class Labels A few works exist on different aspects of validation with noisy class labels in the broader area of LNL. Chen et al. (2021) theoretically prove that for diagonally-dominant confusion matrices, the validation accuracy remains a reliable indicator of true performance. However, in practice, complex types of noise can still pose challenges, especially when the noise is systematic or when not enough data are available to average it out. For example, the empirical findings of Kuo et al. (2023) indicate that even small amounts of (not. necessarily label) noise in the validation signal can significantly degrade HPO outcomes. The observations of Inouye et al. (017)) also confirm that standard validation can be misleading for localizod, systematic label noise. Their proposed solution injects synthotic label noise into the training data (based on an estimated noise model) while koeping validation labels unchanged. This penalizes
models that overfit spurious patterns and improves over standard cross vass-valiia validaiion. Guo et al. 2024 evaluate
4

---

Published in Transactions on Machine Learning Research (11/2025)
LFC approaches with carly stopping using noisy validation data. However, analysis regarding the effects of such an early stopping is reported. Yuan et al. (2024) also reoognize the issues of training and validating with noisy class labels in the context of early stopping. Therefore, they propose solution for early stopping that docs not rely on a separate validation set. However, they do not perform any HPO but instead focus on
demonstrating the robust of their solution across different HPCs. In contrast, Wang et al. 2025 the issue HPO by proposing HPS criteria when learning from partial labels.
tackle
3
Hyperparameter Optimization with Noisy Labels from Crowds
This section first formalizes the problem setting and approaches to LFC, then outlines the basics of HPO, and finally introduces corresponding HPS criteria for handling noisy crowd-labeled validation data.
3.1 Problem Setting
Here, we describe the data generation process to define the objective of LFC approaches.
Data Generation Process Figure 2 depicts the probabilistic graphical model of the commonly as- sumed data generation process in LFC settings (Li et al. 2022 Herde et al. 2024b). Let the multisof X = {o,) he=1 C fx,,N .N E N>1 denote the observed instances, which are independently drawn from Pr(æ). Then, their one-hot encoded true class labels, denoted as the multiset y (yn),= S C == {ee) with C E N22 as the number of classes, are distributed according to Pr(y)(2,,) and latent. Only the multiset N.M
Z = {zmm) f m== S fzz = fy U {0} of one-hot encoded conditionally independent noisy class labels provided by M E N22 crowdworkers obsorvable. Since not every crowdworker is requested label cach instance, some class labels from the crowdworkers are unobserved, denoted an all-zero vector O. An observed class label 2nm with E Mn (m| Zum + M)M is assumed to be drawn from the instance-, class-, and crowdworker-speeinn distribution Pr(z Xn. yn, m).
Instance
y4. True Label
Noisy Label
V/
m E M,,
E [N]
Figure 2: Probabilistic graphical model of LFC. Arrows show dependencies between random variables, while shaded circles indicate observed variables and unshaded latent oncs. The set Mn S [M] = (1...,,M indicates that an instance Ix is not necessarily labeled by all M crowdworkers.
Objective LFC approaches aim to optimize the parameters 0 Re of the data classification model fe 0x + Ac by minimizing its expected risk:
lt == arg min (Epr(ma)) (y..fo(z))]
(1)
where Ac is probability simplex and L : Ac this article, we employ the zero-one loss /Vapnik 1995 to assess the data classification model's predictions 8
A R denoles appropriate loss function. Throughout
arg max (e. (e)v) ) E.
)"}
arg max (e. :))
Lu/1 (y. 9 =
arg
(2)
egeRy
3.2 Approaches to Learning from Crowds
Given the objective in Eq. (1, LFC approaches do not directly optimize the outputs of the data classification model fe due to the lack of true labels y. Instead, the noisy class labels Z are used to train crowdworker
A multiset is set that contain duplicates. 9The dot product one-hot encaded label vectors is one only they represent the same class.
5

---

Published in Transactions on Machine Learning Research (11/2025)
classification model 96 f2x [M] + Ac with parameters & Szg. This model predicts the probability distribution over all class labels for each instance-crowd probability that the given crowdworker will assign that label to the given instance. The estimates of both classification models are typically linked through transformations based on confusion matrices (Tanno et al., 2019) or noise adaplation layers (Rodrigues & Pereira 2018): which try to separate the crowdworkers' noise from the true class label distribution. Furthermore, such noise separation allows defining a crowdworker performance model he Qx [M] + [0, 1] with parameters E Se quantifying crowdworkers' labeling accuracies. These three different models' predictions have the following probabilistic interpretations:
pair, where each label's value indicates the
[fo( = Pr(yn =e enl (..nn:0) 0);
(3)
(2n, m)]e = Pr(znm = Bn, m,, Φ),
(4)
(2 m) Pr L m. 2)),
(5)
where He denotes the e-th element of vector. Throughout the main text, we regard the three models as black- box functions, potentially with shared parameters. Appendix A summarizes concrete LFC implementations and how they estimate the probabilities in Eqs. 35
3.3 Hyperparameter Optimization
Let us define the dataset D = {( In) with 2n = {Zam to encompass only instances with their observed class labels. Then, a learning algorithm Ax (corresponding to an LFC approach) with the HPC À E fl outputs all required parameters (D(D) E Sm, of which model-specific parameters are (possibly overlapping) projections with π (Ax(D)) Ne. Tg (Ax(D)) Ωtg and n. (Ax(D)) Qy. Each dimension in the HP search space a사 corresponds to single HP, e-g-; the number of epochs (integer), the learning rate (continuous), or the type of the optimizer (categorical). Ideally, find the optimal HPC X E Ωx such that our learning algorithm outputs the optimal classification model parameters (see Eq. 1
T (Ax (D)) = 0"
(6)
In practice, finding the optimal solution is difficult due to many challenges, which two critical ones are:
Evaluating each HPC A E ΩA is computationally infeasible for large HP search space Ωs
We only estimate the expected risk (sce Eq. because Pr(æ,, y) is unknown.
In this article, we focus exclusively on challenge bocause the risk estimation difficult and undcrexplorod with access only crowd-labeled validation data (see Section 2 Challenge ① not part of contributions. Instead, we briefly review established solutions context for the HPO loop shown in Figure
① Hyperparaneter Search Strategy
Hyperparameter Selection Criterion
Propose Next Hyperparameter Configuration
Retain Hyperparameter Configuration with Lowest Empirical Risk
E fzy Sobol Sequence
Hyperparameter Optimization Loop
= A iff RI,8, (Ax) < R1,83 (Ax)
Grid
Random
Bayusian Optimization
Resampling S
Loss Function
Side Information I
Zerosane Less
Teu Labele
Biis Stare
Labe Aggrspation
Cross Eatrogy
Label Weighting
Figure 3: HPO loop. In an itcrative process, HPO techniques explore the HP search space by retraining and evaluating the learning algorithm with different HPCs.
1 Hyperparameter Search Strategy Given true labels, research on HPO focuses primarily on improving the scarch stratogy for (iteratively) proposing a set of candidate HPCs A C ΩA by balancing the exploration- exploitation trade-off within the HP search space Ωx given budget of |A| Nş1 evaluated HPCs. For this purpose, random search is popular choice that samples HP values randomly from predefined ranges, often

---

Published in Transactions on Machine Learning Research (11/2025)
Mcanwhile, Sobol sequences (Sobol 1998) and Bayesian optimization (Wang et al. 2023) guide the search of candidate HPCs even more efficiently. Bayesian optimization usually excels because it adaptively selects each new HPC. However, we employ Sobol sequences so that every HPS criterion evaluates the same set A of candidate HPCs. Hence, any performance differences between the HPS criteria come solely from choosing different HPCs.
outperforming exhaustive grid search in high-dimensional spaces [Bergstra & Bengio 2012
Hyperparameter Selection Criterion Ideally, an HP search strategy has access to a reliable empirical risk estimation (Vapnik 1995), which assigns learning algorithm scalar value Rt,s,z(Ax) R. Thereby, L denotes the loss function (see Eq. 22), S resampling technique, and I additional side information. Formally, we represent a resampling technique, e-g., hold-out, pross-validation or bootstrapping, through a set of K E N21 disjoint training (T) and validation (V) splits of the full training set D:
8 = ((Tk) TE UV = D A To nv = 0]
(7)
Side information I encompasses all required inputs beyond the loss function and resampling technique S for computing the empirical risk. Based on such empirical risk estimates, we define an HPS criterion as a rule picking the evaluated HPC with the lowest empirical risk:
Å = arg min (Ax)s, AEa
(8)
We treat the empirical risk measure RI.s,I z as a placeholder, whose explicit definition depends on the respective criterion. For example, suppose the true class labels are obtained from an expert as side information such that I == y. Then, the true empirical risk of the learning algorithm Ax computed as:
(Ax) Σ Σ
I. I.n (mp(A
K K-DVE (Te.V E.V
Since we have only access to noisy crowd-labbled validation data in an LFC setting, the HPS criterion based the true empirical risk RLSs, represents our upper baseline criterion for HPO, denoted as TRUE (plug
Eq. (9) into Eq. (.). ). In contrast, our lower bascline criterion DEF (plug Eq. (10) into Eq. {8) constantly
outpuls a default HPC Aper E Ua which corresponds to a naive risk estimation using the default HPC as side information, i.e., I= APEr, such that:
RL. (Ax) = 6(ADEr * A),
(10)
where d: {false, true} - {0, 1} denotes an indicator function.
3.4 Hyperparameter Selection Criteria for Crowd-labeled Validation Data
If the validation set is labeled only by the crowd, the HPS criterion TRUE cannot be used. We therefore introduce proxs[] HPS criteria. Each proxy plugs a specific empirical risk measure RI,s,I into the minimum- risk selection rule (see Eq. (8)). Again, the side information I is only a placeholder that is instantiated in accordance with each risk definition. Rather than listing formulas, Figure E highlights commonalitics and differences between the HPS criteria by presenting them as leaves of tree. The path from root to leaf records the design choices and assumptions about the empirical risk template, how crowdworker performances are modeled, and how labels are weighted. Which combination works best is unknown and depends on the data and the LFC approach. Hence, we also introduce ensemble of HPS criteria that combines their risk estimates, aiming for more robust selection than any single criterion. Tn the following, we proceed through the trec's levels beyond its root. Finally, we note that this treo-structured overview is illustrative, not exhaustive, and future HPS criteria may introduce additional design choices.
WWe the term proxy avoid confusion with classical surrogate losses that have proven consistency guarantees. For further guidance regarding our empirical viewpoint and its limitations, we refer to Section 4.3 and Section
7

---

Published in Transactions on Machine Learning Research (11/2025)
Selection Rule
Empirical Risk Estimation Template
Crowdworker
Label Weights
Hyperparameter
Performances
Selection Criteria
Confidence: Eq. (24)
ALC
Learned:
Eq. (21)
Uniform: Eq. (23)
ALU
Aggrogation-level: Eq. (11)
Uniform: Eq. (23)
AEU
Equal:
Eq. (20)
Confidence: Eq. (24)
Minimum-risk: Eq. (8)
ENS
Confidence: Eq. (26)
CFC
Equal: Eq. (20)
Uniform: Eg. (25)
Crowd-level: Eq. (13)
CXU
Uniform: Eq. (25)
Learned:
Eq. (21)
Confidence: Eq. (26)
CLC
Figure HPS criteria for noisy crowd-labeled validation data. Each HPS criterion is defined by composing the fixed root equation with the equations at branching nodes along its path. Fixed equations elsewhere are implicit. The criterion's name is formed from the internal nodes' initials on its path. The abbreviation CXU indicates that the paths with the abbreviations CEU and CLU lead to identical HPC selections. The ensemble-based criterion ENS combines all criteria for improved robustness.
Empirical Risk Estimation Template LFC approaches typically involve the joint training of the data classification model fo and the crowdworker classification model gø (sce Section 3.2 whose predictive performances capture distinct facets of the training outcome. We therefore introduce two empirical risk templates that become concrete risk measures when their components instantiated at the tree's subsequent levels. On the one hand, we assess the data classification model fe by computing the aggregation-level empirical risk through:
w(En, Zn), (z(xn, Zn), f (m)(A)()))((n))) with
(AN)
Σ
Σ
(11)
K W+
(T2. (2).2n)
Wk
Σ
w(Kn; 2n, E R>o;
(12)
DEV.
where the two functions in I = (z, w) denote the side information. The label aggregation function z: 2x x B (fyy + Ao aims to infer the latent true class labels with B (fy) referring to the set of all finite multiscts over the class labels in fly. The label weighting function w : 2g x B (Syy Rzo weights individual
loss contributions of the aggregated class labels. On the other hand, we compute the crowd-level empirical risk to assess the crowdworker classification model Iø through:
U(In, m. 2n) L (Zum Ing (A,(( with K.
RL,S.O(A)) (Ax)
Σ
Σ
Σ
(13)
(TE.E)ES (#n.Z.)EV, Znm EZn
VE ==
Σ
Σ
v(an, m. 2n) E Ryo,
(14)
EE.VV From E2,
where the label weighting function v Ωx x [M] x B (Srr R20 as side information := v weights the individual loss contributions the crowdworkers' labels. Intuitively, both label weighting functions are to downweight the influence of labels that are likely false, allowing the more reliable ones to dominate the loss and thereby enhancing robustness to label noise. Both risk templates come with potential downfalls. While the aggregation-level risk requires to estimate the true class labels, and therefore evaluates the model
8

---

Published in Transactions on Machine Learning Research (11/2025)
data we do not have, the crowd-level risk evaluates the model's capacity in predicting the labels provided by the individual crowdworkers, and therefore evaluates the model on different objective. Both can serve proxies under a reasonable modeling of crowdworkers' labeling behavior.
Crowdworker Performances Starting from different assumptions about crowdworker performances, we estimate true class label probabilities. This creates direct connection between crowdworker performance and risk estimation, because these posteriors induce the label aggregation function and the label weighting functions w, parts of the risk estimation template. For this purpose, let assume that have the estimated confusion probabilities Pr(znm Dns Yn, m) for each crowdworker E Mn and instance In E X. Then, estimate the posterior true class label probabilities, i.., after observing the crowdworkers' labels Zn, through:
Pr(yn- Col In. 2n) ②2 Pr (yn, e. x.. Pr( Pm Fn, Yn Ce)
(15)
Pr (y/ =n)
II meM..
Pr(znm (nn, yn eos m),,
(16)
where the transfornation (x) corresponds to Bayes' theorem and (f) to the assumed crowdworkers' conditional independence. Even with strictly disjoint training and validation instances, if we employed the data classification model fo to predict Pr (yn) we would bias the posterior true class label probability ostimation Pr(y,. = eal Zn) toward the data classification model's own predictions (see Proposition in Appendix B. 1,, yielding a model-dependent (circular) validation largel and over-optimistic risk estimates. Instead, we assume uniform prior class probabilities, i.e., before observing any crowdworkers' labels Z,
1
Pr (yn, = =cl=x)) C'
(17)
If we employed the full crowdworkers' confusion probability estimates Pr 2nm Tn. Vn: m), the posterior computation would be vulnerable to class-specific biases (see Proposition 2 in Appendix B.2p. Hence, we model the confusion probabilities (only during validation) via the crowdworker's instance wise performance estimate Pr(zT y/ 1| m) by defining:
1-c
Pr(znm = ex Un - E m) = Pm Un 1 1n m) Pr(2T Un = 0 22, m)
(18)
C-1
By summarizing each crowdworker's behavior on given instance with one scalar performance value, apply the same performance value to every class. All classes are, therefore, treated identically, and no systematic bias toward any particular class ariscs. As a result of Eq. 17 and Eq. 18 the posterior estimation from Eq. (16) reduces to:
Pr nm Un = 2n m))
1-2T
1-2 Rc
Pr(yn = Lz 2n) & II
Pr(xTmmn = 1 1|xn, tmm m.M..
(19)
C'-1
where we distinguish between:
Pr(z Un 1/ m) = p E 1//,
as equal crowdworker performances, (20)
Pr(2TT Un 1| m) = (A m) as learned crowdworker performances.
(21)
In the first case, the exact value of the labeling accuracy p is irrelevant. Instead, p only encodes the assumption that all crowdworkers have the same performance across all instances, which is better than randomly guessing. In the second case, the crowdworker- and potentially instance-wise performances are estimated by the crowdworker performance model (TT))A(TE)) obtained after training with the respective LFC approach Ax on the k-th training fold using the candidate HPC X. In both cases, the label aggregation function outputs the marimum a posteriori (MAP) estimate of the true class label:
2 2(Fn 2n) == arg max (Pr (y = eelann 2n))
(22)
e.Efly
9

---

Published in Transactions on Machine Learning Research (11/2025)
When all crowdworkers assumed to perform equally, the MAP estimate reduces to simple majority voling. In contrast, it naturally becomes weighted majority voting once their performances are learned (see Proposition 3 in Appendix B.3
Label Weights The label weighting functions control class label's impact on the risk ostimate. A uniform weighting gives every label the same weight, whereas confidence weighting scales label's weight in proportion to its estimated correctness probability. Accordingly, the label weighting functions for aggregation-leve risk estimation take the forms:
Wn Zm) =
uniform weighting,
(23)
wxnn nnn 2n)) = max
Prr (y,, celarn, Zn))
as confidence weighting,
(24)
e.Ely
whereas the label weighting functions for the crowd-level risk take the forms:
v( m, 2n) =
as uniform weighting,
(25)
v(xn, m, 2m) = Pr (yn - Znm [nn 2n)
as confidence weighting.
(26)
We evaluate only these two label weighting schemes because they serve simple, assumption-ligat baselines. Neverthcless, other weighting schemes are possible in principle, e.g., discarding highly uncertain labels or lempering confidence scores.
Hyperparameter Selection Criteria The different combinations of risk estimation template, crowdworker performance modeling, and label weighting correspond to J = 7 distinct HPS criteria with their associated risk measures R = (R,,s,,, 1/=11 As an example, let us look at the path of the criterion AEU in Figure 4 This criterion instantiates the aggregation-level risk template from Eq. [1] with the aggregation function from (19) with the equal crowdworker performance model from Eq. (20), and uniform label weights from Eq. (23). The resulting risk mcasure reduces to computing the average zero-one loss between the data classification model's 's predictions with plain majority vote labels targets for the validation instances. Each empirical risk measure in R is only proxy the true empirical risk
Eq. (22). which evaluates the class posteriors from Eq.
(see Eq. (9);:
RL,S.I, (Ax) = (Ax) + CL.S.Y.,, (Ax) with ELS..V (Ax) E E.R.
(27)
The difference arises through multiple sources, e.g., imperfect label aggregation, imprecise crowdworker performance estimates, or even different target models. For example, the crowd-level risk estimation measures the risk of the crowdworker classification model, which different from estimating the risk of the data classification model as in Eq. 9 Because these issues, we propose combining our empirical risk measures into an ensemble-based selection criterion ENS. For this purpose, we adopt the classic Borda count (de Borda 1781,, a agnk-aggregttion rule used in robust meta-evaluation (Abdulrahman al. 2018
Let O (0; ..... [A1))=1 denote the set of ranking functions such that:
og (A) 1+ Σ ((RL,S,,,AA)) > RL.S.I, (Ax)). NEA
(28)
Accordingly, (A) = 1 corresponds to the HPC with the lowest empirical risk. The empirical risk estimate (proxy) based on the Borda count then defined as:
RL.S.(AN) = oA()).
(29)
EO
where the ranking functions serve as our side information such that = O. As a result, the criterion ENS outputs the HPC with the minimum rank sum (plug Eq. 29 into Eq. (8)) Intuitively, summing rankings is expected stabilize decisions by balancing individual biases of each noisy risk measure R, increasing the likelihood of choosing a robust HPC.
10

---

Published in Transactions on Machine Learning Research (11/2025)
4 Experimental Study
This section starts with comprehensive description of our experimental setup. Subsequently, analyze our experimental results to answer RQi and RQ2 as our two central research questions. The corresponding answers serve as basis for formulating a guideline to design realistic and fair experiments when benchmarking LFC approaches in the future.
4.1 Experimental Setup
Our setup covers datasets, neural network architectures, LFC approaches, HP scarch, and HPS criteria. We describe our design choices for each of these aspects in the following.
Datasets Realistic datasets are requirement for meaningful evaluation of LFC approaches. Therefore, we rely only real-world datasets annotated by error-prone humans, mostly actual crowdworkers. Table 2 overviews these datasets by detailing their key attributes. The dataset mgc (Tzanetakis & Cook 2002]
originally contains 30s audio files of songs to be classified according Lo their music genres. A subset of the well-known image benchmark dataset label-me (Russell et al. 2008) concerns the classification of scenes, while the datasct dopanim (Herde et al.) 2024a) targets the classification of doppelganger (groups of highly similar) animals. There are two text datasets, which are subset of the dataset reuters (Lewis) 1987) for
news document classification and a subset of the dataset spc (Pang & Lee 2005) for sentiment polarity classification of movie review sentences. Based on large sets of noisy labels resulting from the datasets' labeling campaigns, we follow the ideas of Wei et al. (2022) and Herde et al. (2024a) by introducing variants of these noisy label sets. These variants simulate different levels of crowdworker performance and varying amounts of label redundancy. For each instance, we either retain only the labels produced by the worst crowdworkers, i.e., we keep false labels if any are available, or we select labels uniformly at random. The suffixes -1, -2, and -v then specify how many labels to keep per instance: exactly one, exactly two, or a variable number, respectively. Because we preserve only a subset of all submitted labels, the total number of crowdworkers contributing labels can change across variants. The variant full refers to the originally published set of class labels from crowdworkers. Together, these dataset variants cover a wide range of different LFC settings. Specifically, the nunber of crowdworkers ranges from a small group of M = 20 people to a large group of M = 203 people. Label noise, defined as the fraction of crroncous labels from crowdworkors ranges from approximately 20 % to %.. When these labels are aggregated via majority voting, the resulting aggregation noise, i.e. the fraction of aggregated labels that are incorrect, ranges from circa 11 % to 87 %. Finally, the datasct variants encompass scenarios ranging from no label redundancy, i.c., only one class label per instance, to those exhibiting substantial label redundancy, i.e., over five class labels per instance.
Neural Network Architectures The original audio files unavailable for the crowd-labeled datasct mgc. Instead, Rodrigues el al. 2013) published features extracted via music information retrieval tool. Similarly, only term counts published by Rodrigues et al. (201)) are available for the crowd-labeled dataset reuters for which we apply a term trequency-inverse inverse document frequency (TF-IDF) transformation. As a result, the instances for these two datasets correspond to plain feature vectors. Thus, we employ multi-lager perceptrons (MLPss) as the data classification model fo- Apart from the input dimension, which depends on the respective dataset, the MLPs share two hidden layers (256 and 128 neurons) enhanced by batch normalization (loffe) & Szegedy 2015) and rectified linear wnil (ReLU, Glorot et al.
2011 activation functions. For all image published, we employ DINOv2 vision transformer (vit-s/14, Oquab et al. [2023 a backbonn model. Analogously,
datasets, where the actual images with their associated noisy class labels from the crowdworkers
we use an MPNet sentence transformer (all-mpnet-base-v22 Song et al. 2020 Reimers & Gurevych 2019) as the backbone for the sentences of the dataset spc. Both backbones pre-trained weights remain fixed to preserve the robust feature representations as inputs to an MLP head as the data classification model fo with the same architecture (apart from the input dimensions) as for the other datasots.
Learning from Crowds Approaches Table I lists the 13 LFC approaches evaluated in our study. We focus on one-stage LFC approaches, whose end-to-end training has yielded state-of-tho-art performances (Nguyen et al. 2024 Herde et al. 2024b: Tbrahim et al. 2023 Of this type, include approaches that model
11

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Dataset overview. The first column indicates the names the datasets, while the remaining columns refer the datasets' attributes. We denote counts by the # symbol, fractions by the % symbol, and means are supplemented by standard deviations.
Datasct
Variant
Labeling Campaign
Training Instances [#]
Test
Classes
Workers
Labels per
Instance [#]
Label Noise
Aggregation
Noise
[#] Andio Data
[#]
[#]
[%]
[%]
1.0g0.0
woret-2
37
1.9.0.3
72.5
69.4
sorst-v
2.5+1.6
59.2
58.6
rand-1
Rodrigues et al.
700
300
10
1.0.0.0
47.1
47.1
rand-2
43
1.9+0.3
45.7
43.9
2.6:1.6
44.6
38.3
full
44
4.2g2.0
44.0
30.3
Image Data
2.520.6
woret-2
59
2.0g0.a
30.8
30.1
worst-v
1.840.5
32.5
label-me
rand-1
Rodrigues et al.
1,000
1,188
8
57
1.040.0
23.9
23.9
rand-2
2.0+0.2
25.3
25.7
rand-y
1.810.
25.5
25.0
full
59
2.5
26.0
23.7
1.0+0.0
77.6
77.6
woret-2
2.0+0.0
62.7
62.2
woret-v
3.0
45.2
46.9
dopanim
rand-1
Herde al.
10,484
4,500
15
1.0.0.0
32.5
32.5
rand-2
2.0+0.0
32.8
33.2
3.0+1.4
32.7
full
5.0g0.2
32.7
19.3
Text Data
L000.0
09.2
woret-2
2.0ga.2
54.0
54.0
2.0
reuters
rand-1
Rodrigues et sl.
1,786
4.217
8
38
1.040.0
38.5
38.5
rand-2
2.0+0.2
39.9
40.9
rand-v
2.0+1.0
40.8
38.4
full
3.0+1.0
40.4
35.5
1.0g0.0
2orat-2
199
2.0+0.0
47.1
47.0
sorst-v
202
3.2+1.
31.6
32.5
spc
Rodrigues al.
3,000
1,999
2
184
1.01100
21.2
21.2
rand-2
200
2.0+0.0
20.8
20.6
rand-y
3.321.6
full
203
5.510.7
20.9
11.0
class-lependent and instance-dependent crowdworker performances. Further, we consider two two-stage approaches, which mv serves as a lower baseline because it trains the data classification model fe on majority vote labols. Implementing the crowdworker classification model gø and the crowdworker performance model hy depends on the respective LFC approach, which Appendix provides more detailed descriptions.
Hyperparameter Search Table lists general (approach-agnostta HPs and approach-speciffc HPs. All approaches share the general HPs. Specifically, we fix R.Adam (Liu et al. 2019) as the optimizer, combined with a cosine annealing learning rate scheduler @Loshchilov & Hutter 2017)) without restarts to gradually reduce the learning rate throughout the training process, thereby promoting stable convergence. For the remaining general HPs, we define suitable search spaces derived from related literature and default values of PyTorch (Paszke et al. 2019) optimizers (c.g., no weight decay). The distributions for sampling HP values are denoted uniform and log-uniform. For approach-speciffc HPs, we adopt default values from the publications codebases. If available, the search spaces also extracted from these two sources. Otherwise, they are defined based reasonable value ranges. The defined HP search spaces are sampled using Sobol sequences [Sobol 1998 as HP search strategy (see Section 3.3 for our rationale). A total of 50 distinct HPCs are generated per combination of LFC approach and dataset variant. Together with the default HPC, this yields set of |A| 51 HPCs, from which the HPS criteria tries to choose the best performer.
Hyperparameter Selection Criteria Each HPC evaluated via a K 5-fold cross-validation given the crowd-labeled training set to obtain risk estimates for the respective HPS criterion. The HPC chosen by
12

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Overview of LFC approaches' general and individual HP search spaces. For each HP, we define a default value and search space the basis for the HPO. The notation not applicable (N/A) indicates that an LFC approach does not introduce additional HPs or that an HP is not optimized. The expressions uniform and log-uniform define the scarch spaces distributions used for generating HPCs.
Approach
Reference
Hyperparameter
Default Value
Search Space
optimizer
RAdam
N/A N/A
learning rase scheduler
cosine annealing
number of epochs
30
uniform([5, 30, 50})
General
N/A
batch size
32
uniform [6. 32, 64))
initial learning rate
10
logunifozu 10-!
weight decay
loguniform 9. 10100 *1)
dropout rate
0.0
Mnifoorn[U.0.. U.5])
Two-stage Approach without Crowdworker Performance Modeling
N/A
N/A
N/A
N/A
wwo-stage Approach with Clase-dependent Crowdworker Performance Modeling
da
Dawid & Skene
N/A
N/A
N/A
One-stage Approaches with Classdeepndeet Crowdworker
Performance Modeling
61
Rodrigues & Pereira
N/A
N/A
N/A
trace
Tanno et al
confusion matrix regularization (A)
10
loguniform [10 3. 9.10 'D)
conal
Chu al.
embedding dimenslon
confusion matrix rogularization (A)
20
10
waiform((20) 40, 60, 80}}
loguntTorm ( [10 6 10 }
union-a
Wei
confusion matrix initialization (c)
10-
loguniform (omnifomm([11555 (110", 10 (0
union-b
geo-f
Ibrahim et al.
confusion matrix regularization (A)
10-3
loguniform (100 10
geo-w
One-stage Approaches with Instance-dependen Crowdworker Performance Modeling
confusion matrix initialization (1)
uniforn (0.75, 0.95))
gamma distribution parameter (a)
1.25
uniforn (1.0
mad1
Herde et al.
gamma distribution parameter (8)
0.25
uniforn [0.25, 0.5))
embedding dimension (Q)
16
wniforn! (8, 16, 32))
croud-ar ar
Can ot al.
loss balancing
0.9
uniforn 00.5, 1.0)
annot- mix
Herde et al.
confusion matrix initialization (n)
mixup (a)
0.9
1.0
uniform 00755 0.95.)
uniforn[0.00 (0.0.
outlier regularization (11)
10-2
loguniform(10 (1000_10001)
coin
Nguyen
volume regularization (222
10
1oguniform(10 1.10 1
the respective criterion (see Eq. S is then tested on the hold-out test set with true labels for five different initializations of the neural networks' weights, ensuring a reliable assessment the data classification model's generalization performance. We include two variants of default HPCs. On the one hand, we use the default values specified in Table 3 across all datasets. This criterion, to which we refer as DEF (see Eq. (T0), ), is the most naive one since there is no consideration of the datasets' individual requirements. A more advanced and commonly used alternative in LFC evaluation studies is to fix one default HPC per dataset, denoted DEF-DATA. For each dataset variant, we first perform a conventional HPO: a vanilla classification model, ignoring the LFC setting, is trained and validated on the true class labels, tuning only the approach-ognostic HPs (Herde et al. 2024a). The best HPC obtained from this search is then frozen and transferred to every LFC approach. Consequently, every approach shares the same general HPC for that datasel variant while its approach-specifii HPs stay at their default values. For well-studied benchmarks, one could alternatively adopt HPCs reported in the literature (Tanno et al.) 2019), yet rely on the HPO variant as DEF-DATA criterion to guarantee consistency across datasets. This DEF-DATA criterion differs from our upper baseline criterion TRUE, where all HPs of an I.FC approach are optimized by training on the noisy crowd-labeled data and validating on the true labels (see Eq. (9). Despite this difference, DEF-DATA and TRUE are the only HPS criteria requiring access to the true validation labels.
4.2 Experimental Results
Our setup encompasses 5 datasets, each with 7 variants, 13 LFC approaches, and 11 HPS critcria. Duc to the numerous combinations, we present only the main results addressing our two research questions here and refer to Appendix CC2 for the complete list of results. These supplementary results also contain more in-depth analyses regarding the impact of different noise levels, numbers of candidate HPCs, and an architecture search for the data classification model.
13

---

We compare the HPS criteria with each other across multiple datasets and LFC approaches. Therefore, we report performance results in the form of rankings and zero-one losses in Table E In addition, this table lists for each criterion the number of significant. wins, significant losses, and non-significant comparisons (ties) against all other criteria. We treated each pair of dataset variant and LFC approach as a block and applied a Friedman test (Friedman 1937) at the significance level = 0.05, which was significant. Thus, performed all pairwise Wilcoxon signed-rank tests with Holm correction (Holm 1979). For each pair of criteria, the comparison was counted as win (or loss) for a critcrion if the adjusted p-value was below a (with the sign of the performance difference determining win versus loss). Otherwise, the comparison was counted as a tie. The complete table of pairwise comparisons is given in Appendix [C2] One central finding is that the criterion DEF significantly ranks poorly, supporting our motivation that using default HPCs leads to underestimating the approaches' potential performances. Morcover, the poor rank of the DEF-DATA criterion indicates that using well-tuned HPCs from standard classification lasks (without any noisy labels) does not. account for the unique requirements of each LFC approach. As anticipated, the TRUE criterion, which utilizes the true class labels for selecting the best HPC per LFC approach and per datasct, performs superiorly to Published in Transactions on Machine Learning Research (11/2025)
RQ1: Given access only to crowd-labeled validation data, which evaluated HPS criterion yields the highest performance, for L,CC approaches?
Table 4: HPS criteria's results. One column per criterion reports the rank, number of significant wins, significant losses, non-significant ties compared to the other criteria, and the zero-one loss reductions (absolute as percentage points [%p] and relative as percentages [%)) compared to DEF as criterion. Means and standard errors are computed over all combinations of dataset variants and LFC approaches (excluding the approach mv that is not compatible with each criterion). The arrows show whether a smaller (4) or higher (1) value is better. The best and second best, values are marked per result type. A marks criteria that had access to the true validation labels.
Baseline DEF-DATA
Aggregation-level
Crowd-level CBC
Ensemble ENS
TRUE"
DEP
ABU
AEC
ALU
ALC
CXU
CLC
(1)
4.64
7.84
9.76
5,66
5.57
5.29
5.60
5.46
5.36
5.91
4.89
0.15
# 0.17
1 0.11
4 0.11
1
0.11
4 0.11
1 0.13
d: 0.11
1
0.11
1: 0.13
+ 0.09
Significant
Wins [#] (f) / Non-siguilice. Ties [#] Significant Losses [#] (J)
9 1/0
1/0/9
0/0/10
2/6/2
2/6/2
3/5/2
2/6/2
2/6/2
2/6/2
2/5/3
9 0/1/0
Absolute Zero-one Loss Reductions Compared
(Amm Loza
[%,] 1)
5.36
+ 0.76
+ 0.00
+ 4.55
4.59
4.73
+ 4.43
+ 4.52
+ 4.57
+ 4.07
5.15
0.24
# 0.31
1 0.00
1 0.26
1 0.26
4 0.26
+ 0.27
4 0.29
+ 0.28
4 0.31
0.26
Relative Zero-one Loss Reductions Compared to DEF (Ans Lozs
()
+18.20
3.65
0.00
+16.18
16:22
+16.48 16.48
+14.66
+15.48
+15.93
+ +13.30
+1..60
1. 0.73
4 1.12
0.00
1 0.76
1 0.76
1 0.80
1 0.81
1 0.87
1 0.84
1 0.91
0.77
Baseline TRUE*
Aggregation-level
Crowd-level
Ensemble ENS
D2F
DEP-
$ ABU
f ABC ALU
ALC
CXU
ł CBC
CLC
+-8
+6
Juce Lazi (%)p)
+4
+2-
-2
trace
conal
union-a
union-b
goo-1 -2
goo-f
mad1
crosd ar annot mix coin
Figure 5: Absolute zero-one loss reductions of HPS criteria. For each LFC approach (æ-axis), the scatter plot displays the mean and standard error of a criterion's 's reduction in zero-one loss (y-axis) as percentage points [%p] compared the criterion DEF (see Appendix C.2 for relative reductions as percentages (%))- Higher reductions correspond to greater improvements. A marks criteria. that had access to the true validation labels.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: LFC approaches' results with ENS as HPS criterion. One column per approach reports the rank, number of significant wins, significant losses, von-significant ties compared the other approaches and the zero-one loss reductions (absolute as percentage points [%)] and relative as percentages [%]) compared to the approach mv trained with its default (DEF) HPC. Mcans and standard errors are computed over all dataset variants. The arrows indicate whether smaller (L) or higher (1) value is better. The best and second best values are marked per result type.
Baseline
Class-dependent
Instanco-dopedentt
trace
conal
on-aa
und on-b
200-w
goo-f
nad1
crexd-ar
annot-nix
coin
Ranks
10.57
9.09
7.90
7.31
7.37
7.60
7.61
5.81
4.10
5.36
8.97
5.34
3.96
0.38
0.53
0.64
0.53
+ 0.53
+ 0.67
0.55
0.50
+
+ 0.63
0.43
+ 0.65
0.50
Significunt Wins [# (t) Non significutt Ties [#] Significant
Losses
(3)
0/3/9
0/7/5
0/11/1
1/9/2
1/9/2
0/10/2
1/8/3
4/870
//4 //4/0
3/9/0
1/6/5
3/9/0
7/5/0
Absolute Zern-one Loss Reductions Compared to ny with nes [Am](se) Tays
[%p] 1)
+ 5.60
+ 6.05
8.04
7.81
+ 7.18
+ 7.0b
7.42
+ 9.70
+10.05
+ 8.52
+ 6.87
+ 9.21
9.88
0.90
1.17
1.19
4 1.09
0.92
4 0.97
1.05
_ 1.28
4 1.08
+ 1.33
0.97
+ 0.93
0.99
Relative Zero-one Loss Reductions Compared to nv with oer (Aniowr) bazs
[%]] f)
+19.31
+21.01
+23.95
+21.99
+24.20
+21.67
+23.82
+28.75
+30.79
+27 .6
+22.68
+28.22 22
+30.86
2.75
3.14
2.95
3.06
2.83
1 2.92
2.86
2.99
4 2.86
1 3.45
2.87
+ 2.60
2.84
every other criterion. Concretely, it ranks best and provides the highest loss reductions compared to the DEF criterion across all dataset variants. Among the criteria relying solely crowd-labeled validation data, the ensemble-based ENS criterion is the clear runner-up: it ranks better and provides higher loss reductions than all other competing criteria apart from the upper baseline criterion TRUE. Notably, ENS is the only criterion without a significant loss compared to TRUE. Reducing the size of the ensemble decreases its performance (see ablation study in Appendix (..).. At the same time, the extra computation demanded by ENS remains negligible (see time complexity analysis in Appendix B.4): the expensive steps are training and inferring the predictions of the models, and once those predictions are available, they can be reused to compute every risk estimate. Finally, although the AEU criterion, which aggregates the noisy validation labels per instance via majority voting, performs worse than the criteria TRUE and ENS, it still yields substantial improvements over default HPCs. Together, these results highlight the benefits of HPO in LFC settings with crowd-labeled validation data. Bosides these results averaged over the LFC approaches, we also examine how each criterion performs in combination with each approach individually. For this purpose, Figure 5 breaks down the absolute zero-one loss reductions according to the LFC approaches. This way, we measure how much specific criterion improves an approach relative to using that approach's default HPC. It does not indicate which LFC approach is superior to others. We observe that for a few LFC approaches, e.g., coin and geo-f, the criterion TRUE does not achieve the highest loss reduction on average. An explanation is that cross-validation uses only subsets of the data for training, so the HPC that minimizes the validation zero-one loss on the subsets may not be optimal when training is performed on the full dataset. This observation also suggests an interdependence between the criterion and the approach. Another example of this interdependence can be found when comparing the results for the union-a approach, where criteria that rely on crowd-level risk estimates outperform those that use aggregation-level risks, to the results for the coin approach, where aggregation-level criteria take the lead. Consequently, when it is unclear which criterion will pair best with a given approach, the ensemble-based criterion ENS offers a robust compromisc.
RQ1: Takeaway
A Aggregation-level and crowd-level HPS criteria, estimating risks only from noisy crowdworker labels, enable effective HPO in LFC. Combining them via the ensemble- based criterion ENS yields the highest performance across datasct variants and LFC approaches.
RQ2: Given the best-evaluated HPS crilerion for crowd-labeled validation data, how do LFC approaches compare in performance?
We compare the LFC approaches whose HPCs have been selected via the best-evaluated criterion ENS across all dataset variants and report the performance results again rankings and zero-one loss reductions in
15

---

Published in Transactions on Machine Learning Research (11/2025)
Baselines
Class-dependent
Instance Instance-dependent
nv
da
c1
trace
conal
union-a $ funion-b
geo-w
go0-1
Fnad
crood-ar $ aanot-mixx coin
+12
+10-
Barfnee) Lo/1 [%6p]
18
14
+2-
TRUE" DEF-DATA
DEF
ABU
AEC
ALU
ALC
CXU
CEC
CLC
ENS
Figure 6: Absolute zero-one loss reductions of LFC approaches. For each HPS criterion (æ-axis), the scatter plot displays the mean and standard error of an LFC approach's reduction in zero-one loss (y-axis) as percentage points [%p] compared to majority voting (mv) trained with its default (DEF) HPC (see Appendix C.2 for relative reductions as percentages [%]). Higher reductions correspond to greater improvements. A marks criteria that had access to the true validation labels.
Table 5 In addition, this table lists for each approach the number of significant wins, significant losses, and non-significant ties against all other approaches. These numbers are obtained via the same test procedure as in RQ1 with each dataset variant as a block. The results underscore the benefits of onc-stage LFC approaches that estimate class- or instance-dependenl crowdworker performances, as the two-slage baseline approaches and ds (Dawid & Skene 1979) attain the worst ranks and have the most significant losses. Moreover, they provide the lowest zero-one loss reductions compared to training my with its default (DEF) HPC. The two one- stage approaches geo-f (Ibrahim et al. 2023) and coin [Nguyen et al. 2024) provide significant performance gains to most of their competitors with an average zero-one loss reduction of around 10%p corresponding to a relative improvement of over 30% compared to training mv with DEF as a criterion. Both approaches' idea is identify the crowdworkers' confusion matrices via special regularization terms, where geo-f estimates a single confusion matrix per worker, while coin relaxes this assumption by modeling instance-dependent outlier terms for the crowdworkers' confusion matrices. Figure breaks down the approaches' absolute zero-one loss reductions compared to training mv with the default (DEF) HPC according to the HPS criteria. We observe that similar absolute reductions of the zero-one loss around 10%p are also achievable when relying on other criteria, such as ALC paired with the annot-mix approach (Herde et al.) (024b).. Moreover, the results indicate that the comparison betweon the approaches' performances is affected by the choice of the criterion. For example, the loss reduction of the approach coin is much inferior to the reduction of the approach annot-mix for the baseline criterion DEF-DATA, but is superior when relying on ENS as a criterion.
To systematically analyze the impact of a criterion comparing the approaches, compute the Kendall rank correlation coefficient [Kendall 1945) to determine hhe the various criteria disagree on the ranking of the approaches. Figure 7 shows violin plots of these coefficients across all dataset variants for 6 example criteria pairs. Each coefficient is a kind of distance measurement between two rankings of the approaches obtained after using two different criteria for the same dataset variant. The coefficients are in the interval [],,1], where -1 indicates a perfect negative ordering, no ordering, and perfect positive ordering. In the absence of ties, a Kendall 7- coefficient of 0.50 between two rankings of the approaches means that if you randomly sample pair of approaches, the probability that both criteria rank the same approach higher than the other exceeds the probability that they disagree by 50%, (75% 25%). The criteria DEF and DEF-DATA paired with ENS or TRUE have an average Kendall r-b coefficient of around 0.25, showing only modest. agreement in the ranking of LFC approaches. Criterion ENS versus criterion TRUE has an average Kendall r- coefficient of about 0.40, reflecting a moderate positive overlap of rankings. The violin plots, however, make clear that these average Kendall 7-b coefficients mask considerable dispersion: dataset- wise coefficients spread widely around each average, and some even take negative values, showing that two criteria more often disagree than agree on the ordering of approaches.
16

---

Published in Transactions on Machine Learning Research (11/2025)
+1.00
+0.75-
Kendall Rank Correlation
+0.50-
Coefficiett 7-b
+0.25
0-
0.25
-0.50-
0.75
TRUE" vs. DEF-DATA TRUE" V8. DEF
TRUE" V8. ENS DEF DEF-DATA" ENS VS. DEF-DATA" ENS DEF
Figure 7: Rank correlation between HPS criteria. Each violin plot shows the mean and distribution of pairwise Kendall r-b coefficients, shown as violet dots and obtained from 35 dataset varianls when comparing the ranking of LFC approaches with their HPCs selected via baseline and ensemble- based criteria. Higher coofficients indicate stronger correlation. marks criteria that had access to the true validation labels.
RQ2: Takeaway
With the HPS criterion ENS, all one-stage LFC approaches, particularly geo-f and coin, outperform the two-stage baselines. The approaches' gains and rankings vary with the criterion, underscoring the importance of criterion for fair and realistic evaluation.
4.3 Hyperparameter Optimization in Learning from Crowds: A Guideline
Based crowd-hpo with its experimental study and related experimental studies (see Section 2 propose guideline to perform HPO given (new) crowd-labeled dataset. Figure g summarizes this guideline as a flowchart that can be used both by researchers, who design and evaluate LFC approaches on benchmark datasets, and by practitioners, who apply an LFC approach to a crowd-labeled target dataset. The flowchart is be applied to each pair of a target dataset and an LFC approach obtain an optimized HPC. Any subsequent steps conducted depend on the specific objectives of the application or research context.
Choose single
HPS criterion.
ennemmble-hssdd
HPS criterion.
Inputs are target datnett, benchmark datsowt(s), LPC approsch, HP soarch spaes, stralogy. condidsse HPS criteria.
ovaluntion on crord labeled bencmmrkk distnoctis)
Eraiuate HTPS criteria lunchrack datroet(4) and chonse the best HPS criterion.
Apply chosen HPS ceiterion with your appzoach to the targrt datacot.
multiple candidate HPS
Yes
Output aptimited HPC.
labels
possible!
Figure 8: Guideline to perform HPO for (new) crowd-labeled dataset. If only one candidate HPS criterion is available, it is applied directly. If multiple criteria cxist, they are either evaluated on benchmark datasets (when available) to choose the best HPS criterion (which may itself be an ensemble) or, in the absence suitable benchmarks, combined into an ensemble. The chosen HPS criterion is then used to optimize the HPC on the target dataset.
For each run of the flowchart, we need to fix the following inputs:
Target dataset: For practitioners, the crowd-labcled target dataset is typically fixed by the application, and the flowchart is run once for chosen LFC approach obtain an optimized HPC. Researchers may consider multiple benchmark datasets and various LFC approaches, and therefore the flowchart.
17

---

Published in Transactions on Machine Learning Research (11/2025)
repeatedly, treating each benchmark in turn as the target datasct, while the remaining benchmarks serve as auxiliary datasets.
Both practitioners and researchers may use lrowedlabeled benchmark datasets with additional true labels to compare candidate HPS criteria. When curating benchmarks, we recommend datasets with noisy class labels from human crowdworkers (see Table 2 rather than purely simulated crowdworkers. Where possible, multiple noisy label variants can be derived from these datasets (Wei et al. Herde et al. 2024a) to study different noise levels and label redundancies.
Benchmark datasets:
2022
Learning from crowds approach: From a practitioner's perspective, an LFC approach is ideally state- of-the-art and empirically robust (see results in Table 5 for orientation). In contrast, researcher repeatedly applies the flowehart to cover multiple LFC approaches that differ in their training principles and crowdworker performance modeling, enabling systematic comparisons. For example, including both two-stage and one-stage training approaches (Li et al. 2022) allows us to contrast label aggregation as a separate step with end-to-end training. Likewise, comparing methods that model class-dependent and instance-dependent crowdworker performance (Herde et al. 2023) reveals when the additional modeling complexity beneficial.
Hyperparameter search space and simieeg:: The efficiency and effectiveness of HPO depend on the HP search space and strategy (Bergstra & Bengio 2012). For example, overly broad spaces waste budget and obscure findings. Practitioners commonly favor tight, well-motivatod spaces around scttings known work robustly, combined with simple strategies such as random search or Sobol sequences. Researchers may explore richer spaces, but should still focus HPs that are critical for the behavior of the LFC approach (see Table 3 for orientation). We therefore advocate that new LFC approaches provide concrete recommendations for their approach-specific search spaces. When such guidance is unavailable, reasonable ranges should be derived from theoretical considerations and the functional role of each HP rather than being chosen arbitrarily.
Candidate hyperporumeter selection criteria: All HPS criteria that are compatible with the chosen LFC approach may serve as candidates. Relying on default (DEF) HPCs typically results in suboptimal perfor- mance of an LFC approach. For a target dataset, where assuming the availability of a separate validation set with true labels is reasonable, validating with those labels (corresponding to the criterion TRUE) is fine. Otherwise, criteria designed for crowd-labeled validation data, such as those provided by crowd-hpo, are to be used. In practice, the optimal HPS criterion depends on the LFC approach and target dataset, so we recommend including at least one simple crowd-based criterion (e-g., AEU) and robust ensemble-based criterion (e.g., ENS) among the candidates. In the future, new LFC approaches are ideally introduced in conjunction with HPS criteria tailored to their specific characteristics, reflecting that HPO under noisy validation labels is a central challenge in related settings, such as partial label learning WWagg et al. 2025
After fixing the inputs, we must choose an HPS criterion. There is only single candidate HPS criterion when, for example, prior evidence already identifies suitable choice for the given LFC approach, or when TRUE is applicable as an HPS criterion because true validation labels are available for the target dataset. If multiple candidate HPS criteria are available, an extra choice is required. When suitable crowd-labeled benchmark datasets with true test labels or sufficient computational resources are unavailable, we recommend using the ensemble-basee criterion ENS as the default. Otherwise, one may instead compare the candidate criteria on these benchmarks, as in our study: run HPO for each criterion on the benchmark datasets and select the one that, on average, yields the best test performance, e.g., the lowest mean zero-one loss
Finally, we obtain an optimized HPC by rumning HPO the target dataset, applying the chosen HPS criterion. Specifically, for each candidate HPC, the LFC approach is trained on crowd-labeled training data and evaluated on (mostly crowd-labeled) validation data using the chosen HPS criterion. Practitioners typically perform this procedure once on their application dataset for deployment, whereas researchers repeat. it across benchmark datasets and LFC approaches as the basis for comparative analyses.
Thii procedure does not necessarily require scparate HPO runs with model retrainings: when using static HP search strategies, e.g., Sobol sequences, grid search, random search, multiple HPS criteria can be evaluated on the same set tried HPCs, so only the decision which HPC best under the chosen HPS criterion needs he recomputed.
18

---

Published in Transactions on Machine Learning Research (11/2025)
5
Limitations
Our three core contributions, i.c., the proposed HPS criteria (see Section 3.4) Section 4.2 and the guideline for HPO in LFC settings (see Section 4.3 are subject to limitations.
the experimental study (see
These proxy HPS criteria, whose underlying empirical risk estimates access only crowd-labeled validation data, offer no theoretical consistency guarantees, i.e., that they will asymptotically select the HPC minimizing the expected risk of an LFC approach. In general, the true validation labels are not identifiable from crowd labels alonc: different combinations of latent true labels and crowdworker performance can induce the same distribution over observed noisy labels (Tanno et al. 2019). Any general guarantee, therefore, requires explicit structural assumptions on the noise in the labels, e.g., crowdworkers being better than random or identifiable noise models. As we do not impose such assumptioss, we can not claim universal guarantees. Instead, we provide a comprehensive empirical evaluation of the proposed HPS criteria as practical proxies for validation performance with true labels on real crowd-labeled datasets.
Our experimental study does not investigate aspects, such as the HPO search stratogy, the number of folds K for cross cross-validation. and fine-tuning pre-trained backbones. These aspects impact the computational complexity of HPO. Furthermore, our analysis based on computing means across all datasets does not, account for the influence of specific dataset attributes, including the number of crowdworkers and label rodundancy. While our obscrvations refer to the zero-one loss function, alternative loss functions such the Brier score (Brier 1950) may also be relevant when assessing probabilistic estimates. An initial analysis (see Appendix C.2) confirms the benefit of employing the proposed HPS criteria for the Brier score as the target loss function, while the choice of the best HPS criterion changes. Finally, we refer to the number of candidate HPCs |A| as the computational budget, which does not account for different training and inference complexities between the LFC approaches.
Our guideline for HPO in LFC settings is limited to empirically motivated advice on how choose an HPS criterion for given LFC approach and (new) target dataset. In particular, selecting an HPS criterion based experiments with related benchmark datasets does not guarantee that this criterion will also be optimal on the target dataset. A possible direction for future work is to exploit latent structure in dataset characteristics to inform the choice of HPS criterion, or to adaplively tune the weights in the ensemble-based criterion ENS using meta-learning techniques (Brazdil et al. 2022). Finally, while the guideline outputs an optimized HPC, does not address the practical requirement for a reliable, independent test set to confirm the classification model's performance before deployment.
6 Conclusion
We introduced crowd-hpo as a framework to enable more realistic and fairer benchmarking of LFC approaches by leveraging HPO with only access to crowd-labeled validation data. We started with exemplary results demonstrating notable zero-one loss reductions and changes in the rankings of LFC approaches when performing HPO with true class labels in the validation set compared to default HPCs. Subsequently, we ified a lack of research regarding HPO with crowd-labeled validation data. Therefore, we proposed and evaluated HPS criteria accounting for the potential noise in class labels from crowdworkers. Across extensive experiments, our proposed HPS criteria strongly reduced the losses of the LFC approaches relative to their default HPCs. This applies particularly to the ensemble- based criterion ENS, which is also easily extensible by including future empirical risk measures for crowd-labeled validation data. However, the ranking of LFC approaches shifted with the criterion applied. These findings grounded our guideline for future experimentation and benchmarking in LFC settings. To further improve the HPS, future work should rigorously explore advanced HP search strategies, particularly Bayesian optimization (Wang et al. 2023 and examine how they interact with criteria accounting for crowd-labcled validation data. Progress can also be achieved through HPS criteria with alternative loss functions, label weighting, label aggregation, or resampling schemes, e.g., by allowing per-class crowdworker biases during validation. Finally, crowd-hpo serves as a starting point for developing empirical risk mcasures that are not only suited for improving HPS given crowd-labeled validation data but ultimately should provide reliable estimates of the approaches' generalization performances given only crowd-labeled test data.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Broader Impact Statement
In broader context, we identify impact on real-world applications and crowdworkers two branches of potential societal consequences of crowd-hpo. (1) On the one hand, the validation with noisy class labels from the crowd makes the LFC approaches" employment more practical in real-world applications because potential users do not need to rely on default or manually picked HPCs when training the LFC approaches. However, selecting the best HPC is different from accurate risk estimates. In the first case, we only need to rank the HPCs, while the latter case, we require precise estimate of the risk associated with the resulting data classification model. In other words, despite being able to optimize the selection of the HPC, do not know the actual generalization performance of the data classification model after training. Correspondingly, there is still no solution to how practitioners can obtain such an estimate without access to a separate test set with true labels. Therefore, practitioners must only employ LFC approaches in safety-critiall applications after thorough evaluation based on a separate test set with true labels. (2) On the other hand, wider adoption of LFC approaches can raise demand for crowdworkers. However, crowdworkers often endure difficult working conditions (Bhatti et al.) 2020), e.g., insufficient payments and job insecurity. Therefore, collecting crowd-labeles data for training LFC approaches should always be coupled with explicit provisions for fair working conditions.
Acknowledgements
This work was funded by the ALDeep (P/681) and CIL (P/710, P/1082) projects through the University of Kassel.
References
Salisu Mamman Abdulrahman, Pavel Brazdil, Jan N. Van Rijn, and Joaquin Vanschoren. Specding up algorithm selection using average ranking and active testing by introducing runtime. Mach. Learn., 107: 79-108, 2018
Anthony Bagnall and Gavin C. Cawley. On the Use of Default Parameter Settings in the Empirical Evaluation of Classification Algorithms. NXY:::703306777 2017.
James Bergstra and Yoshua Bengio. Random Search for Hlyper-Paramete Optimization. J. Mach. Learn. Res., 13(2):281-388 330,, 2012.
Shahzad Sarwar Bhatti, Xiaofeng Gao, and Guihai Chen. General framework, opportunities and challenges for crowdsourcing techniques: A comprehensive survey. J. Syst. Softw., 167:110611, 2020.
Pavel Brazdil, Jan N. van Rijn, Carlos Soares, and Joaquin Vanschoren. Metalearning for Hyperparameter Optimization. In Metalearning: Appl. Autom. Mach. Learn. Dala Min., pp- 103-122. Springer, 2022.
Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability. Mon. Weather Rev., 78(1):1 3, 1950.
Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds. In Int. Conf. Learn. Represent., 2019.
Zhi Cao, Enhong Chen, Ye Huang, Shuanghong Shen, and Zhenya Huang. Learning from Crowds with Annotation Reliability. In Int. ACM SIGIR Conf. Res. Dev. Inf. Retr., pp. 2103 2107, 2023.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-An Heng. Robustness of Accuracy Metric and its Inspirations Learning with Noisy Labels. In AAAT Conf. Artif. Intell., pp. 11451-11461, 2021.
Ziqi Chen, Liangxiao Jiang, and Chaoqun Li. Label augmented and weighted majority voting for crowdsourcing. Inf. Sci., 606:397-409, 2022.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Zhendong Chu, Jing Ma, and Hongning Wang. Learning from Crowds by Modeling Common Confusions. In AAAT Conf. Arlij. Intell., pp.
2021.
Alexander P. Dawid and Allan M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.), 28(1):20-25) 1979.
Jean-Charles de Borda. Mémoire sur les élections au scrutin. Histoire de I' l'Académie Royale des Sciences, 1781.
Milton Friodman. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. J. Am. Stat. Assoc., 32(200):675 701, 1937.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. In Int. Conf. Artif. Intell. Stal., pp- 315-323, 2011.
Hui Guo, Grace Yi, and Boyu Wang- Learning from Noisy Labels via Conditional Distributionally Robust. Optimization. In Adv. Neural Inf. Process. Syst., pp. 82627-8267777 2024.
Bin Han, Yi-Xuan Sun, Ya-Lin Zhang, Libang Zhang, Haoran Hu, Longfei Li, Jun Zhou, Guo Ye, and Huimei He. Collaborative Refining for Learning from Inaccurate Labels. Adv. Neural Inf. Process. Syst., pp. 92745 92768, 2024.
Marek Herde, Denis Huseljic, Bernhard Sick, and Adrian Calma. A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification. IEEE Access, 9:166970 166989, 2021.
Marek Herde, Denis Huseljic, and Bernhard Sick. Multi-annotator Deep Learning: A Probabilistic Framework for Classification. Trans. Mach. Learn. Res., 2023.
Marek Herde, Denis Huseljic, Lukas Rauch, and Bernhard Sick. dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans. In Adv. Neural Inf. Process. Syst., pp. 11187-51117, 2024a.
Marek Herde, Lukas Lührs, Denis Huseljic, and Bernhard Sick. Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via Mixup Extension. In Eur. Conf. Artif. Intell., pp. 2910-291111 2024b.
Sture Holm. Simple Sequentially Rejective Multiple Test Procedure. Scand. J. Stat., 6(2):65-70, 1979.
Shahana Tbrahim, Tri Nguyen, and Xiao Fu. Deep Learning From Crowdsourced Labels: Coupled Cross- Entropy Minimization, Identifiability, and Regularization. In Int. Conf. Learn. Represent., 2023.
David I. Inouye, Pradeep Ravikumar, and Pradipto Das. Hyperparameter Selection under Localized Label Noise via Corrupt Validation. In Learn. Limit. Label. Data Workshop, 2017.
Sergey Toffe and Christian Szegedy. Batch Normalization: Accelcrating Deep Network Training by Reducing Internal Covariate Shift. In Inl. Conf. Mach. Learn., pp. 448 456, 2015.
Liangxiao Jiang, Hao Zhang, Fangna Tao, and Chaoqun Li. Learning From Crowds With Multiple Noisy Tabel Distribution Propagation. TEEE Trans. Neural Netan Learn. Syst., 33(11):6558 6568, 2022.
Maurice G. Kendall. The meet of Ties in Ranking Problems. Biomelrika, 33(3) 251, 1945.
Ashish Khetan, Zachary C. Lipton, and Animashree Anandkumar. Learning From Noisy Singly-labeled Data. In Tnt. Conf. Learn. Represent., 2018.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Master's thesis, University of Toronto, 2009.
Kevin Kuo, Pratiksha Thaker, Mikhail Khodak, John Nguyen, Daniel Jiang, Amect Talwalkar, and Virginia Smith. On Noisy Evaluation in Federated Hyperparameter Tuning. In Annual Conf. Mach. Learn. Syst., pp. 2023.
21

---

Published in Transactions on Machine Learning Research (11/2025)
David Lewis. Reuters-21578 Text Categorization Collection. UCI Machine Learning Repository, 1987. URL https://ddi.orrg /0. 24432/C52G6M
Jingzheng Li, Hailong Sun, and Jiyi Li. Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Mach. Learn., pp. 1-23, 2022.
Shikun Li, Tongliang Liu, Jiyong Tan, Dan Zeng, and Shiming Ge. Trustable Co-Tabel Learning from Multiple Noisy Annotators. IEEE Trons. Multimed., 25:1045 1057, 2021.
Shikun Li, Xiaobo Xia, Jiankang Deng, Shiming Gey, and Tongliang Liu. Transferring Annotator- and Despndcc- Transition Matrix for Learning From Crowds. IEEE Trans. Pottern Anal. Mach. Intell, 46(11):7377-7333 2024.
Liyuan Liu, Haoming Jiang, Pengeheng Hc, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the Variance of the Adaptive Learning Rate and Beyond. In Inl. Conf. Learn. Represent., 2019.
Nya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In Inl. Conf. Learn. Represent., 2017.
Tri Nguyen, Shahana Ibrahim, and Xiao Fu. Noisy Label Learming with Instace-Deeendenn Outliers: Identifiability via Crowd Wisdom. In Adv. Neural Inf. Process. Syst., pp.
2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Trans. Mach. Learn. Res., 2023.
Bo Pang and Lillian Lee. Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. In Annu. Meet. Assoc. Comput. Linguist., pp. 115-124, 2005.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Porformance Deep Learning Library. In Adv. Nearal Inf. Process. Syst., 2019.
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from Crowds. J. Mach. Learn. Res., 11(44)::297-1322, 2010.
Nils Reiners and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Netword Conf. Empir. Methods Nat. Lang. Process. Int. Jt. Conf. Nat. Lang. Process., pp. 3982-30 3992, 2019.
Filipe Rodrigues and Francisco C. Pereira. Deep Learning from Crowds. In AAAI Conf. Artif. Intell., pp. 1611-1618, 2018.
Filipe Rodrigues, Francisco C. Pereira, and Bernardete Ribeiro, Learning from multiple annotators: Distin- guishing good from random labelers. Pattern Recognit. Lett., 14(12))1428-1443 2013.
Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C. Pereira. Learning Supervised Topic Models for Classification and Regression from Crowds. IEEE Trons. Pattern Anal. Mach. Intell., 39 01:220992222 2017.
Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. LabelMe: Database and Web-Based Tool for Image Annotation. Tra. Comput. Vis., 77(1-33:157-173 2008.
Ilya M. Sobol. On quasi-Monte Carlo integrations. Math. Comput. Simul., 47(2-5):103-112 -12, 1998.
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac-Gil Lec. Learning From Noisy Labels With Deep Neural Networks: A Survey. IEFE Trans. Neural Netu. Learn. Syst., 34(11):8135 1158555553,, 2022.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. MPNet: Masked and Permuted Pre-training for Language Understanding. In Adv. Neural Inf. Process. Syst. pp. 16867-16667, 2020.
Ryutaro Tamno, Ardavan Sacedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Silberman. Learning from Noisy Labels by Regularized Estimation of Annotator Confusion. In Conf. Comput. Vis. Pattern Recognit., pp. 11244-11253, 2019.
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. IEEE Trans. Speech Audio Process., 10(5):293-302, 2002.
Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
Jennifer W. Vaughan. Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research, J. Mach. Learn. Res., 18(193):1-46, 2018.
Wei Wang, Dong-Dong Wu, Jindong Wang, Gang Niu, MingLing Zhang, and Masashi Sugiyama. Realistic Evaluation of Deep Partial-Label Learning Algorithms. In Int. Conf. Learn. Represent. 2025.
Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent Advances in Bayesian Optimization. ACM Comput. Surv., 55(13)):1-366 2023.
Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. Deep Learning From Multiple Noisy Annotators A Union. IEEE Trans. Neural Netw. Learn. Syst., 1112):1055521056222 2023.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with Noisy Labels Revisited: Study Using Real-World Human Annotations. In Int. Conf. Learn. Represent., 2022.
Suqin Yuan, Lei Feng, and Tongliang Liu. Early Stopping Against Label Noisc Without Validation Data. Int. Conf. Learn. Represent., 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding Deep Learning Requires Rethinking Generalization. In Int. Conf. Learn. Represent., 2017.
Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. Coupled Confusion Correction: Learning from Crowds with Sparse Annotations. In AAAI Conf. Artif. Intell., pp. 16732-16740, 2024.
Jing Zhang, Xindong Wu, and Victor S. Sheng. Learning from Crowdsourced Labeled Data: A Survey. Artif. Intell. Rev., 46(4):543-577 576, 2016.
Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander. Disentangling Human Error from Ground Truth Segmentation of Medical Images. 2020.
Adv. Neural Inf. Process. Syst., pp.
A Inference Mechanisms of Learning from Crowds Approaches
This appendix overviews the common inference mechanisms of LFC approaches to better understand the connections between the data classification model fe, the crowdworker classification model Jø, and the crowdworker performance model hy. Moreover, the probabilistic estimates of Eqs. 3- are required to evaluate our presented HPS criteria. To explain the inference, we distinguish between two architecture types employed by LFCs approaches: the ones with probabilistic confusion matrices and those with non-probabiliste noise adaptation layers.
23

---

Published in Transactions on Machine Learning Research (11/2025)
A.1 Probabilistic Confusion Matrices
Many LFC approaches (Dawid & Skene 1979 Tanno et al.
2019 Ibrahim et al. 2023 Cao et al. 2023
Herde
2024) estimate crowdworker performances through confusion matrices, et al. 2023 2024b Nguyen et al.
which we formalize as a function Qa Ωx x [M] + Ag with parameters 3 E S2p. Thereby, confusion
matrix entry has the following probabilistic interpretation:
QQ m)]c. == Pr (Zmm = Ex yn Ec, m,
(30)
This confusion matrix entry row c E [C] and column the class label 6j to instance ain with e, as its true class label. Depending the assumptions of the LFC approach, there are confusion matrices differing in their degrees of freedom v E Ny50 (Herde et al. 2023) Here,
[C] is the probability that crowdworker m assigns
we distinguish between class-independent (v = 1) and class-dependent (v C(C - 1)) confusion matricos. Moreover, the confusion matrices can be modeled as instance-inddeennden::
Van, a7 Ωx (m m) Qa(an,
(31)
or as an instance-dependent function. Despite different assumptions about confusion matrices, the LFC approaches share the inference schemes for their crowdworker classification model with parameters = (0, B):
(Tn; m) = Q2 fo(x.)
(32)
and for their crowdworker performance model with parameters +b = (0,
( m))= = Σ
[fo(an)]e (Q m)lc,e-
(33)
cE|C]
A.2 Non- non-probabillitic Noise Adaptation Layers
In contrast to probabilistic confusion matricos, we refer to noise adaptation layers in LFC approaches (Ro-
drigues & Pereira 2018 Chu et al. 2021 Wei et al. 2023) as unconstrained transformations of the estimated
true class probabilities. For this purpose, the approach cl (Rodrigues & Pereira 2018) introduces a noise
adaptation layer Wm E RBCC for each crowdworker m E [M]. Then, the crowdworker classification model
with parameters Φ (0, W WI...., WM) performs inference via:
9 (2, m) = softmax (W.f)(
(34)
The approach conal (Chu et al. layer W E C'C modeling common confusions across crowdworkers, which leads to:
2021 extends the crowdwwrkker-seeddfic noise adaptation layers by another
(G m) = (1 - sy( m)) softmnx ((w)) + sy(2,, softmax (w" fa(wa))
(35)
where Sy : 22x x [M] [0, 1] is an auxiliary network with parameters y Sr. It outputs an instance- and nrowdworker-dependderr scalar the estimated degree to which a crowdworker's class label distribution follows common confusions across crowdworkers. Accordingly, the crowdworker classification model in Eq. (35) has the parameters Φ == (0, W...., WM, W. -7) Another variant of noise adaptation layer is implemented by the LFC approaches union-a and union-b WWei et al. 2023) Instead of treating the crowdworkers independently, the two approaches model the crowdworkers as union via single noise adaptation layer W CC(CM))) Therelore, they do not directly implement crowdworker classification model but a classification model go : Ix + AC-M parameterized by & = (0, W) and treating the crowdworkers class labels as union:
(x) = softmax (WT fo(xx)
(union-a).
(36)
To (xx)) = softmax (w) fo (x
(union-b),
(37)
24

---

Published in Transactions on Machine Learning Research (11/2025)
where the softmax is applied row-wise in the case of union-b. As workaround for approximating the crowdworker classification model, we normalize the outpuis associated with each crowdworker, which corresponds to:
I (T, m) == normalize (2m))( (0)(
(38)
where Hay denotes the entries from index to index in vector. For all these LFC approaches, which do not explicitly implement probabilistic confusion matrix per crowdworker, resort to using marginal alignment accuracy, which is computed as the agreement between the predicted crowdworker distribution and the predicted true label distribution as an instance-level proxy measure for crowdworker accuracy. Consequently, the crowdworker performance model with parameters up == (0, σ) performs inference via:
(a m) = Sō (æ.) (Za,
( m).
(39)
B Theoretical Analysis of Hyperparameter Selection Criteria
This appendix expands on our design decisions underlying the HPS criteria framework introduced in Section 3
Using simple propositions, we show that richer modeling assumptions can inject class-specifie bias into the posterior computation of Eq. (16). We then prove that the computation we adopt in Eq. (17) is immune this issue and remains class-agnostic. Finally, overview the time complexities of the HPS criteria.
B.1 Prior Class Probabilities
Proposition ! motivates our design choice not to use non-uniform prior class probabilities when computing the posterior class probabilities in Eq. (16). Intuitively, if the prior class probabilities are sufficiently biased toward particular class, this bias will dominate in the posterior regardless of the observed class labels from the crowdworkers. Therefore, relying on the class probabilities estimated by the data classification model fo as prior can bias the aggregated class labels towards the data classification model's predictions.
Proposition For an instance In E N, let us assume strictly positive likelihoods for the class labels such that Ve, E Qy and Vm Mn:
Pr(znm En, Un = Cc, m²) > 0.
(40)
Then, for any non-emply allocation of observed class labels 2, E B(srr) \ {0} and for any class label E Sry there erists constant (0,1) such that from Pr(y = ex(2rn) follows:
2(xnn Bn) = Ck-
(41)
Proof For fixed instance 2n X' and class label ex Nr, we first define
1e = II
Pr(znm (nn Un = Ec, m);
(42)
M..MA
1max
max (E(C])(A)
(lo)
(43)
1max
(44)
Imax +
Now, let Pr(yn- ex|xxn) > s. Our goal is to show that for any e, E Gy Mee} we have Pr(y. Ex [x, Z,)) >
Pr(y.. ee ecl 2n), since then we get 2(m,, 2n) et- Starting from Bayes' theorem in Fq.. (15 and the
25

---

Published in Transactions on Machine Learning Research (11/2025)
conditional independence assumption in Eq.
16 have
Pr(yn- Ex In. Zn)
Pr ex (n
II meM,.
Pr Inn yn ek. m)
(45)
Pr(2n| km))
fanax Imax-
lmax
Pr( 2m (2 Lmax LK
(46)
Imax.
(47)
Pr(Zn1 (2.m
/mmax + bx
While the first. factor Eq. 47
with Pr(Zn, (mn) > for normalization, we get for the second factor
4
Iuxx Imax
(48)
mmx + 44
Imxx + 4x
Liuax
=1
(49)
Imax + 4
1 Pr(yn ek (m))
(50)
Pr(yn * (n)
(51)
2 Pr(y = ec xn))
(52)
where Eq. 50 follows by definition of as -Pr(yn ex (m)) < s-=
funx Imaxtly
and Eq.
52
follows by
monotonicity as yn = ec = yn # ex. For the third factor, we have
mmx 2 40
II Pr(zum In, Un Ce, m))
(53)
mEM,,
by definition of max. Incorporating both inequalities into Eq.
47
we get
l Linax
Pr(yn = ee (an))
II
Pr(znm Fn. Ce: m.)
(54)
Pr(Zn|(zn) Iumx 14
Pr(Znl xxn)
meM,,
Pr(yn = ec @cl Zn.
(55)
and therefore Pr(yn- ek |. 2n)
Pr(yn c. Fn, Zn) desired.
B.2 Crowdworker Confusion Probabilities
Proposition motivates our design choice not to use full confusion probability estimates for computing the posterior class probabilities in Eq. 16). Intuitively, suppose the full confusion probabilities for the crowdworkers are sufficiently biased toward a particular class (see sufficient condition in Eq. 58 ). that case, this bias will dominate the posterior regardless uniform prior class probabilities and observed class labels from the crowdworkers. Therefore, relying on full confusion matrices, as estimated by many LFC approaches in the form of Qa in Appendix A. can bias the aggregated class labels towards their own predictions.
Proposition For an instance x,, X, suppose uniform prior class probabilities according to Eq. Then, for any non-empty allocation of observed class labels Z,, B(Qy) \ {0} and for any class label ex E Qy there cxist confusion probability estimates Pr(zum Kn. yn m.) such that:
2(m.ns Zn) ek-
(56)
26

---

Published in Transactions on Machine Learning Research (11/2025)
Proof With uniform prior class probabilities and Pr(Zn ( > 0 for normalization, the posterior probability for class label E fy from Eq. 115 and Eq. [66 is given by:
Pr(yn = enlon, -n)=
Pr(zn (.n) C II
Pr(zum| Rn Yn = Cc. m).
(57)
mEM.
Therefore, by requiring that Ve, E fy \ fer} and vm Mn
Pr(znm Xn. Un = ex, ex, m) > Pr(znm on, Un = Ec, m),
(58)
we obtain Ve, E Ωy fext:
Pr(y,- = ex/2n. Zn)=
II mEM..
Pr(zem In. Un = Ex: m)
(59)
Pr(Zn (..)
II
Pr( Pr(Znm| yr = Cc, m))
(60)
Pr(Z,, x
mEMp
Pr(y. = en ans 2n);
(61)
which implies that e₁ E fy is our MAP estimate and, thus, our aggregated class label.
B.3 Maximum-a-posterio Estimate as Weighted Majority Vote
Proposition motivates design choice to uniform prior class probabilities in Eq. 17) and scalar performance- model for the instance-wis crowdworkers' confusion probabilities in Eq. {18) Intuitively, the posterior class probabilities correspond to soft weighted majority vote, so the MAP estimate is the label with the most. "soft votes". This weighted majority voting treats all classes symmetrically, and any difference in posterior probabilities or the MAP estimate arises only from the class labels provided by the crowdworkers, never from a built-in preference for a particular class.
Proposition 3 Given the posterior class probability computation from Eq. 19 for instance H, X with any non- non-empty allocation of observed class labels Z, E B(Qy) and assuming Pr(z yn = 1 (x,, m) E (0,
for all m Ma, the aggregated label in Eq. 12 equals weighted majority vote such that:
z 2 2n)
arg max e,Efly
Σ In
Pr(2T Un [.,; (Bn:
Pr(2T Un 0) xn. m) (z eed)
(62)
M.MMa
Proof For case of notation, let us define:
Oyym Pr(z Ur 1 1| m) (0, 1),
(63)
as the estimated (e.g., via the crowdworker performance model) labeling accuracy of the crowdworker given instance a. Then, rewrite (with the help of basic logarithmic laws) the un-normalized posterior
27

---

Published in Transactions on Machine Learning Research (11/2025)
class probability from Eq.
19
according to:
--2
Pr(yn- - eo en|2n, 2n) & exp
In II (arm)
) fa 6c
0am
C
mE..M
1 anm
1-2 mm
Σ In (amm) Fum mEM,,
exp
C
I
exp
Σ In (anm
( ) e0
In
(
1 Omm
MEM,a
1 am
exp
Σ
(zmm Ce) In(anm) + (1
Znm Cc) In
C
mEM.
exp
Σ
(= ee) [n((am)) In C
Ommm
+ In
- O'ymm
C
meM..
Σ
(2 ea)
(I((aam) In
anm
X exp
MEM,
exp
Σ In
O'ym
C ---1)
(z/mm ea)
(64)
1 Oyum
M.M..
Replacing the labeling accuracy Oym with its probabilistic definition from Eq. arg max function in Eq. 662
63
we gel the input of the
B.4 Worst-case Time Complexity Analysis
We analyze the time complexity when evaluating an HPS criterion as one part of the HPO pipeline. More specifi- cally, Table B presents the worst-case time complexity of individual steps involved by the non-default HPS criteria in O-notation. Typically, the training time complexity is the most expensive part of HPO. Figure g provides an example of this. Here, the training times are almost 100 times higher than computing the empirical risks (including previous steps such as prediction computation) for evalu- ating the HPS criterion ENS. Even when using GPU, the difference can be higher for more complex neural network architectures, such as when fully fine-tuning backbone. Accordingly, differences in time complexity when evalu- ating the various HPS criteria are of minor importance. Since the training lime complexity identical for each HPS criterion, we do not further analyze this complex- ity and instead focus on the steps related to the actual evaluation of an HPS criterion in the following.
180
154.22
120
Time
60
1.66
0-
Training
Empirical Risk
Figure 9: Computation times. Given AMD
Ryzen 9 7950X as CPU, the sum of training times for the LFC approach coin (Nguyen et al. 2024 with its default HPC and the times for comput- ing all empirical risks in R (see Section 3.4 are reported in seconds across K 5-fold cross- validation for the dataset variant dopanim-full.
Prediction Computation Computing model predictions represents the first step of an HPO criterion's evaluation. This step's complexity is allected by which of the three models, i.e., data classification model fo, crowdworker classification model gø. or crowdworker performance model hg, is required to make predictions. If only the data classification model is involved, we need class probability predictions for each validation instance. When performing a K-fold pross-validttion each instance used for validation once. Accordingly, this step scales linearly with N as the number of observed instances and |A] as the number of candidate HPC'.
28

---

Published in Transactions on Machine Learning Research (11/2025)
Table Worst-case time complexity analysis. Each row corresponds to one HPS criterion, with its name indicated by the first column. The other column headings refer the steps of evaluating an HPS criterion. The colors distinguish between baseline, aggregation-leved crowd-level, and ensemble- criteria. There are N observed instances, M crowdworkers, | candidate HPCs, and C classos. The variable T denotes the worst-case time complexity of performing a single forward pass for any model trained by LFC approach. A * marks criteria that had access to the true validation labels.
Hyperparamcter Selection Criterion
Prediction
Posterior Label Probability Estimation
Empirical Risk
Winner Selection
Computation
Computation
TRUE"
O N
N/A
O[]A] N C)
O(AA
ABU
-
M.C) O[AA
N
N C
O(AA
QQA
OQA
O[
C)
O(A)
ALD
[[]A] N M.
OQA N M.C
O(A
N C
O(A
ALC
( T
OQA M.
[
O(A)
CXU
( N M
N/A
( N. M
O(A)
CBC
(
M
O[AI N M.C)
( N. M
O(A
([A]
N. M
о N
M.C)
( N
O(A)
ENS
O ([A) MT)
O N M.
O A CCC
O(J [A] In A.)
The variable T denotes the time complexity for obtaining a prediction from one of the three models. The
step's 's complexity increases if the crowdworker classification or crowdworker performance model is involved, because then we need to compute for each of the N instance-croodwworke. pairs a prediction in the worst case, where each crowdworker has labeled every observed instance.
Posterior True Label Probability Estimation The HPS criteria TRUE and CXU do not require any posterior probabilities. For the other criteria, the posterior probabilities are obtained according to Eq. (19 by iterating over all of the M (worst case) observed class labels per instance for each of the C classes. Accordingly, this computation has complexity of O(M C') and must be repeated for each of the N observed instances and for each of the |A| candidate HPCs. For the masemble-based criterion ENS, this process is additionally repeated for each of its J members.
Empirical Risk Computation Given the probabilistic model predictions and the targets, this step refers to computing the zero-one losses, which are then averaged to obtain the empirical risk measurement. For the HPS criterion TRUE and the aggregation level HPS criteria, we need to find one of the C class labels with the maximum probability predicted by the classification model for each of the N observed instances. In the case of the crowd-level criteria, this step extends to each of the N M mnstancecrroddworke pairs. We need to repeat this step for each of the |A] candidate HPCs. For the HPS criterion ENS, this step must also cover each of its J members.
Winner Selection For nearly all HPS criteria, we find the winner HPC by selecting the one with the lowest empirical risk from the |A| candidate HPCs. Only for the ensemble-based HPS critcrion, we need to compute the Borda count. this case, the complexity of sorting the |A] candidate HPCs according each of their J empirical risk measurements dominating the subsequent sunmation and finding of the minimum. In other words, the reported worst-case complexity corresponds sorting J lists of |A] elements with the merge sort algorithm.
C Experimental Evaluation
This appendix describes our computational resources for experimentation and provides additional results the experimental evaluation presented in Section
C.1 Computational Resources
Table lists the results for all 35 dataset variants, 13 LFC approaches, and 11 HPS criteria. Morcover, we report the results for training with the ground truth (gt) class labels the upper baseline approach. Each test zero-one loss value is the result of determining the selected HPC from candidate set of |A] 51 HPCs
29

---

Published in Transactions on Machine Learning Research (11/2025)
via K = 5-fold cross-validation, of which 50 HPCs have been generated via Sobol sequences (Sobol 1998
and one corresponds to the default (DEF) HPC. Subsequently, each selected HPC is tested with 5 different. initializations of the respective neural network architecture. In total, this corresponds to almost
14
51
5
11
= 151,900
(65)
datasets
variants approaches
HPCs training & validation criteria
training & testing
training and evaluation runs. We executed all runs on a compute cluster equipped with NVIDIA A100 and V100 GPU servers, which we used to pre-compute the image and text embeddings. Almost all other computations were executed with AMD EPYC 7742 CPU servers. The time measurements from Appendix B.4
are the only exception because they refer to an AMD Ryzen 9 7950X as the CPU of local workstation.
C.2 Supplementary Results
We present supplementary results regarding the HPS criteria' and LFC approaches' performances. Further, we ablate the members' impact on the ensemble-based criterion ENS, analyze the impact of the noise level, the number of candidate HPCs, an architecture search for the data classification model, and the loss function.
(analogous to Figure 5) depicts the criteria's relative zero-one loss reductions compared to the lower baseline criterion DEF. The results confirm our observat ions for the absolute zero-one loss reductions, namely, that a criterion's benefit varies across the LFC approaches, while ENS the most robust one for noisy crowd-labcled validation data.
Hyperparameter Selection Criteria Figure 10
Baseline
Aggregation-level
Crowd-level CEC
Ensemble
DEF
TRUE*
DEF-DATA
AEC
ALU
ENS
+25
+20
Aor Lo/1 [%]]
+15
+10
+5
0-
-3
-10
12'
da cl
trace
co0a1
union-a
union-b
geo-v
geo-I
mad1
crowd-ar ennot-miix coin
Figure 10: Relative zero-one loss reductions of HPS criteria. For each LFC approach (x-axis), the scatter plot displays the mcan and standard error of a criterion's reduction in zero-one loss (y-axis) as a percentage [%]] relative to training with default (DEF) HPC. Higher reductions correspond to greater improvements. A * marks criteria that had access to the true validation labels.
Beyond reporting average reductions in zero-one loss relative to the default HPC, we also examine how performance is distributed across pairs of HPS criteria. Figure II] therefore, presents pairwise win-rate matrix, including the results from the pairwise significance tests, where each cell shows the proportion of experiments which the row criterion beats the column criterion. The matrix shows that the TRCE criterion dominates all alternatives: its win-rate is consistently higher than its loss-rate. Several cell pairs do not sum to one, indicating ties in which the two criteria selected HPCs with identical performance. For the default baselines, DEF loses to every other criterion, confirming that single global default HPC per approach is inadequate. DEF-DATA, which transfers HPCs optimized on classification tasks without any noisy labels, performs somewhat better yet still lags behind. Accordingly, optimizing the HPC of a standard classification model per dataset with access to true validation labels does not satisfy the LFC approaches' individual requirements. Among criteria that explicitly account for noisy crowd-labeled validation data, the ensemble- based criterion performs best: against every rival except TRUE, it wins more often than it loses. Together with TRUE, it is the only criterion whose loss-rate versus DEF stays below 10%.
30

---

Published in Transactions on Machine Learning Research (11/2025)
Row criterion wins more often than column criterion. Comparison is significant.
Row criterion loses more often than column criterion.
TRUE
N/A
78.46
93.85
41.76
41.10
44.05
47.62
48.33
46.43
53.10
39.12
DEF-DATA"
19.78
N/A
72.97
25.93
26.37
25.00
29.76
26.67
24.76
30.95
22.86
DEF
5.93
27.03
N/A
10.77
10.11
8.81
11.43
12.86
10.71
13.81
7.03
AEU
28.13
72.75
88.57
N/A
5.27
22.62
34.76
28.57
25.95
37.62
15.60
ABC
28.57
71.87
89.23
7.47
N/A
22.86
34.76
28.57
25.00
37.62
15.38
ALU
31.43
73.57
89.05
28.10
26.90
N/A
29.29
29.52
31,43
40.00
18.57
ALC-
28.81
68.10
86.67
37.38
35.95
24.76
N/A
30.48
31.90
25.71
22.38
CXU
32.38
72.14
85.00
34.76
32.14
24.76
33.33
N/A
12.14
30.95
16.90
CEC
32.38
73.57
87.14
34.52
30.95
28.33
35.24
12.62
N/A
30.71
15.95
CLO
26.90
67.62
84.05
34.76
33.33
28.33
19.52
22.86
20.95
N/A
17.38
ENS
34.73
75.16
91.65
30.55
27.69
26.67
36,67
26.67
25.00
37,86
N/A
TRUE"
DEF-DATA"
DEF
AEU
AEC
ALU
ALC
CXU
CEC
CLC
ENS
Figure 11: Pairwise win-rate matrix for HPS criteria. Across all datasets and LFC approaches, a cell reports the percentage [%]] which the row criterion selects HPC outperforming the HPC selected by the column criterion. A cell's color decodes whether the row criterion has more wins loses than the column criterion, whereas circle indicates whether this comparison is significant. The diagonal shows not applicable (N/A) because a criterion is not compared with itself. A * marks criteria that had access to the true validation labels.
(analogous to Figure 6 depicts the approaches relative zero-one loss reductions compared to the lower bascline approach mv mv(DEF), which corresponds to training with the majority vote labels and the default HPC. The results confirm our observations for the absolute zero-one loss reductions, namely, that one-stage approaches can achieve high performance gains, while the choice of the criterion affects the comparison of the approaches with each other.
Learning from Crowds Approaches Figure 12
Baselines
Class-dopendent
Instance-depondent nadl
[..e)v[ngee do
trace
conal
$ union-a bunian-b goo-x geo-f
croud ar annot coin
Aar [Daa] Lo/1 [%]]
+22
+20
+15
+10
+5-
TRUE" DEF-DAIA'
DEF
AEU
AEC
ALU
ALC
CXU
CEC
CLC
ENS
Figure 12: Relative zero-one loss reductions of LFC approaches. For each HPS criterion (æ-axis), the scatter plot displays the mean and standard error of an LFC approach's reduction in zero-one loss (y-axis) percentage [%] compared majority voting (mv) trained with the default (DEF) HPC. Higher reductions correspond to greater improvements. A marks criteria that had access to the true validation labels.
Beyond reporting average reductions in zero-one loss relative to the lower bascline approach mv nv[DEF], we also examine how performance is distributed across pairs of approaches. Figure [3] therefore, presents a pairwise win-rate matrix, including the results from the pairwise significance tests, with ENS as the HPS criterion, where cach cell shows the proportion of dataset variants on which the row approach beats the column approach. Several cell pairs do sum to one, indicating ties in which the two approaches reached identical performance. The two-stage baseline loses to every other approach, confirming that estimating
31

---

Published in Transactions on Machine Learning Research (11/2025)
Row approach wins more often than column approach Comparison is significant.
Row approach loses more often than column approach.
8V
N/A
22.86
34.29
20.00
11.43
31.43
17.14
14.29
5.71
20.00
25.71
22.86
2.86
da
48.57
N/A
45.71
28.57
28.57
45.71
25.71
25.71
11.43
28.57
45.71
25.71
17.14
65.71
54.29
N/A
51.43
37.14
48.57
45.71
25.71
20.00
37.14
57.14
37.14
28.57
trace
80.00
71.43
48.57
N/A
42.86
48.57
54.29
51.43
25.71
25.71
62.86
31.43
25.71
co001
88.57
71.43
62.86
57.14
N/A
42.86
51.43
37.14
20.00
22.86
62.86
28.57
17.14
68.57
54.29
51.43
51.43
57.14
N/A
48.57
31.43
22.86
34.29
68.57
34.29
14.29
union-b
82.86
74.29
54.29
45.71
48.57
51.43
N/A
22.86
11.43
37.14
60.00
25.71
22.86
200-N
85.71
74.29
71.43
48.57
62.86
68.57
77.14
N/A
31.43
42.86
71.43
45.71
37.14
ge0-r
94.29
88.57
80.00
74.29
80.00
74.29
88.57
68.57
N/A
51.43
88.57
51.43
48.57
80.00
71.43
62.86
74.29
77.14
65.71
60.00
57.14
48.57
N/A
82.86
48.57
34.29
croxd-ar
74.29
54.29
42.86
37.14
37.14
31.43
40.00
28.57
11.43
17.14
N/A
20.00
8.57
annot-mix
77.14
74.29
62.86
68.57
71.43
65.71
74.29
54.29
48.57
51.43
80.00
N/A
37.14
coin
97.14
82.86
71.43
74.29
82.86
82.86
77.14
62.86
51.43
65.71
91.43
62.86
N/A
ds
c1
trace
conal
unioa-a
union-b
geo-v
geo-f
madl
crowd ar annot -ix
coin
Figure 13: Pairwise win-rate matrix for LFC approaches with ENS as HPS criterion. Across all dataset variants, a cell reports the percentage [%] on which the row approach outperforms the column approach. A cell's color decodes whether the row approach has more wins or loses than the column approach, whereas a circle indicates whether this comparison is significant. The diagonal shows not applicable (N/A) because an approach is not compared with itself.
crowdworkers' performances is beneficial. The other two-stage approach ds (Dawid & Skene 1979). estimating crowdworker performances for label aggregation in the first stage, performs somewhat better yet still lags behind. Accordingly, one-stage LFC approaches combining the crowdworker performance and true label estimation in one joint training lead to superior performances. Among these one-stage approaches, the coin approach [Nguyen et al. 2024] dominates all alternatives: its win-rate is consistently higher than its loss-rate. Together with geo-f (Ibrahim et al. )023), it is the only approach whose loss-rate versus mv stays below 10%.
Ablation Study In Table ablate the individual members' impact the ensemble-based HPS criterion ENS. Therefore, demonstrate the effect of removing risk measures from R (see Section 3.4 in the ranking order of their respective criterion from Table 4 starting with the worst one. We observe that removing members mostly leads to a lower absolute and relative reduction of the zero-one loss compared to the DEF criterion. This observation confirms the importance of individual members. Nevertheless, there might be unexplored combinations of multiple members leading to higher reductions than the full ensemble.
Table 7: Ablation study for ENS. In comparison to DEF as HPS criterion, each column reports the zero-one loss reductions (absolute as percentage points [%,] and relative as percentages [%]) for one subset of risk measures in R when employing the HPS criterion ENS. The full set is given in the leftmost column, while each succeeding column shows the results after removing one criterion with its associated risk measure. The rightmost columm refers to the case when only one member is remaining, corresponding the criterion ALU. The colors distinguish between baseline, aggregation-level crowd-level, and ensemble- criteria. Means and standard errors are computed over all combinations of dataset variants and LFC approaches (excluding the approach mv that is not compatible with each critcrion's 's empirical risk mcasure). The arrows indicate that higher (f) values are better. The best and second best values are marked per result lype.
ENS
-CLC
ABU
ALC
ABC
CXU
-CBC
Absolute Zero-one Loss Reductions Compared to DEP (A bo/s [%6
t)
5.16
5.08
+ 5,09
4.96
4.84
+ 4.96
4.73
+ 0.26
$ 0.26
0.27
4 0.26
0.27
4 0.27
+ 0.26
Relative Zero-one Loss Reductions Compared to DEF (Aow Lo/1 [%] 1)
+17.60
+17.49
-17.43
+17.77
+16.56
+17.14
+16.48
+ 0.77
+ 0.76
= 0.78
4 0.77
0.78
4 0.78
+ 0.80
32

---

Published in Transactions on Machine Learning Research (11/2025)
Analysis Stratified by Noise To assess how label noise impacts the performances of HPS criteria and LFC approaches, we group the dataset variants into low and high noise via class-count-invarii correction of the aggregation noise. Let X E [o, 1] be the aggregation noise rate, ie., the fraction of instances whose majority vote label disagrees with the true label, from Table 2 noise:
We define the class-corrocted aggregation
X
X
(66)
1-1/0"
This normalization removes the chance-level dependence of 1/c, making noise levels more comparable across datascts with different numbers of classcs. We use a median split across all datasct variants (modian Xo.5 == 0.439) such that 18 dataset variants with X X Xo.5 are assigned to the low-noise group, and 17 dataset variants with X > Xo.5 to the high-noise group. Note that this grouping does not purely reflect the noise level, as confounding factors such as the number of labels per instance or the inherent difficulty of a dataset may also play a role. Table S breaks down the results from Table 4 according to these two groups of noise levels. We observe that the benefit of HPO is higher in the case of high-noise dataset variants, especially in combination with the HPS criteria TRUE and ENS. For both groups of noise levels, the results confirm that ENS is the best performing HPS criterion with only access to crowd-labcled validation data, For Table $ with the results of the LFC approaches, an analogous breakdown is given by Table g Here, the performance gains of one-stage over two-stage LFC approaches are higher for the high- in comparison to the low-noise group of dataset variants. Accordingly, the ranking of the LFC approaches is also affected by the noise level. For example, cl (Rodrigues & Pereira, 2018), union-a (Wei et al., 2023), and annot-mix (Herde et al. 2024b achieve much better ranks for the high-noise level. Figure 14 extends our stratified noise level analysis to the violin plots from Figure 7 Here, Kendall r-b coefficients between the two groups are differently distributed. In particular, there is a higher spread of the Kendall r- coefficients for the group of high-noise dataset variants when comparing rankings for the default (DEF, DEF-DATA) HPCs with the rankings after HPO (via TRUE or ENS), although their mcans remain similar.
Table 8: HPS criteria's results stratified by noise levels. One column per criterion reports the rank compared to the other criteria and the zero-one loss reductions (absolute as percentage points [%p] and relative as percentages (%]) compared to DEF as criterion. Means and standard crrors are computed over all combinations of dalaset variants in noise level and LFC approaches (excluding the approach mv that is not compatible with each criterion). The arrows show whether smaller (L) or higher (1) value is better. The best and second best values are marked per result type and noise level. A * marks criteria that had access to the true validation labels.
Noise Level
Baseline DFF-DATA"
Aggrogation-level
Crowd-level
Ensemble
TRUK"
DRP
ARU
AFC
ALA
ALC
CXU
ORC
CLC
RXS
Ranks (4)
High
4.60
+ 8.62
9.56
5.71
5.35
5.22
5.58
5.35
5.40
5.70
1.70
0.21
0.16
0.17
4 0.17
+
4
4: 0.15
: 0.15
0.18
4.69
7.10
9.94
5.62
5.60
5.36
5.62
5.57
5.33
6.11
5.07
Low
+ 0.21
+ 0.26
+ 0.14
4 0.15
+ 0.14
+ 0.17
+ 0.18
+ 0.18
+ 0.16
+ 0.18
0.
Absolute Zero-one Loss Reductions Compared
DEF (Anr
Lay1 [%,,]
1)
High
7.24
+ 1.01
0.00
5.99
+ 6.08
+ 6.43
+ 5.97
+ 6.23
6.15
5.64
7.06
+ 0.41
+ 0.42
+ 0.00
+ 0.47
+ 0.48
+ 0.47
$ 0.50
+ 0.52
+ 0.52
+ 0.55
0.47
Low
3.59
+ 0.53
0.00
+
+ 3.18
+ 3.13
+ 2.97
+ 2.91
3.09
2.58
4 0.19
+ 0.45
+ 0.00
+ 0.18
+ 0.18
+ 0.20
4 0.21
+ 0.23
+ 0.21
+ 0.25
0.19
Relative Zero-one Loss Reductions Compared to DEF (Agm
Lay1 [%]]
1)
High
+ 19.84
+
+ 0.00
+17.63
+17.81
+ 18..58
+ +15.85
+17.64
+17.51
+15.11
-19.76
+ 1.17
+ 1.08
+ 0.00
$ 1.32
+ 1.33
+ 1.32
$ 1.34
+ 1.43
$ 1.43
$ 1.45
1.31
Low
+ 16.66
+ 3.05
+ -.00
+14.81
+11.82
+11.49
+13.51
+13.44
+14.11
+11.59
+15.56
1 0.88
4. 1.92
0.00
4 0.78
4 0.76
4 0.92
$ 0.93
+ 0.99
0.89
1.13
0.81
Analysis Stratified by Hyperparameter Budget To assess how the size of the candidate set |A| (our proxy for the HP budget) affects the performances of non-default HPS criteria and LFC approaches, we complement the main setting |A| = 51 in Section 4.2 with evaluations at |A] E 311. Table 10 breaks down the results from Table A according to these two additional HP budgets. As one might expect, the benefit of the non-default HPS criteria is lowered by reducing the HP budget, while still achieving notable improvements over the default. HPS criteria. This is likely because the selection is restricted due to the lower
33

---

Published in Transactions on Machine Learning Research (11/2025)
Table LFC approaches' results stratified by noise levels with ENS as HPS criterion. One columm per approach reports the rank compared the other approaches and the zero-one loss reductions (absolute as percentage points [7pp and relative as percentages (%)) compared the approach my trained with its default (DEF) HPC. Mcans and standard errors are computed over all datasct variants of the respective label
noise level. The arrows indicate whether smaller (J) higher (1) value is better. The best and second best values are marked per result type and noise level.
Noise
Baseline
Class-dpeendent
Instance-dependent
mv
conal
union-a
union-b
ge0-1
gea-f
croud-ar annot-nix
coin
Ranks
High
11.32
10.03
6.68
7.35
6.59
6.26
7.88
5.15
4.35
5.59
9.06
4.35
4.38
0.71
+
0.83
+
+
+
0.91
+ 0.51
0.86
+
+
+
Low
9.86
8.19
9.06
7.28
6.22
8.86
7.36
6.44
3.86
5.14
8.89
6.28
3.56
4 0.68
+ 0.73
+ 0.76
0.71
+ 0.76
+ 0.80
+: 0.82
+ 0.74
+: 0.75
+ 0.95
+ 0.59
: 1.01
: 0.63
Absolute Zero-one
Reductions Compared
(Aartoar)
La/1
1%,
f)
High
6.23
6.83
+1.553
9.81
8.34
9.30
9.41
+13.44
13.15
+10.45
8.18
+12.16
+12.64
+ 1.71
+ 2.32
+ 2.04
1.96
- 1.73
4 1.64
+ 1.95
+ 2.16
- 1.74
- 2.54
+ 1.79
: 1.41
: 1.60
Low
+
5.32
+
5.93
+
+
6.17
+ 7.13
+
+
+
+ 0.71
+ 0.71
+ 0.68
0.87
+ 0.73
+ 0.86
+ 0.70
+ 0.82
+ 0.89
= 0.86
+ 0.77
= 0.82
+ 0.84
Relative
Zero-one
Loss Reductions Compared to mv with
DEr (A
(mm) Lnz1
[7]
High
+16.86
+18.81
+27.71
+25.16
+22.25
+22.97
+21.03
+31.90
+31.71
+26.92
+21.65
+30.05
+31.30
+ 4.92
5.85
+ 5.27
5.46
+ 5.12
4 4.94
+ 5.28
5.27
+ 4.91
+ 6.38
+ 5.13
+ 4.47
+ 5.00
Low
+21.63
+23.15
+20.40
+21.83
+26.04
+20.41
+23.61
+25.77
+29.92
+28.35
+23.65
+26.50
+30.45
+
3.14
+
+1.00
Kendal Rank Correlation
+0.75
+0.50
Coefiiiintt 7-b
+0.25
0-
0.25
-0.50
HighsNNise
Low-NNoise
-0.75
TRUE* vs. DEF-DATA TRUE" Vs.
TRUE V8.
DEF VS. DEF-DAIA ENS vS. DEF-DATA" ENS VS. DEF
Figure 14: Rank correlation between HPS criteria stratified by noise levels. Each violin plot shows the mean and distribution of pairwise Kendall 7-b coefficients, visualized as dots and obtained from the high- and lowsn groups of dataset variants when comparing the ranking of LFC approaches with their HPCs selected via bascline and ensemble-based criteria. Higher coefficients indicate stronger correlation. A marks criteria that had access to the true validation labels.
number of candidate HPCs. The results also confirm that ENS is the best performing HPS criterion with only access to crowe-labeled validation data across all tested budgets. For Table containing the main results of the LFC approaches, an analogous breakdown is given by Table 11 Here, the performances of the LFC approaches also decrease with a decreasing HP budget, while the one-stage approaches still take the lead over the two-stage approaches. The ranking of the approaches is also affected by the HP budget. For example, the approaches conal (Chu et al. 2021) and madl (Herde et al.) 2023) achieve worse ranks for lower HP budgets. Figure 13 extends our stratified HP budget analysis to the violin plots from Figure 7 The rankings obtainod via the criterion TRUE and ENS get more similar for lower HP budget, as indicated by higher Kendall 7-b coefficients. This can be explained by lower selection variability for smaller set of candidate HPCs.
Hyperparameter Optimization with Data Classification Model Architecture Search In the main experiments of Section | the architecture of the data classification model fo fixod, tuning only the optimizer and LFC approach-speciffic HPs. This design keeps the HP space ΩΛ tractable for our budget of || = 51 candidate HPCs. To demonstrate feasibility rather than to redefine the benchmark, we additionally
34

---

Published in Transactions on Machine Learning Research (11/2025)
Table 10: HPS criteria's results stratified by HP budgets. One column per criterion reports the rank compared to the other criteria and the zero-one loss reductions (absolute as percentage points [% and relative as percentages [%]) compared to DEF as criterion. For the given HP budget [A], means and standard errors are computed over all combinations of datasct variants and LFC approaches (excluding the approach mv that is not compatible with each criterion). The arrows show whether a smaller (4) or higher (t) value is
better. The best and second best values are marked per result type and HP budget. A * marks criteria that had access to the true validation labels.
HP
Baseline DEF-DNTA"
Aggregation-level
Crowd-level
Ensemble
Budget
TRUE
ABU
ABC
ALC
CXU
CBU
CLe
Ranks (1)
|N|
31
4.59
7.78
9.77
5.62
5.58
5.41
5.57
5.57
5.36
5.90
4.8G
+ 0.11
+ 0.17
0.11
0.11
0.11
0.11
0.13
= 0.12
+ 0.11
+ 0.12
0.09
|||
11
+ 0.12
4.75
+ 0.19
7.24
0.13
9.36
0.10
5.71
5.62
0.10
0.10
6.41
5.84
0.12
= 0.10
5.43
= 0.10
5.42
+ 0.12
5.99
0.07
5.22
Absolute Zero-one
Loss Reductions Compared
(Aper
Lays [76p]
f)
||| 31
+ 5.18
0.76
0.00
4.41
4.40
+ 4.56
4.19
4.38
4.51
+ 3.98
5.00
0.21
+ 0.31
0.00
0.25
0.25
0.26
0.30
0.29
= 0.28
+ 0.31
0.26
||] = 11
4.50
+ 0.76
0.00
3.70
3.74
3.94
3.36
3.94
3.96
3.14
4.33
1 0.23
1 0.31
1 0.00
1 0.25
1 0.25
1 0.27
1. 0.32
1 0.27
1 0.27
1 0.33
0.25
Relative Zero-oue Loss Reductions Compared to DEF (A..,
Le/1 [%]]
1)
||| = 31
+17.70
+ 3.65
+ 0.00
+15.76
+15.73
+16.04
+14.09
-14.87
+15.65
+13.31
+17.24
4: 0.72
+ 1.12
4 0.00
+ 0.76
+ 0.76
+ 0.80
+ 0.95
4 0.87
+ 0.82
+ 0.91
0.74
||] = 11
+15.42
+ 3.65
0.00
+13.35
+13.55
+13.71
+11.17
13.41
+ 13.69
+ +0033
+14.76
1: 0.72
+: 1.12
0.00
0.74
0.74
0.89
1.05
0.80
1 0.78
1. 1.08
0.71
Table 11: LFC approaches' results stratified by HP budgets with ENS as HPS criterion. One column per approach reports the rank compared to the other approaches and the zero-one loss reductions (absolute as percentage points [%p] and relative as percentages [%]) compared to the approach mv trained with its default (DEF) HPC. Means and standard errors are computed over all dataset variants for the respective HP budget. The arrows indicate whether a smaller (4) or higher (f) value is better. The best and second best values are marked per result type and HP budget.
Noise Level
Baseline
Class dependent
Instance-dependent croud-ar annot-mix
nv
ds
cl
trace
conal
union-a
union-b
geo-v
gec-f
madl
coin
Ranks (4)
[A] = 31
10.49
9.33
7.90
7.17
7.74
7.17
7.40
5.96
3.63
5.86
9.24
4.99
4.13
+ 0.38
= 0.50
= 0.61
# 0.60
+ 0.48
+ 0.68
4: 0.50
+ 0.60
+ 0.36
= 0.05
+ 0.15
: 0.58
+ 0.49
|A] == 11
10.10
8.16
8.13
7.23
8.63
7.56
6.97
5.41
3.71
6.51
8.63
5.49
2.17
1: 0.51
+ 0.60
1 0.66
0.53
4 0.52
4: 0.65
+. 0.49
1. 0.46
4. 0.44
1 0.72
+ 0.48
+ 0.70
d: 0.44
Absolute Zero-one Loss Reductions Compared to
mv with DEF (A,,/om)
La/1
[7hp]
T)
|A] = 31
5.19
+ 5.61
+ 7.77
7.80
+ 6.97
+ 7.17
+ 7.37
+ 9.51
+ -10.06
8.10
+ 6.57
+ 9.46
+ 9.66
0.76
1 1.07
1,21
0.98
1 0.88
0.93
1. 0.96
1.31
1. 1.06
1.34
0.90
d. 1.00
1.00
|A] 11
3.93
+ 5.01
+ 6.92
7.04
+ 6.11
6.41
+ 7.16
+ 9.20
+ 9.50
7.75
+ 6.09
+ 8.16
+ 8.65
1: 0.56
1. 1.00
1.21
0.89
1.
0.94
0.99
1.32
4. 1.08
1.31
0.90
4. 0.98
0.90
Relative Zero-one Loss Reductions Compared to with
DEF (A..
bo/1
[%]]
f)
|A] = 31
+18.10
+ +19.53
+3.11
+24.91
+23.50
+22.36
+23.85
+28.33
+30.96
-26.65
+21.93
+28.87
+30.39
2.51
1 2.89
3.09
2.83
1 2.77
2.93
1 2.72
1 3.09
1
3.44
2.75
1. 2.60
2.92
[A] = 11
+14.21
+ 18.04
20.81
22.74
+0032
+1.70
+23.03
+7.35
+29.27
23.00
+20.95
+24.97
+27.33
- 1.80
= 2.61
= 2.97
+ 2.64
+ 2.73
+ 2.79
+ 2.68
+ 3.07
+ 2.88
= 3.23
+ 2.79
+ 2.71
+ 2.66
run a targeted study for the dataset variants of label-me and dopanim that includes the classification head architecture the search. Concretely, instead of using a fixed MLP with (256, 128) neurons its two hidden layers, we now extend our search to different numbers of layers and neurons sampled according to uniform (f(25),, (512), (256, 128), (512, 25 Because of the more complex HP space Ωi, we also increase the number of candidate HPCs to |A| 101, of which one HPC corresponds to the default one from the main experiments. Table 12 reports results in the same format as Table 4 Nevertheless, both tables' zero-one loss reductions are not directly comparable because the aggregation of Table A additionalyy encompasses the results from variants of the other datasets mgc, reuters, and spc. Moreover, our goal is not to assess the benefit of potential architecture scarch, but investigate whether our main conclusions persist under heterogencous head architectures. In this context, we observe non-default HPS criteria remain beneficial, and only TRUE surpasses ENS in rank and in zero-one loss reductions.
35

---

Published in Transactions on Machine Learning Research (11/2025)
+1.00
+0.75-
Kendall Rank Correlation
+0.50-
Coefiicient 7-b
+0.25
0-
0.25
-0.50-
A 31
A
00.5
TRUE" vs. DEF-DATA TRUE" V8. DEF
TRUE" V8. ENS DEF DEF-DATA" ENS VS. DEF-DALA" ENS DEF
Figure 15: Rank correlation between HPS criteria stratified by HP budgets. Each violin plot shows the mean and distribution of pairwise Kendall r-b coefficients, visualized as dots and obtained from all dataset variants when comparing the ranking of LFC approaches with their HPCs selected via baseline and ensemble-base criteria. The HP budget |A] is only relevant for the non-default criteria. Higher coefficients indicate stronger correlation. A * marks criteria that had access to the true validation labels.
Table 12: HPS critcria's results with architccture search. One column per criterion reports the rank compared to the other criteria and the zero-one loss reductions (absolute as percentage points [%p]] and relative percentages [%]) compared to DEF as criterion. Including simple classification head architecture search, means and standard errors are computed over all combinations of label-me and dopanim dataset: variants and LFC approaches (excluding the approach my that is not compatible with each criterion). The arrows show whether a smaller (4) or higher (t) value better. The best and second best, values are marked
per result type. A * marks criteria that had access to the true validation labels.
Baseline
Aggregation-level
Crowd-level
Ensemble
TRUR"
DEF-DATAA
DEF
AEU
AEC
ALU
ALC
CXU
CBC
CLC
ENS
Ranks (L)
3.59
8.87
9.27
5.74
5.88
5.21
5.58
5.06
+ 0.23
+ 0.21
+ 0.17
+ 0.20
+ 0.19
+ 0.19
0.22
+ 0.19
+ 0.17
+ 0.21
+ 0.15
Absolute Zero-one Loss Reductions Compared to pe9 (Anp Tays
[%p]]
1)
0.22
3.81
$ 0.17
+ 0.30
+ 0.00
0.00
+ 0.18
+ 2.31
I 0.18
2.18
- 0.19
2.64
0.50
2.20
2.67
0.23
+ 0.22
2.73
# 0.54
1.37
2.76
0.18
Helative Zero-one Loss Reductions Compared to DEF (Aner Lnz1
[%]
1)
+15.17
+ 0.45
0.00
+10.46
9.98
+11.5
8.82
+10.73
+11.09
+ 5.65
+11
4 0.70
$ 0.61
$ 0.00
+ 0.76
I 0.76
0.78
2.41
4 0.84
+ 0.77
+ 2.70
0.71
So far, we have only focused on the zero-one loss (see Eq. as our larget performance measure. However, trained data classification model fo is often also required to output meaningful probabilities, which can be evaluated using the Brier score Brier 1950 as loss function:
Loss Functions Beyond Zero-one Loss
2
LBS (y. = (y (T 9).
(67)
At the same time, the data classification model Se ideally achieves a low zero-one loss. In this case, we have a kind of multiiobjective optimization problem, whose solution beyond our scope. Instead, we present a brief analysis demonstrating that the search for the best performing HPS criterion depends on the target loss function(s). Specifically, we compare the non-default criteria using the Brier score as a loss function for selecting the best HPC, whereas the two default criteria remain unchanged. Table 13 shows that conducting HPO with alternative loss function remains beneficial in the LFC setting. In particular, the proposed criterion ALC using estimates of the crowdworker performance model he for label aggregation and weighting excels. Overall, the criteria based on aggregation-level risk measures strongly outperform those only relying on crowd-level risk measures. A likely reason is that crowd-level measures overemphasize the ability of the crowdworker classification model 9s assign high probabilities the workers' noisy labels. Consequently, HPS criteria cannot be transferred naively from one loss function another. Despite not being the overall best criterion, the ensemble-based approach ENS remains appealing, its flexible design allows combining
36

---

Published in Transactions on Machine Learning Research (11/2025)
risk measures derived from different loss functions. Exploring such combinations represents promising direction for future work.
Table 13: HPS criteria's results with Brier score as loss function. One column per criterion reports the Brier score reductions (absolute unitless [-] and relative percentages [%]) compared DEF criterion. Means and standard errors are computed over all combinations of dataset variants and LFC approaches (excluding the approach mv that is not compatible with each criterion). The arrows show whether smaller (4) or higher (f) value is better. The best and second best values are marked per result type. A marks criteria that had access to the true validation labels.
Baseline
Aggregation-level
Crowd-level
Ensemble
CXU
Ranks (1)
3.93
8.16
9.13
5.45
1.98
4.52
4.46
6.58
6.73
6.81
5.25
4
Absolute Brier Score Reductions Compared to DEF [Amn Las
+0.092
-0.024
+0.000
++0 068
+0.072
+00.078
+00.080
+-0. +30
+0.030
+0.020
+0.067
+0.005
+0.008
#0.000
+:0
20.004
+0.001
+0,001
#: 10.006
=0.006
20.007
#0.001
Relative Brier Score Reductions Compared DEF (Ane
Las [%]]
+18.70
2.59
+ 0.00
+3.24
-1.32
+13.78
+ 116.43
+ 7.74
7.65
5.08
+ +13.88
1.10
37

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part 1). The first column lists the LFC approaches and the remaining columns the HPS critcria. Each critcrion selects the estimated best HPC per approach, and results are reported as means with standard deviations. The best-performing approach per column (excluding gt training with true labels) and the best-performing selection criterion per row (excluding TRUE validating with true labels) are highlighted. The symbol * marks criteria with access to true validation labels. Some criteria are not applicable (N/A) to all approaches.
Raseline
A Aggrogation-level
Crowd-level
Ensemble
Approach
TRUE"
DEP-DADS"
DFF
ABU
ABU
ALU
ALC
UXU
CHC
CLU
ENS
ngc-sorst-1
gt
20.27
20.27
24. 24.60
21.00
31.001100
N/A
N/A
N7A
N/A
N7A
21. 21.00
81.27+1.32
86.93+1.30
81.4740.00
79.73
1.75
79.73
=170
N/A
N/A
N/A
N/A
N/A
79.73
1.76
da
cl
31.2741.32
44.33 29
36.93
85.6721.90
81.4710.96
79.73-11.67
79.73
1:77
80. 50.53
79.73
1:
80,53
79.73 11 15
1.54
74.33
79.73
14.29
1.75
80.5311.54
79.73
11.75
79.73
80.53
21.53
11.75
79.73
74.33
74.33
79.73
11775
70.60+2.29 86.73+0.5a
81.80+0.77
82.27
82.27
82.27
+2.60
82.27+0.98
82.27+0.98
70.60
-2.29
82.27 +0..5
conal
79.53+1.11 85.87=1.50
82.07+1.85
80.87
60.87
80.87
79.53
11.71
80.8T41.10
80.87
79.53
1:
10.87+1111
union-a
71.07+3.07
84.13+1.39
79.47+0.77
79.
75.93+3.65
75.93-
75.93+
74.20+2.34
75.93+3.86
75.93+a.65
71.07
35.83+3.86
unionbb
78.8711.63 85.7311.04
80,20
78.93
19.63
78.93
78.93
85.80
12.63
78.93
19.63
87.93_0.15
78.93
geo-x
75.6011.06 81.8711.50
80.0012.0s
74.33
12.95
74.33
1.96
75.
10001.000
74.33
12.90
74.33
42.96
74.33
12.95
74.33
2
74.33
geo-f
71.73+4.79 84.60+1.5a
79.27+1.01
69.80
3.18
69.80
69.80
69.80
+3.18
73.60+3.40
73.60
+3.46
09
78.60+3.46
madl
72.07+4.01 87.53
82.27
80.67
+2.36 60.67
=2
80.67
+
69.13
13.04
81.27
81.27
11.60
69.13
80.67 2..36
croud-ar
80.1311.a 84.67
81.3310.85
78.53
11.35
78.53
11 15
78.53
11.30 15
89.07
12.65
78.53
11.55
15.35 78.53
89.07
78.53 11
annot-nix
72.40+s.u6
87.5
80.27411.7
79.87
+2.87
+2.84
79.87
22.04
79.87
72.40
78. 22.00
78.20
26.00
72. 40
78. 15.20+ 5.50
coin
83.6044.96
43.0044.17
83.53:46.08
80.20
80.20=1.80
80.20+1.80
79.67
+1.43
66.80
82.80=6.00
82.73
75.40
ngc-2orat-2
gt
19.27+1.44
19.27
24.00+1.12
10.27+1.44
19.27
=1
N/A
N/A
N/A
N/A
N/A
19.27 1
av
58. 38.534+2.64
68.27
58.53
+2.54
58.60
58. GO
N/A
N/A
N/A
N/A
N/A N/A
58. 60.
da
22.93+2.18 79.93
72.47
73.27
73.27
73.27
73.27
70.40
73.40=0.85
73.27
73.27
cl
52.9311.09 50.73
57.80
60.87
60.87
53.60
57.07
52.9311.00 on
52.93
53.60
533601
trace
17. 17.93
60.6041.ss
39.33
59
59.17
59. 59.17
17.93
+.65 17.93
47.93
59.47
17.93
1.82
17.93 1.66
conal
53.33+1.08 55.53+2.41
50.80+0.00
50.33+1175
54.67 54.67
57.60+1.77
85.73+1.59
57.60+1.77
54.67
54.67
54.67
union-a
52.0711.38 38 51.6710.87
54.7311.14
52.07
52.07
59.20
52.07
48.93 12.15
48.93
19.15
52.07 a6
51.47 11.2
union-b
20.
61.17+2..29
58, 1.28
59.87
58
80
58.80+2.22
59.53+2.64
59.
00.13
58
88.8022.22
56.73
+0.05
59.53.2..4
58. 80
geo-x
56.2711.12 49.07
07.4721:73
56.87
56.27
56.27
56.27
13.13
56.27
11.95
56.27
49. 19.9311.12
goo-!
24.40:2 45.20
+1.44 55.87
57.13
17.13-139
53.00
54.40
54.9311.60
54.93
54.40=2.25
48.33
madl
47.40+3.a7
58.20 58.2044.74
59.93+0.55
47.40
3.87
47.40
3.87
47.40
47.40
47.40
+3.57
47.40
+3.87
47.40
-8.87
47.40
8.88
47.40
8.88
croud-ar
57.00
56 53..00
56.67
57.47
11
54.33
18
14
11.77
57.47
11:7
57.47
1.
minot-iix
11
45.6011.22
37.07
47.87
47.87
23.50
44.87
1.
19.00
13.87
47.87
13.50
44.87
1.77
47.87
13.50
coin
45.93+1.02
51.27+6.24
61 61.80
57.53
53.53-1.74
46,
93+1
53.67+3.97
45.98+1.62
46.80
+2.12
03.67-3.97
3.99
45.80
+2.12
ngc-woret-vv
gt
18.b340.18
18.53+0.73
24.60+1.12
10.0340.60
19.93+0.00
N/A
N/A
N/A
N/A
N/A
19.93+
mv
53.73:1.u1 56.07
50.53+2..x
50.53
+2.18
50.53
N/A
N/A
N/A
N/A
N/A
53 1.44
da
cl
51.87+0.38 58.80+1.08
42.60 47.53
52.67+1.02
48.4741.a5
45.27
50.93
+1.94
45.27
50.93
1.04
50.13
50.93
50.1312.50
50.93
+..0 53.93+1.58
46.8711.68
53.93+1.63
45.27
53.93
45. 45.80_1.20
45.27 1.
53, 53.93+144
trace
10,0012.10 11.58
13.33
18.00_0.62
48.
18.00_0.e2
10.00
10.00
47.6012.00
17.60_2.00
10.87_1.07
17.6012.00
conal
44.00+0.58
42.53
+1.70
46.27+1.48
45.67
45.67
43.73+1.19
45.4742.15
45.6741.25
45.67+1.25
44.00-0.65
44.00+0.53
union-a
41.93+0.5a
43.67+2.16
46.87
43.07
20.07
43.07
41.67
43.07
43.07=0.77
42.13
40.07
union-b
41.3311.11 19.4011.08
11.67
46.33_1
18.1311
48.1311.35
43,6010.50
43.53
43.53
44.13_2.19
18.13
geo-M
10.07+1.u9 22.20
47.47
11.47
41.47=u.5
12.93
40.07
39.80
12.13
39 80 0.84
42.93+1.00
gao-f
38.0012.78 38.93
45.38
41.13
11.13
41.60
41
41.6012.11
30.8011
30.80
58
39.
00.801156
41.13go.0
madl
39.20+s.16
42.40+1.55
19.80+1.19
39.20
39.20
3.16
39,20
39.20
39.20
39.20
+3.16
39.20 3.16
39.20+
croud-ar
43.3340.97
44.87
20.30
48.07
50.13
50.
13
50.13
50.13
13
50.
13.
13.13
50.13
annot-mix
37.2711.67
18.07
38.07
48.00
38.13
80
38.13
20 80
38.13
39.93
39.13
39.
39.60...
38.13
38.13
12.07
10.53
51.27
39.73
1.00
39.72
39. 39.73+1m
16.00+2.44
39.73,1.00
21.02
39.72 19.73...
39.73,,
ago rand- 1
gt
18.67+1..1
18.67+1.11
24.60
21.13+1.99
13.13:1.5
1.98
N/A
N/A
N/A
N/A
N/A
21.13+ 1.99
mv
19.2011.71
40.07 23.1212.22
22.10
40.07
20.10
40.07
20
N/A
N/A
N/A
N/A
N/A
40.07 2.110
da
39.20+1.71 40.13+2..2
40.07
40.07
+2.10
40.07
2.10
40.07
40.07
40.07
40.07
40.07-2.10
40.07
el
38.67+2.30 19.6744.50
41.60+1..s
38.67
38.67
38
11.0044
19.67
41.2043.00
20.20
49.67-4.60
11.00+44.12
trace
11.07 36.73
11.35
40.0012.44
39.00
39.00
11
39.00
41.07
39.0011.45
39.00
41.07_1.67
39.00
conal
37. 37.53 38.8721.90
10.33
37.07
+2.01
37. 37.07
2.03
37.07
38,00
37.07
37.07+2.01
37
38.
38.00
37.07
union-a
33.53+0.96
40.80+4.75
36.6043.01
36.53
13.00
35.53
23 9.
+
45.93+2.15
43.00
36.53
+3.06
44.00-
union-b
39.40 49.67
40,80
38.80
11228
38,80
11 98
38.80 11
52.47
38.80
11
38.80
11.25
52.47
38.80
11.28
geo-x
37. 37.10
11.0714..1
40.
40.00
37.40
37, 10
37. 10
13.10
39.40
39,
39.40-1.00
45.
15.60_1.77
37. 17.76
geo-f
35.3340.41
39.07+1.91
37.73+1..4
37.13
13.94
37 13-144
37.13
38.60+1.50
35.87
36.87
1.20
38.60-1.69
13+144
madl
35.07 36.00
40.20
36.27
36.
36.
35.07
1:48
40.20
40.20.1.54
35.07
1 36.27
crowd-ar
39.07+2.88 44.60+2.13 39.73++1.0
36
87
36 87
36,87
39
49.07
36.87
36.87
39.07-2.88
36, 36.87
smnot-nix
33.27 a2 37.1313.32 39 19.00
32
32
36,53
36.5340.51
33.27
11
33.27
36.53
36.53_0.61
33.27 11.32
coin
35.5811.17 10.9313.7H 44.8714.31
35.53
11.77
55.53
77
35.53
T7 39.27
35.53
11.77
35.53
+1.77
39.27
35.53
1.
Contimued on The next page.
38

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part II). Continued from the previous page.
Approach
Baseline
Aggregation-leeel
Crowd-level
Ensemble
TRUR
DEP-DATA"
DEF
ARC
ALU
CXU
CRC
CLC
EN8
Rgc-rand-2
gt
19.07
19.07 20
24.00
20.9
08.93=0.76
N/A
N/A
N/A
NZA
N7A
18,53 76
av
40.67+
43.60+1.21
38. 38.20
40.67
40.67
N/A
N/A
N/A
N/A
N/A
40.67 40.67+0.78
da
23.5342.00
44.5
41.07
38.40
20.4
38.40
39.53
09
20.27
42.1340.77
42.13
42.
07
39.53+2.00
cl
34.8010.17 37.4012.16
39.0711.46
33.60
10.09
33.60
2..00
34.8010.17
35.2011.13
34.9311.50
34.0311.50
35
26.20_1.12
34.8010.77
trace
33.47+1.07 35.00+1.on
35.87
33.47
11.07
33.47_100
33.47
1.007 33.47
11.07
13.47 36.00
36.53-1.45
33.47
36.53+1.45
conal
$1.0711.19 35.13
37 22.00
35,00
35.00
35,
00..01
35.00
35.6010.08
33.00
41.07
35,
00.077
35.601..00
union-a
33.00+1.05
33.67+1.03
35.40+2.57
33.27+1.29
33
33.27-1.26
33.27+1.26
33.80+0.11
+2.00
32.00
-0.47
33.80 03.80-0.17
33.80+0.77
union-b
34.47+1.48 40.00
17.91
35.47
35.47
35.87
34.87
36.5341.30
36.50
37.00
34.87 1
geo-x
33.601o.55
34.2710.70 36, 36.80
32.73.0.98
32.
33.13
33.13
32.40
32.40
11.85
34.27 10.50
33. 33.13 1.....0
geo-f
34.33+4.97 34.87+1.07
34.40
33.53+1.39
33.07
33.20
33.20+1.28
33.40+1..0
33.40-1.09
33.07
1.80
33.20+1.25
madl
31.1042..6 38.60+1.38
35.53+2.20
33.53
20.90
33.53
0.00
33.53
31.1
33.53
31.87
33
53
33.53 0.00
croud-ar
34.53+1.09
36.47
37
37.40+3.04
36.00
+0.97
35.00
0.07
35.00
35.00
40.97
35.00
+0.97
35.00
00.07
35.00
0 0.97
35.00
annot-nix
13.07
11.33
35.6722.51
30.80
30.80
31.00
31.00
1.00
30.80
30.80
30.80
30.80
coin
32.93 as
31.53 1.001
41.9314.8z
33.00
11.48
33.
33.0011.0
33.00
32.93
32.93
32.93
31. 60
33.0011 1.43
agc-rand-v
gt
19.40+0.36 36
10.40
24.60
=0.00
20.00=0.00 91
N/A
N/A
N/A
N7A
N/A
20,00+
da
av
36.47+1.50
38.73+0.5a
35.40+2.36
35.40
36.87
1. 37.67+0.91
+2.35
36
00.00
38.73
36.67 +3.57
36. 47.61.2
03.65
38.00
N/A
78
38.00
N/A
38.00
N/A
38.00=1.75
N/A
38.
28.00=1.: 1
N/A A
18.00
36. +..52
cl
31.8010 39.8011.26
36.0013.50
36.40.0.00
36.40
31.80
10
31.80
20.51
33.0T11-16
33.07
33.80
33.07
trace
31.07+1.30 32.67
35.60
31.53
31.53
31.53
31.07
36.27:2.22
36.27
31.53=0.e
36.27
conal
$1.8012.02 35.2011.71
33.67 33.07
31.33
31.33
2255
34.33
31.4011.11
33.8011.07
31.33.2.23
31.80_2.44
31.33
union-a
29.58+1.04 37.20+3.35
35.13+2.55
31.73
+0.08
31.73
0.99 31.73+0.95
31.73+0.05
31.73++.00
31.78
+0.98
31.73-0.98
31.73
0.96
+1.93+00.96
union-b
geo-x
$2.2010.s3
34.53+1.32 19.53
34.7312.0z
35.40
35.33
32.20
33.20
11.29
10.93
33.20
32.
40
11 28 38.07
32,20
10.83
32.6711.11
34.00+2.17
32.4011.50
36.0740.63
32.40
32,
36.07
34.53
32, 40
32.
36.07
geo-f
31.13+1.07 31.07
34.87+1.52
33.00
33
00.00
32.67
31.20
32.67
32.07
+1.05
31.13
-.07
1..57
madl
32.93+1.12 32.00 1..62
36.27=0.80
31.13+2.15
35.
35.67+2.96
32.93+1.12
36.
36.4042.25
36.40
32, 12.93-11.22
31.1342.115
croud-ar
33.60+1.19 36.87+1.95
35.33+1.00
36.87
35.87
35.87+1.1
34.13+1.15
35.87+1.19
36.87 +1.19
34. 13.18--1 1.15
33.60
1.119
annot-nix
10.40+1.50
21.93
36.00
31.93
31.93
0.00
31.93
0a 20.9
31.60
31.7320.88
30.60
31.93
0.808
coin
33.9312
37.6014.75
40.0013.44
33.93
$3.9312
02 33.2013.0
33.2013.00
32.0011 oo
32.00
22.00
32. 22.00
30.87
1.0.77.
agc-full
gt
20.20+0.06 06
20.20+0.90
24.00
20.60
20.
28
N/A
N/A
N/A
N/A
N/A
20,60+ 0.28
av
34.67
37. 1.
36
31.07
+1.62
34.67
11.62
N/A
N/A
N/A
N/A
N/A
34.67 1.02
da
30.40+1.38
33.00
a3.20
32.80=1.33
31.33=0.02
31.33
31.33
31.67
31.67
31.00
31.67 0.002
cl
31.4011.4 10.90
37.27
31.4011.04
31.40
31.40
33.2011.2
31.4011.0
31.4011.04
31.40
31.4011
trace
29.20 35.33
34.07:1.19
30.07
30.07
30.07
29.20+1..
33.8741.57
33.8721.57
30.07
30.07
conal
31.60+1.54 30.47
33.47+1.15
31.60
31.87 31.87=1.19 32.00+1.08
32.67+1.27
32,60+1.44
32.60+1.44
32, 12.60-1144
32. 1.44
union-a
31.20+o.1a
30.07+1.79
34.47+2.s1
31.20+00.75
31. 21.20
30.53+1.o7
32.00+0.22
31.20+0.77
31.20+0077
31.20-11.4
31.20+0.75
union-b
31.07+0.12
30.93
1..01
35.47
31.07
31.07
31.00=1.05
32.87+2.39
31.0740.22
31.07
31.07 31.07
geo-x
20.9312
22 30.4011.00
35.13
30.60
30.60
30.33
10.7
31.1311.32
30.33
30.60
31.27
30.60
geo-f
28.67 30.
34.53
30.20
13
30.27
30.73+t.44
31.13
30.43.0.45
30.93-1122
30,20
madl
29.13+1.17 31.00+1.03
34.93+1.82
31.33+1.35
29.73
32.27
32.27+2.06
29.73
29.73
#1
19.73... 1.16
29.73 1.16
croed-ar
31.6711.33
31.20
+0.38
35.00
31.73
31.73
31.
31.4741.74
31.67
31.67
11.33 33
31.73
32
31.73 1.32
annot-nix
27.20+2.04
28.13
23.90+2.24
20.47
29.47
00.00
20.47
2.096
27.80
26.80
26.80
00. 00.0
27.60
26,20,
coin
31.601a.0
30.2011.80
40.00
31.8011.24
$1.80
30.001o.71
28.80
$1.58
30.001o.71
28.80
$1.59
28. 88.80
28.80
1.59
label-ne-veree
gt
6.40
6.40
6.31 10.00.00 27
9.48 95
1.48
N/A
N/A
N/A
N/A
N/A
9.48 10.85
ai
10.8
32.
76
34.49.20.44
31.07
+0.70
31.07
0.70
N/A
N/A
N/A
N/A
N/A
31.67 +0.70
da
30.86+1.00 32.76+0.99
34.49
31.67
31.67
31.67
31.67
31.67
31.67
31.67
31.67
cl
27 33.7211.51
33.2710-34
31.50_1.41
31.50
31.5011
27.17
10.31
31.501141
31.501141
27.59
31.5011.4
trace
31.20 32.48go.7u
34.38-10.a3
31.25
#0.s1
31.25
0.81
31.25
31.25
11.25
31.25
40.51
31.25
31.25 0..8
conal
31.26+1.50 31.11+2.5a
34.33+0.44
31.35+0.73
31.35
01.35=0.75
31.35+0.78
31.26+1.50
31.3540.78
31.35+0.75
31.26-1.60
30,98
uniona
22.6441.90
29.07+1.90
32.41+0.75
29.98
19.98-177
29.98
29.98+1.72
29.41+0.s7
29.41+0.a7
29.73-
union-b
26.50+2.28 32.04+1.29
34.3440.68
31.48
11.48
31.48
21 24 31.48
11.48
31.48
21.24
31.48
11
31.48
1.24
31.48
1
31.48
geo-x
28.1610 sa 32.1711.30
34.411n-35
31.57
11.5711.27
31.57
31.57
29.02
31.57
31.57
01771.33
29.02
31.57 11
geo-f
28.4841.27 31.90
31.0110.70
31.40=1.25
31. 10
31.40
25.40
+2.58
31.40
31.40.1.25
25.40
2.85
31.40:1.26
madl
29.02+7.49 32.54+1.37 34.78-0.78
30.30+1.62
30.30
30.30
29.02
30.30+1.02
30.30+1.62
29.02
7
30.30+100
growd-ar
32.0240. 24 31.03
1.003
34
11.18
32.37
10.97
32
37 37
32.37 10
32.37
10.87
32.37
32.37
10.97
32.37
32.37 10.00000
annot-nix
30.10+1.22
30.88+2.53
38.1840.7a
30.86
30.8
30.86
31.50
30.86
30.86
10.00
30.86
30.86
coin
31.7315.11
30.15
30.98 10.08
30.25
10.7
30.25_ 20 30.25
26.70.22.0
26.70,
$2.00
30.2510.71
30.25
20
26.7
2.04
30.25
Contimued on the next page.
39

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part III.. Continued from the previous page.
Appronch
Baseline DEP-DATA"
Aggregation-level
Crowd-level
Ensemble
TRUR
DEF
ABC
ALU
CXU
CRC
CLC
EN8
label-ne-voree
gt
6.44
6.43-0..4
27
6.75
6.15+0.14
N/A
N/A
N/A
N/A
N7A
6.75-2.14
av
18.08+0.17
24.97+0.47
22.20
22.20+0.95
17.41
17.41
N/A
N/A
N/A
N/A
N/A
17.41
da
18.00
25.57
22.44
18.13
17.59
20
17.66g..07
17.66
17.6640.07
17.66=0.67
17.6620.07
17.66
cl
16.5810.89
17.29
20.8210.26
17.7
17.712117
17.71
17.71
17.71
17.71
17.7121-17
11-17
trace
conal
16.03;1.11 23.4240.an
18.141a.a
22.2120.70
22.76:20.45
22.1910.97
17.73
17.07
11:51
17.07=0.4s
17.73
17.07
17.73
19.0210.43
1:
18.8611.18
17.73
11.5
17.73
17.07
17.73
17.07
11.51
18.13
17.73
1:51
17.73
17.07
17.73 11551
union-a
14.02+1.30
16.79+0.00
20.79+0.45
20.09
18.82-1.45
16.80
16.80+0.72
17.3140.75
18.03+0.55
21.99-0.96
21.09+0.56
union-b
16.55+1.93 17.91 20.37
10.21
16.38
07
16.38
16.38
10.37
17.0740.88
17.07
17.07=0.88
16.38
1. 5.
geo-x
15.7411.03
18.0610.51
21.321n.42
17.3220.00
17.12
15.74
15.74
17.1211.05
17.1221.05
18.70
17.1211.05
geo-f
17.5240.s6 18.16+0.22
21.46++0.ss
17.20=0.50
16.36=
16.21+0.09
16.36+0.85
+5.67
15.67
17.98-0.75
16.36+0.05
sadl
15.72+0.94 20.00+0.07
23.2140.01
19.41=0.44
19.41=
18
18.00
20
19.4140.74
19.41
19.
19.41
19. 19.41...44
croud-ar
18.06+1.18 20.72+0.05 21.82+0.09
20.03+0.37
20.03+0.37
18.66
20.03+0.37
20.03+0.37
20.03+0.37
20.03 0.37
20.03+0.a7
annot-nix
18.75
20.88
21.72 16.95
16.95
20.37
2
20.37
16.95
16.95
16.95
18.75
coin
16.0310.81
16.7510.38
19.93
17.5911.28
17.59
18.20
18.20
18.2011.50
16.60
18. 20
18.20
label-ne-voree
gt
5.33
5.30
+.47
20.2
N/A
N/A
N/A
N/A
N/A
7.47+0.20 20.25
av
19.41+0.87
24.44+0.30
24.21+0.45
19.41
+0.57
19.41+0.87
19.41
N/A
N/A
N/A
N/A
N/A
19.41 00.87
da
21.0440.01 24.90
24.8220.46
19.36
10.02
21.04=0.01
19.36
21.04
21.0440.91
21.04
21.04
+4.04.0.00
cl
17.9310.5 22.2610.15
22.3710.47
22.51
22.5110
22.07
19.85
18.72
18.9220.70
18.92
19.85
22.51
trace
19.66:1.15 23.15go.72
23.11
20.57
20.57=0.a0
19.66
19.66+1.10
19.6641.15
19.66-1.110
19.66
19.66
conal
18.1910.02
23.1810.11
23.6211.15
18.18
19.1920.42
18.18
19
18,08
18.1810.49
19.1920.42
18.18_0.49
18.1810.09
union-a
17.29+1.20 21.23+0.58
21.70+0.47
18.96+1.62
18.96-1.62
16.33
+6.93
18.2340.95
18.23+0.95
18.23+0.95
18.23-0.95
2..23
union-b
19.55 15.4641130 47.63
22.8110.02
18.9220.71
18.92_0.71
19.83
18.59
19.5840.30
19.58
18.9220.71
18.92
geo-x
se 22.63
22.90
18.8410.80
18.84_0.00
21.1410.
17.56
19.7610-46
19.70
19.70_0.61
21.14go.m
geo-f
16.60 22.49+o.us
22.5640.58
18.01
18.01
19.68
19.68
4v.68
19.68
t0.es
19.68
18.01
madl
19.1140.03 22.76+0.03
23.6040.70
19.38
19.38=0.70
18.37
2 66
18.37
60
18.37
10.60
19.14+0.63
18.37
0.09
19.38+0.70
croud-ar
18.70+0.31 23.37
23.25+0.51
20.06++.644
20.06+9.04
23.25+0.5.
19.90+1.00
18.97
18.97
+0.71
20.05++0644
18.97
-0.71
20.05+0.04
annot-nix
18.28 20.00
22.56.0.00
19.95
19.95_1.01
19.92 10
20.56
20.56
20.56
20.56
0. 41
22.05
coin
16.9013.19
21.2110.41
20.8910.49
18.13
18.13
18.30
16.90
$3.19
18.13 o.
18.13
18.15 oo
18.13
labol-ne-rand-1
gt
6.28
0.28
60.1
7.36
7.3660.30
N/A
N/A
N/A
N/A
N/A
7.36
ai
14.7640.00
19.50+0.57
18 18.45-40.51
14.49
+0.60
14.49
+0.00
N/A
N/A
N/A
N/A
N/A
14.49 +0.09
da
14.76+0.60
19.50
18.4540.51
14.49
14.49
14.49
14.49
14.49
14.49
14.49
20.00
14.49
14.49
cl
15.5610.84 a4
17.2611.00
18.9710.47
15.00
10.63
15.00
20.63
15.00
10.00
15.00
10.30
15.00
10.63
15.00
20.63
15.00
15.00_0.63
15.00
1..00.0.00
trace
14.34
19.58
18.38-20.47
17.52
+0.74 17.52
7.74
17.52
17.52
+0.74
17.52
17.52
to.74
17.52
0..74
17.52
07.52
conal
15.05+0.08
18.00+1.50
19.24+0.57
14.48
20.38 14.48
14.48 14.48
14.48
10.38
14.48
14.48
14.48
10.38
14.48
14.48
union-a
15.07+0.59
16.84+0.73
18
48.65
15.07
+0.59 15.07
-..09
15.07
16.06+0.71
15.0740.50
15.07
14.70-0.00
14.70
15.07
union-b
geo-x
15.37+0.5a 17.8840.42
15.7411.30
17.6610.a1
19.1110.63
19.07
15.40
15.39
10.68
10.6s
15.40
15.39
20.00
15,40
15.39
1 68 15.39
0 68
15.39
20.08
10.88
13.40
15.39
6o
15.40
15.39
10.68
10.08
15.76 D 86
15.39
68
15. 15.40
15.39
geo-f
14.46 17.8620.30
19.044t0.9
+0.79
15.30
-0.99
15.30+
15.54
15.30
+0.79
15.30
+0.79
15.54_1.15
15,30, +5.30+
madl
14.83+0.52 5a
20.91+0.s1
19.0140.00
18.11
0
18.11
0.002
18.11
13.11
00.02
13.11
60.02
13.11
20.62
13.11
0
13.11 13.111+000
croed-ar
15.7710.51
18.1010.
19.4110.22
16.41
+0.72
16.41
0.72
16.41
+0.72
16.41
+0.72
+0.72 16.41
16.41
+0.72
16.41
0.77
16.41
0..12
annot-nix
14.46.0.50
17.91
18.2740.34
15.76
15.76
15.76
15.76
15.76
15.76
15.76
15.76
15.76
coin
12.5810.10 to
17.4611.00
18.08 10.28
13.75
g0.51
13.7520.61
13.75
12.58
11.12
13.8710.25
13.87
20.25
12.56
1.12
13.87
Labol-ne-rand-2
gt
1.21
6.21_0.26
6.31 27
6.21
6.21 26
N/A
N/A
N/A
N/A
N/A
6.21 122110.22
ai
15. 15.42+0.52
16.60+o.us
19.02
16.16
+0.41
16.35=g.41
16.35
N/A
N/A
N/A
N/A
N/A
16. 16.35
da
15.13+0.0a
15.02+0.41
17.2440.40
14.41
15.13=0.05
15.02+0.41
15.02+0.41
15.02+0.41
15.02=0.41
15.02
15.02+0.41
cl
13.1111.23 14.6010.35
15.8210.41
15.25
15.1320.49
15.27
14.11
15.2716.50
15.6210.79
13.77_0.47
15.27
trace
14.53ga.55
14.4%
17.95
15.15
15
15.152a.s5
15.15
15.15
15.15
15.15g0.95
15.15
conal
15.62+0.71 14.29+0.02
17.12+1.05
15.67+0.68
15.67=0.05
15.67+0.08
15.72+2.19
13.92
13.92
13.94-0.78
0.36
uniona
15.4444.74
14.4440.75
15.7440.50
14.78
14.78=u.a2
15.44+4.74
13.72+0.su
18.37+0.20
14.1640.55
13.72=0..6
14.16
union-b
12.95+1.11
14.3440.00
16.6040.41
14.34-0.62
14.98=0.97
13.16
14.3440.02
17.0440.40
15.7940.93
15.79=0.08
15. 15.794...0
geo-x
13.1010.09
14.331o.77
16.3710-35
15.76
15.76
04
13.10
19
15.76
14.7810.46
14.78
14.78 46
15.76
geo-f
13.57
14.094o.73
16.3640.49
14.01
11.01_u.7
12.21+0.32
11.36.o.m
15.61
14.01.0.75
13.91
14.36
madl
14.2840.04
14.19+1.00
18.1340.38
13.55
20.33
13.55
14.28
30.55
13.53
13.55
20.33
13.55
0.88
13.55
growd-ar
16.2510.52 sa
14.73
16.38
14.26
+0.38
14.26
0.38
16.31
14.26
14:26
+0.58
14:26
+0.38
14.26
0.88
14.26
00.36
annot-nix
14.43+0.46
13.64
10
16.5240.77
13.82
13.82
0.
15.29
15.29
18.5440.14
18.54_0.14
17.84_0.8
13.82+0..4
coin
13.6110.15 18 13.01
10.88
14.48
10.5a
13.05
10 57 13.05
13.82 13.05
15.71
13.05
10.57
14.09
56
13.05
Contimued on the next page.
40

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part IV). Continued from the previous page.
Appronch
Baseline DEP-DATA"
Aggregation-leeel
Crowd-level
Ensemble
TRUR
DEF
ARC
ALE
ALC
CXU
CRC
CLC
EN8
mabel-ne-rand-
gt
60.3
2.35
27
6.35
6.43
N/A
N/A
N/A
N/A
N7A
0..21
av
15.19+0.50
17.96+0.50
19. 19.68
16.19
16.19
N/A
N/A
N/A
N/A
N/A
15.19
0..80
da
14.3440.65
17.4220.73
18.60
15.54
15.54=0.27
15.54
25
13.91
13.91
17.58=0.09
14.34
cl
14
15.9410.48
18.1810.71
19.08
19.98
14
15.6910.a5
15.7410.63
15.3910.95
15.69 ss
15.6910
trace
13.824o.11
16.75+o.an
19.31
16.21:1.05
16.21=1us
16.67:0.82
16.67
13.82
13.82
10
13.82
10.91
16.1840.51
conal
11.161..50
16.8420.70
19.23:20.95
16.01
16.01
17.7Lg0.59
14.71
46.46
14.46
14.46
14.7110.46
11.16
union-a
16.50+6.75
15.15+0.s8
17.93++0.22
16.50+0.75
16.50-a.75
16.50+a.75
15.69+2.48
15.22+0.37
15.22-00.7
14.90
18.87 +0.80
union-b
14.65 sc 16.38
18.97
20.39
18.6220.5
15.98
14.85
20.34
18.6240.58
18.62
14.85
L0.34
18.62:0.05
geo-x
15.7612.80 so 16.3610.58
18.8410.61
15.20
10.57 15.20
2.577
15.20
15.20
10.57
20.2410-42
18.57
70.78
15.20
10.57
15.20
geo-f
14.76+0.51
16.03+o.34
18.77++0.4
13.28
+0.45
13.28-0.45
17.44:0.16
+3.28
13.28
+0.45
13.28
+0.45
13.28
0.44
13.28, +0.45
sadl
11.7540.98
17.00+0.28
19.41+0.60
14.75
20.95
11.75
18.61
18.6140.79
14.73
18
14.75
20.98
17.
17.19=0.4
11.75
croud-ar
15.08+1.34
16.33+0.05
19.33++0.49
20.17
20.17+0.05
15.08+1.34
15.08+1.34
14.65
14.65
40.70
14.66
+0.70
14.06
15.08+1.34
annot-nix
14.5810.42
16.08
18.4320.61
14.58
10.42
14.58
20
14.58
14.5840.41
14.58
10.43
19.55
19.55
10.55
15.52 0.
coin
12.0010.17
13.96
20.57
10.77
14.5420.43
11.440-44
14.5410.0
14.5410.41
14.5410.43
14.5410.43
11.54g
14.54ga.
label-ne-tul
gt
2.02
6.01
00
6.60
6.60
N/A
N/A
N/A
N7A
N/A
6.60
av
14.76+0.50
16.89+0.71
18.
18.42+00.47
14.03
14.38
00991
N/A
N/A
N/A
N/A
N/A
14.38 +..91
da
13.1340.76
15.07
15.9640.20
14.81
14.8
13.13
24.8
12.95
12.95
12:95
12.95
13.13
cl
12 13.7T10.78
15.0210.48
14.38
14.38_0
14.1111:4
13.8610.00
14.7310.a1
14.7310-31
13.82
13.8220 68
trace
14.1640.54 15.1740.a5
16.534o.17
14.34:1.0s
14.34_105
14.34;1.s
12.63.0.74
14.3441.05
14.34:1.0s
14.34=1.0
14.34
conal
13.5410.59
14.80g1.01
16.80
13.54
18.54
13.51
16.95
13.54
18.54
16.95_2.17
13.54,
union-a
12.73+0.49
13.72+0.70
15.12+0.24
14.16+0..54
14.16-0.44
15.64+0.00
14.06+0.58
44.29
13.02
+1.02
17.69-2.a5
14.06+0.55
union-b
12.98 so
13.97=0.s3
16.0340.20
13.86
13.8620:1
13.37
13.57
14.5840.16
14.58=0.16
13.57=0.02
12.95 1.
geo-x
15.1912.1
13.9210.51
16.03
10.51
14.51g0s3
13.69
13.69
14.81
13.42
10.49
15.69_0.63
14.81 1.0
geo-f
26.58 13.7440.73
15.824o.21
12.8620.4s
12.58-
36.
26.58
14.76
12:00
+0.78
13.33-1
12.00+o.78
madl
14.21+0.3a
15.07+0.58
16.72+0.48
12.9
12.95
12.98+1.68
12.98+1.5a
25.22+1.80
15.
12.95
20.37
15.
15.22-1.80
12.95
croud-ar
14.66+0.58
14.78+0.90
15.79+0.55
13.77+0.61
13.77
14.53+0.78
13.77+0.51
13.65
13.66
+0.41
13.77+0.61
13.77
annot-nix
13.6440.38 38
14.95
16.03
14.90
14.
16.43
13.64
10.38
16.35
16.35 16.35
07
14.90
coin
11.0610.00 s6 12.95
13.57
13.28_...0
13.28
13.43
13.43
15.6210.M
12.07
20.04
15.62 as
12.07 1.0.54
depan-n-woreet
gt
10.59+0.14
10.59+0.14
10.52+0..2
28.15
15.1511122
N/A
N/A
N/A
N/A
N/A
28. 15.11+1..8
av
66.30+1.14
52.55+0.30
73.28
73
08.61
+0.70
68.61
+0.79
N/A
N/A
N/A
N/A
N/A
68.61
0.
da
66.30+1.14 22.55
73.28440.38
68.6110.75
68.61
68.61
68.61
68.61
68.61
18.61
68.61
68.61
68.61
cl
62.6713 sa
69,6312.58
68.4112.28
67.77
67.77
67.77
8o
62.85
12.37
62.85
12.57
62.85
22.57
62.65
1 2.57
trace
52.79:3.43 21
73.16:0.44
70.62
70.6220.3
30
64.9221.33
64.9211.53
64.9221.63
52.79
3.43
64.60
conal
68.01+0.52 72.02+0.34
72.35+0.35
70.78
20.0s
70.78
70.78
70.81+0.50
70.78
70.78
70.81=0.86
70.78
union-a
63.49+1.07
67.42+2.31
67.65
71.16+0.35
71.16-0.35
67.6544..11
67.65+0.21
67
65..21
07 07.06-40.21
67. 07
67. 66.62+0.21
union-b
geo-x
63.9012.1 67.28
63.01 66.4220.29
69.01
71.33
70.56
71.05
70.56.00.0
71.05g0::
71.05
70.56.0.00
63.01
66.15
17
66.13
66.7040.25
63.01 66.70.0.25
66.15 10.19
66. 15
10.17
66.15
66.70+0.25
geo-f
58.00 28
70.99.0.53
70.71
70.71=u.46
70.71
65.81
67
67.
07.62233.7
63.67
4.08
15.81
madl
57.6043.11 71.98+1.18
13.53+1.01
70.93=0.68
70.93=0.05
70.93
63.87
67.17+3.40
67.17+3.40
63 83.87 3.33
68. 08.88+1.00
croed-ar
70.15 22.05
72.0110.45
72.00.0.54
72.00
12.00
70.44
40.71
12.00
72.00
70.44
00771
72.00
annot-nix
59.4244.15
1..63
67.8240.7a
60.75.0.04
69.
70..5
0.
59.42
14.15
60.35
20.25
65.74
coin
67.1512
67.04
68.3811.13
69.37
69.37
69.37 11 -
65.33
68.
18.0611.02
68.0615.02
68. 06
67.72
dopanin
gt
11.0910.11
11.0910.11
10.5210.22
12.49
N/A
N/A
N/A
N/A
N/A
12.49.0.m
av
52.77+0.35
54.39
56
20.43
52.42
+0.40
52. 23
N/A
N/A
N/A
N/A
N/A
52.42,
da
45.64+0.56 45.34
48.75+00.4
00.2
20.27
06.28+0.27
46.28
40.41
20.44
46.91
44
46.91 +.044
cl
50 55.63 71.9212.7
.88
55.87
55.87
56.59
52.91
52.91
12:00
58.3314.0
58.3314.03
58.33
03
53.5211.00
trace
48.98 51.31g3.71
54.61
52.08
52.082n
48.341a.45
42.17.0.m
42.17
19.42...11
39.42
18.16
67.87
48.34.1..4
conal
52.94+1.01 70.58+3.30
53.94+0.78
52.99+0.22
52.99-0.22
52,99+0.22
53.51+1.43
32.99+0.22
52.99
53.51+1.43
52.95
uniona
52.35+2.34
51.89 73.77+4.48
+4.25
53.53
13.58-1..
51.89
51.89+4.11
51.89.4.25
01.89+4.25
51.89
4.22
51.89
+4.25
union-b
51.40+3.11 72.28+2.00
50.55
54.16
52.82_0.1
51.40+3.11
51.4043.11
52.9544.05
51.40=3.11
51.40=3.11
51.40
51.40+3.11
geo-x
48.0510.01
10.58
30.
50.4610.72
53.02
53
52.1812.00
52.1812.20
48.45
49.43
49.
13.4311.0
1.
49.43 1.33
geo-f
52.0142.09 61.45
30.36:1o.18
51.99
20.33
51.8
81.86
19.77,
51.57
51.5741.77
51.57
51.5721
19.77++1.65
madl
46.3842.18
72.72+2.50
32.96+3.88
49.65
49.65=181
49.65+1.81
48.53
49.6541.81
49.65
48.53
-3.03-000
49.65+1.91
growd-ar
55.1311.44
64.5814.46
54.
62.1240.42
54.27
54.27
54.27
53.93
+0.71
54.2741.73
(4.2721.77
53.93
0.71
53.93,
annot-nix
44.1643.16
50.74+1.70
40.75
1..08
1
47.60
49.95
47.35
4..60
0.001
47.35
coin
44.57
18.23
50.1710.22
51.91.o.pa
50.09 80 50.09
51.3513.61
45.5T
14.95
50.09
10.80
50.09
50.09
Contimued on the next page.
41

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part v). Continued from the previous page.
Appronch
Baseline
Aggregation-level
Crowd-level CRC
Ensemble
TRUR
DEP-DATA"
DEF
ARC
ALU
CXU
CLC
EN8
Sopanin-voret-
gt
10.74
20.74
10.5220.22
10.01
10.14
N/A
N/A
N/A
N/A
N/A
11.03
av
34.12+0.48
36.47+0.40
41.50+0.e0
32.1240.43
34.09
0.78
N/A
N/A
N/A
N/A
N/A
34.00
+..78
da
29.7340.05 05
20.58
35.2240.38
29.05
29.05
29.05
29.05
29.05
29.05
20.05
29.05
29.05
cl
35.0313.68 43.2615.96
38.4611.71
35.03
13.05
35.03
os
37.2211
35.03
44.1312.10
44.13
35.03
35.03
4..66
trace
21.1640.38 29.82
34.89
28.17
28.
17:0
53.564t..m
21
1o.35
21.16
10.35
53.56
21.16
10 38
conal
30.7511.47
32.5220.42
33.93
32.
32.40
10.32
32.
20
32, 32.10
32.08
32.3410.36
30.86
20.40
32.
02.08811
1.88 92
33.17 1:0.22
union-a
33.6b+2.42
43.92+5.72
37.73+0..6
36.84.4.34
36.84
4.84 37.13+0.05
37.73+0.98
37.73+0.0s
37.73
37.73-0.46
37.73+0.05
union-b
31.69 31.85+1.45
33.35
31.69
70.79
31.69
78 32.62
31.85
36.76.1.72
32.62
31.85
31.69
1o
geo-x
27.41g0.26 29.8420.15
32.761n.67
27.41
27.41
27.
10
30.08
30.7813.17
30.78
13.17
30.05
00.088...8
27. 27.41 11.441:
geo-f
21.91 26.32+o.71
40.49
25.44
29. 0.55
23.16+1.32
25.84
24.40+2.51
23.16
+1.32
25.
0.
23,16 1..32
madl
croud-ar
31.59+0.57
20.74+0.35 29.60+2.70
31.89+0.41
34.07
31
87
32.60+2.17
27.58
32 22.6672
29.90=1.18
20.74
32.66+2.17
a5
31.96+1.40
20.14
20.74 00
30.98
20.30
+0.58
30.98
20.74
2..35
+0.58
31.96-
20.74
20.71 3.
31.5b++1.97
annot-nix
21.61 24.09
26.29
28.09
28. 18.09=1.20 1.20
22.32
22.32
23.45
2..2
22 58
23.45
coin 21.2613.:1
28.21
23.20
10.33
29.51.0.40
29.51_0-10
20.11
F 22
26.1410.77
20.11
10.22
20.11
40.22
25.79
20.11
dopanim- raad-1
gt
10.97
10.97
20
10.52:0.22
11.28:0.26
11.28
0.20
N/A
N/A
N/A
N/A
N/A
11.28 0.26
av
20.79+0..2
21.50+0.a0
27
20.56
+0.30
20 56
0.30
N/A
N/A
N/A
N/A
N/A
20. 10. 0..30
da
20.79+0.02 21.50 27.66.0.33
20.56
20.56
20.56
20.56
20.56
20.56
20.56
20.56
cl
23.0112.1 60.001s.17
26.9212.65
23.04
12.10
23.04
22.10
23.04
12.11.10
29.2412.73
23.04
12.10
23.04
12.10
29.24_
26.4914.0
trace
22.33+3.13 23.53;2.m 27.48-2..6
17.34
+0.44
17.34
0.44
20.08
27.47
17.3440.44
17.34
+0.44
46.79=2.8
17.34
conal
19.6111.00 18.4213.
23.1440.60
19.61
19.61
1
19.61 11:11:000
19.61
+..00
19.61
19.61
19.61
19.61
19.61 61.61+1.000
union-a
20.36+2.11 68.81+3.45
25.32+3.71
20:30
+2.11
20.30--.11
25.32+3.71
25.32+3.71
25.32+3.71
26.32+a.71
26.32-a.71
22.43+1.77
union-b
20.0440.34 45.95
22.01
20.18
+1.55
20.18
20.18
1
21.7442.a0
20.3242.01
20.32=2.01
21.74=2
20.32
geo-x
18.8810.57 15.08
22.50
18.88
10.57
18.
18.88
17.57
19.68
19.20
19.20
20.35
28.6882.00
19.20 10
geo-f
16.59;u.se 23.57+3.20
21.95
20.54
16.59-
16.59
16.59
16.45
+0.21
16.45
+0.21
16.59
16.59
madl
16.78+0.98
24.20+0.94
27.09
16.78
16.78
20.06
16.78
2 08
16.78
00
16.78
20.99
16.78
00.95
16.78
0.00
16.78. +
croud-ar
19.95+0.43
61.13+6.01
22,
18.4..2
19.90+0.45
19.96+0.43
19.95+0.48
18.89
40.52
19.95
19.95
18.80
-0.0
18.80
00.62
annot-nix
17.7940.92 22
32
21.40
17.79
17.79
17.79
10
18.39
17.79
17.79
18.81
10.0000 40
17.79
coin
17.0912.00 69 21.0442.90
19.1610.c1
17.78
17.78_0
17.78
17.09
42.89
17.09
+2.60
17.09
42.00
17.09
4-2
17.09 2.2.9
dopanim-rant
raand22
gt
10.85+0.16
10.85
10.52+0.22
10.75=0.07
10.91
=0.39
N/A
N/A
N/A
N/A
N/A
10.91 0.330
av
20.7640.30
23.31
28.22
20.76
+0.30
20.76
0.30
N/A
N/A
N/A
N/A
N/A
20. 70.76,
da
19.78+0.4a 21.36+0.40
24.43:20.30
20.13
20. 03
20.38
19.78 20.02+0.00 20.02=0.c0
13 2.
20.
03.
cl
23.72ga.45 33.3314.77
24.2613..9
23.72
23.7223-4
23.7213
13.48
31.0212.75
31.0219.75
76 23,80
23.72
trace
16.10 16.982o.31
23.59
19.19.2.74
19. 19.17
16.77:0.50
16.10+0.30
16.774v.0
19.19.0.74
16.77
16.77:0..0
conal
18.59+0.30 18.90+0.39
20.42+0.17
18.77+0.41
18.
18.48
19.79+0.23
18.59+0.30
18.59
19.79=0.28
18.59+0.30
union-a
21.30+4.52
34.20+4.o0
23.75+3.17
21.30+4.32
4..20-4332
23.75+3.11
23.75+3.17
23.7543.17
23.75+4..17
23.75-3.17
20.20++1.9 1..0
union-b
18.80 36 24.79+2.22
19.17
19.48
19.48_0.05
20.03
18.80
20.0342.05
20.13
19.48
19.484...
geo-x
18.0210.2 22 19.7012.11
19.61
18.02
18.02
18.02
18,8110.45
18.02
18.02
18.02
18.02
geo-f
15.29 15.98
19.07
15:29
-0.25
15:29
0.22
15.29
17.4240.14
15.29+0.25
17.49.0.33
17.49_0.1
15.29 +.
madl
17.30+0.5a
16.37 77
22.
22.1840.07
17.30
16.72=0.41
16.61+0.47
16.61+0.47
16.6140.77
16.72=0.41
16.72=0.4
16.72+0.41
croed-ar
18.7410.48
21.1011.s1
19.
18.15
18.74
40.45
18:74
-0.45
18.74
18.74
60.56
18.74
40.48
18.74
18.74
-0.80 0.66
18.74
annot-nix
17.1840.37
16.4310.40
18.1220.33
17.18
0.
17.18
16.92
16.92
16.02
20.22
16.92_0.68
0.00
16.92
0.
coin
15.5710.31
14.75
40.31
17.15 10.41
17.23
17.23
16.1810.44
17.02
16.221n.18
17.23
17.15
17.23
dopantn-rama rand-y
gt
10.4310.15
10.4310.15
10.3210.22
10.23
11.2320.15
N/A
N/A
N/A
N/A
N/A
11.2310.15
av
18.51
21.26
21.
23.83
23.83-20.55
18.50
+0.41
18.56
-0.41
N/A
N/A
N/A
N/A
N/A
18.55
+..41
da
17.59+0.42 19.34+0.52
21.08+0.30
17.59
17.50
17.59
17.59
17.59
17.5910.42
17.59
17.50
17.59
cl
20.54ga.11 32.4415.70
22.8411.37
20.54
13.71
20.54
23
20.54
20.5413.11
31.4017.11
31.4017.1
20.54_..17
20.54
13.71
trace
14.62ga.17 17.75go.28
20.36
17.83
17.83gu:1
15.8710.11
14.62
16.891v.27
16.8920.27
16.89_0.17
16.89.0.27
conal
17.49+0.08
17.15
18.32+0.20
17.26+0.18
17.26-0.16
17.26
17.51+0.20
18.63+1.56
17.51+0.20
17.51+0.20
18.65
18.63-1.88
17.26+0.16
uniona
18.8642.34
30.60+6.42
22.97+4.49
19.05
+2.07
19.05
22.97+4.40
22.97+4.40
22.97:4.4u
22.97+4.40
22.97=4.4
24.97+4.44
union-b
17.7340.21 18.5442.32
16.9440.27
17.73
17.73=0:21
19.01
17.75
17.8440.32
16.57
10.43
17.75=0.50
16.57
+0.4
geo-x
17.1410
18.7012.01
17.4416-30
17.14_p.3s
17.14
18.81
17.23
17.9710.48
17.97
17.14_p.3t
17.14
geo-f
14.624t..32
15.35
16.89.0.22
10.90.0.11
11.90_0.12
14.51+a.12
16.15
14.90
15.73
15.
15.73_0.21
13.03:0.21
madl
14.24+0.3a
15.61+0.12
20.12+1.24
14.88
14.88=0.05
14.58
14.88+0.08
40.07
14.73
20.07
15.11=0.47
o0.07
crowd-ar
16.7910.8 38
18.2812.30
17
17.58
16.79
+0.38
16.79
0.36
16.79
+0.38
16.79
+0.38
16.79
+0.38
16.79
+0.38
16.79
0.88
16.79.
annot-nix
14.82+0.47
15.65
16.37
14.82
05
14,82
14.82
15.46
1.46620.0
15.91
15.91
0..8
15.46.0.20
coin
14.27 13. 16.101o.75
15.63
16.32
10.31
16.32
14.36 ka.
16.17
15.8610-44
16.32
16.17
16.52
Contimued on the next page.
42

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part VI). Continued from the previous page.
Approach
Baseline DEP-DATA"
Aggregation-level
Crowd-level CRC
Ensemble
TRUR
DEF
ARC
ALU
CXU
CLC
EN8
dopanin-full full
gt
11.02+0.28 28
11.0220.28
10.5220.22
10.57
20.26
10.57 20.26
N/A
N/A
N/A
N/A
N7A
10.57 0.2.6
av
17.32+0.58
17.82+0.25
20.39+0.17
17.32
17.32-0.66
17.32
N/A
N/A
N/A
N/A
N/A
17.32
da
16.82+0.51
17.16
19.0740.17
17.45=0.29
17.45=0.29
16.82
16.82
17.33
17.33
17.45+0.29
cl
18.2010.52 52
42.6611.87
22.3614.39
20.20
28.20
22.0915.16
19.6312.30
33.9013.31
25.0611.05
19.63
18.20
trace
13.804o.20 15.1940.15
18.55420.4
14.12
10
15.89=0.17
14.12
14.12
16.85+0.25
17.20=0.75
17.20=0.76
15.8940.17
15.89
conal
16.6220.14
19.2122.28
17.31
16.76
16.76
16.76
16.76g0.28
19.3912.07
16.581o.13
16.5820.13
17.09
16.16;
16.16;:..
union-a
20.12+3.93 34.56+4.02
23.1443.22
20.12+3.08
20.12
23.14+3.22
23.14+3.22
23.1443.22
23.14+3.22
23.14-3.22
20.12
union-b
17.82+1.94 20.57+4.03
16.26
17.82
17.82=1.04
18.88
18.03
17.7342.70
17.82=1.04
18.03=1.68
16.28 2.....33
geo-x
16.06go. 29 19.94
16.
16.06
20.29
16.06
16.54
17.48
16.591o.15
16.06
16.06
16.06
geo-f
13.71+0.49 14.90+
16.25+0..0
10.80
15.80=0.19
+3.71
16.77+0.20
14.93+0.42
15.52-1.331
15.52=0.31
14.93+0.42
madl
11.1540.28
11.99
17.86
16.41+0.12
16-012
15.26=1.17
15.26+1.17
14.91+0.92
17.95+1.87
14:22
00.2
11.91+0.02
croud-ar
16.27+0.19
21.47+1.74
16.63++..18
16.
16.27
+0.15
16.27
-0.19
16.27
16.27
+0.19
16.27
+0.17
16.27
16.27
-0.10
16.27
annot-nix
14.3840.28
28
10
15
20.35
0..44
15.54
14.35
15.76
15.40
15.40
15.54
coin
11.1210.12
21.4119.11
14.72
20.251
14.90.0.19
14.90=0.11
15.1210.31
15.87
15.12
14.90.0.19
11.90_o.11
14.901.
reuters-vorets
gt
1018
3.18-0.17
4.14+0..0 07
7.25
1.25
N/A
N/A
N/A
N/A
N/A
7.20++0.22
av
48. 18.80+2.4
59.46+1.77
58
10.1..8
49.41
+3.00
49.41
3.00
N/A
N/A
N/A
N/A
N/A
19.41, 5.00
da
48.80+2.40 59.48+1.77
18.90
49.41
49.41
49.41
49.41
49.41
49.41
13.00
49.41
49.41
49.41
cl
32.0612.17 58.1011.61
57.20
32.66
12.17
32.66
27
32.66
12.17
34.9219.0
32,66
12.17
32,06
12.17
34.02
32.66
1.
trace
48.77+2.32 58.6940.17
18.50
51.75
+5.58
51.76
6.68
51.75
+5.58
51.75
52.534u.61
52.53.0.e1
52.53
51.75, 5.66
conal
17.8915.64
60.6121.18
39.65.1.61
51.03
51.08
51.03
51.03
51.03,1.54
51.03
41.44
51.03-1.44
51.03
51.03
union-a
47.38+4.54 58.64+0.48
59.13+1.64
43.80
43.80
43.80
43.80
51.62+4.91
61.62+4.91
61.62-4.91
43.80
union-b
48.51+3.03 59.30+1.90
57.8% 11.32
48.30
48.30 7.
48.30
37.18
13.44
37.18
13.44
37.18
20.44
37.18
3
37.18
17.18
geo-x
48.4712.66 17.97
37.7711.30
51.26
51 51.26
51.26
14
36.11
36.11
36.11
36.11_1.43
36,
36.11 11.11111.40
geo-f
24.56+3.29 58.07
58 :8.69
42.19
45
12.89
42.63
4..63
42.03-44.00
4.60
37.01+1.97
madl
51.41+0.78 59.77+1.02
39.32+1.10
51.41+0.78
51
51.
+1
51.41
41.45
20
11.45
20.27
11:45
0.27
11.45
15.15.00 0
croud-ar
44.60+126
58.86+1.30
58 68.62
44.60
1.20
44.60
1.26
44.60
44.60
1.20
44.60
44.60
+1
44.60
1.26
44.60
1.226
annot-nix
47.1144.60
61.35
62.35
48.4622.38
26
48.46
3.38
44.36
44.36
44.36
44.36
coin
48.0611.59 s9
59.2410.51
62.95
48.06
$1.59
48.06
1.69
18.06
12.1215.07
72.12
53.01
48.06 1.59
neuters-voret-2
gt
3.79
3.70=0.14
4.14+0.0r 07
4.14
1..40
N/A
N/A
N/A
N/A
N/A
1..40.007
ai
26. 26.22+1.05
2.
43. 12.0079
26.22
1.65
26.
22
N/A
N/A
N/A
N/A
N/A
26. 22,
26.22+1.00 1.05
da
23.34+0.58 33.63
34.38
23.34
23.34_0.68
2
23.34
23.12
23.12
23.12
+0.55
cl
19.3211.52 27.11 31.0310.00
20.20
20.20
19.32
1121113
19.32 11.52
19.32
11.52
19.32
11221.333
19.32
19.32 11.52
trace
conal
22.11+1.20 32.99+0.71
20.58 36.63
35.76-20.53
35.70+0.53
22.11
20.58
+1.8
22.11
20.58
1.85
22,11
20.58
1.
11:20
27.75+0.96
26.25
22.11
26.25
22.11.1.20
26.25
26.25
27.75=0.06
20.58
22.11
union-a
19.33+1.97 34.01+2.sa
41.08+0.58
19.33
1.97
197
39.77+20.3
25.2543.30
26.07+1.40
26.07
39.77=20.5
39.
+
union-b
geo-x
20.8612.02
18.17+1.00 32.73+1.an 33.81
30.1911.00
33.3410.50
21.12
20.86
20.86
22.12-2004
86
18.17
17.23
13.02
18, 18,49
18.17
18.49
18.17
11.3911.0
+1.80
18.49
18.17
11.9911.3
11.89
18.
18.17
19.4991000
1
17.23
18.17
1.80
geo-f
20,53+1.. 27.00
20.53
+..0
20.53
20.53
10.60
40.60
10.60.1..7
10.60-1176
22.59 1.05
madl
23.3840.01 35.00+5.58
37.6448.49
20.38
23.38=0.00
21.60+3.10
19.57
19
19.37
20.84
19.57
19.57 +
croed-ar
22.3611.02 32.4910.64
33.08
22.36
1.62
22.36
1.60
22.36
+1.00
22.36
1.02
22.36
+1.02
22.36
+.62
22.36
62
22.36
1.00
annot-nix
21.87+1.28
35.45
43.81
26.73=1.54
26.73
1.64
20.91
20.91
20.91
10.81
20.91
26.73+1..6
coin
23.791111
31.7812.40
36.86
23.79
$1.18
23.79
16
23.79 +
29.
19.43111
1.77
69.49
69.4912.90
69.49
69.
so
26.07 11.39
reuters-vorete
gt
3.88
3.88_n.09
4.1410.07
4.13...11
3.7510..1
N/A
N/A
N/A
N/A
N/A
131310011
da
ai
20.55+4..7
18.92+0.96
38. 38.8440.41
30.49+1.1n
40.
32.04:1.14
40.1340..22
20.56
18.02
+0.97
20.
18.92
0..97
18.92
N/A
18.92
N/A
23.31+1.00
N/A
24.64
N/A
24.64=0.44 4.
N/A
20.
18.92
00.97
cl
21.1112.09 17.88
29.0610.71
20.4011.00
20.40
04
20.40
21.1112.08
21.1112.00
21.1112.04
21.11
20.40
trace
18.481a.51
31.181o.47
31.48:1:24
18.48
+0.51
18.48
n.601 24.66
24.6612.0
24.6612.05
24.6622.03
24.66_2.05
18.48
conal
17.93+2.19 29.04+1.22
32, 12.16+1.02
18.52+0.78
17.93
18.52+0.78
17.93
18.52+0.78
17.93
42.19
17.93
17.93
union-a
40.03+2a.7 37.20+4.41
36.41+0.79
17.96=1.41
17.
17.96=1.4:
40.03
40.03
16.93.2.53
16.93
42.93
16.98
2.68
16.93 16.93+2.68
union-b
17.9941.sc 22.21 30.00
17.99
11.6
17.99
1.86
21.96
24.43
21.9642.70
21.96=2.79
24.43=2.02
21.9642.79
geo-x
18.2011.s1
17.25
10.41
30.25
18.20_1.84
18.20
18.20
19.85
19.85
19.8511.28
19.85
25.73_9.07
19.85
geo-f
14 18.8141.11
30.05:2o.67
18.05=1.77
14.76
15.62
34.91:17.
31.91
27.
27.70_2.0
15.62
madl
17.4840.22
29.76*8.49
32.6642.06
17.48
17.48-0.8
17.06
a 88 17.00
17.06
17.06
20.83
17.06
-0.
17.06
growd-ar
18.5711.50
23.4921.07
31.4511.35
18.57
21
18.57
18.57
1. 55 18.5741.50
16.02
+1.01
18.5721.65
18.57
1
18.57 11.55
annot-nix
18.62+0.
34.92+4.11
37.7448.37
16.01
16.91
16.91 11.1 20.07
+0.80
20.07
20.07
16.91
coin
16.861o sa 27.14
33.6810.53
16.86
0..33 16.86
19.13
19.13
19.13
19.13
19.13
19.13
Contimued on the next page.
43

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part VII). Continued from the previous page.
Approach
Baseline
Aggregation-leeel
Crowd-level
Ensemble
TRUR
DEP-DATA"
DEF
ARC ALU
CXU
CRC
CLC
EN8
reuters-rann
gt
1.
2.14
4.14g0.07
4.05
4.05
N/A
N/A
N/A
N/A
N7A
4.05.
av
13.29+0.07
25.86+0.90
26. 26.55+0.38
16.16
+0.69
16.16
00.00
N/A
N/A
N/A
N/A
N/A
15.16
0.06
da
13.29
25.86
26.55
15.16
16.16g0
15.
15.16
10
15.16
15.1610.00
15.16
10.06
15.16
15.16
cl
14.6410.84 22.4713.56
24.7210.78
14.28
11.18
14.28
16
14.2811.16
21.3711..0
14.6410.84
14.64.0.84
21.37
14.64
trace
conal
13.8511.13 22.2214.01
13.8640.47 15.17
25.33
26.6440.74
13.85
14.91
10.41
13.85.1.1
14.91
13.85
10:4
13
14.91
13.85
10.4
18.23
13.85
13.85
14.91
13.85
14.91
10.41
11.15
14.91
13.85
10.41
13.85
18.23
13.85 13.85
14.91
10.41
union-a
14.79+1.28 34.62+14.2
26.58+0.77
14.79+1.25
14.79-1.2
12.99
+0.88
12.99 +0.58
+0.88
12.99
+0.68
12.99
12.99
+0088
union-b
geo-x
14.41go.94 20.6811.08
14.3440.92 22.25
24.7540.08
25.00
13.92
13.88
10.68
13.92
13.88
20 0.
13.92
13.88
1.0 68
20.6613.02
19.53
13.2411.83
13.88
10.65
13.88
15.24
10.05
20.66
19.53
2.80
15.2111.83 1.83
13.88
geo-f
12.6940.44 18.31+2.1
24.39
12.69
40.44
12.69
0.44
12.69
12.69+0.44
23.23
12.69
12.69
+0.44
23.23-1.79
12.69 +o.69
madl
11.97+1.44 2 27. 27.26:1.41
18.36
18.36=0044
18.36
21.78+2.97
+1.34
14.97
11.44
21.78=2.0
11.98
croud-ar
14.90+0.93
21.62+1.10
24.87+1.00
14.46
+0.61
14.46
00.01
14.46 14.46
60.61
14.46
+0.61
14.46
14.46
14.46
+0.61
14.46
-0661
14.46
annot-nix
13.2240.40
15.4
29.56.0.51
13.22
13.22
13.22
17.33
13.58
13.58
10.80
17.33
0.00
13.22
coin
11.0110.77
53.8919.38
28.4310.88
11.01
$0.57
11.01
11.01
37.92
11.01
t0.s7
11.01
10.07
37.92
11.01 4..37
11.01
routers- rand-
gt
3.84
4.14 3.84-0.09
07
4.14
4.147
N/A
N/A
N/A
N/A
N/A
1..47
av
13.05+0.70
27.22+0.a7
29. 29.22++0.98
13.00
+0.70 13.06
0.77
N/A
N/A
N/A
N/A
N/A
13.05
da
12.87+0.95
19.7740.00
20.5340.c0
12.87
12.87
12.87
14.99
14.39
14.7640.64
14.76=0.64
14.76
14.99 0.33
cl
11.4510.17 13.1710:33
15.31
11.45.0.77
11.45_0.77
11.41
15.0411.08
11.77
11.45
15.04_ os
11.41
11.47
trace
conal
11.22go.58 20.91
11.51go.57
15.58
18.4410.94
18.9240.es
11.22 +0.5s
11.36
11.22
0.88
11.22
12.1910.5
11.55
12.1910.38
11.554u.73
11.36
11.36
11.22 +o.s8
11.55
11.51.0.57
11.36
11.22
+0.68
union-a
11.45+0.48 27.37+2.44
28.28+1.02
11.40+00.4
11.45+0.44
10.07
10,67+1.06
10.67
10.07
+1.07
10.07
10.67
10.67+11003
union-b
11.53 15.46
17.3640.64
11.06
20.73
11.06
73
11.06
1
11.85
11.8541.30
11.06
10.73
11.85
11.85
geo-x
11.5310.0 14.6311.3
17.30
11.06
11.06
10
11.53
18.55
11.9810-48
11.98
10.48
18.55
25.55...00 09
11.53
geo-f
11.1540.44 12.98+o.4u
16.96
11.15:20.4
11.15:0.44
10.72+0.55
15.95+2.50
14.40+1.72
10.72
16.90=2.ve 60
10.52
madl
11.58+0.12 20.23+6.01
21.27+2.00
12.36+1.05
11.58=0.72
9.55
9.55
1164
9.55 64
1.64
9.55
+1.4
9.55 1 1.66
croud-ar
11.87+0.50
14.6440.79
17.
44.447
11.00
+0.34
11.00
-0.84
11.00
+0.34
21.6640.58
11.00
+0.34
11.00
11.00
-..4
11.00
0..34
annot-nix
12.02+1.07
19.9420.49
29.09
12.02
12.02
02022100
11.74
11.74
10.61
14.08
14.08
20.62
14.08
0.002
11.74
coin
9.25
19.8210.77
21.23
9.53
9..8
9.0310.62
9.25
20.44
9.25
0.44
9.25 4
9.25
4.
9.25
$0.45
9.25
0..44
routers-rand-
gt
3.86.20.14
3.86-0.14
4.14
4.1420.07
4. 4.14
N/A
N/A
N/A
N/A
N/A
4.14.00.7
da
ai
14.1140.47
13.6340.50
24.86+0.7
22.15+0.99
26.32
26.
22.66
14.11
13.63
+0.47
13.63_0.00
13.63
14.11 0..4
17.25
N/A
17.25
N/A
17.25
N/A
17.25
N/A
17.25_1.18
N/A
17.25+1.18
14.11 0..4
cl
13.0910.02
15.8012.7
18.3010.73
13.09
10.42
13.09
20-42
15
18.8413.45
14.53
15.2211.79
18.84
15.22
trace
13.7540.36
21.16:1.2
19.91
16.1720.64
16.1720.44
13.75
16.3420.30
13.75
13.75
+0.39
16.34_0.95
13.75
conal
13.72+1.02
16.28+0.95
20.40+0.50
13.72
18.72
13,72
13.72
13.72
13.72
13.72
13.72
union-a
15.4141.2
20.84+2.02
24.98+1.02
16.41+1.42
15.41-1.4
12.67+1.02
11.92+1.64
13.60+1.00
13.60=1.00
13.60-1.00
13,60+1.09
union-b
geo-x
10.8610.a9
14.9440.08
16.09
14.2010.0
18.9010-45
19.50
14.8511.0t
14.04
10.98
14.85
14.94
20.0
15.12
10.56
19.49
10.86
15.12+2.58
15.1242.88
12.39
12.3911.01
12.57
14.94
115711.84
10.98
12.57
19.61=1.40
86
15.12
12.57
geo-f
12.03gu.5a 19.19:2.04
18.744u.67
12.03
12.08=0.b5
10.78+1.14
14.37
10.78+1.14
10.78
+1.14
10.78-1.14
10.78
10.78., 10.78411.14 1.14
madl
croed-ar
12.341o.7o 15.43
13.01+0.48 20.61+10.2
22.02+2.45
19.88
10.88
10.99
14.16
+0.88
11.99=0.07
14.16
0.66
13.46+2.44
14.16
60.68
13.46+2.34
21.99
13.4642.34
14.16
+0.58
14.16+0.58
14.16
13:01
+0.58
20.48
21.99
13. 16.4--2444
13. 1642.88
14.16, 0..6
annot-nix
13.59+1.05
21.78
27.5140.70
15.26
15.42_0.08
15.26
15.26
15.37+1.38
10.4220.68
15.37=1.88
15,26
coin
10.1711.32
34.3618.17
24.0441.00
10.17
11.32
12.47
10.17
32
10.17
$1.32
10.17 11.52
10.17
11.32
10.17
1.22
10.17 1.1.2
reuters-full
gt
3.80
3.80_0.15
4.1410.07
4.20
25
3.80
N/A
N/A
N/A
N/A
N/A
1..6..0.2
da
ai
16.7140.32
11.6440.22
22.844o.441
17.78+0.42
24.324t.22
19.88+0.02
13.96
11.64_0..3
11.64
+0.0s
13.96
11.64
0.99
11.64
N/A
to
11.64
N/A
11.6440.33
N/A
16.01
N/A
17.68
N/A
13.90
cl
10.521o 12.2011.51
14.9010.39
10.52
10.79
10.52
20.79
12.54
16.9310.39
12.5412.00
10.52
10.52
16.93 as
12.54
trace
11.50ga.84 16.801o.77
16.61
11.50
+0.64
11.50
0.64
18.4413.40
18.4413.40
18.4413.40
11.50
18.44_3.40
11.50
conal
11.53+0.14 16.32+0.52
17.07+1.02
11.46+0.78
11.26
11,26
10.00
11.58+0.74
11.53+0.74
11.53+0.74
11.53-
11.53++0.74
union-a
15.53+13.4
20.36+3.28
23.3640..57
11.08
20.64 15.53-13.4
15.58
15.53+154
10.95+0.72
10.95
10.95
n.12
10.95+
union-b
geo-x
12.1110.00
12.1340.56 13.3440.42
11.4210.
15.10
15.7940.25
12.11
13.
12. 13.33.0.0
12. 12.11go.m 40
10.47
11.61
10.24
17.15
10.2411.92
14.9040.72
12.01
10.81
1.0110..05
10.88
17.15=1.04
10.24_
10.29 29
13.79
geo-f
10.224o.31
9.58
14.9940.52
10.16.2..1
10.46=u.63
10.35
12.9642.48
12.96422.4
10.35-10.a5
02.9622.00 48
10.35
madl
9.45+1140
14.38+1.00
20.1643.18
13.12=0.70
17.47+0.40
9.45
9.45
1.40
9.45
9.45
9.45
1.40
0.45
10.40
9.45
1.45+1.40
crowd-ar
11.7810.30
14.5520.37
16.
16.11
11.78
+0.30
11.78
0.30
1L.78
0..0
11.78
+0.30
11.78
+0.30
11.8
+0.30
22.59
06
11.78
+0.30
annot-nix
10.33+1.1a
17.37+1.5a
27.0940.76
10.95=0.55
11.95
10.33
11
10.33
10.33
10.33
10.33
11.13
10.33
10.33
coin
10.1111.01
28.1813.01
20.5010.83
10.11
11.01
10.11
1.
10.11, 1111,100
10.77
24.2
13.44_1.01
27.27
10.11 1.003
Contimued on the next page.
44

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part VIII). Continued from the previous page.
Appronch
Baseline DEP-DATA"
Aggregation-level
Crowd-level
Ensemble
TRUR
DEF
ABC
ALA
ALC
CXU
CRC
CLC
EN8
spc-vorst-1
gt
15.47
20.33
17.2740.31
20.17
10.17 20.08
N/A
N/A
N/A
N/A
N/A
16.17+0.22 0.33
av
53.87+0.48
51.47
+0.47+
+0.35
51.51+0.97
03.4442.67
53.44-2.67
N/A
N/A
N/A
N/A
N/A
53. 53.44+2.67
da
53.87+0.48 51.47
10.50
51.51
53.4442.57
23.44_2.07
53.44
23.44
23.4442.07
53.4442.07
28.44=2.07
53.44+2.07
cl
32.29127.6 51.7211.00
52.5610.77
32.29
32.29 227.6
32.29
127.6
22.
42..
32.29
32.2
82.20
12.6
trace
52.37+4.u5
51.73++1.11
51.46:21.04
50.76
13.09
50.76
0.00
52.37
52.3744.ah
52.3744.o5
52.37+4.e5
52.37
52.37+4.05
conal
53.1811.43 12.03
51.82
20.75
52.03
52.
03 26
52.03
52.03
52.0311.25
26
52.03
52.
02.03.1:2
52.0311.26
union-a
39.57+20.2
52.34+0.50
02.69
42.69
81.50+0.58
81.50+0.60
+2..2
39.5
22..
39.67
39.07
26.
+2..2
union-b
geo-x
49.9840.00 11.67
18.2110.04 11.9911.08
51.9610.94
52.23:1.77
49.98
18.00
10.00
18.00
49.98
00 00
18.00
49.98
1. 00
18.00
49.98
10.00
10.23
18.00
49.98
10
18.00
49.98
10.0
10:25
49.98
18.00
10
49.96 10.00
18.00 10
geo-f
42.60+54.9 51.78+0..u
51.86
31.33
+2..1
31.33
20.1
91.33
20.1
31.33
+20.1
20.1
31.33
31:33
+2..1
+2..1
31
20.1
31.83+20.1
31.33,
madl
croud-ar
62.25427.0 52.03+0.92
52.92+2.03
62.25+0.17
51.21
47.80
05
50.69
02.33
52. 02.33+0.99
50.69
69-
69.72+
50.10
20
50.
36.28
20.29
69
52.33
72427.1
02.33
69.72+27.7
52.33 02.33-
69
61.61
69.61+25.0
50.10
annot-nix
14.33
23.09 93
10.13
42.61
42.61
26.1
31.50
31.50
40.36
40.36
31.50 16.9
22.61
coin
4 21.85
32
10.86
31.77
+20.5
31.77
20.5
31.77
20.0
31.T7
+20. +20.5
43.31
43.31
43.31
31.77
+22.55
spc-
spc-worst-2
gt
0..77
15.67 20
+0.27
10.03
06.09=0.19
N/A
N/A
N/A
N/A
N/A
16.03+
av
28.9 28.93+0.84
30.45+1..00
38
05.55
28.93
+0.64
28. 93
0.64
N/A
N/A
N/A
N/A
N/A
28.92
+0.04
da
20.85+0.s8 20.85
28.28
19.63
19.63
19.63
19.63
19.63
19.63
19.63
19.63
19.63
cl
25.8111112 25.8411-12
31.5410.83
25.28
25. 28
20 72
16.21
10.52
16.21
10.52
17.1510.76
17.1510.78
17.15
16,21,
10.52
trace
25.75 28.7241.30
35.13
25.75
28.95-1.11
19.36
+..3
19.36
+0.33
19.49
19.49-1.08
19.49=1.0e
19. 19.49
conal
25.6111.36
29.5220.52
35.80
15.80
23.85
23 85
23.85
23.85
10.87
23.85
23.85
10.87
23.85
23.85
union-a
16.78+1.16 22.49+0.54
14.14
16.73
+1.15
16.73
1.16
16.73+1115
16.73+1.11
16.13+1.11
16.73
+1.15
16.73
1.15
16.73+1.11 1.15
union-b
geo-x
22.8211.21 25.4711.17
23.25+1113 26.03
34.0440.58
32.03
26.89
20.95
4477
10.92
20.95
16.20
4.76
16.20
20.95 1.4 76
10.78
16.20
20.95
24.70
10.78
16.20
20.95
10.70
20.95
16,20
4477
10.78
16.20
20.95
-0.78
4.76
10.20
20.95 44
geo-f
22.79+1.10 25.26+1.11
31.47
17.44
17.
17.41=u..44
16.55
16.55
16.55
16.53+
16.00
+0.97
17.
17.44_0.64
07.4440..4
17.
madl
21.78+1.16
28.71+6.19
28.61+12.2
16.20
23 16.20
=
16.98
17.1640.74
18.044
18,
18.10
18. 18.10=0.4
18.10+0.47
croud-ar
24.85+1.27 29.46+1.00
35.72+0.78
24.80
1.27
24.85
- 17
76
25.
25.
25.20+3.84
24.85
+1.07
24.86 +1
25. 25.20-33.4
25.20+
annot-nix
17.3940.81
26.30
20.48
16.94_0.40
16.94_0.44
0..4
18.46
26
16.70
16.94
16.94_0.40
16.70
16.94
coin
24.6511.41
23.6811.28
31.0011.50
22.20.1.01
22.
20
20 0. 16.35 ja
16.35
$0.35
16.35
16.33
40.35
16.35
40.35
17.28
16.35
spc-zorst-v
gt
15.85+0.48
15.85+0.43
17.27=+0.31
16.00
15. 15.0000000
N/A
N/A
N/A
N/A
N/A
15.1640.000
av
20.4440.70
25
22.8
27. 91 40.72
18.50
+0.53
18.
0.68
N/A
N/A
N/A
N/A
N/A
18.50, +0.68
da
18.6440.40
18.6420.49
22.72+0.03
16,08
16.98
16.98
16.98
16.98
10.31
16.98
10.38
16.98
16.98
cl
18.0910 2s
18.0910.28
21.5010.s0
16.54
16.54
16.13
16.13
16.13
16.13
16.54_0.36
16.22
16.13 10.53
trace
16.50
20.20
26.0240.62
16.50
+0.40
16.50
0.40
16.8610.25
17.4420.38
16.50
16.50
+0.40
16.50
-0.0
16.50
conal
17.36+0.00
20.30+0.74
25. 23.4340.54
16.86
16.86
16,86
10.37
16.86
17.36+0.00
17.36+0.60
17.36=0.00
16,86
union-a
16.27
18.06+0.25
20.43++..51
16.13
+0.49
16.18
16.18-
25.12+14.0
16.13+0.40
12.1244..0
26.12-11.0
25.12-14.0
25.12+14.0
union-b
17.81
18.31
22.50
17.96
17.96-0.61
16.20
16.20
16.03
17.24
17.24_0.38
16.03
1
geo-x
17.72
26 18.01
21.33
17.72
17.7220.1
15.74
10.00
17.53
15.74
1o.0s
17.78
17.78
15.74
geo-f
17.69
18.03
21.22
17.69
20.31
17
15.78+0.00
15.78
+0.60
16.
12.12.4.22
17.69
17.74
16,
o0.23
madl
18,1041.46
18.59+1.49
18.1540.79
16.10
16.
16.1660.88
16.16+0.38
17.83+1.14
16.16+
16.05
16.87=0.02
16.16+0.38
croed-ar
17.8220 ss 20.38
25.34
17.66
+0.47
17.56
0.47
17.56
19.5611.40
17.56
#0.47
17.56
+0.47
17.56
00.7
17.56 0..47
annot-nix
15.96 30
19.61
19.31
15.96
16.87
0.
15.96
10
16.26
15.96
15.96
16.87
20.72
15.96
15.96
coin
16.8310.00 69
17.5810.31
20.9310.27
16.71.0.35
16.71
16.68
16.29
16.29...52 40.52
17.39
.22
16.7110.35
16.71
16.71
spc-rand-1
gt
15.1810.24
18.1810.21
17.27
18.1820.24
18.0
N/A
N/A
N/A
N/A
N/A
15. 15.1810.22
av
16.214o.33
16.21
16.
22.89
16.07
+0.61
16.07
0..0
N/A
N/A
N/A
N/A
N/A
16.07
+0.01
da
+0.21
+0.33
22.80+0.41
16.07
16.07
16.07
16.07
16.07
16.07
16.07
16.07
16.07
cl
15.58 29
10.58
21.6010-00
15.58
10
15.58
20.39
15.58
15.58
15.58
10.28
10.29 15.58
15.58
10.29
15.58
15.58_0.19
15.58 1.
trace
18.881
16.21
23.1210.49
16.15
+0.67
16.
n.67
16.15
16.15
16.
16.
+0.07
16.15-p.ez
16.15
conal
16.05+0.40
16.14+0.42
22.35+0.31
16.05
16.05
20.40 16.05
16.05
16.05
16.05
10.40
16.05
16.05
16.05
uniona
17.3640.48
15.64+0.21
21.5644.77
17.36=00.5
17.30=0.45
17.36
17.3640.44
17.3640.44
17.36=0.45
17.36
45
17.3640044
union-b
17.90+0.52 5a
15.59
21.9240.40
17.42
17.42_0.4:
15.42
ao.44
15.42
10.44
15.42
15.42
20.44
16.70=0.52
15.42 +
geo-x
17.9710.65
15.57
21.8410.52
17.40
17.40_o.1
17.4010.4
44
17.40
15.40
10.34
15.40
10:34
15.40
_0.34
15.40 10
geo-f
17.9140.59
15.51+0.a7
21.7240..1
17.36-0.45
17.36=u.45
17.36
17.03
16.
16.28
16.28
20.33
17.03_2.45
17.361a.4s
madl
16.67+0.60 50 16.49
73 19.01+1.07
10.79
16.79+
16.79
16.67
16.7940.87
16.79+0.87
38.
18.75415.4
16.79
crowd-ar
16.3210.52 52 16.20+o.3a
s0 22.9210.37
16:2110.36
16:21
36 16, 16.32
16.3210.12
16.21
16:21
16.32_o.50
16.32 5.5
annot-nix
16.37+0.81
16.16
10
21.43
16.63
16.63
16.63
16.63
16.36+0.01
16.36
16.36
16.63 0.01
coin
15.6610 za
15.35
20
21.0210.31
15.66
15.66 93 15.66 10 23 16.45
15.6610-23
15.66
16.45
3s
15.66
Contimued on the next page.
45

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Zero-one loss results [%] (part IX). Continued from the previous page.
Approach
Baseline
Aguregation-level
Crowd-level
Ensemble
TRUE"
DEP-DATA"
ABC
E238
apo-rand->
gt
15.19
15.19+0.22
17.27+0.51
15.93+0.21
15.19-0.22
N/A
N/A
N/A
N/A
15.85+0.21
19.5340.08
16.58
22.8240.40
16.58
16.58
N/A
N/A
N/A
N/A
N/A
16.58
58
da
15.8710.16
15.8720.16
20.2810.50
10.28
15.87
15.87
15.87
15.8710.16
15.87
16.27 29
15.87
el
17.022a.0e
15.69+0.30
18.9940.42
15.97
15.09
0.33
15.75:0.14
15.75
15.69
+0.30
15.69
+0.39
16 16.46-0.34
15.69 15.00
trace
15.21
15.79
20.26+0.72
16.22
16.22=0.45
16.05+0.39
16.05+0.39
16.22+0.45
16.2220.45
16. 16.22=0.4
16.22+0.45
conal
16.4110.17
15.4620.55
19.8320.54
15.20
15.20
15.20
16.21
15.20
15.20_0.70
16.21 as
15.20
16.3410
20.00
18.70
16.34.0.25
16.34_0
16.3410.25
18.00
16.651o.12
16.6520.12
18.00
16.3410.2
union-b
16.3919.3
15.65
19.0610.50
16.39
15.6720
15.6710.30
15.67
15
15.67
16.39_o.a
15.67
geo-w
15.65+0.25
15.6020.32
18. 28.63-20.28
15.46
+0.28
15.0
15.65=u.2s
15.65-0.22
15.65+0.22
15.65+0.28
15.65=0.22
16. 2..588...44
15. 15.65++0.2
goo-f
15.62+0.27
15.61
20.3s
18.9340..1
15.62
15.62=0
15.62+0.27
15.62+0.27
15.6240.27
15.02=0.27
15.62
15.62+0.27
madl
17.8712
15.85
11
17.90
17.43
17.43g0.5
17.52
17.52
16.591071
16.59_0.71
16.59_0.71
16.59
croud-ar
16.98
15.49
10.59
19.81
16.62.0.51
16.0
16.06gu.22 22
16.48g0.71
16.48
16.48
16.48:0.71
16.48_0.73
16.48
16.10+0.54
15.21
10
18.79+0.28
15.99+0.26
15.99-0.26
16.05+0.07
15.99+0.20
10.99
15.99+0.22
15.99-
15.99+0.26
coin 15.77+0.30
15.63
+0.42
19.15-40.51
15.77.0.30
15.77=0.00
15.77+0.30
15.77+0.30
15.7740.50
16.77+0.30
16.33 16.33
15.77+0.30
epo-rand
gt
15.76+0.29
15.76+0.29
17.27+00.51
15.28+0.25
15.28-0.28
N/A
N/A
N/A
N/A
15.28+
mv
16.89+0.78
16.32
18.89.0.117
16.69
16.32
N/A
N/A
N/A
N/A
N/A
26.89
ds
15.7210.26
15.71g0.10
18.2910.34
15.72_0.26
15.72
15.44g0
15.7220.20
15
14.89
20.43
14.69
14.89_a.1
15.44go
el
15.72
15.91
16.37:0.49
15.72.0.a1
15.6920.eu
15.61
15.61
15.614o.63
15.40
+0.31
15.54
15.69 0
trace
15.2240.41
16.31
18.3440.38
16.22 0.41
15.22
15.22
15.22
15.22
15.99
15.22
15.22
conal
16.4710.41
15.0820.39
17.4520.35
16.04.0.29
16:04.0:29
15.68
15.42
16.0410:20
16.0420.29
16.04
29
16.0410.0 20
15.5710.32
15.8310.08
16.4310.52
15.57
15.57
#0.32
15.57
20.32
15.57
15.57
15.57
15.57
to.52
15.57
20.32
15.57
2..32
15.57 15.52.
union-b
15.301o.a1
15.901o:1
16.4810.65
15.30
20.41
15.30
20:11
15.5810
40.30
15.30
10.41
15.30
10.41
15.30
15.30 200
geo-x
15.78
15.80
16.41:0.41
15.61
15.612u37
15.0
15.32
15.32+0.37
15.32
15.78
16.7820.a3
15.78
15.61,
15.
15.61-t-..7 t..37
goo-f
15.7640.57
15.50
16.1440.60
15.63
24
15.81
40.35
15.81
15.8140.58
16.03 15.76=0.57
15.63
madl
15.8810.54
16.41
16.2410.46
15.59
15.59
15.19
10
15.19
17
15.3710.00
15.59_0.63
15.59
15.59 1.000.00
croud-ar
16.7410.01
15.94
17.70:0..33
16:29
16.29 so
16.29
16.29
16.29
16.29
16.29 16.29
16.29
annot-nix
15.71+0.40
16.15+0.30
16.38+0.777
15.66+0.37
15.66-
15.66+0.a7
15.66+0.37
15.14
15.52+0.25
15.52-
0
15.14 0.
coin
15.6740.30
15.8040.48
16.15-40.25
15.07
15.67
15.54 +0.26
15.67
15.7240.50
15.55
15.67
15.67
spo-full
gt
15.2440.18
15.24+0.15
17.27:0.55
16.09
15.09-0.20
N/A
N/A
N/A
N/A
15.09+0.20
mv
15.6440.28
15.10
17.93
15.2440.30
15.24=0.30
N/A
N/A
N/A
N/A
N/A
15.24+0.30
da
15.231o.10
15.14
16.8010.50
15.23_o.10
15.23_0
15.23 10
15.23
15.231o.10
15.26_0.39
15.26 29
15.23
cl
14.89
14.87
15.241o.58
14.891.0.17
11.892011
15.33
15.33
15.3340.31
14.87 +0.37
15
15.23_n.19
11.87++a.31
trace
18.56+0.58
14.66
16.7440.38
14.73=0.33
14.78-0.35
14.7340.35
14.78+0.38
14.7840.35
14.9440.35
14.94-0.38
14.94+0.38
conal
15.60
14.79
20
16.8110.24
15.60
15.6020.45
14.85
15.6020.43
15.60
15.6020.43
14.85 39
14.85
15.70
14.79
44.79
15.30.10.23
15.63
15.63_022
15.6310.27
14.93
15.3310-47
15.33
15.63
15.63
unien-b
15.3310.5
14.8210.47
15.111o.33
15.28
15.28g0:42
14.744o.13
15.28
15.2810-42
15.28_0.42
15.17
15. 15.28
geo-x
15.30
14.71+0.40
15.34
15.30.0.30
15.30=u.06
15,1110.47
11.97
13.21
14.97:1..55
15.
15.15
11.97
goo-f
15.3240.84
14.86
40.40
15.26400.21
14.86
14.86
- 49
14.86
14.86
15.154
14.86
15.43
14.86 0
madl
15.53
15.09
15.71
15.53
15.53
15.07
15.53
15.0810.37
15.20
14.57
15.53
crowd-ar
16.60
14.68
t0.31
16.28
15.06
10.53
15.06
06
53
15.30
16.20
13.41go-43
15.41.0.43
15.30_o.mr
16.20g..00
anno-mix
14.73+0.25
14.25
+0.36
15.83+0.20
14.73+0.25
14.73+0.28
14.73+0.25
14.73+0.25
14.73+0.25
15.66+0.60
15.66-
14.73+0.26
coin
14.994..30
14.75
+0.42
15.35
14.99
11.99-u.00
14.99:10.00
11.99
15.2540.75
15.
15.12
15.12=n.33
11.99
46


Reviews:
[{"review_id": "1jjnSku9Ek", "reviewer": "Reviewer_gS1P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper attempts to solve the common issue in learning from crowds, namely how to tune models when the only available validation labels are noisy crowd annotations. They (1) propose a hyperparameter-selection (HPS) criteria that estimate validation risk directly from crowd labels and recommends a simple rank-ensemble over these criteria to make selection more robust; and (2) build a benchmark, including 13 LFC approaches across 5 real datasets with 7 noise variants for realistic, fair comparison.\n\nKey strengths: This work provides empirical evidence that the proposed criteria (especially the ensemble) reliably beat defaults in zero-one loss. The benchmarking setting is reasonable with the same candidate set, the same CV, and multiple seeds. It is a large, reproducible benchmark.\n\nKey weaknesses: these experimental results emphasize zero-one loss, a notable limitation, but the authors have already discussed it in the manuscript. As a benchmark, the sensitivity analysis in this benchmark is also insufficient; for example, they fix a Sobol, $|\\Lambda|=51$ candidate set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly yes for the paper’s central claims, the evidence is careful, repeated, and clearly presented.", "audience_yn": "Yes", "audience_explanation": "This paper can attract the readers who care about weak supervision, crowdsourcing, practical HPO/AutoML, and reproducible benchmarking.", "requested_changes": "1. This paper attempts to compress baseline methods used $C \\times C$ confusion matrices to scalar throws, this disadvantages some methods whose core strength is modeling per-class worker biases (e.g., DS and GLAD style methods).\n\n2. This paper freezes strong pretrained backbones (e.g., DINOv2 and MPNet) and put a small MLP head on top for fair comparison. The author can report an end-to-end fine-tuning setting with light fine-tuning or at least vary the head size to show conclusions are stable.\n\n3. The statistical significance should be reported, add paired tests.", "broader_impact_concerns": "I have no additional ethical concerns beyond what the authors already discuss.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Kh0LG4u1o", "reviewer": "Reviewer_7Mew", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the critical problem of hyperparameter optimization (HPO) in the Learning from Crowds (LFC) literature, where methods are often evaluated unfairly using hyperparameters tuned on clean labels or unrealistically with fixed default settings. The central contribution is `crowd-hpo`, a novel framework for selecting hyperparameters using only the available noisy, crowd-labeled validation data. This removes the need for either suboptimal defaults or an idealized clean validation set. The authors systematically propose several selection criteria based on different risk estimation strategies and worker performance assumptions, culminating in a robust ensemble method (ENS).\n\nA major strength is its comprehensive and large-scale experimental study. It provides compelling evidence for the framework's effectiveness while also yielding critical insights, such as how the choice of selection criterion can drastically alter the final performance and ranking of LFC methods across different datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is structured as a comprehensive investigative study, and its conclusions are directly grounded in the results of its extensive and well-designed experiments.\n\n- **The claim that existing HPO practices are suboptimal and lead to unfair comparisons** is clearly supported. Figure 1 provides a stark visual demonstration of how performance and rankings change between default (`DEF`) and optimized (`TRUE`) hyperparameters. Table 4 further quantifies this by showing that both `DEF` and `DEF-DATA` baselines consistently rank the lowest and offer minimal performance gains, validating the paper's central motivation.\n\n-  **The claim that the proposed `crowd-hpo` framework effectively addresses hyperparameter selection using only noisy data** is convincingly demonstrated. Table 4 shows that all proposed criteria achieve significant zero-one loss reductions compared to the `DEF` baseline. Crucially, the proposed ensemble criterion (ENS) stands out as the top-performing realistic method, significantly narrowing the performance gap to the TRUE upper-bound baseline.\n\n-  **The claim that the choice of selection criterion is itself critical and can alter the ranking of LFC methods** is also well-supported. The low Kendall's rank correlation coefficients presented in Figure 7 provide direct, quantitative evidence that rankings produced by naive baselines (`DEF`, `DEF-DATA`) differ significantly from those produced by more robust criteria (`ENS`, `TRUE`). Figure 6 further illustrates this by showing how the relative performance gains between different LFC approaches shift depending on the HPS criterion used and the datasets.", "audience_yn": "Yes", "audience_explanation": "Its systematic review of literature from top ML venues (Table 1) reveals that prior studies are often less experimentally thorough and commonly rely on flawed protocols, such as using unrealistic clean validation sets or datasets with simulated, non-crowdsourced labels. By demonstrating that these common practices can alter method rankings and lead to unfair conclusions, the paper provides an indispensable guide. Its methodology is essential for any researcher aiming to develop or benchmark LFC algorithms with real-world relevance and scientific rigor.", "requested_changes": "The authors highlight the considerable performance dispersion of HPS criteria across datasets, as in the end of Section 4.2. Since Table 2 shows that these datasets also feature a wide range of noise levels, the paper would be even more insightful with an analysis stratified by noise. For instance, presenting the Absolute Zero-one Loss Reductions or Rank Correlations for low-noise versus high-noise variant groups could offer a clearer understanding of how noise severity impacts the effectiveness of different selection criteria. This analysis would likely further strengthen the necessity of the robust ensemble (ENS) strategy.", "broader_impact_concerns": "The paper includes a \"Broader Impact Statement\" section. The authors have acknowledged the primary societal and ethical implications of their work, including the practical limitations of the framework and the crucial importance of fair labor conditions for crowdworkers.\n\nThe provided statement is sufficient and well-addressed. I have not identified any further broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MzT10Rl841", "reviewer": "Reviewer_q474", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Hyper-parameter optimization is a central challenge in label noise learning due to the lack of clean validation data subsets. However, this issue has been overlooked by the research community in noisy label learning and hence, not been properly studied. The paper proposes *crowd-hpo* as a framework to evaluate different label noise learning methods, while selecting hyper-parameters following certain criteria on noisy validation sets. Empirical demonstrations have shown that selecting hyper-parameters based on these criteria improves the performnace of many state-of-the-art methods in label noise learning.", "claims_and_evidence_yn": "No", "claims_explanation": "Despite the significance in the introduction and related work about hyper-parameter optimization, the methodology presented in section 3 is too vague and unclear in terms of addressing the challenge. In addition, as the paper presents several formulation, the notations are abused, causing more confusion than clarification, and hence, significantly reduces the readability of the paper. After going through the whole section 3, I cannot understand the main point presented there personally. The main idea presented in Figure 4 is unclear, especially for each component.\n- It is hard to understand the meaning of Eqs. (11) and (13). To what I can understand, Eq. (11) calculate the risk on the validation set given some label aggregation and weighting mechanism. However, it is unclear what exactly those aggregation and weighting mechanisms are. Is this only an illustration that the authors want to use to demonstrate the risk calculated on the validation set? If this is the case, the introduction of these two things cause confusion.\n- Eq. (13) is to present the risk of the model $g$ on the validation set. It is unclear why this risk is taken into account because according to Eq. (4), $g$ is used to model the prediction of each annotator, while the target in noisy label learning is to have the highest performance when predicting ground truth labels. Do the authors mean to enforce consistency to match with the annotations made by humans? This case is only helpful if $g$ and $f$ are independent, but it has not been specified in the paper.\n- The presentation about *Crowdworker Performance* is unclear. Eqs. (15) and (16) are fine, but the justification for Eq. (17) is vague. Could the authors elaborate more why predicting $\\Pr(y | x)$ will result in *double-dipping*. To what I understand, it happens when we use the same samples for training and validation. However, here, the authors already slit the dataset into two disjoint subsets.\n    - The proof for Proposition 1 should be revised significantly. In its current form, it is unclear why one can go from $l\\_{max}$ and $l\\_{nk}$ to $\\Pr(y = e\\_{c} | x)$. It would be helpful if the authors could provide detailed explanation of each step.\n    - In Eq. (18), it is unclear why there is a division by $C - 1$. Due to that division, it is no long a Bernoulli distribution.\n    - Similar to the above two points with Eq. (19).\n- Nevertheless, it is unclear about the estimation of posterior of the ground truth label given the annotations in (15) and (19).\n- For *Label Weights*, it is unclear why there are only two ways to weight samples: uniform and confidence. Could the authors elaborate further more about this?\n- For the *Hyper-parameter Selection Criteria*, could the authors elaborate further on how to select hyper-parameter based on such criteria? Does it reply on gradient based methods or the the classic Borda count as mentioned in the paper?\n\nDue to these concerns, I have not proceeded further but stopped right before section 4. I believe that it should be clear in the methodology before moving to the empirical demonstration.", "audience_yn": "Yes", "audience_explanation": "The most valuable contribution of the paper is to study the hyper-parameter optimization in label noise learning. In standard machine learning, such a hyper-parameter optimization relies on a validation set consisting of samples with their corresponding ground truth labels. However, the setting in noisy label learning often assumes the absence of such a validation set. Instead, only noisy label validation set is available. This poses a central challenge in label noise learning as most current methods often fine-tune their hyper-parameters directly on testing sets, resulting in bias reported results. Hence, the motivation of this study is worthy and significant.", "requested_changes": "It seems that the paper contains too much of information without giving a big picture to explain what it is going to do. I would like to request the authors to elaborate, add discussion and explanation for each step in section 3, so that reader can understand why we need to go through such formulations and why such formulations make sense.\n\n---\n### **Minors**\n- The abbreviation \"HP\" is often used in the paper, but it has never been defined.\n- Figure 2 could be improved further to increase the readability of the paper. In its current form, it is too thin but long. In addition, it is helpful if the model parameter, such as $\\theta$, is included into the graphical model to describe the data generation process more clearly.\n- In section 3.1 - paragraph Data Generation Process, the terminology \"multiset\" is used without being properly defined. A set is defined to contain multiple elements, but it is unclear why in such a context, it is called \"multiset\".\n- The paper considers the case of missing labels at the early part of section 3, but later on, discard that case. Hence, I suggest to specify that explicitly at the beginning, saying that only the case with annotations would be considered. That reduces the confusion because of the graphical model presented in Figure 2. If missing labels are considered, then the random variable $z$ is observed for some samples, while being hidden for others.\n- The definition of 0-1 loss in Eq. (2) is inaccurate. As $\\mathbf{e}\\_{c}$ and $\\mathbf{y}$ are defined as one-hot vectors, their dot product $\\operatorname*{argmax}\\_{\\mathbf{e}\\_{c}} \\mathbf{e}\\_{c}^{\\top} \\mathbf{y}$ is eventually $\\mathbf{y}$. Why would one go through a complicated definition without writing that: $L\\_{0-1}(\\mathbf{y}, \\hat{\\mathbf{y}}) = 1 - \\mathbf{y}^{\\top} \\hat{\\mathbf{y}}$\n- Although the authors mention the abuse of notation in footnote 2 on page 5, using the same notation $\\theta$ for the functions f, g and h causes confusion. Please update the manuscript with a clearer notation.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)