Realistic Evaluation of Deep Partial-Label Learning Algorithms
Authors: Wei Wang
Dong-Dong Wu
Jindong Wang
Gang Niu
Min-Ling Zhang
Masashi Sugiyama
Venue: ICLR 2025 Spotlight (2025)
Keywords: Partial-label learning
weakly supervised learning
benchmark.
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: The first partial-label learning benchmark with a new dataset of human-annotated partial labels.

Abstract: Partial-label learning (PLL) is a weakly supervised learning problem in which
each example is associated with multiple candidate labels and only one is the
true label. In recent years, many deep PLL algorithms have been developed to
improve model performance. However, we find that some early developed
algorithms are often underestimated and can outperform many later algorithms
with complicated designs. In this paper, we delve into the empirical
perspective of PLL and identify several critical but previously overlooked
issues. First, model selection for PLL is non-trivial, but has never been
systematically studied. Second, the experimental settings are highly
inconsistent, making it difficult to evaluate the effectiveness of the
algorithms. Third, there is a lack of real-world image datasets that can be
compatible with modern network architectures. Based on these findings, we
propose PLENCH, the first Partial-Label learning bENCHmark to systematically
compare state-of-the-art deep PLL algorithms. We investigate the model
selection problem for PLL for the first time, and propose novel model selection
criteria with theoretical guarantees. We also create Partial-Label CIFAR-10
(PLCIFAR10), an image dataset of human-annotated partial labels collected from
Amazon Mechanical Turk, to provide a testbed for evaluating the performance of
PLL algorithms in more realistic scenarios. Researchers can quickly and
conveniently perform a comprehensive and fair evaluation and verify the
effectiveness of newly developed algorithms based on PLENCH. We hope that
PLENCH will facilitate standardized, fair, and practical evaluation of PLL
algorithms in the future.

Published as a conference paper at ICLR 2025
REALISTIC EVALUATION OF DEEP PARTIAL-LABEL LEARNING ALGORITHMS
Wei Wang 1.2
Dong-Dong Wu Jindong Wang"
Gang Niu 2,3
Min-Ling Zhang Masashi Sugiyama
2,1
The University of Tokyo, Chiba, Japan 3 Southeast University, Nanjing, China wangwêg ecc. u-tokyo. ac. ip jwang8 O@wm.edu zhangml @seu. cn sugi@k.. bugi@k.u-tokkee w-t.kyo..ccc....
RIKEN,, Tokyo, Japan
William & Mary, Williamsburg, VA, USA
(dongdongwu1230, gang. mml @gmail.com com
ABSTRACT
Partial-label learning (PLL) is a weakly supervised learning problem in which each example is associated with multiple candidate labels and only one is the true label. In recent years, many deep PLL algorithms have been developed improve model performance. However, we find that some early developed algorithms are often underestimated and can outperform many later algorithms with complicated designs. In this paper, we delve into the empirical perspective of PL. and identify several critical but previously overlooked issues. First, model selection for PI.I. is non-trivial, but has never been systematically studied. Second, the experimental settings are highly inconsistent, making difficult to evaluate the effectiveness of the algorithms. Third, there is a lack of real-world image datasets that can be compatible with modern network architectures. Based on these findings, we propose PLENCH, the first Partial-Label learning bENCHmark to systematically compare state-of-the-art deep PLL algorithms. We investigate the model selection problem for PLL for the first time, and propose novel model selection criteria with theoretical guarantees. We also create Partial-Label CIFAR-10 (PLCIFAR 10), an image dataset of aumaa--nnotated partial labels collected from Amazon Mechan- ical Turk, to provide a testbed for evaluating the performance of PLL algorithms in more realistic scenarios. Researchers can quickly and conveniently perform a comprehensive and fair evaluation and verify the effectiveness of newly developed algorithms based on PLENCI. We hope that PLENCH will facilitate standardized, fair, and practical evaluation of PLL algorithms in the future
INTRODUCTION
Partial-label learning (PLL) is a weakly supervised learning problem that has attracted much atten- tion recently (Sugiyama et al. 2022; Wang et al., 2022a Tian al. (023). In PLL, each training example associated with multiple candidate labels (Jin & Ghahramani] 2002] Cour et al. 201
The true label for each example is hidden in the set of candidate labels, but not accessible to the learning algorithm. PLL has been successfully applied to computer vision (Liu Dietterich
2012]
Zeng et al. 2013 Chen et al. 2018 Gong et al. 2018 Tang al. 2023 Wang al. (024c). natural language processing (Garrette & Baldridge 2013 Zhou et al. 2018 Ren et al. 2016a b), web min-
ing (Luo & Orabona 2010), ecoinformatics (Briggs al. 2012 Wang al. 2019 Li al. 2021
Lyu et al. 2022), etc.
Among various strategies to address this problem, deep learning-based PL.I algorithms have demon- strated satisfactory generalization performance due to the strong representation learning capabilities of deep neural networks (Lv et al. 2020 Wang al. 2022b). Despite the abundance of algorithms in this area, we find that there are several Fundamental and critical issues that have received less at- tention in the PLL literature, including neglected model selection issues, inconsistent experimental settings, and lack of real-world image datasets, which will be discussed point by point.
'The code implementation of PLENCH is available at https: thw.gghthuuu... com wwtphhwwwwwwwwwwwwww...eecc
PLENCH The PICCIFAR10 dataset is available aathttst/////t chu.. com/ www.///wwawwitteeeee.eaa tsel

---

Published as a conference paper at ICLR 2025
Paper
Paper
Superviadd
Val calinn Trakkigg
Superviaed
Valdattonn Tran
POP
SCL-ML
Accurnsy
Accuracy
Accuncy
Accuracy
CAVL
PRODEN
72
CAV
CAN
(a) Aggregate
(b) Vaguest
(c) Soccer Player
(d) Yahoo! News
The two left panels show the differences in using an ordinary-label dataset for valida- tion (lighter colors) and training (darker colors) for given algorithm. For validation (lighter colors), we searched for the best hyperparameter configurations with the validation set for a given algorithm. For training (darker colors). we considered the validation set as partial-label examples with a single partial label and added them to the training set for training, using default hyperparameters without tuning. For fair comparisons, we trained all models with the same number of iterations. The two right panels show the classification accuracies of some PLL algorithms on Soccer Player and Yahoo! News from papers A (Zhang et al. 2022) and B (Xu et al. 2023b), respectively.
Figure 1:
Neglected model selection issues. Most PLL algorithms select their hyperparameters by using
clean ordinary-label validation set (Qiao et al.] 2023a, Xu et al., 2023a b). However, the original definition of PLL does not allow the existence of an ordinary-label dataset (Jin & Ghahramani] [2002] Cour et al., 2011 Zhang et al., 2017), indicating mismatch between the problem definitions and experimental settings in the literature. This problem can even lead to unfair comparisons if some al- gorithms follow the classical protocol of PLL to prohibit the use of ordinary-labod data, while some algorithms do not. Moreover, if we have a clean ordinary-label dataset, why not use it for training? In many cases, the use of clean labels is more valuable for weakly supervised learning than for ordi- nary supervised learning (Hendrycks et al.) 20188] Yu et al. 2003).. A pilot experiment was conducted compare different approaches to using validation data with ordinary labels. Figures I(a) and [ illustrate the results of the experiment on two versions of our collected PLCIFARII 10 dataset. We can observe that for many PLL algorithms without much need to tune hyperparameters, it is more
beneficial to use validation data with ordinary labels for training. Therefore, it is imperative to stan- dardize the use of validation data and model selection criteria to ensure the integrity and practicality of empirical studies.
Inconsistent experimental settings. We have found that the experimental settings used in different papers are often quite different, creating a dilemma when comparing the performance of different algorithms. We illustrate this point with an example Figures [(c] and[TT) We reported experi- mental results of several PLL algorithms on Soccer Player and Yahoo! News from papers A (Zhang ct al. 2022) and B (Xu et al. 20236). respectively. The data points for the same algorithm are connected by line, and the number of intersections between different lines indicates the times of inconsistency in the performance ranking. In particular, the algorithm implementations, network architectures, and datasets are identical. However, the relative ranking of the classification perfor- mance differed significantly between the different papers. It is hypothesized that the discrepancy is due to subtle variations in experimental settings, such as data partitioning and preprocessing, as well as hyperparameter configurations. Such an obstacle can hinder objective comparisons of different algorithms, making it difficult to determine the effectiveness of a developed technique.
Lack of real-world image datasets. Existing PLL works have mainly conducted experimental results on real-worle tabular datasets or synthetic image datasets to demonstrate the effectiveness of the proposed methodology (Lyu et al., [021]] Wang et (022bb). However, on the one hand, tabular datasets may not be compatible with modern network architectures, such as convolutional neural networks (Oquab et al. 2015 He et al. 2016). On the other hand, synthetic image datasets are generated by a numan--adde generation process (see Section 2 which may not be consistent with complex annotation mechanisms in real-world applications (Jiag al.) 2020 Wei et al.) (2022).
This may lead to potential concerns about whether an algorithm that works well on synthetic image datasets will still work well on real-world complex image datasets (Xu al. 2021
2

---

Published as a conference paper at ICLR 2025
Contributions. To this end, propose PLENCH, the first Partial-Label learning bENCHmark to standardize the experiments PLL. The main contributions are as follows:
We systematically investigate the model selection problem in PLL for the first time and propose new model selection criteria that are both theoretically and empirically validated.
We create PLCIFARIO 10, a new PLL benchmark dataset with human-annotan annotated partial labels. PL- CIFARIO provides an effective testbed for evaluating the performance of PLL algorithms in more realistic scenarios.
We present the first PLL benchmark that includes twenty-sevy algorithms and eleven real-world datasets, to systematically compare state-of-the-art deep PLL algorithms.
Takeaways. Based on our research, we suggest the following key takeaways:
Many concise algorithms can outperform or be comparable to complicated algorithms with strong regularization techniques and higher computational requirements.
There is no algorithm that can outperform all other algorithms in all cases, suggesting that the
algorithmic design should be tailored for different cases.
Model selection is non-trivial for PLL, and should be specified when proposing a PLL algorithm or comparing different algorithms.
BACKGROUND
Problem setting. Let X IRd denote the d-dimensional feature space and 2 .
denote the label space with class labels. Let (x, S) denote a partial-label example where X' feature vector and = is candidate label set associated with æ. The basic assumption of PLL is that the ground-truth label of æ is concealed within its candidate label set S, i.e., S. The task of PLL to learn a multi-class classifier f : X [0, 1]a from a partial-label training set D = (-. ) where f(x) = [fi(z), (e(xz).2ee fq (x)] is the estimated class class-posterios posterior probability vector for x.
an fi-1
Data generation process. There are mainly two ways to generate synthetic instance-independer partial labels, i.c.. the uniform sampling strategy (USS) (Feng al. (2220b a) and the flipping prob- ability strategy (FPS) (Zhang et al. 2002). Feng et al. (2020b) proposed the USS that the candidate label containing the ground-trute -ruth label is sampled from uniform distribution, i.e.,
1 (( 20-1
TI (y E S)
(1)
where [(()) returns predicate ı holds and returns otherwise. Wen et (2021) proposed the FPS that each false positive label is independently drawn into the candidate label set with a flipping probability p(2 Sly). Then, the class-conditional probability distribution of the candidate label set could be formulated
p(S|x,
II m.S.myy
p(m E Sly) I (1-p(t ply))
(2)
Although FPS seems more practical, the flipping probability is unknown. Many papers assume that the flipping probability is a constant value for different labels, which is often not true in real-workd scenarios (Wei et al.] 2022). Xu et al. (202T) proposed instance-dependent PL.., which assumes that the generation of partial labels also depends on the feature. However, the datasets are still synthesized by using the model outputs of an auxiliary network. In Section we present a more partial labels.
realistic benchmark dataset with
REALISTIC MODEL SELECTION CRITERIA FOR PLL
Model selection critical component in machine learning problems. However, it has received little attention in the context of PLL. Most PLL work assumes the existence of a clean ordinary- label dataset, which may be neither consistent with the original definition of PLL nor practical, as discussed in Section [ To promote fair and practical evaluation of PLL algorithms, it is important to systematically examine the model selection procedure for PLL. We follow the original definition of PL. to have only single partial-label training set (Cour al. 2011
Zhang al. 2017). Then,
3

---

Published as a conference paper at ICLR 2025
following widely used validation procedure in machine learning Lopez-Paz 2021). we divide a partial-label validation set V traiing set for model selection. Next, we introduce each model selection criterion in turn.
(Raschka 2018 Gulrajani &
{ (at S. SYal) 3i-1 from the
3.1 COVERING RATE
Definition (Covering Rate (CR)), The Covering Rate of a multi-class classifier f on the partial- label validation set D is defined as:
CR(F)
Σ I
arg max fz(a) )
(3)
CR indicates the fraction of validation data whose predicted label is included in its candidate label set. It is a natural metric in PLL, but its effectiveness may depend on the size of the candidate label sets. When the number of partial labels of each example is equal 1, i.e., |S| 1, PLL is reduced to ordinary supervised learning and CR is reduced to the validation accuracy. However, as [S] increases. more false positive labels are included in the candidate label In the most extreme case, when |.S] information. Before analyzing the gap between CR and the validation accuracy, we introduce the following definition.
PLL is reduced to unsupervised learning and CR does not convey effective
Definition (Ambiguity Degree), The Ambiguity Degree r is defined as
sup
p(ū E S),
where p(z, y) is the joint distribution over x and
y < 1, the small ambiguity degree is satisfied and the ERM learnailiity for PLL is guaran- teed (Cour et al. 201T Liu & Dietterich 2014). We also define the expected accuracy
ACC = pE (arg max fi (x) = y).
(5)
Then, have the following proposition.
Proposition 1. Suppose that there is constant E (0, 1) such that the expected accuracy of classifier f satisfies ACC (f) 3 e. Then, we have E [CR(f) - ACC (f) (1 (1-6)7:
The proof is in Appendix A The gap between CR and the accuracy is affected by both the accuracy and the ambiguity degree. If the classifier is more accurate and the partial-label dataset is less ambiguous, the gap between the CR metric and the accuracy will be smaller. Furthermore, we show that the minimizers of both are the same under certain conditions.
Theorem 1. Suppose that the partial labels are generated by following the USS or the FPS with constant flipping probability. Then, for any two classifiers fi and fz that satisfy E [CR(fi)) < [C)( we have ACC (fi) < ACC (f2).
The proof in Appendix A Theorem D shows that when partial labels are generated by using the USS or the FPS with a constant flipping probability, the classifier that minimizes the expectation of CR will also minimize the expected accuracy. Therefore, the CR metric will serve consis- tent model selection criterion for PLL under certain data distribution assumptions. However, this conclusion may not hold when partial labels are not generated by either strategy.
3.2 APPROXIMATED ACCURACY
Next, introduce the definition of the Approximated Accuracy metric. Definition 3 (Approximated Accuracy (AA)). The Approximated Accuracy of multi-class classi- fier of on the partial-labe label validation set Val is defined as:
kessy (zo arg max Si(xe [((=: = =5)
AA(F)
Σ JeSVal
(6)
Val
4

---

Published as a conference paper at ICLR 2025
Then, have the following theorem.
Theorem 2. Suppose there is function C P X 2 - R. such that the condition p(S) x. y) Cxx, S)I (ye S) holds for partial-labe data. Suppose further that the multi-class classifier f( is consistent with ply)ar) a.). Then, under mild conditions (A( is stalistiallly consistent with the expected accuracy, ie.. [[AA(f)]= ACC
The proof can be found in Appendix A The introduction of AA is inspired by data-generation based strategies for PLL (Wu et al., 2023). Theorem illustrates that AA can be a consistent metric w.r.t. the expected classification accuracy on test data under certain assumptions. In particular, the data distribution assumption can hold for wide range of types of partial labels (Liu & Dietterich, 2012 Wu et al. 2023). However, there are two factors that may affect its effectiveness. First, may not be suitable for certain algorithms (Wen et al. (021) where the loss function is not strictly proper (Gneiting & Raftery 2007 Charoenphakdee et al. 202T) and its modeling output is not calibrated to the posterior probabilities. Second, it requires that the modeling output to be accurate, which may not be satisfied in the early stages of training.
3.3 ORACLE ACCURACY
Finally, we present the definition of the Oracle Accuracy metric.
Definition (Oracle Accuracy (OA)). The Oracle Accuracy of multi-class classifier on a partial- label validation set Val is defined as
Val
OA(F)
Σ
arg max fi(2)
y)
n Val
where Vi Val is the underlying true label of Vo!al.
The OA metric is natural if we have access to true labels in supervised learning. However, such a condition is not realistic in real problems of PLL. In fact, the availability of true labels contradicts the original motivation of PLL, which is to reduce labeling costs at the expense of true label ignorance. Inspired by Gulrajani Lopez-Paz (20TT) we restrict the use of true labels by allowing only one query (the last checkpoint) for each hyperparameter configuration. This means that we do not allow early stopping when using the OA metric. We try to compensate for the unrealistic access to true labels by restricting the model selection space. We also include the results of OA with early stopping (ES), which can be considered as an upper bound of the model selection performance, for reference. Actually, OA with ES may be the most common model selection criterion for PLL papers.
4 PLCIFARIO 10: A DATASET WITH HUMAN-ANNOTATER PARTIAL LABELS
In this section, we present novel image dataset with for the lack of real-world image datasets for PLL. We used Amazon Mechanical Turk (MTurk) as crowdsourced annotation platform. CIFAR-10 (Krizhevsky & Hinton 2009) was chosen the base dataset because it has been widely used in the PLL literature. In addition, the difficulty and workload are relatively moderate, which can facilitate our experimental design and analysis. We posted the annotation tasks of the images from the training set of CIFAR-10 10 as Human Intelligence Tasks (HITs), and the crowdsourced workers received salaries by completing the HITs. We created 5000 HITs, each containing 10 random images. Workers were allowed to select multiple candidate labels, which could include the true label for an image. We then asked 10 different workers to perform the same HIT at the same time. As result, each image could be annotated with multiple candidate label sets, and each candidate label set could contain multiple labels. More details on the data collection can be found in Appendix B
partial labels to compensate
In summary, we collected 502, 190 candidate label sets with total of 712,109 partial labels. Fig- ure 2(a) shows the distribution of our collected partial labels. The imbalanced distribution of partial labels may not match the commonly used USS or FPS with a constant probability. Furthermore, find that the noise rate, i.e., the proportion of training examples whose candidate label sets do not contain the true label, is high with few annotators, as shown in Figure 2bb] As the number of annotators increases, the aggregation of partial labels may become less noisy. We consider two
5

---

Published as a conference paper at ICLR 2025
0.20
1.6
Notte
0.05
0.00
Class Index
$
AAnotators
(a) Class Distribution
(b) Noise Rate Distribution
(c) Aggregate
(d) Vaguest
Figure 2: (a). The distribution of the collected partial labels of PLCIFARIO. (b) The noise rate with the increasing number of annotators. (c) The flipping probability matrix computed PICCIFARIO- Aggregate. (d) The flipping probability matrix computed on PLCIFARIO Vaguest.
versions of PLCIFARIO for experiments. The first is PLCIFARIO-Aggregate which assigns the aggregation of all partial labels from all annotators to each example. The second is PLCIFARIO- Vaguest, which assigns to each example the largest candidate label set from the annotators. Since PLCIFARIO Vaguest has a high noise rate, actually serves as dataset for noisy PLL (Xu al.)
2023a Lv et al. 2024a), which is more practical setting since it is hard to ensure that annotated partial labels always include the true label in real-world applications.
((c) shdw the flipping probability matrices p(j Sly i) computed based on the two versions of PLCIFARIO 10. We can see that the flipping probabilities are not equal and the diagonals are not all I, which is more challenging and practical than current synthetic datasets generated with the USS or the FPS.
5
SETUP OF PLENCH
5.1
BENCHMARK ALGORITHMS
We have divided the benchmark algorithms into four main groups. The detailed descriptions and hyperparameter settings can be found in Appendix
Vanilla deep PLL algorithms. We considered deep PLL algorithms that used simple loss func- tions or disambiguation strategies without strong regularization techniques as vanilla deep PI.I. al- gorithms. Identification-basse strategies included PRODEN (Lv et al. 2020), CAVL (Zhang al.) 2022], and POP (Xu et al.) 2023b). Note that RC (Feng et al. 2020b) is in the same form as PRO- DEN, we only included PRODEN here avoid repetitive comparisons. The averaging-base strategies included ABS-MAE (Lv et al.; 2024a) and ABS-GCE (Lv et al.) (2024a), which showed favorable performance among the losses the family. The data-generation-basee strategies cluded EXP (Feng et al. 2020a), MCL-GCE (engg et al. (020a)). MCL-MSE (Feng et al. 2020a).
CC (Feng et al. 2020b), LWS (Wen et al., 2021) and (Qiao et al. 2023a). Also, LOG (Feng
ct al. 2020a) is analogous to CC, so we only included CC here.
learning (CLL) is special case of PLL with only one label excluded from each candidate label set of training examples, i.c., |S| = q (Wang al. 2024b). If we consider each label outside the candidate label set as a complementary label, it possible to apply CLL algorithms to solve PLL problems as well (Feng et al., (2020a). Therefore, we
Vanilla deep CLL algorithms. Complementary-Bie
included several vanilla CLL algorithms with simple loss functions in PLENCH. The vanilla CLL
algorithms included PC (Ishida et al., 2017). Forward (Yu et al.) 2018). NN (Ishida et al. 2019).
GA (Ishida et al.] 2019), SCL-EXP (Chou et al. 2020), SCL-NL (Cou et al. 2020). L-W (Gao &
Zhang 2021) and OP-W (Liu et al. 2023).
Holistic deep PLL algorithms. We considered PLL algorithms that employ strong representation
learning or regularization techniques to be holisitic PLL algorithms. We used five algorithms, includ-
ing VALEN (Xu et al., (002). PiCO (Wang et al. 2022b), ABLE (Xia et al., 2022). CRDPLL (Wul and DIRK (Wu et al. 2202)..
al. 2022
Deep noisy PLL algorithms. We also included three noisy PLL algorithms that explicitly con- sidered tailored strategies to handle the cases where the true label might be outside the candidate

---

Published as a conference paper at ICLR 2025
label set. The deep noisy PLL algorithms included FREDIS QQiao al. 2023b ALIM (Xu et al. 2023a), and PiCO+ (Wang et al. 2024a).
5.2 BENCHMARK DATASETS
To comprehensively evaluate the model performance of all algorithms, we included eleven real- world PLL benchmark datasets, including nine widely used tabular datasets and two image datasets that we collected, The tabular datasets included Lost (Cour et al., 2011), Soccer Player (Zeng al.) 2013), and Yahoo! News (Guillaumin et al. 201O) for the automatic face naming task, MSRCv2 (L.u & Dietterich 2012) for the object classification task, Mirflickr (Huiskes & Lew 2008) for the web image classiication task, Birdsong (Briggs et al. (2022 for the bird song classification task, Mala-
gasy (Garrette & Baldridge| 2013), Italian (Johan et al. (209). and English (Zhou et al. 2018) for
the POS tagging task. A detailed description of the datasets can be found in Appendix
5.3 EXPERIMENTAL SETTINGS
In this paper, we mainly investigate the hyperparameter tuning problem in the context of model se- lection. For tabular datasets, we first divided a test set from the entire dataset. Since the datasets were not explicitly divided into training and validation parts, manually divided them into partial-label training set TV and partial-label validation set Val, Then, we trained a model with DJ and eval- uated its validation performance D with the model selection criteria proposed in Section 3 as well as its test performance on the test set Do with ordinary labels. We then selected the checkpoint with the best validation performance and returned the corresponding test accuracy as the final result. We randomly selected set of hyperparameter configurations from a given pool for a given data split and recorded the mean accuracy as well as the standard deviations for the best performance with different dataset splits, We used ResNet (He et al. 2016) and DenseNet (Huang et al., (017) for image datasets and multilayer perceptron (MLP) with a hidden layer width of 500 equipped with the ReLU (Nair & Hinton 2010) activation function for tabular datasets.
6 EXPERIMENTS
6.1 EXPERIMENTAL RESULTS ON TABULAR DATASETS
Figure 3 shows the box plots of vanilla deep PLL and CLL algorithms on real tabular datasets, not satisfactory. Moreover, most holistic deep PL.L algorithms are based on different data augmentation strategies and cannot be applied here. Detailed experimental results can be found in Appendix FF] We can observe that AA does not work well in some cases where the classifier is not accurate enough. The CR metric is sometimes more effective and can serve as an alternative model selection criterion. Therefore, we may need to prepare different model selection criteria to effectively determine the hyperparameters in different cases. The classification performance of PRODEN is very strong in most cases, which clearly confirms its effectiveness. However, it still cannot outperform all other algorithms all cases. In addition, some early vanilla CLL algorithms, such as Forward, SCL- EXP, and OP-W, can also achieve good performance and deserve to be considered when conducting comparative experiments for PLL.
where NN and GA not included in the figure because their performance
6.2 EXPERIMENTAL RESULTS ON IMAGE DATASETS
Tables I and report the experimental results on A.CIFARIO-Aggrenn and PLCIFARIO- Vaguest with ResNet. Tables 3 and report the experimental results on ELCIFARII-Aggree
and PLCIFARIO- Vaguest with DenseNet. We do not include the results of VALEN because it requires a very large computational and memory budget. The following conclusions can be drawn. First, com- pared to the results with OA and ES, CR is very effective in most cases. The performance decreases in some cases. We speculate that this is because the modeling results may not be reliable, and the results of AA may deviate from the true accuracy. Second, PRODEN and its variants are still strong in performance, but there is algorithm that can outperform all other algorithms in all cases. Third, noisy partial labels have a significant impact on the model performance of most algorithms. Therefore, it is still promising develop effective noisy PL. algorithms in more practical scenarios.
AA
7

---

Published as a conference paper at ICLR 2025
Accuracy
70
8
60
8
56
PRODEN
CAVE
PGP
ABS-MAE
ABS GCE
EXP
MCL-GGE
MCL. 0C
LWS
IDGP
PC
Forward
SCL-EXP
SCL-NL
L-W
OP-W
VALEN
Accuracy
80
80
40
PRODEN
CAVL
POP
ABS-MAE
ABS CCE
EXP
MCL-GCE
MCL MCL-MSE MSE
CC
LWS
IDGP
PC
Foreard
SCI-EXP
SCL-N
L-W
OPW
VALEN
Accuracy
8
865
8
60
8
55
PRODEN
CAVL
POP
ABS ABS-MAE
ABS GDE
Exp
MCL-GCE
MCL MSE
LWS
IDGP
Foneard
SCL-EXP
SCLANL
L-W
OP-W
VALEN
Accuracy
20
65
8
PRODEN
CAV.
POP
ABS-MAE
ABS-GDE
ExP
MCL-GCE
MCL-MSE
6C
LWS
IDGP
Foread
SCL-EXP
SCL-NIL
L-W
MP-
VALEN
ABS
Figure 3: Experimental results of different algorithms on tabular datasets. The top, middle, and bottom figures correspond box plots experimental results using CR, AA, OA, and OA with ES for hyperparameter tuning, respectively. The colors of the bars indicate the mean accuracy.
Table 1:
Classification accuracy (tea of each algorithm on PLCIFARIO-Aggreee with ResNet, where the best performance w.r.t. each metrie is shown bold.
Algorithm
Venue
w/ CR
w/ AA
w/OA
w/ OA & ES
PRODEN
ICML 2020
Lv al. 2020
85.95=0.08
85.77+0.11
85.91=0.188
86.03++0.13
CAVL
ICLR 2022
Zhang ct al.
2022
68.09
65.92+
68.23+0.17
69.12+0.47
POP
ICML 2023
Xu al. 20236
84.94-0.08
85.2741.34
85.04:1.27
85.53-10.24
ABS-MAE
TPAMI 2024
LLV a.. 2024a
15.13=1.113
45.51
54.31
1..3
ABS-GCE
TPAMI 2024
eL al. 2024a1
7..21
1..2
74.68
76.00
EXP
ICML 2020
(Feng el al. 2020a
10.00:1.000
10.00
10.00
10.00
MCL-GCE
ICML 2020
Feng èt al. 2021ka
60.57+0.60
18.26
32.31
62.26+00.72
MCL-MMR
ICML. 2020
Feng el al. 2021ka
61.03+1.25
10.00
55.09
63.17+0..9
NeurlPS (U20(Fengg ot al.
2020b
80.66+0.23
80.774:046
81.4410.29
8188700111
ICML 2021
(Wen al. 2021
55.31=0.32
26.4445..55
55.5010.58
56.1740.51
IDGP
2023
Quao al. 2023a
82.8010.30
80.72.10.66
83.43.0.117
83.65.10.44
NeurlPS 2017
(Ishuda al 2017
71.45=0.71
70.34:10.39
71.06
72.24-10.75
Forward
RCCV 2018
Yu al. 2013
81.19:0.24
79.51
80.98
81.57
NN
ICML 2019
(Isinda el al. 2019
30.68
25.97.11.17
29.5010.51
31.65
GA
ICML 2019
Ishida el al. 2019
1.08
37.51
36.8411.37
00.79
SCL-EXP
ICML 2020
79.50+0.33
79.57+0.37
79.34+0.53
00.30
SCL-NI.
ICML. 2020
al..
81.87+0.07
81.09+0.55
79.96
81.75+00.14
L-W
ICML 2021
& Zhang 2022
76.76=00.4
74.04+0.88
74.76+00.00
76.76-t0.49
OP-W
AISTATS 2023
2023
78.91
78.71+0.39
79.64+0.21
81.15+0.499
PICO
ICLR 2022
Wang el al. 202.5
79.2010.61
15.1910.599
79.37
10.77
ABLE
UCAI 2022
Xia
2022
85.86
86.30.10.14
86.07
86.46-10.10
CRDPLL
ICML 2022
Wü
2022
81.60
79.82+0.116
81.66+0.52
82.36
DIRK
AAAT 2024
Wu
2024
85.9010.31
84.96.10.3
85.7411.42
85 8..60-10.28
FREDIS
ICML 2023
Uiao
2023b
85.01
84.94+0.07
85.66+0.22
85.74+00.09
ALIM
NeurIPS 2023
Xu ct al.
202.6
63.51
11.76+1557
61.42
64.72+00.27
PICO+
TPAMI 2024
Wang et 20244)
60.07+0.13
57.63+0.84
59.01
62.66-10.33
6.3 COMPLEXITY ANALYSIS
Figure shows the running time and GPU memory usage for each running step of different PLL algorithms PL.CIFAN 10- Vagusst with DenseNet. We can that some holistic deep PL.. al-
8

---

Published as a conference paper at ICLR 2025
Classification accuracy (mean+ssd)) of each algorithm PLCCIFARO-VVagee with ResNet, where the best performance w.r.t. each metric is shown in bold.
Table 2:
Algorithm
Venue
w/
AA
OA
OA ES
PRODEN
ICML 2020
(L el al. 20201
74,95 10.09
68.5411.399
74.7810664
76.9410.41
CAVL
ICLR 2022
(Zhang al. 2022
03.6
61.76+0.75
63.70+0.62
64.68
POP
ICML 2023
Xu a. 20250
75.17
67.7110.55
74.35
76.08:10.18
ABS-MAB
TPAMI 2024
(Lv al
2124a
55.58
32.2547.18
0.660
55.84
ABS-GCE
TPAMI 2024
(Lv
2024a
75.17
72.8610.17
74.1310.68
7..95
EXP
ICML 2020
Feng et al. 2020a
03.93
61.77+0.16
63.04
64.14+0.36
MCL-GCE
ICML 2020
Frong ct al. 2020a
71.74:0.211
64.5341167
70.78+0.32
73.28:10.19
MCL-MSE
ICML, 2020
Feng ct al. 2m
0.999
65.91
64.94+0.277
69.98
CC
NeurlPS 2020
(Teng el al. 2020bb
71.78
61.05510.22
70.1110.443
13.26
ICML 2021
Wenn ct
2021
60.21
14.12210.76
60.98+0.46
61,78+0,67
IDGP
ICLR 2023
Qta
20266
76.14:10.18
7133442.22
76.05
77.74::0.19
NeurIPS PSPS222 2017
[[shida
2017
6.84611.88
66.02
66.5910.34
68.13-10.29
Forward
ECCV 2018
ef
70.01
13.09112.96
70.5210.07
70.98
NN
[CML 2019
Ishida al.
2019
33.23
25.31+
30.35+0.37
33.12++0.48
GA
ICML 2019
Ishida
33.94111001
33.6411.200
0.86
34.00-L0.98
SCL-EXP
ICML 2020
Chou al.
20201
70.13
65.45
7..3
72.86
SCL-NL
ICML 2020
Chou
2020
69.66
60.4611.54
69.5610.47
71.52:10.14
L-W
ICML 2021
(G0 Zhang
202
60.59
19.19
71.12#0.19
72.44+00.45
OP-W
AISTATS 2023
(L.u ct a 2023
70.53
69.64412.77
73.20+0.26
73.21
PICO
ICLR 2022
Wang al.
A1226
74.06
68.33++0.58
73.27
74.30++0.33
ABLE
UCAL 2022
Xia el al. 2022
10.58
68.14100.5
74.87
76.27-10.13
CRDPLL
ICML 2022
Wi ct a.. 2022
76.21
70.99+0.119
75.70+0.099
77.68+00.46
DIRK
AAAI 2024
Wu al. 2024
80.32
75.02:12.36
80.10-1.33
81.08
FREDIS
ICMI. 2023
QQao al 2023b1
70.57
67.7240..13
73.45
76.944
ALIM
NS 2023
(Xu al. 202.a
1.05
66.51
1..57
7..59
PICO+
TPAMI 2024
(Wang cl.
20244
61.59
59.6540.84
61.27+0.12
61.75:00.30
Table 3: Classification accuracy (mean+ of each algorithm PLCIFARIO-Aggreee with
DenseNet, where the best performance w.r.t. each metric is shown in bold.
Algorithm
Venue
w/ CR
w/ AA
OA
OA ES
PRODEN
ICML 2020
(Ly ct al. 2020
80.94
81.54 0.72
80.9410.90
1..70
CAVI.
ICLR 2022
(Zhang al.
2022
21.12
20.17
1..
63.82+1.0
POP
ICML 2023
Xu al
20256
80.99
80.50
81.20-:0.41
01.38-10.43
ABS-MAE
TPAMI 2024
(Lv
2024.
53.93
50.56 1.85
1.88
1..56
$3.85
ABS-GCE
TPAMI 2024
(Lv
2024a
72,48 10.34
71.5010.46
73.05
74.28
EXP
ICML 2020
Feng et al. 212
10.00
10.00
10.00++0.00
10.00++0.00
MCL-GCE
ICML 2020
[Fong ct al. 2020a
58.82+0.31
57.81
36.47
61.34:0.03
MCL-MSE
ICML 2020
Feng el al. 20
57.29
10.00
$4.27
59.37
CC
NeurlPS 2020
(Teng et 2020bb
78.28 10.82
77.64.0.66
78.52 10.38
78.97
LWS
ICML. 2021
Wen ct al.
2021
47.57
41.64+1991
48.48
48.90+00.41
IDGP
ICLR 2023
Qtao ali
2026m
17.69
70.07
78.4140.84
79.03-40.86
NeurIPS 201
[Ishida el
2017
65.60-10.13
65.95
66.12.10.4
66.55
Forward
BCCV 2018
Yu 2013
78.74.0.48
78.02
78.38
19.3910.119
NN
ICML 2019
Ishida
31.51
26.01
30.22
32.97+
GA
[CML 2019
Ishida
37.21110.58
36.85
37.28
37.86-10.20
SCL-EXP
ICML 2020
Thou 2020
78.6710.71
78.2710.69
18.38
7..32
SCL-NL
ICML 2020
Touu 2020
10.59
78.90
78.29 10.27
79.26 10.59
L-W
ICML 2021
Gao Zhang 202
+0.53
71.77
72.24+00.69
73.7
OP-W
AISTATS 2023
(Lu al 2025
19.35-0.33
20.43
80.04+10.16
80,36-10.32
PICO
ICLR 2022
Wang al
A122b
74,89 1:.14
0..6
75.81
76.37-1
ABLE
UCAI 2022
Xıa el al. 2022
81,38 1 1.333
1..34
81.21
81,28 0.661
CRDPLL
ICML 2022
Wi et al. 2022
74.97
74.90
75.36++0.59
75.67++0.66
DIRK
AAAI 2024
Wu al. 2024
77.74:10.64
77 77.83
77.66-10.67
77.85
FREDIS
ICML 2023
Quao al 2023bi
81.25
81.08 1.880
81.11
81.66 0.51
ALIM
NeurlPS 2023
eX
2025a
56.61
$7.29 10.32
58.46 L0.38
59.75 0.64
PICO+
TPAMI 2024
(Wang al.
20244
57.05
54.01
56.02+ 1.02
58.45++0.115
gorithms with strong regularization terms and complicated training strategies can achieve better performance, but they also take more time and memory. Therefore, it is important

---

Published as a conference paper at ICLR 2025
Table 4: Classification accuracy std) of each algorithm PLCIFARIO- Vaguest with
DenseNet, where the best performance w.r.t. each metric is shown in bold.
Algorithm
Venue
w/
w/
OA ES
PRODEN
ICML 2020
(Lv al.
2020
0.67
67.42.1..00
72.5310.44
73.98 1.111
CAVL
ICLR 2022
Zhang
2022
15.93
58.85
59.01
19.81
POP
ICML 2023
Xü
2023b
7.87
19.18+1.64
71.90
72.85+0..21
ABS-MAE
TPAMI 2024
eL
21244
27.07
5..50
57.69 1.98
57.94+22.01
ABS-GCE
TPAMI 2024
(LY ct
20248
73.40=0.46
6..85+1.27
72.75=00.48
73.63++0.42
EXP
ICML 2020
Feng al.
20210
61.50=0.69
55.26++2.27
61,44=0.54
62.56+0.552
MCL-GCE
ICML 2020
Teng et al. 2020a
6..57
6..90.11.36
6..88
69.57-10.70
MCL-MSE
ICML 2020
Feng et al. 4020a
63.37=1.09
63.79:10.92
63.08:00.42
65,26-10.55
CC
NeurIPS 2020
(Feny ei al.
2020b
68.77:10229
60.92
69.62:10.3
71.49
LWS
ICML 2021
Wen el al 202T
17.8511447
54.6511.65
58.1611..38
59.09 12.00
IDGP
ICLR 2023
1ia at 21255
72.02=0.66
70.55+0.91
7..9
74.19+00441
PC
NeurTPS 20
[shida of all 2017
59.99+0.93
60.28
63.11
63.93-+0.00
Forward
ECCV 2018
Yū ct a 2018
66.35=00977
60.12
66.54
67. 0.
NN
ICML 2019
Uhida ct all 2019
31.08=0.91
26.52+0.37
30.23+0..3
33.54+00112
GA
ICML 2019
Ishia et al 2019
3..5
35.52.1046
34.5210.555
35.75 0.330
SCL-EXP
ICML 2020
Chou ci al 2020
68.9910.34
65.5910.82
69.41
70.72:10.49
SCL-NL
ICML 2020
Chou ct al 2020
66.90=0.37
61.42:0.22
65.87
0..7
L-W
ICML 2021
Gap & Zhang
202
68.28-0083
63.5240.71
68.20
6..66
OP-W
AISTATS 2023
Luu el
20233
1.491
70.25
11.55
10.45
PICO
ICLR
Wang ci al.
20226
71.76+0.28
69.30++0.93
70.89+0.59
72.27+0.48
ABLE
UCAT 2022
Xia al.
71.68 1.01
68.224
72.10+0.20
73.4400.51
CRDPLL
ICML 2022
Wu ct 2022
70.73+0.75
70.6940.117
71.99+0.67
72.46+0.299
DIRK
2024
Wu al..
70.47=0.32
71.06:10.48
71.26::0.19
7..60
FREDIS
2023
Qhao el
2023b
71.3810.41
65.92.10.13
71.3310.30
72.95
NeurlPS 2023
Xu
202.a
6.29
62.04.1..10
63.8410.49
65.37
PICO+
TPAMI 2024
Wang et al.
20248)
62.45-0.74
59.59-
60.67
62.82
2.00
1.0
1.75
Memory
Time
1.00
0.75
0.2
0.6
0.
PRODEC
pOR
EXP
MCL- OCE
NCL MBE
LING
PC
SOL SOLEE
SOLU
op
Pico
ABLE
CROPLL
nink
FREEDS
ALM
0.00
PRODEN
CAVL
PO
ABS-002
00
EX
NCL-EE GEL-GE
NCL
Fowarr
ARUS
CROPLL
pink
mer ALM nicoe
Ats
Running Time
(b) GPU Memory Utilization
Figure 4: Running time and GPU memory utilization for each running step of different PL.I algo-
rithms on PLCIFARIO- Vaguest with DenseNet.
sider the tradeoff between performance and resource consumption when deciding which algorithm use given limited computational resources.
CONCLUSION
In this paper, we proposed the first PLL benchmark to standardize the performance evaluation of state-of-the--rr deep PLI algorithms. We proposed new model selection criteria to fill in the gap of model selection problems for PII.. We also introduced PLCIFARIO, a novel image dataset of human-annotated partial labels. We hope that the availability of this benchmark can promote fair, practical, and standardized evaluation PLL algorithms in the future. limitation of our work is that the datasets relatively small, and will leave the exploration of large-scale PLL datasets as our future work. In addition, our experiments are conducted with simple deep neural networks. is also interesting explore the incorporation of foundation models to improve the model performance of PLL algorithms in the future.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors thank Wei-I Lin, Takashi Ishida, and Yu-Jie Zhang for their helpful discussions and suggestions, Wei-Xuan Bao for his help with the datasets, and Yuko Kawashima for her help with the funding management. The authors thank the anonymous reviewers for their helpful comments. WW was supported by the SGU MEXT Scholarship. by the Junior Research Associate (JRA) program of RIKEN, and by Microsoft Research Asia. MLZ was supported by National Science Foundation of China (62225602). MS supported by the Institute for AI and Beyond, UTokyo and by grant from Apple, Inc. Any views, opinions, findings, and conclusions or recommendations expressed in this material those of the authors and should not be interpreted as reflecting the views, policies or position, either expressed or implied, of Apple Inc.
ETHICS STATEMENTS
This paper does not raise any ethical concerns. This paper contains a human subjects study where the data collection process was conducted on Amazon MTurk. The privacy of the annotators was strictly protected by Amazon MTurk. We strictly followed the Code of Ethics during the dataset collection process.
REFERENCES
Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 542, 2012.
Nontawat Charoenphakdee, Jayakor Vongkulbhisal, Nuttapong Chairatanakul, and Masashi Sugiyama. On focal loss for class-posterior probability estimation: A theoretical perspective. In Proceedings of the 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5202-5211, 2021.
Ching-Hui Chen, Vishal M. Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7):1653-1677, 2018.
Yu-Ting Chou, Gang Niu, Hsuan-Tien Lin, and Masashi Sugiyama. Unbiased risk estimators can mislead: A case study of learning with complementary labels. In Proceedings of the 37th Inter- national Conference Machine Learning, pp. 1929-1938. 2020.
Katherine M. Collins, Umang Bhatt, and Adrian Weller. Eliciting and learning with soft labels from every annotator. In Proceedings of the 10th AAAI conference on human computation and crowdsourcing, pp. 40-52, 2022.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12(May):1501-152) 2011.
Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with mul- tiple complementary labels. In Proceedings of the 37th International Conference Machine Learning. pp. 3072-3081, 2020a.
Lei Feng. Jiaqi Lv. Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial-label learning. In Advances in Neural Information Processing Systems 33, pp.
2020b.
Yi Gao and Min-Ling Zhang. Discriminative complementary-len learning with weighted loss. In Proceedings of the 38th International Conference on Machine Learning, pp. 3587-3597, 2021.
Zhengqi Gao, Fan- Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Mare Engeler, Antonio Burazer. Linda Wildling, Luca Daniel, and Duane S. Boning. Learning from multiple annotator noisy labels via sample-wise label fusion. Proceedings of the 17th European Conference on Computer Vision, pp. 407-422, 2022.
11

---

Published as a conference paper at ICLR 2025
Dan Garrette and Jason Baldridge. Learning part-of-speech tagger from two hours of annotation.
In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 138- 2013.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation.
Journal of the American statistical Association, 102(477):359-37, 2007.
Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang, Jie Yang, and Dacheng Tao. A regularization approach for massdnce-based superset label learning. IEEE Transactions on Cybernetics, 48(3): 967-978, 2018.
Xiuwen Gong, Dong Yuan, and Wei Bao. Partial label learning via label influence function. In Proceedings of the 39th International Conference on Machine Learning, pp. 7665-7678, 2022.
Mononito Goswami, Vedant Sanil, Arjun Choudhry, Arvind Srinivasan, Chalisa Udompanyawit, and Artur Dubrawski. AQuA: A benchmarking tool for label quality assessment. In Advances in Neural Information Processing Systems 36, pp. 79792-79807, 2023.
Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multiple instance metric learning from automatically labeled bags of faces. In Proceedings of the 11th European Conference on Com- puter Vision, pp. 634 647, 2010.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In Proceedings of the 9th International Conference on Learning Reprexentations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
nition. In Proceedings the
Shuo He, Chaojie Wang, Guowu Yang, and Lei Feng. Candidate label set pruning: A data-centric perspective for deep partial-label learning. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Dan Hendryeks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances in Neural Information Processing Systems 31, pp. 10477-10486, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp.
2017.
Mark J. Huiskes and Michael S. Lew. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, pp. 39-43, 2008.
Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. Advances in Neural Information Processing Systems 30, pp. 5644-5654, 2017.
Takashi Ishida, Gang Niu, Aditya K. Menon, and Masashi Sugiyama. Complementary-label learning for arbitrary losses and models. In Proceedings of the 36th International Conference Machine Learning. pp. 2971-2980, 2019.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on con- trolled noisy labels. In Proceedings of the 37th International Conference on Machine Learning, pp. 4804-4815, 2020.
Rong Jin and Zoubin Ghahramani. Learning with multiple labels. Advances in Neural Information Processing systems 15, pp. 897-904, 2002.
Bos Johan, Cristina Bosco, Alessandro Mazzei, et al. Converting dependency treebank to catego- rial grammar treebank for Italian. In Proceedings of the 8th International Workshop on Treebanks and Linguistic Theories, pp. 27-38, 2009.
Ashish Khetan, Zachary C. Lipton, and Anima Anandkumar. Learning from noisy singly-labelen data. In Proceedings of the 6th International Conference on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2025
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Changchun Li, Ximing Li, Jihong Ouyang, and Yiming Wang. Detecting the fake candidate in- stances: Ambiguous label learning with generative adversarial networks. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 903-912, 2021.
Ximing Li, Yuanzhi Jiang, Changchun Li, Yiyuan Wang, and Jihong Ouyang. Learning with partial labels from semi-supervised perspective. In Proceedings of the 37th Conference on Artificial Intelligence, pp. (666-8674, 2023.
Liping Liu and Thomas Dietterich. Learnability of the superset label learning problem. In Proceed- ings of the s1 International Conference on Machine Learning, pp. 1629-1637, 2014.
Liping Liu and Thomas G. Dietterich. A conditional multinomial mixture model for superset label learning. Advances in Neural Information Processing Systems 25, pp. 548-556, 2012.
Shuqi Liu, Yuzhou Cao, Qiaozhen Zhang,
Lei Feng, and Bo An. Consistent complementary-laber learning via prder-presrrving losses. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, pp. 7748, 2023.
Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems 23, pp. 1504-1512, 2010.
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identifica- tion of true labels for partial-label learning. In Proceedings of the 37th International Conference on Machine Learning, pp.
2020.
Jiqqi Lv, Biao Liu, Lei Feng, Ning Xu, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):2569-25) 2583, 2024a.
Jiaqi Lv, Yangfan Liu, Shiyu Xia, Ning Xu, Miao Xu, Gang NIu, Min-Ling Zhang, Masashi Sugiyama, and Xin Geng. What makes partial-label learning algorithms effective? In Advances in Neural Information Processing Systems 37, 2024b.
Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. GM-PLL: Graph matching based partial label learning. IEEE Transactions on Knowledge and Data Engineering, 33(2): 521-535, 2021.
Gengyu Lyu, Songhe Feng, Tao Wang, and Congyan Lang. self-paced regularization framework for partial-label learning. IEEE Transactions on Cybernetics, 52(2):899-911, 2022.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference Machine Learning, pp. 807-814, 2010.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Is object localization for free? Weakly- supervised learning with convolutional neural networks. In Proceedings of the 2015 IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 685-694, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8026-8031 2019.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision pp. 9617-9626, 2019.
13

---

Published as a conference paper at ICLR 2025
Congyu Qiao, Ning Xu, and Xin Geng. Decompositional generation process for ristance-dependee partial label learning. In Proceedings of the 11th International Conference on Learning Repre- sentations, 2023a.
Congyu Qiao, Ning Xu, Jiaqi Lv. Ren, and Xin Geng. FREDIS: A fusion framework of refinement and disambiguation for unreliable partial label learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 28321-283366 2023b.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection machine learning. arXiv preprint 1rX.v:1811.12000 2018.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. AFET: Automatic fine- grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pp. 1369-1378, 2016a.
Xiang Ren, Wenqi He, Meng Qu, Clare R. Voss, Heng Ji, and Jiawei Han. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1825-1834, 2016b.
Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke. Anna Valros, Nina Volkmann, and Reinhard Koch. Is one anno- tation enough? a data-centric image classification benchmark for noisy and ambiguous label estimation. In Advances in Neural Information Processing Systems 35, pp. 33215-33232, 2022.
Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai, and Gang Niu. Machine learning from weak supervision: An empirical risk minimization approach. MIT Press, 2022.
Wei Tang, Weijia Zhang, and Min-Ling Zhang. Disambiguated attention embedding for multi- instance partial-label learning. In Advances in Neural Information Processing Systems 36, pp. 66756-56771, 2023.
Shiyu Tian, Hongxin Wei, Yiqun Wang, and Lei Feng. CroSel: Cross selection of confident pseudo labels for partial-label learning. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19479-19489 2024.
Yingjie Tian, Xiaotong Yu, and Saiji Fu. Partial label learning: Taxonomy, analysis and outlook. Neural Networks, 161:708-734, 2023.
Deng-Bao Wang, Min-Ling Zhang, and Li Li. Adaptive graph guided disambiguation for partial label learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(12):8796 8811, 2022a.
Haobo Wang, Ruixuan Xiao, Sharon Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In Proceedings of the 10th Interna- tional Conference on Learning Representations, 2022b.
Haobo Wang. Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO+: Contrastive label disambiguation for robust partial label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):318333198 2024a.
Hsiu-Hsuae Wang, Mai Tan Ha, Nai-Xuan Ye, Wei-I Lin, and Hsuan-Tien Lin. CLImage: Human- annotated datasets for complementary-label learning, 2024b. URL net/forum?id==3eehee
Qian-Was Wei Wang, Yu-Feng Li, and Zhi-Hua Zhou Partial label learning with unlabeled data. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3755-3761, 2019.
Wei Wang and Min-Ling Zhang. Semi- semi-supervised partial label learning via confidence-ratee margin maximization. In Advances in Neural Information Processing Systems 33, pp. 6982-6993, 2020.
14

---

Published as a conference paper at ICLR 2025
Wei Wang and Min-Ling Zhang. Partial label learning with discrimination augmentation. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 1920-1928, 2022.
Wei Wang, Takashi Ishida, Yu-Jie Zhang, Gang Niu, and Masashi Sugiyama. Learning with com- plementary labels revisited: The selected-completely- caappleeely-at-rnddoo setting is more practical. Pro- ceedings of the s1st International Conference on Machine Learning, pp. 50683- 50710, 2024c.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. In Proceedings of the 38th International Conference on Machine Learning. pp. 11091-11100 2021.
Dong-Dong Wu, Deng-Bat Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In Proceedings of the 39th International Conference on Machine Learning, pp. 24212 24225, 2022.
Dong-Dong Wu, Deng-Bar Bao Wang. and Min-Ling Zhang, Distilling reliable knowledge for instance- dependent partial label learning. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15888-15896, 2024.
Zhenguo Wu, Jiaqi Lv, and Masashi Sugiyama. Learning with proper partial labels. Neural Compu- tation, 35(1):58 81, 2023.
Shiyu Xia, Jiaqi Lv, Ning Xu, and Xin Geng. Ambiguity-induced contrastive learning for instance- dependent partial label learning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, pp. 3615-3621, 2022.
Mingyu Xu, Zheng Lian, Lei Feng, Bin Liu, and Jianhua Tao. ALIM: adjusting label importance mechanism for noisy partial label learning. In Advances in Neural Information Processing Sys- tems 36, pp. 38668 -3668,, 2023a.
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In Advances in Neural Information Processing Systems 34, pp. 27 27119-27130, 2021.
Ning Xu, Biao Liu, Jiaqi Lv, Congyu Qiao, and Xin Geng. Progressive purification for instance- dependent partial label learning. In Proceedings of the 40th International Conference on Machine Learning. pp. 38551-38565, 2023b.
Yao Yao, Jichui Deng, Xiuhua Chen, Chen Gong, Jianxin Wu, and Jian Yang. Deep discriminative CNN with temporal ensembling for ambiguously-labeled image classification. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp. 12669-12676, 2020.
Chenglin Yu, Xinsong Ma, and Weiwei Liu. Delving into noisy label detection with clean data. In Proceedings of the 40th International Conference on Machine Learning, pp. 40290-40305, 2023.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In Proceedings of the 15th European Conference on Computer Vision, pp. 68-83, 2018.
Zinan Zeng. Shijie Xiao, Kui Jia, Tsung-Han Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 708-715. 2013.
Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Ex- ploiting class activation value for partial-label learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions on Knowledge and Data Engineering, 29110)2155-2167, 2017.
Deyu Zhou, Zhikai Zhang, Min-Ling Zhang, and Yulan He. Weakly supervised POS tagging without disambiguation. ACM Transactions on Asian and Row-urs Language Information Process- ing, 1744) 2018.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
A.1 PROOF OF PROPOSITION T
E[CR(f) (f)
S)
arg max max f3(2) E $ -
-EE Ep(a.v) 11
(ra max f5(2)) = ]
Dag [5(x)) E :S,1v) ]
Eple.sss)
arg max f5(x)) E shin)) I arg max fz(x)) # v) ]
= =Ep(x.0.5)
I
=p arg max fa:) E (((y), max [5(x) + j
arg max fx())) + v arg max fn( E Sitvll arg max 15(20) # y
j
(1 ACC (f)) p arg f)( e SMyll arg max fs(x) # y
<1 max ply S1 94y
(1 - 6)7
Here, the third equation can be obtained by traversing the cases of arg max, fz(x) = y. arg max; fx) E Sy and arg max, fa(²) $ S. The last inequality results from the defini- tion of the ambiguity degree. The proof is completed.
A.2 PROOF OF THBOREM
Under the USS assumption or the FPS assumption with a constant flipping probability, the proba- bility p(y E S|æ, * y) a constant value for different y. Suppose the constant is c. Then, we have
E B[CR(f) (F)
=(1 - (F) p arg max f5(x) SMlyil arg max fz(2) * v)
=c ACC(F))
Therefore, we have
ACC (f) = E [[CR(f)]-3
(8)
Since we have e < 1, for any two classifiers fi and fz that satisfy [[CR(()) < [CR((2)),, we have ACC (fi) ACC (fz). The proof completed.
A.3 PROOF THBOREM 2
The proof of Theorem is mainly based on the theoretical results from Wu et al. (023). We
introduce the following lemma.
Lemma 1 Wu et al. 2023 ) Assume that there exist function C X X 2 R such that the
condition p(S y) C(., S)1 (y E S) holds for partial- label data. Then, the classification risk
Epter [f(f(x)
(9)
is equivalent to
p j|ar)
Ep(an.))
Σ
((f(2),
(10)
E p(y = k) xx)
JeS
16

---

Published as a conference paper at ICLR 2025
Then, provide the proof of Theorem
Proof of Theorem 2 We set (()(),, k) (arg max, [(x) = v). Then, the multi-class clas-
sifier ((x) consistent with p(y x)) AA(f) is statistically consistent with the expected aceuracy. The proof is completed.
B MORE DETAILS OF BENCHMARK DATASETS
In this section, we first describe the data sheet of PLCIFARIO, a novel dataset collected by us. Then, we provide more information of real-world datasets used in the paper. The summary of all the datasets shown Table
Table 5: Characteristics of woaldworld PLL datasets used PLENCH.
Dutaset
Examples
Features
Classes
CLs
Noise Rate
Type
Task Domain
0%
automatic
Courr
MSRCV2
1,758
48
23
3.16
0%%
tabular
objoct class/fication dI & Distersh 2m
Mirflickr
2,780
1,536
14
2.76
0%
tabular
web image classification prikk
Birdsong
4,998
38
2.18
0%
tabular
bird song classiscation Brings
2012
Malagaty
5,303
384
0.04%
tahular
Tagging (Garvette
Soccer Player
17,472
279
2.09
0%
tabular
automitic face namunn Zen cral 201
Italian
21,878
519
90
1.60
0%
tabular
POS Tagging (Jobah el al 2009 face
Yahan' News
22,991
0%%
labular
2010
English
24,000
0.97%
tabular
POS Tagging ef 2013
PLCIFARIO lo-ggrateeeeeatte (Ours)
50,000
3,072
4.87
0.13%
image
image classification
Kribeeske & Himen
2009
PL CIEARRR 10. Vaguest (Ours)
50,000
3,072
3.49
17.36%
image
image classification
Krizeessy & Himbe
2019
B.1 MORE DETAILS OF PLCIFARIO
For each example, we have a list of lists, where each sublist contains partial labels given by single annotator. Each image resized 256 x 256 for easy annotation. We also imposed several requirements on the annotation task to ensure the quality of the annotations. First, we asked the annotators to choose labels for all ten images. Second, did not allow the same annotator to select the same label for more than threshold number of images. Third, we did not allow annotators to select many partial labels tain little supervision information. We set the thresholds to 6 for a HIT of 10 images. Crowdworkers were involved in the data collection process. We paid $0.02 for HIT, where $0.01 was given the crowdworkers and S0.01 was given to the MTurk platform.
than threshold for single image since the annotation may
B.2 MORE INFORMATION OF TABULAR DATASETS
For face age estimation, we considered ten crowdsourced labels of age numbers along with the true label as partial labels for given human face. For automatic face naming, we considered the names in the corresponding captions or subtitles as partial labels for face cropped from an image. For object classification, we considered object classes as partial labels for segmentation part in an image. For web image classification, we considered tags on a web page as partial labels for given image. For bird song classification, we considered bird species appearing in a ten-second bird song fragment as partial labels for the singing syllables of the fragment. For POS tagging, considered all possible POS tags as partial labels for given word with its contexts.
C USAGE OF PLENCH
PLCIFARIO is located in pleecch/cata/pploifari in the code package. Tabular sdu.euu
datasets can downloaded from easily add newly developed algorithms PLENCH to ver-
Bieaangm/RResource
htmiddtal Researchers ity their effectiveness. We mainly need to inherit the Algorithm class and implement the update and predict functions for the new algorithm in a algorithms. .py. For
17

---

Published as a conference paper at ICLR 2025
Table 6: Summary of benchmark algorithms.
PRODEN eL 2020) CAVL
Zhang al 2022
POP (Xu etal 20236 ABS- Lv 2024a
Vanilla Deep PL.L Algorithms
ABS-GCE Ly ar 2024a EXP Fenr ar. 20205
MC1 MCCAGGEET Fen Feir 3020a
Feig 0 ar 2020a
Fong 2020b
20235
Vanilla Deep CLL, Algorithms
PC
Ishida al 2017
Forward
Yo ot 2018 NN
[shada ctal. 2019: GA Ishida etal 2019
SCL-LAP
2020
SCL
20201
2023
Holistic Deep PLL Algorithms
VALEN
Xu al
2021 PICO Wang et al, 20226
ABLE Xia et al, 2022
CRDPLL Wa et 2022
DIRK
Wird
Deep Noisy Algorithms
Olao
20236
Xuetal.
(Wang 2024a)
example, to implement CAVL Zhang et al.
2022 we can write the following code:
class CAVL (Algozichmm
def _nntt_(seltt (self, input_shapes train_given', hparama)
super (CAVI,, self)._ing iinit_tnputt_____i.iiiiii (input_shaagee shape, train_givent, hparams)
self.featurizer networks. networks.Peatarizze fanpct_shape, self. self.hparmms) self.cllassi fier networks. Classisfccaasiiir........o htl.ffeeturizee.nnoottee telf.num_classe))
nelf.netwurk nn. Sequent isl featurizer, self.classifier) self.optimizer torch.optim.Aden. self welffmetwork. ptt.etowrr....seeesseee
hr-seat.hparame (par (*1r"), keiqitt_ecay=seell .hhpaaaa
hparams (etarammswweidiiie ['weight_decay']
train_givent from..frroomumpm (train__geee]
tempY train_givent wennessum (dim-1) unsqueere (.eeeee repeat (1, train_giveny shappe [1])
sabel_coafldence grain_givenn. fleat (1/emmp
self.label_confideer self.labei_confideen
label sabel_confidence
self. htt.i_s_oooofdencaa......
def update minibatcheal:
x, strong_x, partial_y, index minibatches
Toss = self. (self. (x), index)
solFFaattimizer.zzrogguuuu
loss basssaacwward(1
self.
self.con"idence_upp" _pppetee_ partial_y, index)
return ("loss': loss.item!)]
def re (cess,, outputs, index):
device "cuda" 1f index.1s_cus else "opu" 1ale. 1oll _confidence self. 1aFelabbl_oooo donca.to (device)) pogen_outputs T.leg_sootmme (outputs, dim-1)
tinal_outputs Logam_outgure self, lalf.aaabl_cooiiiee [Lndee, average_ loss return average_lose
[Einal_outpul sum (dim=l)) .eean
def predict (self, x): return self.network
def sartidence_uddaeeeee batchX, batchY, badeh_index):
with torch.no_grad()- batch_cutputs self. self.predict (batchX)
cav ootatchhuttputa-torch abe (1 (l-batcct_outputs) <batchy
cav_pred torch.man (cav, dim=l) [1]
gt__labe1 P_ono_hoe (cav (cav_pred, www.hh..aaappeeeee..
self. aaffllabel_ooofideee (batch_index,
st_label.doubll
After implementing the algorithmic code, we can specify the necessary hyperparameters for the algorithm aparams_registry.p rrggistryppy. Then we can train the model using the new algorithm with the following script:
python -m Mtextsc|Plendee Plw.chh..rsiiiiii. trsin - dir data --algorithm CWVL dataset AacIFARio_Aggreeee cotpputddir-yooo puttrutouuuutttt path s-epp 60000
D RELATED WORK
There three main categories of deep PLL algorithms, including identification- based strategies, averaging- strategies, and data- setitd-ennraatio--aee bbaed strategies. Identification based strategies try to find the true label from the candidate label set and train classifier simultaneously (aoo al.
18

---

Published as a conference paper at ICLR 2025
2020
Li al. 2023 Xu et al. 2023b Gong et al.] 2022 He et al. 2024 Tian et al. 2024 Lv et al. 2024b).
Lv al. 2020 Wang & Zhang 2020 Zhang et al. 2022 Wang & Zhang 2022
Averaging based strategies consider equal contributions from all candidate labels and use the aver- age modeling output as the final output (Lv et al. 2024a). Data-generation-bann baseed strategies model the generative relationship between partial labels and true labels, and derive loss functions with the- oretical guarantees (Feng et al. 22020b Wen et al. 2021 Qiao et al., (023a).. In addition, many works use strong representation learning techniques improve model performance based on these basic strategies, such as contrastive learning and consistency regularization (Wang et al. 2022b Wu al., 2022 Xia al., 2022, Wu et al., 2024). There are some work that also handles data annota- tions with multiple labels (Khetan et al. 20188 Peterson et al. 2019 Collins et al. 2022 Schmarje
et al. 2022 Gao et al. 2022 Goswami et al. 2023)
E
DETAILS OF DEEP PLL ALGORITHMS
In this section, we first give detailed descriptions of the PLL algorithms used in this paper. Then, we provide the hyperparameter configurations for the algorithms. Table shows the summary of the deep PLL algorithms included in this paper.
E. DESCRIPTIONS OF ALGORITHMS
The vanilla deep PLL algorithms include:
PRODEN (Lv et al. 2020): An identification-basel strategy that uses self-training to progres- sively estimate the true label distribution from the candidate label set.
CAVL (Zhang et al. 2022): An identification-basee strategy that uses the class activation value to directly identity the true label from the candidate label set.
POP (Xu et al.
2023b): An sitiificatiin-baseee strategy that progressively filters out false positive labels from the candidate label set based on PRODEN.
ABS-MAE (Lv al.) 2024a): An everaging-baser strategy using the mean absolute error (MAB)
loss function.
ABS-GCE (Lv al.
2024a An averaging- based strategy using the generalized cross-entropy
(GCE) loss function.
CC (Feng al. 2020b): A data- generation-bee -bseed strategy using classifier-cnnsitenn loss func-
tion based on the uniform distribution assumption.
data-generation-baase strategy using exponential loss under the uniform distribution assumption.
EXP (Feng et al. 2020a):
MCL-GCE (Feng et al. 2020aa: A kata-generattonn-aasee strategy using generalized cross-entropy
(GCE) loss under the uniform distribution assumption.
MCL-MSE (Feng al., 2020aa::
setiaa-nnerattnbbbeeeeee strategy using mean squared error (MSE) loss based on the uniform distribution assumption.
LWS (Wen al. 202T): A lata-generation-basee strategy that uses a leveraged weighted loss function to account for losses from both candidate and non-candidate labels.
IDGP Qiao et al., 2023a): A sitiaa-gneraatio-baae based strategy that performs Maximum A Poste- rior (MAP) using decomposed probability distribution model.
The vanilla deep CLL algorithms include
PC (Ishida et al. y0
A risk-consistent CLL algorithm using the pairwise-comparison loss
based the uniform distribution assumption.
Forward (Yu et al.) 2018): A classifier-consisten CLL algorithm using a transition matrix to generation process based on the biased distribution assumption. NN (Ihida et al., 2019): A risk-consisted CLL algorithm using a non non-neeative risk estimator based on the uniform distribution assumption.
model the complementary-leee
GA (Ishida et al., 2019): A risk-consistent CLL algorithm using the gradient ascent technique
based on the uniform distribution assumption.
SCL-EXP (Chou al. (2000]: A discriminative CLL algorithm using exponential loss.
SCL-NL (Chou et al. 2020): A discriminative CLL algorithm using negative loss.
L-W (Gao & Zhang 2021): : A discriminative CLL algorithm using weighted loss.
OP-W W (Liu et al.] 2023): A classifier-consistent CLL algorithm by using the opposite number of
logits to compute the loss.
19

---

Published as a conference paper at ICLR 2025
Table 7: Classification accuracy (mean+sst)) of each algorithm Lost with different model selec-
[ion criteria.
Algorithm
Venue
w/ CR
w/ AA
w/ OA
w/ OA & ES
PRODEN
ICMI 2020
(L al. 2020
2..33+2.45
70.09
21.35+2411
71.33
CAVT.
ICLR 2022
Zhang et al. 2022
60.00+3.30
58.41 +:2.9
24.42
2.440
POP
ICML 2023
Xu al. 2023b
69.38-1.990
69.38
70.44+245
2..10
ABS-MAE
TPAMI 2024
(Lv al
20246
68.50-2.70
67.43
68.8513.07
69.03
ABS-GCE
TPAMI 2024
cL
202448
62.65+2.64
62.30+2.04
62.48+2666
67.4342.68
EXP
ICMI 2020
Feng et al.
20,20a
71.33+1.27
70.27
72.21
70.97
MCL-GCE
ICMI, 2020
Feng et al.
20,20a
60.71+2.41
58.94442.62
1..95
6,.49
MCL-MSE
ICML 2020
Feng et al.
20/20a
58.58=2.26
59.47-2.711
58.7642.52
5..47
CC
NeurIPS 2020
(Feng et al. 2020b)
7..33+2.60
70.62
72.57
72.21
LWS
ICML 2021
WWnn ct al. 2021
72.92+3.59
71.33+3.75
72.21+241
74.3442.68
IDGP
ICLR 2023
Qiao ét al.
202.5
69.73+2.87
10.97
7..04+1.611
7..68
PC
NeurIPS 20
(1shida el al..
66.37-2.54
63.01
66.37+2.33
2.296
Forward
ECCV 2018
al 2018
73.68
71.50
73.27
7.574
NN
ICML 2019
[shida ct 2019
26.73=1.04
17.52+1.49
22.8++1.24
2.037+1.04
GA
ICML 2019
[shida et 2019
18.94+3.40
13.63+2.57
6.73+1.14
2..00+2..6
SCL-EXP
ICMI 2020
CCho ct al.
2020
69.73+2.50
70.60
1..8
73.81 1.85
SCL-NI.
ICMI. 2020
CChnu et al.
2120
71.6
73.80
74.69
72.74 1.93
L-W
ICML 2021
(Gao & Zhang
2021
63.72-2.36
63.36+1.119
63.01+2.81
65.3142.33
OP-W
AISTATS 2023
[Lu ct al.
2026
73.10+3.14
62.30+2.11
76.28+2.78
76.99
VALEN
NeurIPS 2021
(Xu 2021
66.02-2.48
65.31
64.4242.98
6..19
The holistic deep PLL algorithms:
VALEN (uu et al.
2021): ) An identification based strategy that uses variational inference to esti-
mate the true label distribution.
PiCO (Wang et al.
2022b): : A PI.I algorrthm that uses the supervised contrastive learning module
to improve model performance.
2022); A PLL algorithm that uses an umbiguity-induce contrastive learning ABLE (Xia et al.
module to improve model performance.
CRDPLL (Wu et al. 2022):
A PLL algorithm that uses consistency regularization improve
model performance.
DIRK (Wu et al. 2024): A PLL algorithm that uses knowledge distillation and contrastive learn- ing to improve model performance.
The deep noisy PI.. algorithms include:
FREDIS QQiao al. 2023b); A noisy PLL algorithm that filters false positive labels while
including false negative labels.
ALIM (Xu al. 2023a): noisy PLL algorithm that uses a weighted sum of the labeling confi- dence of candidate and non-candidate label sets the target. We assume that the true noise rate is accessible.
PiCO+ (Wang et al. 2024a): A noisy PLL algorithm using distance-based clean sample selection semi-supervised contrastive learning. We assume that the true noise rate is accessible.
E.2 IMPLEMENTATION DETAILS
All the algorithms were implemented in PyTorch Paszke et al., 2019) and experiments were conducted with a single NVIDIA Tesla V100 GPU. We used the Adam optimizer (Kingma & Ba 2015). We ran 60,000 iterations for the image datasets, 20,000 iterations for the Soccer Player, Ital- ian, Yahoo! News, and English datasets, and 10,000 iterations for the other datasets. We recorded the performance on validation and test sets per 1,000 iterations. We used three random data splits for PLCIFARIO and five random data splits for tabular datasets. For each data split, selected 20 random hyperparameter configurations from a given pool. rameter configurations for all algorithms.
the details of the hyperpa-
20

---

Published as a conference paper at ICLR 2025
Table 8: Hyperparameters, their default values, and distributions for random search.
Condition
Parameter
Default Value
Random Distribution
learning rate hatch size weight decay
0.001
10Uaioom! 4.5,-2.6)
ResNet
256
20tomm65.))
0,00001
Untoom! 6,-3)
learning rate
0.001
10 10
MLP
batch size
128
2 20)omm(3,))
weight decay
0,00001
10 6,-33
window size warm up epoch initial threshold
5
RandomChoice (3. 4,5,, SandomChoicce/f 15, 20J)
POP
0.001
Unitom 4.5 2.5)
step size
0.001
10 4., 2.5)
ABS-GCE
4
0.7
0.7
MCL-GCE
9
0.7
LWS
leveraged weight
2
RandomChhoice(1,2
IDGP
up epoch
RandomCChoice 15, 10])
up iteration
5000
RandomChoocee 10000 2000, 3000, 4000, 5000))
VALEN
number of nearest neighbors
3
RandomChoice [3, 4])
warm up iteration
178
178
of output representation
RandomChoiice([ 128, 256])
size of the queue
8192
RandomChoiced 4096, 8192)
PICO
model update momentum
0.999
RandomChoice (0.9,,..90) 0.999]
prototype calculation momentum
weight of the contrastive loss
0.5
RandomChhoice(0.5,,
size of output representation weight of the ambiguity-induu loss
128
RandomChoicee [64, 128,256]
ABLE
1
RandomChoice [.)]
temperature
Rundom@Choice((0.00,..00 0.00
CRDPLL
weight the consistency
DIRK
teacher model update momentum
0.99
RumbomCChoiice(355503.....00))
initial refinement threshold
0.000001
0.000001
initial disambiguation threshold
1
size of refinement threshold
0.000001
0.1
step size of disanbiguation threshold number changed entries
0,000001
0.1 [knd0onchhee(k55.33......
FREDIS
500
RandomChoice((5)) 300, 800, 1000))
time the number of chunged entries
2
(mndomChoicee1,,,23...
epoch of updating intervals weight the consistency loss
20
RandomChooceel 10.20])
RandomChoiced 10])
weight of the supervised loss
1
up iteration size of cutput representation size of the queue
178
178
128
RandomChoice 256)
8192
RandomChoicee [092)
model update
0.999
RundomCheiice (,9,, 0.999]
ALIM
prototype calculation momentum
0.99
RandomChoocee (..99)
weight the contrastive loss starting epoch of denoising weight of the mixup loss
0.5
RandomChoiced 1.0,
RundomChoice [20, 80, 140))
1
warm up iteration
250
RandomCChoice(11.. (1.
size of output representation
128
RandomChoice! [64, 125, 256))
of the queue
8192
RundomChoice [1096, 8192)
model update momentum prototype calculation momentum
0.999
0.99
RandomCCoiiee(11P9,, (0.99.0.]]
weight of the contrastive loss selection ratio for cleun sample
0.5
RunãomCheiice(O...0
PICO+
1
RundomChoiice((I.. 0.,,,
start iteration of I-NN augmentation
5000
Random(Choice!! 22, 100, 100,))00)
number nearest neighbors
RandomChoice// 1G])
temperature of label guessing
0.07
RundomChoice! (aloomCCooice(01....... (..1,
weight for the losses of unreliable examples weight for losses of mixup loss
0.1
Handom(Chiceed [..1)
RandomChoice// (2)
F DETAILS OF EXPERIMENTAL RESULTS
Table and Tables to T6 show the experimental results on tabular datasets.
21

---

Published as a conference paper at ICLR 2025
Table 9: Classification accuracy (tda of each algorithm MSRCv2 with different model
selection criteria.
Algorithm
Venue
CR
AA
ES
PRODEN
ICML 2020
Lv et al.
2020
0..5
13.11+1111
51.82+0.65
52.39 1.16
CAVL
ICLR 2022
Zhang al
202
0.84
45.00+
48.64+1.56
46.14+ 1.63
POP
ICML 2023
Xü al.
2023b
5.93
52.05+0.733
52.61
54.09
ABS-MAE
TPAMI 2024
(Lv
2024a
1.82
44.09+1.80
46.48+1991
48.18=
ABS-GCE
TPAMI 2024
(LV
2024a
1..30
46.93
49.55+1.31
1.13
EXP
ICML 2020
(Feng al.
2020a
44.77-1.73
45.57:0.94
1.31
45.57
MCL-GCE
ICMI, 2020
(Feng ct al.
2020a
44.43-+1116
44.77#1.199
47.16+0.28
46.93+1.10
MCL-MSE
ICML 2020
(Feng cl al.. 2020a
4..6
33.6442.08
43.75+1.27
45.80
CC
NeurlPS 2020
[reng ct al. 20220b
52.16=1.32
$2.27
$2.27+1.60
53.07 1.25
LWS
ICML 2021
(Wen et al.
2021
47.50
46.59+0.53
19.77
49.32 1.03
IDGP
ICLR 2023
Qiao al. 2025a
52.61
50.45+2.07
52.05+1.14
52.84 1.56
PC
NeurlPS 2017
(Ishida
2207
41.82+1.33
43.98+2.02
44.09+0.73
44.89+1.24
Forward
ECCV 2018
Yu 2018
49.89
5..5911.64
51.1410.53
50.57 1.80
NN
ICML 2019
(Ishdda
2019
1..86
16.8211.33
16.70+0.83
20.23 1.14
GA
ICML 2019
[shida
1..0
13.52+0.67
10.91
14.55 1.22
SCL-EXP
ICML
(Chou ct al
20200
18.86
49.8941.20
48.98+1.70
49.32+1.67
SCL-NL
ICML 2020
(Chou ct
2020
51.25+1.02
$1.70::141
49.32+1.11
51.36 1.24
L-W
ICML 2021
Gao & Zhang
202
1..77
41.9310.74
45.34+0.90
0.855
OP-W
AISTATS 2023
[Tu 202
49.89
27.50+9.01
11.59+1.55
$1.93+1.42
VALEN
NeurlPS 2021
[1 et
2021
48.30+1143
49.09+1.29
48.4140.93
49.89+0.81
Table 10: Classification accuracy (meant:st)) of each algorithm on Mirflickr with different model selection criteria.
Algorithm
Venue
CR
w/ AA
w/ OA
w/ OA & ES
PRODEN
ICMI. 2020
LL al.
2020
66. 1.61
65.40
66.47+1.63
66.69 1.29
CAVL
ICLR 2022
Zhang
2022
63.68:10.69
6..75+1140
1..73
1.17
POP
ICML 2023
Xu
2023b
6.14
64.53
6..62+1.05
6..40+
ABS-MAE
TPAMI 2024
(L
63.60+0.99
58.49+1.92
63.81
6..04+
ABS-GCE
TPAMI 2024
al.
20245
13.96+1.71
53.02+0..8
54.17+0.90
18.27+
ICML
|eng ci al
2020a
64.46+1.39
57.55+3.13
64.82+1.32
6..53+1.76
MCL-GCE
ICML 2020
Feng ci al.
2020a
55.40+1.35
53.17#1.29
54.10+0.89
56.62+0.22
MCL-MSE
ICML 2020
Feng el al.
2020a
12.23+
35.0415.62
1..68
1..27
CC
NeurIPS 2020 [Tng et al.
20220b
65.04
63.6741.37
64.3210.65
6..98
LWS
ICML 2021
WWn ct al 2021
64.96+1.30
38.78+5.29
6..04+1.23
65.54+
IDGP
ICLR 2023
Qiia al.
20256
66.12+2.07
68.35+1.222
68.13+0.98
69.06++091
PC
Ne 2017
(Ishida et al. 2017
6..45+1.114
18.78+1.411
62.37+1.07
1..02+1.61
Forward
ECCV 2018
YY al. 2018
65.32+1.45
64.68:2.116
64.96+1.119
1..5
NN
ICML 2019
Ishida
2019
19.93
9.28+1119
18.06+1.22
24.96
GA
ICML 2019
Ishida
2019
21.37+10.2
11.3744.01
17.8410.39
1.
SCL-EXP
ICML 2020
Chou
13.31+1.229
59.42+4.338
6..60+1103
65.32++0.86
SCL-NL
ICML 2020
63.60+1.13
63.74+1.17
67.12+1.15
66.83+0.62
L-W
ICML 2021
Gao Zhang 2021
17.55+1.31
$2.09+3.57
57.3442.71
60.07+2.18
OP-W
AISTATS
Liu 2023
64.32+1.47
15.32+9.24
64.53+1.24
65.83+1.21
VALEN
NeurIPS 2021
[Xu et al.
2021
18.56
59.42++1.10
18,78+1..0
60.58
22

---

Published as a conference paper at ICLR 2025
Table 11: Classification accuracy (mean+sstd) of each algorithm Birdsong with different model
selection criteria.
Algorithm
Venue
w/ CR
w/ AA
OA
w/ OA & ES
PRODEN
ICML 2020
Lv et al
2020
71.64:0.66
70.16+0.88
71.44+0.65
72.32+ 0.89
CAVL
ICLR 2022
Zhang al.
202
0.79
66.04=0.336
66.56+0.41
6..08=0.68
POP
ICML 2023
Xu et al. 20.5b
71.80
69,96:10.58
72.60+0.92
72.24=0.73
ABS-MAE
TPAMI 2024
(LV al
2024g
6..20
6..48
68.2010.70
68.28
ABS-GCE
TPAMI 2024
(L.v
69.04=0.55
67.96++0.79
69.4410.66
70.16+0.94
EXP
2020
(Feng
2020a
64.16=0.74
6..64+1.222
65.20+0.70
65.24
MCL-GCE
ICML TTng at.
2020a
65.92+1.40
64.96:10.95
64.92.10.3
67.56
MCL-MSE
ICML 2020
(Feng et al.,
2020a
65.64+0.92
18.48
67.04+0.52
66.68+0.32
CC
NeurIPS (Feg ct al. 20220b
70.88=0.85
68.68
72.24+0.57
71,96+0.55
LWS
ICML 2021
(Wen et al. 2021
66.04=10.3
56.48+5.22
65.8440.61
06.04
IDGP
ICLR 2023
Qiao et al.
20358
72.48
69.72+0..0
73.40+0.64
0..24+0.62
PC
NewrIPS 2017
(Ishdaa al. 20017
68.92
1..
70.08+0.52
0.722
Forward
BCCV 2018
Yu et al.
2018
69.84=0.77
69.80+0.41
70.00+0.50
0.68
NN
ICML 2019
[shida
2019
18.68
16.04.10.80
18.0010.70
20.7
GA
ICML 2019
Ishida
2019
1.20
1..04
13.2410.73
19.84 1.15
SCL-EXP
ICML 2020
CCou et al.
2020
7..12
68.84
70.72+10.38
7.60
SCL-NL
ICML 2020
(Chou ct al 20200
70.22=0.62
70.1640.995
70.28+0.50
70.28=0.78
L-W
ICMI. 2021
GGao Zhang
2021
62.48=0071
61.80:10.33
66.4840.82
67.60+0.81
OP-W
AISTATS 2023
Lau et al.
2025
69.60
51.72.9.37
69.6010.74
71.0
VALEN
NeurlPS 2021
Xu et al.
2071
66.76=1.62
66.76+1.07
66.76+0.67
68.44+1.00
Table 12: Classification accuracy (mean-ist)) of each algorithm Malagasy with different model
selection criteria.
Algorithm
Venue
w/ CR
w/ AA
OA
w/ OA & ES
PRODEN
ICMI. 2020
Lv et al.
2020
67.19
66.93+2.41
71.83+0.81
71.15
CAVL
ICLR 2022
Zhang al.
2022
6..4
64.71+1.21
68.5540.98
69.34 1.08
POP
ICML 2023
Xu al. 2025b
62.52+1.89
65.39++2.12
70.47+0.65
70.43+0.81
ABS-MAE
TPAMI 2024
(124(LV
2024g
58.46+3.15
54.39+3..29
66.52+00.2
66.06+0.44
ABS-GCE
TPAMI 2024
(Lv
2024a
60.64
59.70++..10
63.47+1.60
6..76+1.13
ICML 2020
(Feng al..
2020a
0.23+0.10
0.23+0.100
0.23+0.10
0.23=0.10
MCL-GCE
ICML 2020
(Feng al.
2020a
0.23+0.10
0.23+0.10
0.23+0.10
0.23=0.10
MCL-MSH
ICMI. 2020
(Fnng al., 2022
0.2340.10
0.2340010
0.23++0.10
0.23=0010
NeurlPS ( ct al 20220b
57.82+0.78
60.53+2.32
70.66+0.66
71.41
LWS
ICML 2021
(Wen el al.
2021
17.34
45.50:14.77
63.54+1.55
65.08 1.02
IDGP
ICLR 2023
Qiao et
20258
69.94-0.70
1..50+145
70.06+0.62
70.70+0.83
PC
NeurIPS 2017
(IShida et al. 2017
6..01
1..52+
19.15+
6..38
Forward
ECCV 2018
Yu ct 2018
61.51
62.79+2.60
69.7240.44
69.38+0.79
NN
ICML 2019
Ishida
2019
1..8
14.43+2.04
2..34+1.000
25.54
GA
ICML 2019
Ishida
2019
18.83
19.47+1.68
9.04
19.55 1.62
SCL-EXP
ICML. 2020
Chou et al.
61.73+2.40
57.48+1.38
69.76+0.36
69.72+0.87
SCL-NL
ICML
Chou cl al
2020
57.36.00.99
61.7742.34
68.66+1.05
68.89
L-W
ICML
Gao Zhang
2021
17.33+1.74
58.3842.46
64.71+1.11
66.70=1.03
OP-W
AISTATS 2023
Liu et al.
2025
58.95
0.64:10.18
70.2110.39
69.3440.74
VALEN
Ne 2021
Xu Lu al.
2021
65.76+1123
60.2342.71
68.7810.75
68.63+0.64
23

---

Published as a conference paper at ICLR 2025
Table 13: Classification accuracy (meantstd) of each algorithm on Soccer Player with different model selection criteria.
Algorithm
Venue
w/ CR
w/ OA
ES
PRODEN
ICML 2020
Lv el al. 2020
0..9
54.65=0.55
55.02=0.52
05.38::0.49
CAVL
ICLR 2022
Zhang et al 202
51.89 1.08
11.52+1.09
51.26=1.04
51.67=1.02
POP
ICML 2023
Xu
2025b
0..7
54.57+0..2
54.82=0.56
55.02:+0.40
ABS-MAE
TPAMI 2024 LL 2024g
48.90
48.75:10.54
4..90
48. 0..90210.62
ABS-GCE
TPAMI 2024
al.
55.33=0.48
48.95:+0.55
55.70=1.68
55.90+0.67
EXP
ICMI. 2020
Feng et al.
49.05=0.55
48.78+0.54
49.04
48.92+0.500
MCL-GCE
ICML 2020
Feng et al.
2020a
53.64+0.57
49.51
5..56=0062
53.40+0.57
MCL-MSE
ICML 2020
Feng cl al. 2020a
52.22:0.555
$1,41+0.28
53.03=0.59
53.65
CC
NeurIPS 2020
(Fenng et al. 22020b
54.1520.64
54.86+0.33
54.93-0.38
44.71+0.42
LWS
ICMI. 2021
Wen al.
2021
52.55=0.48
48,9940.51
52.60=0.45
02.6
IDGP
ICLR 2023
1Qao cl al. 2023a
54.39=0.71
54.1610.58
54.97=0.42
08.13
PC
NeurIPS 2017
(Tsurd el al. 20177
54.85=0.76
48.75+0.54
54.70=0.72
54.45=00.64
Forward
ECCV 2018
el 2018
50.35=0.93
49.93+1.01
50.57=0.86
50.47
NN
ICML 2019
[hidd c
2019
11.10
3.04
9.97
11,10+0.38
GA
ICML 2019
IShida et 2019
6.06+1046
4.95
5.06
6,02+00.50
SCL-EXP
ICML 2020
Chou
20:0
49.82.1.87
49.35++0.87
49.51
49.42:10.84
SCL-NI.
ICMI. 2020
Chou
2020
00.5
49.98+1.06
50.58=0.91
50.55
L-W
ICMI, 2021
Gao & Zhang 202
49.41=0.69
48.84+0.59
49.60=0.60
49.83+0.66
OP-W
AISTATS 2023
(Liu 2024
50.77
10.54
51.12=0.61
50.93
VALEN
NeurIPS 2021
(Xu ct 2021
52.1320.54
51.97+0.52
52.15=0.54
52.30+0.59
Table 14: Classification accuracy (mean std) of each algorithm on Italian with different model selection criteria.
Algorithm
Venue
w/ CR
w/ AA
w/ OA
w/ OA & ES
PRODEN
ICML 2020
Lv et al.
2020
68.89 00.49
69.32-10.00
71.80+0.95
72.98 0.66
CAVL
ICLR 2022
Zhang el al.
2022
68.83-0.96
68.1240.72
70.71
7..62+0.60
POP
ICML 2023
Xu eu al.
20.3b
68.48+0.67
70.19+0.36
71.56+1.03
71.78+0.91
ABS-MAE
TPAMI 2024
(Lv et al
2024a
67.60=0.32
66.92-0.84
70.11+1.31
1.207=1.26
ABS-GCE
TPAMI 2024
(Lvv
2024a
66.76=0.64
65.36:10.68
69.60+0.57
70.90=0.54
EXP
ICML 2020
[Feng
2020a
6.65
67.28
71.43:10.4
7..6410.46
MCL-GCE
ICMI. 2020
(Feng
2020
66.22.0.72
66.03+00.5
69.6440.41
70.14=0.31
MCL-MSE
ICMI. 2020
(Feng
66.48=0.116
65.86+1.112
68.29+0.52
69.16=0.47
CC
NeurIPS 2020
[Teng el al. 2020b
6..96-11.03
66.89:1.11
71.40+0.32
72.09
LWS
ICML 2021
eW al.
2021
71.65+0.45
58.70+5.74
73.36+0.50
73.68+0073
IDGP
ICLR 2023
Qiao al. (Ishida et al 2017
202.5a
68.43-10.50
67.43+0.82
70.08+0.19
7..73+0.42
PC
Ne 2017
70.10=1..69
67.41+1.23
71.22+0.33
71.73+0.42
Forward
ECCV 2018
ef al. 2018
68.28+1.29
18.28+1.29
71.20+0.86
7..63+0.66
NN
ICML 2019
[shida al.
2019
3..92=1.92
17.62+5.35
23.7810.57
33.75
GA
ICMI. 2019
Ishida
2019
11.62+1.61
11.98+1..4
5.88+0.25
11.98 1.43
SCL-EXP
ICML 2020
Chou ct
202()
67.34+0.31
67.34+0.31
69.03+0.54
69.08+0.69
SCL-NL
ICML 2020
Chou al.
2020
67.59
67.59.0.33
68.8220.69
7..23+
L-W
ICML 2021
Gao Zhang
2021
67.43+0045
68.78+0.43
68,90+0.20
69.5610.33
OP-W
AISTATS 2023
Liu al.
2025
65.70+11110
1.11+0.38
1.004+
71.30+
VALEN
NS 2021
[Xu al.
2021
67.97-0.86
67.29-1.42
69.3140.64
71.12+0.80
24

---

Published as a conference paper at ICLR 2025
Table 15: Classification accuracy (mea each algorithm on Yahoo! model selection criteria.
News with different
Algorithm
Venue
w/ CR
OA
ES
PRODEN
ICML 2020
Lv et al.
2020
0.58
6..29+0.54
66.71+0.58
06.51
CAVL
ICLR 2022
Zhang al 2022
1..77
57.14+1.64
57.30+1.55
58.30+ 1.57
POP
2023
2023
Xu
66.42.00.4
65.76:10.66
66.02+0.57
66.17:0.64
ABS-MAE
TPAMI 2024
2024a
55.70
48.5313.62
56.11+1.63
1..15
ABS-GCE
TPAMI 2024
2024a
58.03=0043
57.79410.43
58.5410.95
58.4810.36
EXP
ICMI. 2020
(Feng ct al.
2020a
50.37=2.06
47.01+3.14
50.13+2.11
50.4442.05
MCL-GCE
ICML 2020
Feng cl al.
2020a
53.91=0.34
53.91
53.9940.50
54.1340.34
MCL-MSE
ICML 2020
(Feng cl al.. 2020a
54.49=0.64
10.46:+1.07
52.12+0.70
54.7440.70
CC
NeurlPS 2020
(Feng et al 20:20b
67.02=0.32
65.43+0.56
66.49+0.45
06.69+0.40
LWS
ICML 2021
(Wen et al. 2021
67.64.0.73
39.87+9.64
67.18+0.55
67.46=0.32
IDGP
ICLR 2023
Qiao al.
2023a
66.10=0.63
65.66:10.69
62.55+0.75
65.99+0.31
PC
NeurIPS 2017
[shida al 22017
58.19=0.41
54.07
58.2210.46
18.17
Forward
HCCV 2018
Yu
50.70=
50.56+1.33
50.62+1.40
50.70
NN
ICML 2019
(shida
23.80=0.24
1.7740.16
23.23+0.27
24.19+0.20
GA
ICML
[shida
2019
13.30-0.31
13.3040.31
12.91
13.30+0.31
SCL-EXP
ICML 2020
CCou et
2020
10.30+1.266
50.35+1.23
50.57+1.28
50,70+1.21
SCL-NL
ICML 2020
(Cou et al
2020
1.25
1..90
50.76+1001
50.85
L-W
ICMI. 2021
Gao & Zhang
202
43.95=0067
42.98+0.66
42.71+0062
44.60+0070
OP-W
AISTATS 2023
[Liu al
2025
57.37
45.21+9.93
57.35+1.05
57.49
VALEN
NeurIPS 2021
Xu et al.
2021
56.97
56.9811.45
56.0221..2
57.8210.50
Table 16: Classification accuracy (meantstd) of each algorithm on English with different model
selection criteria.
Algorithm
Venue
w/ CR
w/ AA
w/ OA
w/ OA & ES
PRODEN
ICML. 2020
(L 2020
2020
7..01
03.78
73.62=0.15
03.70:+0.26
CAVL
ICLR 2022
(Zhang et al
2022
73.7
73.58
74.15
74.1910.25
POP
ICML 2023
Xu 20233
7..04
73.99+0.17
73.78=0.31
74.44-0.36
ABS-MAE
TPAMI 2024
LL al 20,248
73.07
72.37+0.44
73.20-0.13
73.31
ABS-GCE
TPAMI 2024
(L al.
2024a
73.52+0.30
73.12+0.26
74.47=0.16
74.57+0.23
EXP
ICML 2020
Feng et al 2020a
73.18+0.23
72.73+0.22
73.75=0.27
73.61
MCL-GCE
ICML 2020
Feng el al 2020a
73.52+0.24
73.52+0.51
74.09=0.26
74.35
MCL-MSE
ICMI. 2020
(Fong ci al. 20,20a
73.70=0.24
73.37+0.48
74.22-0.399
73.96+0024
CC
NeurlPS 2020
(Feng et al.
20/20
73.83=0.25
73.77+0.29
73.62=0.08
73.87+0.19
LWS
ICML 2021
(Wen el al. 2021
73.31
64.1113.99
73.98
73.91
IDGP
ICLR 2023
Quao el ul. 20233
74.01=0.26
73.82+0.24
73.90=0.35
74.44+00223
PC
NeurIPS 2017
(Ishida
2017
73.47=0.29
72.82+0.30
73.38=0.45
73.74+0.35
Forward
BCCV 2018
el 2018
73.40+0.21
73.60
0..40
NN
ICML 2019
IShida al. 2019
54.18
53.82-10.76
1.42
14.18
GA
ICML 2019
Ishida
1.97
34.77+1.91
0.84
1..77+1.91
SCL-EXP
ICMI. 2020
Chou
73.84
73.44+0.40
74.06=0.26
73.84+0.25
SCL-NL
ICML 2020
al.
2020
74.01
73.78+0044
74.04-0.31
74.17+0.24
L.W
ICML 2021
(Ga & Zhang
73.84
73.13+0.54
74.02-0.39
74.1440.229
OP-W
AISTATS 2023 (Liu ct al
2023
73.88+0.39
31.8++15.01
73.97-0.35
00.05
VALEN
NeurlPS 2021 (Xu et al.
2021
73.20=0041
72.86+0.40
73.39-0.23
73.7740.24
25


Reviews:
[{"review_id": "lBM8vMXHGK", "reviewer": "Reviewer_oKAk", "summary": "The paper proposes the partial label learning (PLL) benchmark PLENCH that standardizes experiments of PLL. The related work in PLL mostly selects hyperparameters based on a normally labeled validation dataset, which is highly unrealistic. The authors  propose three model selection criteria that are theoretically analyzed and empirically investigated. Furthermore, the authors propose a new dataset PLCIFAR10 that consists of human-annotated partial labels. It serves for a more realistic evaluation, as  partial labels are not synthetically generated. In their experiments, they include an excessive amount of PLL algorithms across several datasets.", "questions": "**Questions**\n- Can you provide feedback to the weaknesses I mentioned above?\n- I was wondering if there is access to the identities of the annotators and how the authors ensured their privacy?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written and accessible.\n- Model selection in PLL is a relevant, underexplored topic.\n- The proposed dataset in a realistic PLL setting adds significant value to the PLL literature.\n- Despite the model selection criteria's simplicity, the authors establish a solid theoretical link to expected accuracy and demonstrate its benefits.\n- The experiments are extensive, covering many PLL algorithms from top venues.", "weaknesses": "While I think that this paper is really good, it would be nice if the authors can clarify the following weaknesses:\n- **Model selection criteria:** While the criteria are intuitive and theoretically analyzed, presenting Oracle Accuracy as a contribution is problematic. As also noted by the authors, OA is standard in the literature and, thus, it should be presented as a baseline, not as a novel contribution. IMO the contribution statement in the introduction needs to be slightly adapted.\n- **Use of early stopping:** For me, it is unclear how ES is used in the experiments. Lines 245-247 suggest it is applied to everything except OA. So, my question is: Is ES used in the case of CR and AA? Maybe the authors can clarify this a little better in the paper?\n- **Difference between Aggregate and Vaguest not clear:** I can not find a difference in the explanation between the Aggregate and the Vaguest version. The authors state:\n    \n    > The first is PLCIFAR10-Aggregate, which assigns **the aggregation of all partial labels from all annotators to each example**. The second is PLCIFAR10-Vaguest, which assigns **to each example the aggregation of all partial labels from all annotators.**\n    > \n\n    Considering this explanation, there should be no difference between the versions. It would be nice if the authors could clarify (or fix) this.\n\n- **Minor Stuff:**\n    - The subfloats in Figure 2 are not aligned at the top.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8501B87DkK", "reviewer": "Reviewer_RWfx", "summary": "This paper introduces PLENCH, the first Partial-Label learning bENCHmark for comparing\nstate-of-the-art deep PLL algorithms.\n\nAuthors create Partial-Label CIFAR-10 (PLCIFAR10), an image dataset of human-annotated partial labels collected from Amazon Mechanical Turk. Furthermore, paper investigates the model selection problem for PLL, and proposes model selection criteria with theoretical guarantees.\n\nKey takeaways indicate that simpler algorithms can sometimes match or exceed complex ones, no single algorithm excels in all scenarios, and model selection practices are crucial for fair comparisons in PLL studies.", "questions": "What changes do you expect to see on larger datasets and with more complex deep neural networks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors make a significant contribution by being the first to systematically investigate model selection problems in partial-label learning (PLL), addressing a gap in the existing literature. The paper presents a comprehensive PLL benchmark that includes 27 algorithms and 11 real-world datasets.\n- A notable strength of the paper is the introduction of PLCIFAR10, a new benchmark dataset for PLL featuring human-annotated partial labels. This dataset provides an effective and realistic testbed for evaluating the performance of PLL algorithms in scenarios that closely mimic real-world conditions.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRn2XupwEQ", "reviewer": "Reviewer_v9rL", "summary": "The paper investigates the inconsistent evaluation of different partial-label learning methods and proposes a new benchmark for a fair evaluation. In particular, the standard setting in PLL does not have a validation set that consists of ground truth labels, while many recent studies employ such a clean validation set to tune their hyper-parameters. The paper, therefore, proposes \"surrogate\" accuracy metrics to replace the conventional prediction accuracy (because the conventional accuracy is intractable in training) in the validation to fine-tune hyper-parameters. Those alternative metrics are theoretically proved to be \"close\" to the exact one. Empirical evaluation shows that the many prior methods in PLL can achieve high performance or even out-perform most recent methods in the same benchmark.", "questions": "As explained in the **Weaknesses**, could the authors clarify the following concerns:\n- Provide a comprehensive and coherent description of the toy example in Figures 1a and 1b (or the one at line 75)\n- Clarify further the proof of the Proposition 1 because it seems to be the main building block for the subsequent theoretical results.\n- Clarify the differences between the two synthetic datasets at line 286.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has pointed out a critical issue in PLL research where many recent studies try to achieve the state-of-the-art results by considering an unfair setting when benchmarking with other prior methods. This potentially misleads the research direction where the performance of those prior methods can even out-perform many recent PLL approaches.\n\nIn the standard setting of PLL, the ground truth labels are not available in validation sets. Hence, it is difficult to tune the hyper-parameters of the model of interest. Hence, the paper proposes some alternative metrics as a drop-in replacement of the standard accuracy to optimise the hyper-paramters of interest. Empirical evaluation shows that this approach results in high-performance models in many benchmarks. Note that this is a strong contribution of the paper because the proposed alternative metrics are proved to be \"close\" to the standard prediction accuracy under certain conditions.", "weaknesses": "**Confusing terminologies**: *model selection* vs *hyper-parameter tuning*\n\nIn the paper, the authors argue that the mismatch of the validation setting results in bad model selection. In fact, to what I understand, what that means is hyper-parameter tuning, not model selection. Hyper-parameter tuning is indeed a subset of model selection, but not the other way around. Model selection is a terminology in machine learning and also means things like variable selection and actual model choice (functional form, for example, should it be ResNet or DenseNet or a transformer), whereas the one discussed in the paper is to find the best model within a family. Thus, I suggest the authors to use the correct terminologies to reduce the confusion.\n\n**Confusing toy example in Figures 1a and 1b**\n\nThe bar colors are inconsistent with the legend, making it hard to understand. In addition, why shouldn't the x-axis include the name of each method, instead of letting them floating inside the plotting area. The pilot (or toy) experiment mentioned at line 75 with the current description is hard to understand and should provide further details. For example, what are training and validation in Figures 1a and 1b mean? What are the meaning of the subcaptions in Figure 1? Are they the name of a method to obtain labels or the name of a dataset? I understand that the purpose is to demonstrate that evaluating methods on two different settings: with and without clean validation set leads to different performance. However, the explanation is non-coherent and hard to understand.\n\n**Complicating notations**\n\nEq. (5): accuracy is already an expected (or average) value itself as shown in the right hand side of Eq. (5). Thus, the notation: $\\mathbb{E}[\\mathrm{ACC}(f)]$ seems cumbersome, and can be simplified to ACC only.\n\n**Proof of Proposition 1:**\n\nThe second equality is unclear. Why is it possible to make the subtraction of two expectations be a single expectation? This is quite problematic because according to Lemma 1, S is dependent on x and y.\n\nAnother issue related to Proposition 1 is Lemma 1. A lemma is introduced as a stepping stone to use its **conclusion** to prove meaningful results (e.g., a theorem). Although Lemma 1 is introduced to prove Proposition 1 at line 192, I do not see any connection to the proof of the Proposition 1. In particular, the proof just uses the assumption (or the definition to be exact) of Lemma 1 about the *ambiuguity degree* to include in the result of Proposition 1. Hence, stating Lemma 1 to lead to the result in Proposition is misleading.\n\n**Unclear distinction between two synthetic datasets**\nLines 286 and 287: It is unclear about the differences between the two approaches: aggregate and vaguest. In the text, they are almost equivalent or identical. Could the authors provide detailed clarification about their differences?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdK9IA7b1W", "reviewer": "Reviewer_GD1c", "summary": "This paper presents three key contributions: (1) an investigation of the model selection problem in the partial-label learning (PLL) setting, proposing several metrics such as covering rate (CR), approximated accuracy (AA), oracle accuracy (OA), and OA with early stopping (ES); (2) the creation of a partial label dataset based on CIFAR10, named PLCIFAR10; and (3) a benchmarking of multiple algorithms across several datasets. Overall, this paper serves as a comprehensive benchmarking study and does so effectively.", "questions": "I have a few questions and suggestions that I believe could help improve the paper:\n\n1) Could the authors comment on the differences and similarities between their work and [1,2]? Both papers appear to address benchmarking in the PLL setting and should be discussed to provide a clearer context.\n\n2) My major concern is that one of the main contributions is the proposed model selection criterion, while there is no experimental results to support its validity. Could the authors provide experimental evidence showing that the proposed criteria (CR, AA, OA, OA w/ ES) lead to better performance on the test dataset? Specifically, for a given ML method, does the model selected with CR (or AA, OA, OA w/ ES) outperform the model without selection? \n\n3) While I understand there is no universally best model selection criterion, would it be possible for the authors to include a numerical comparison showing how often each criterion (CR, AA, OA, OA w/ ES) achieves the best performance? For instance, in Table I, there are 27 algorithms. How many times does the best performance come from models selected with CR, AA, OA, or OA w/ ES?\n\n4) To improve clarity, could the authors provide further explanation of $p(x,y)$ and $p(S|x,y)$ in Eq (4)?\n\n5) I want to bring two particular papers [3,4] into the author's sight.\n\n[1] Lars Schmarje, et al., \"Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation,\" NeurIPS 2022.\n\n[2] Mononito Goswami, et al., \"AQuA: A Benchmarking Tool for Label Quality Assessment,\" NeurIPS 2023.\n\n[3] Zhengqi Gao et al., \"Learning from multiple annotator noisy labels via sample-wise label fusion,\" ECCV 2022.\n\n[4] Ashish Khetan, et al., 'Learning from noisy singly-labeled data,' ICLR 2018.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive and easy to follow.", "weaknesses": "See my comments in 'Question'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)