Candidate Label Set Pruning: A Data-centric Perspective for Deep Partial-label Learning
Authors: Shuo He
Chaojie Wang
Guowu Yang
Lei Feng
Venue: ICLR 2024 oral (2024)
Keywords: partial label learning
label disambiguation
candidate label set pruning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We pioneer a new task called candidate label  set pruning (CLSP) for the problem of deep partial label learning, and propose a new CLSP algorithm.

Abstract: Partial-label learning (PLL) allows each training example to be equipped with a set of candidate labels. Existing deep PLL research focuses on a \emph{learning-centric} perspective to design various training strategies for label disambiguation i.e., identifying the concealed true label from the candidate label set, for model training. However, when the size of the candidate label set becomes excessively large, these learning-centric strategies would be unable to find the true label for model training, thereby causing performance degradation. This motivates us to think from a \emph{data-centric} perspective and pioneer a new PLL-related task called candidate label set pruning (CLSP) that aims to filter out certain potential false candidate labels in a training-free manner. To this end, we propose the first CLSP method based on the inconsistency between the representation space and the candidate label space. Specifically, for each candidate label of a training instance, if it is not a candidate label of the instance's nearest neighbors in the representation space, then it has a high probability of being a false label. Based on this intuition, we employ a per-example pruning scheme that filters out a specific proportion of high-probability false candidate labels. Theoretically, we prove an upper bound of the pruning error rate and analyze how the quality of representations affects our proposed method. Empirically, extensive experiments on both benchmark-simulated and real-world PLL datasets validate the great value of CLSP to significantly improve many state-of-the-art deep PLL methods.

Published as a conference paper at ICLR 2024
CANDIDATE LABEL SET PRUNING: A DATA-CENTRIC PERSPECTIVE FOR DEEP PARTIAL-LABEL LEARNING
Shuo He' Chaojie Wang* Guown Yang'*
Lei Feng2*
'University of Electronic Science and Technology of China
Nanyang Technological University
shuohel2388 com, chaojie.wangentu edu. guowuêuestc. edu...dduucco cn, f 1 com
ABSTRACT
Partial-label learning (PLL) allows each training example to be equipped with a set of candidate labels where only one the true label. Existing deep PL.L research focuses on a learning-centric perspective to design various training strategies for label disambiguation i.e., identifying the concealed true label from the candidate label set for model training. However, when the size of the candidate label set becomes excessively large, these learning-centnn centric strategies would be unable find the true label for model training, thereby causing performance degradation. This motivates us to think from a data-centric perspective and pioneer a new PLL- related task called gandidate label set pruning (CLSP) that aims to filter out cer- tain potential false candidate labels in a training- free manner. To this end, we propose the first CLSP method based on the inconsistency between the represen- tation space and the candidate label space. Specifically, for each candidate label of training instance, if it is not a candidate label of the instance's nearest neighbors in the representation space, then it has high probability of being a false label. Based on this intuition, we employ a per-example pruning scheme that filters out a specific proportion of high-probability false candidate labels. Theoretically, we prove an upper bound of the pruning error rate and analyze how the quality of rep- resentations affects our proposed method. Empirically, extensive experiments on both benchmark-simulated and real-world PI.L datasets validate the great value of CLSP to significantly improve many state-of-the-an deep PLI. methods.
INTRODUCTION
The effective training of modern deep neural networks (DNNs) commonly requires a large amount of perfectly labeled data, which imposes great challenge on data annotations. However, better data quality normally at odds with lower labeling costs in many real-world applications (Zhou, 2018). To achieve rational trade-off, partial-label learning (PLL), a coarse- grained labeling scheme that allows assigning a candidate label set for each training example (Cour et al., 2011), has attracted increasing attention in recent years (Wang et al., 2022b; Xu et al., 2023b). PLL naturally arises in many real-world applications, such as web news annotation (Luo & Orabona, 2010), automatic image annotation (Chen et al., 2017), and multi-media analysis (Zeng et al., 2013).
Existing deep PLL research focuses on a learning-cent centric perspective to design various training strategies for label disambiguation, i.e., identifying the true label from the candidate label set for model training, e.g., progressive purification (Lvv et al., 2020; Wu et al., 2022), class activation value (Zhang et al., 2022), class prototypes (Wang al., 2022b). However, the excessively large magni- tude of the candidate label set inevitably brings an obstacle to label disambiguation (i.c., identifying the concealed true label) for these learning-centric deep PLL methods, because the misidentified labels could have profound negative impacts on model training throughout the training process.
In this paper, we pioneer a new PLL-related task called candidate label set pruning (CLSP). CLSP maining-free aims to filter out potential false candidate labels in a training free manner, instead of learning DNNs
Corresponding authors.

---

Published as a conference paper at ICLR 2024
with various training strategies for label disambiguation in conventional deep PLL research. To this end, we propose the first versatile training-free CLSP method, based on the inconsistency between the representation space and the candidate label space. Specifically, for each candidate label of training instance, if it is not a candidate label of the instance's nearest neighbors in the representation space, then it has a high probability of being a false label. Based on this intuition, we employ a per- example pruning scheme that filters out a specific proportion of high-probaie probability false candidate labels. Theoretically, we prove an upper bound of the pruning error rate and analyze how the quality of representations affects the proposed algorithm. Empirically, we evaluate the task of CLSP on both
-imulated and real-worke datasets across various PLL settings with eleven state-of- the-art deep PLL methods. Extensive experiments clearly validate the effectiveness of our proposed CLSP method to improve existing PI.L methods.
Our main contributions can be summarized as follows:
A new data-centric perspective for deep PLL. Different from the conventional learning-centric perspective in deep PLL research, we pioneer a new PLL-related task called candidate label set pruning (CLSP) to improve existing deep PLL methods.
A versatile efficient algorithm. We propose the first versatile training- free CLSP algorithm that prunes certain proportion of candidates based on the inconsistency between the representation space and candidate label space.
Theoretical analysis. We theoretically prove upper bound of the per-example pruning error rate and analyze how the representation quality affects the proposed algorithm.
Significant experimental improvements. We perform comprehensive experiments on four bench- marks under various PLL settings with eleven state-of-the-ar deep PLL methods. Significant improvement validates the superiority of the proposed CLSP method.
2 RELATED WORKS
Conventional partial-label learning. Early exploration of PLL before the trend of deep learning techniques focused on small-scale datasets with hand-crafted features (Gong et al., 2022). There are mainly two different strategies to handle candidate labels: averaging and identification. The former treats all candidate labels equally (Cour et al., 2011), while the latter aims to identify the concealed true label from candidate labels (Zhang al., 2016; Xu et al., 2019: Lyu al., 2019). The drawback of this line of work lies in its limited ability to scale to modern large datasets due to its heavy reliance on hand-crafted features, native linear models, and cost-prohibitive optimization algorithms.
Deep partial-label learning. To address the above limitations, recent research of PLL has been focusing on end-to-end training of DNNs (Yan Guo, 2023b; He et al., 2022). Early deep PLL works studied the consistency of the classification risk and the learned classifier (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021). Due to the unsatisfying practical performance of these meth- ods, subsequent works focused on designing various powerful training strategies for effective label disambiguation, based on class activation value (Zhang et al., 2022), class prototypes (Wang et al., 2022b), consistency regularization (Wu et al., 2022). Moreover, recent works started consider more realistic PLL scenarios, including lahel-dependent (Wang et al., 2022b), instance-dependent (Xu al., 2021; Xia et al., 2022; Qiao al., 2023; He et al., 2023b), and noisy (Yan Guo, 2023a; He et al., 2023a) candidate label set generation, and long-tailed data distribution (Wang et al., 2022a; Hong et al., 2023). In particular, recent work (Xu et al., 2023b) proposed a theoretically grounded algorithm to filter out false candidate labels progressively in every epoch. In summary, existing works in both conventional and deep PLL focus on a learning-centerr centric perspective and inevitably suffer from performance degradation when the size of candidate label sets is excessively large.
Dataset pruning and distillation. Dataset pruning (distillation) aims to select (synthesize) a small but representative subset from the entire training data as proxy set without significantly sacrificing the model's performance. Existing methods generally achieved this goal by certain sample-related characteristics, e.g., diversity (Sener & Savarese, 2018), forgetfulness (Toneva et al., 2018), gradient norm (Paul et al., 2021), training trajectories (Zhao et al., 2021), and the gradients and features (Zhao et al., 2021). These methods focused on reducing the number of training data and meanwhile maintaining the data utility, while our studied CLSP aims to reduce the magnitude of candidate label sets and facilitate existing deep PL. methods.
2

---

Published as a conference paper at ICLR 2024
k-NN for partial-label learning. The k nearest neighbors (k-NN) algorithm is well-known parametric classification technique. which also has been applied to PLL (Hüllermeier & Beringer, 2006) by aggregating the candidate labels of nearest neighbors predict the label for each instance. Besides, some conventional PI.. methods (Zhang & Yu, 2015; Zhang et al., 2016; Xu et al., 2019) also employed k-NN-based topological structure in the feature space to refine candidate labels and thus identified the true label for model training. However, these methods focus on learning-centric solutions and still suffer from the excessively large magnitude candidate label sets.
3 A DATA-CENTRIC APPROACH FOR CANDIDATE LABEL SET PRUNING
In this section, we first formally define the task of CLSP, then introduce the proposed method for CLSP, and finally present corresponding theoretical analyses of the pruning error.
3.1 PROBLEM SETUP
We start by introducing some necessary symbols and terminologies to define the task of PLL and CLSP formally. Let X IR be the d-dimensional feature space and y 21.2..... c} be the label space with class labels, respectively. In PLL., each instance æ; E X' is associated with candidate label set Y; E 22" that consists of true label y; y and false candidate label set Y. Yi 3 containing ((Y,l 1) redundant false candidate labels 3/4 V. In particular, the PLL assumption is that the true label y; must be the candidate label set Y. Given a PLL dataset D {æ,, YJ the objective of deep PLL is to learn deep neural network, which can predict the label of test data as accurately as possible. the candidate label set towards each training example that the pruned candidate label set can be leveraged by existing deep PI.L methods to achieve better performance. Below we introduce two important metrics evaluate the pruning performance of any pruning method.
contrast, CLSP aims to reduce the size of
Definition 1 (a-error and 3-coverage pruning). Given a PLL dataset D = {ær, Y:)i= for each candidate label set Y, let Y, denote the set of eliminated candidate labels from Y, and Y; denote the prumed candidate label set of Y; (i.e., Y; =YA Y.). The pruning method is a-error where
[[y.VV:] and B-coverage where 3 M-M-Y
Idcally, the optimal pruning of CLSP is to achieve 0 and 3 1, which can perfectly find out all false candidate labels and thus finally identify the true label.
3.2 THE PROPOSED PRUNING METHOD
Here, we introduce the proposed CLSP method that aims to eliminate potential false candidate labels in the candidate label set of each training PLL instance. Differently from conventional lcarning- centric deep PLL methods that focus on training desired deep neural network, pursue data- centric solution that delves into the PLL data itself. Motivated by the clusterability setting in the conventional supervised learning (Zhu et al., 2022), setting on the PLL dataset, i.c., nearby PLL instances are likely to have the same true label (vs. false candidate label) inside their candidate label sets with high probability (vs. limited probability), which could be formally defined below.
further focus a label distinguishability
Definition 2 ((k, 6k, Pk: llabll distinguishabilit).. A PLL dataset D {(xi, Y satisfies (k, 5x, Pk) label distinguishabiliiyy if: V(z,, Yi) E D, the true label yi E Y, is inside the candi- date label set v,3)) of its each k-NN example (m! v(O)), with probability at least Ōr. and each false candidate label v/. E Yiyys is inside the candidate label set v(5) of its each k-NN example (x) v((i) with probability no more than Pt-
Definition characterizes the candidate label distribution in the local representation space, which has two-fold implications. First, the true label of instances in the local representation space appears in their candidate label sets with a high probability. Second, each false candidate label of instances in the local representation space appears in their candidate label sets with limited probability.
Intuitively, the candidate label of an instance that appears most frequently in its k-NN instances' candidate label sets more likely to be the true label, and the candidate label that rarely appears in
3

---

Published as a conference paper at ICLR 2024
its N--N instances' candidate label sets has high probability of being false label. Motivated by this intuition, we introduce a per-instance label-wise metric Ozy towards the i-th instance (æ,, Y.) associated with the j-th candidate label, to measure the possibility of the j-th candidate label is nor the true label of the instance Ri, which implies that we should prune the j-th label from the candidate label set if Oij is large. Concretely, Oij is calculated by counting the times of its j-th candidate label not appearing in its k-NN instances' candidate label set v/(1;
Oi - -"- Tun o Dij ( V.jeY.
(1)
where [[[]] is the indicator function and Yij is the j-th candidate label of æ;. In this way, the value of Oig denotes the number of the k-NN instances of æ; whose candidate label set does not include the j-th candidate label of z;, Particularly, the calculating process in Eq. (1) is like a "down-voting" procedure where each nearest neighbor instance of the concerned instance serves a voter to down- vote the candidate labels that are not inside its candidate label set y,(k), Furthermore, we define a specified parameter T to control the pruning extent. Specifically, the number of eliminated candidate labels of z; is
% = [ 1)].
where |-| is the ceiling function that returns the least integer greater than or equal to the given number. Then, for each instance (z,, Yi), we can eliminate a subset of candidate labels that possess a high down-voting value:
y= Top-7y-argmaxxayy0000 (Oi)).
where Targga- returns a subset of indices (i.e., candidate labels) that have the highest 7 down-voting values of Oig for E Y, After eliminating the selected subset of candidate labels Y; from Y₁ for each instance Tg. we can obtain the pruned PLL dataset D {(z,, Y.) where Y, = YAV, The pseudo-code of the proposed algorithm for CLSP is shown in Appendix A. Notably, an incorrectly pruned PLL instance (æt, Y.) whose true label yi is inside the eliminated candidate label set YÚ would become a noisy PI.L instance (Yan & Guo, 2023a; Xu et al., 2023a; Lv et al., 2023; Wang et al., 2024), which is more challenging for conventional PLL methods. To alleviate this issue, we would like to analyze the upper bound of the pruning error in the proposed method.
3.3 THEORETICAL ANALYSIS OF PRUNING ERROR
Given an example (ar:, Y;) and its k-NN examples ((a))) y 3 in PLL dataset DD that satisfies the (k, (k) label distinguishabiliiyy the probability of the true label y; E Y; appearing in each k- NN instance's 's candidate label set y-1) is denoted by > 1- -k, and correspondingly the probability of each false candidate label 3/ E Y = YA(U appearing in each k-NN instance's candidate label
y
set y-U) is denoted by q S Ph- We assume that the true label and false candidate labels of each PLL example appear in its k-NN examples' candidate label sets independently. Then, the dowi-voting statistic of the true label Oiy follows binomial distribution Bkk, 1-1), and the down-voting statistic of each false candidate label Oug follows a binomial distribution B(k, 1 - 9). In this case, there are one random variable Oiy B(k, t) and [Y]|| i.i.d. random variables Oug" B(k, 1 q)- Given the number of eliminated candidate labels E [1, |Y/]], an incorrect pruning event for PLL example (a,, Yi) occurs in the proposed algorithm when O:y Top-7;- arg max, (O,)). In other words, the incorrect pruning implies that the down-voting statistic of the true label On, is larger than the -t-th highest down-voting statistic of the false candidate label. Formally, based on the definition of the k-th order statistic (,56) which is the k-th item in the ascending order statistics ( (1k)
J. the incorrect pruning event can be defined as [0 (Ix))=+++1) < -Oyyl An upper bound for the probability of getting such an event is shown below.
by ..... (.D)"
izy
Theorem I. Assume that the (k, dy, pu)-label distinguishability is satisfied. For each PLL example
(æi, Yd), let us denote that the y-th label in the candidate label set Y; is the true label, and the y/tth label in the false candidate label set Y! = =Y== Vilu is an arbitrary false candidate label, i.e., y' # y. Given the number of eliminated candidate labels % then the probability of getting an incorrect
pruming can be upper bounded by
1 < <Oiy)) TH >
IK'
(14:l) m n7 (1 m7 (1:1 (k:
1 2m=s,
(4)
4

---

Published as a conference paper at ICLR 2024
- 0.8, =1
0.8 Xi] 1..
2- 60,
2-
20,4 0... P2
45
5.2
Upper Sound
Lbpon Bound
64
-0004
6, 0.4
Pa-
Upper
0 0.6
00
0, 0.6
Upper
0.2 p.- 0.6
Aum 0.6
00.8
2.
6,- 0.8
02
200
Figure 1: Numerical simulation experiments to show the effect of k and for the upper bound.
where &. = - (*) = rlln-rj is the combination combinatton.fommula, b6g
n!
(g 62)
denotes the probability mass function of a binomial distribution B(k, 5g), and 17 = Iyx (k (k-j+ where 1p. (k: j) S0 11 I is the regularized incomplete beta function.
The proof is provided in Appendix B. Note that the above upper bound is too complicated to perform the mathematical quantitative analysis for four key factors k %. 82, and Pk. Hence, we aim to conduct an empirical analysis of the four factors to derive technical insights. Specifically, in the practical scenario of utilizing the proposed algorithm, given a PLL dataset (including the employed feature extractor), 55. and Pk are fixed (or can be estimated on the validation set). We need to choose the appreciate values of and 7/ (7). Before formally introducing the empirical experiment of evaluating the effects of varying k and Yi let us additionally come up with another conclusion shown below.
Theorem 2. Given the same assumption of the (ki, du, pua-label distinguishability and notations in Theorem I, when increasing the number of eliminated candidate labels (i.e., 7f > 7/ the extra pruning error can be bounded by
P(O by
- - is
< Oiy) & EØ_1 m ??
(7 xi1- bs (k,
(I m
(5)
where ct in Theorem /.
(IV)) - + 1), c (V)) ?? + 1), and other notations are the same as those used
The proof is provided in Appendix C. The Theorem implics that the extra pruning error caused by increasing the number of climinated candidate labels is also bounded by an upper bound associated with the four key factors. Now, the key challenge is to choose the appreciate values of & and % (r), thereby decreasing the upper bound of the pruning error. To this end, we conduct the numerical simulation experiment to empirically evaluate the effects of varying k and 7: for the upper bound.
Numerical simulation. Here, set various values of dg and A where a small value of ōg (pk) implies high-quality representations (candidate label sets of low label ambiguity). First, we aim to explore the effect of varying is on the upper bound. Suppose a PLL instance æ; possesses three candidate labels (i.c., |Y| = 3) and the number of eliminated candidate label = 1, we set various values of 6z [0.2, 0.4, 0.6, 0.8] and Pk = [0.2, 0.8J. By taking these values into the formulation (4), we can empirically calculate the upper bound as shown in Figure 1. The experiment result shows that under small Pk. as increases, the upper bound generally decreases except for a large 6R and under a large Pk. the upper bound maintain stability only with a small 6. when k increases. Moreover, to explore the effect of increasing the number of eliminated candidate labels y,, assume PLL instance æ; equipped with larger size of candidate label set [Y]] 6, thereby leading to a wider pruning range of 7 5]. The experiment result implics that when increasing the number of eliminated candidate labels, the upper bound maintains stability with small d and Pk: Based on these observations, we can draw two empirical conclusions: (1) a small value of can achieve relatively low upper bound of the pruning error imperfect features or high label ambiguity; (2) eliminating more candidate labels is feasible by using high-quality representations under low label ambiguity.
EXPERIMENTS
In this section, we conduct extensive experiments to verify the effectiveness of the proposed method for candidate label set pruning (CLSP). We first introduce the experimental setup and then present comprehensive experimental results. Based on these results, we conduct in-depth empirical analyses.
5

---

Published as a conference paper at ICLR 2024
Table 1: Test accuracy comparison CIFAR-10 and CIFAR-100 datasets under uniform, lahel- dependent (LD), and instance-dependee (ID) PLL settings. The row in gray color indicates the PLL method using the pruned candidate label set. The better result is highlighted in bold.
Dataset
4
CC
PRODEN
CAVL
PICO
ABLE
IDGP
POP
81.67
81.11
84.85
78.14
94.20
94.53
92.
95.19
0.4
86.45
82.07
86.68
81.25
94.44
97.24
94.97
93.38
95.64
71.16
79.81
81.67
54.52
92.96
96,47
93.69
89.48
94.57
0.6
84.62
82.42
85.87
80.99
94.32
97.21
94.92
92.52
95.48
C-10
89.57
81.83
86.18
80.43
94.59
97.24
94.77
92.47
95.63
LD
90.81
82.49
87.54
82.17
94.49
97.58
95.19
92.82
95.87
73.92
78.03
78.70
67.21
91.08
87.89
91.17
84.45
93.63
ID
77.57
81.92
84.67
77.97
93.41
95.90
93.99
92.08
95.05
64.05
48.68
$1.18
41.00
72.31
83.16
74.43
68.39
76.35
0.05
64.36
49.72
$2.69
49.62
72.66
83.53
75.08
68.86
76.85
62.31
46.26
45.37
21.34
56.80
82.51
74.80
67.62
74.38
0.1
C-100
64.05
48.38
51.62
45.48
72.51
83.39
74.76
68.55
75.95
63.72
29.28
51.29
48.76
72,46
82.93
74.11
22
74.90
H-0.5
65.56
40.92
53.40
48.88
73.10
83.38
75.59
68.53
75.32
63.06
49.83
53.18
47.35
71.04
80.76
74.04
66.71
73.30
ID
63.30
50.11
52.74
48.24
71.78
80.93
74.49
67.23
74.26
Table 2: Test and transductive accuracy comparison on VOC dataset. The row in gray color indicates the PLL method using the prumed PLL data. The better result highlighted in bold.
Dataset
CC
PRODEN
LWS
CAVI.
CRDPL.L
Sol
RECORDS
34.21
43.59
17.78
32.25
21.26
64.53
62.52
Test
50.81
47.85
28.08
49.45
38.53
65.42
65.38
VOC
72.15
77.26
64.46
77.43
67.09
76.36
40.33
Trans.
77.51
79.01
71.07
86.88
75.09
82.35
68.32
4.1
EXPERIMENTAL SETUP
Datasets. We use three benchmark datasets, i.e., CIFAR-10 (Krizhevsky al., 2009), CIFAR-100 (Krizhevsky et al., 2009), Niny-ImageNet (Wu et al., 2017), and real-world PLL dataset PASCAL VOC (Everingham al., 2015). Besides, we consider long-tailes PLL setting (Wang et al., 2022a; Hong et al., 2023) CIFAR-10 and CIFAR-100 denoted by CIFAR- 10-LT and CIFAR-100-LT, respectively. Following the previous work (Hong et al., 2023), we employ the imbalance rate & [50, 100] that denotes the ratio of the example sizes of the most frequent and least frequent classes, ic., Ø = /eee/N M(1). where [NN(e), ( ,N is in the descending order.
Candidate label generation. To simulate the real-world generation process of the candidate la- bel set, existing deep PLL research commonly considers various generation models candidate labels. We consider three general candidate generation models that involve different types of flip- ping probability: uniform generation, label-dependent (LD) generation, and instance-dependent (ID) generation. Specifically, consider uniform probability q [0.4, 0.6] on CIFAR-10, = [0.3, 0.5] on 10-AR- = [0.05, 0.1] on CIFAR- 100, = [0.03, 0.05] on CIFAR- 100-LT, and q [0.01, 0.05] on Emne-emaeeNet respectively. For the label-dependent generation, we consider candidate probability vector q = [0.5, 0.4, 0.3, 0.2, 0.1] for each label on CIFAR-10, and generate hierarchical candidate labels that belong to the same super-class with probability 0.5 on CIFAR-100. For the rustance-depended generation, following the previous work (Xu et al., 2021), we use the prediction of neural network trained with original clean labels as the sample- and label-wise flipping probability.
Evaluation metric. To evaluate the proposed CL.SP method, we can use a-error and 8-coverage in Definition as two metrics. The smaller (larger) the value of (3), the better the performance. Besides, we also employ the Fl score to evaluate the pruning. Specifically, the precision pruning: Precision
and the recall of pruning: Recall 3. Thus, F1 2((1 a) The larger the value of the Fl score, the better the performance.
B)-
Feature extractor. In this paper, consider various visual feature extractors including visual- only ResNet-18- ba-baeed models: ResNet-S that performs conventional supervised learning with orig- inal clean supervision, ResNet-SSL. that leverages self-supervised learning (SimCLR (Chen al.,

---

Published as a conference paper at ICLR 2024
Table 3: Test accuracy comparison class-imbalanced CIFAR-10 and CIFAR-100 datasets under long-tailed PLL settings. The row in gray color indicates the PLL method using the prumed candidate label set. The better result is highlighted in bold.
Dataset
CC
PRODEN
LWS
CAVL
CR
SoLar
RE
75.31
76.73
77.28
44.18
71.53
84,48
79.95
77.51
78.66
78.79
44.21
78.37
84.69
79.80
0.3
67.36
63.38
63.52
43.39
82.61
75.53
70.86
100
70.48
71.23
72.49
43.41
84.35
76.82
71.43
C-10-LT
59.90
62.95
62.22
42.84
47.92
82.41
75,48
50
66.85
65.30
64.13
48.36
64.94
83.57
76.48
0.5
55.36
55.37
57.19
44.85
60.43
71.50
65.73
100
63.05
62.19
62.13
42.59
68.05
70.85
67.08
43.70
43.93
43.40
32.39
45.06
48.31
39.91
44.67
44.69
45.24
33.33
45.80
49.27
40,30
39.32
38.71
38.07
30.35
51.24
42.76
36,42
100
40.14
39.52
39.24
31.75
52.52
43.81
37.44
C-100-LT
42.37
39.53
38.89
28.43
42.92
46,39
44,82
42.66
41.18
40.70
29.46
44.45
47.01
46.06
0.05
35.59
34.94
34.43
26.16
48.91
40.94
40.03
100
37.52
37.16
36.31
26.76
49.75
42.33
40.44
The magnitude change of the candidate label set, a-error (%), and ß-coverage of the pruning method under various PLL settings, O (vs. P) Avg. C means the average number of examples' candidate labels on the original (vs. pruned) PLL dataset.
Table 4:
CIFAR-I0
CIFAR-100
ImageNe 0.05
VOC
Dataset
0.4
0.6
LD
ID
0.05
H-0.5
ID
0.01
x
O Avg.
4.6
6.4
2.5
3.3
6.0
10.9
3.0
4.8
3.0
11.0
8.3
2.5
Avg.
2.1
2.9
1.2
2.6
2.9
5.0
1.3
4.4
2.3
7.2
1.8
a-error (%)
.18
.16
.66
.22
.43
.50
2.9
.54
.46
.65
.38
5.2
8-coverage
.69
.64
.89
.32
.22
.82
.08
.49
.37
.16
.47
2020)) without supervision, ResNet-I that directly uses the pre-trained model on the ImageNet-V dataset, and visual encoders in the pre-trained risual-language model: CLIP (Radford et al., 2021), ALBEF (Li et al., 20 BLIP-2 (Li al., 2023b). More details are shown in Appendix D.
Deep PLL methods. In this paper, consider eleven state-of-thee the-art deep PLL methods, in- cluding six conventional deep PLL methods, e.g., CC, PRODEN, LWS, CAVL, PiCO, CR, three instance-dependent deep PLL methods ABLE, IDGP, and POP, and two long-tailed deep PLL meth- ods SoLar and RECORDS.
Implementation. We employ BLIP-2 the feature extractor based on the open-source library LAVIS (L.i et al., 2023a). For k-NN searching, we employ Faiss (Johnson et al., 2019), a library for efficient similarity search and clustering of dense vectors. The parameters k and used in the proposed algorithm are shown in Table 5. For each PLL method, we keep the same training scheme with both original and pruned candidate label sets. More details shown in Appendix D.
4.2 EXPERIMENTAL RESULTS
CLSP performance. As shown in Table 4, we present the average number candidate labels on the PLL datasets before and after the pruning respectively, as well as the defined a-error (%) and 3-coverage of the pruning. As whole, we can see that the proposed method significantly reduces the size of candidate label sets at relatively low pruning error rates. Specifically, a-error (%) of the pruning maintains a relatively low level which is mostly less than 1%, and 3-coverage of the pruning reaches high level. The largest value of a-error is 5.2% on PASCAL VOC. This is because the feature quality of images on the dataset is relatively low due to their complicated visual appearances. CIFAR-100 (ID). This is because label ambiguity CIFAR- 100 (ID) is knotty (i.c., false candidate labels always co-occur with the true label), which is challenging for the CLSP task. Besides, also explore the effects of different feature extractors as
The smallest value of 8-coverage is 0.08

---

Published as a conference paper at ICLR 2024
Unilorm
LD
ID
WeodlWorld
CDve CPPAR10 50
LtCoee 0EN0.30
cal um 0110333
CRAVEL CFA0111
Truns Are
Trans, Aco
328
300
103
150
20
200
220
150
20
30 125
1.
Thane Are
Thans
85
Trans
Thanss
2
Cocch
Epach
Cpesh
520ch
Cpcc
1.4
+RODS/ ttyyenagaee
1.3
1.2
CRSPLLOO
Trans Acc
Trans Are
Trans Acc
Tranr
0
PRODE on HOS
cD VDC
V0c
Tronk ACC
Acc
Trans. Ace
03
180
33
Pyach
Paesh
Figure 2: Transductive accuracy of PRODEN, CC, LWC, CAVL, and CRDPLL algorithms on CIFAR-10. CIFAR-100, Imageem, and VOC datasets. Bold (vs. thin) curves indicate the corresponding algorithm using the original (vs. pruned) PLL data. Best viewed in color.
shown in Appendix E. On the whole, the performance of pre-trained vision-language models, i.e., CLIP, ALBEF, and BLIP-2, is better than visual-only models ResNet-SSL and ResNet-I.
PLL performance comparison. To show the effect of the proposed CLSP method on improving deep PLL methods, we conduct each PLL method on the original and pruned PLL datasets respec- tively under the same training scheme. From comprehensive experimental results, we can draw the following conclusions:
Overall performance improvement. On the whole, from the results of Tables 1, 2, and 10, we can see that test accuracy of PL.L methods under almost all cases (145/149**7.399) is sig- nificantly improved after training on the pruned PLL dataset. This result directly validates the effectiveness of the proposed CLSP method in improving deep PLL methods. Furthermore, we also present the transductive accuracy of PLL methods in Tables 7, 8, 9 and Figure 2. We can see that the proposed CLSP method also definitely increases the transductive accuracy by a sig- nificant margin, which implies that more PLL examples are identified correctly after eliminating their potential false candidate labels. This observation validates our claim that the proposed CLSP method can reduce label ambiguity and boost label disambiguation effectively. Besides, there are a few bad cases (4/149 2.7%) in the experimental results where the performance of certain PLL methods has slightly degraded after training with pruned PLL data. We argue that this is because the involved PL.L methods (i.e., ABLE and SoLar) in bad cases have a cime-consuming training procedure, i.e., 500 and 1000 epochs respectively, hence they tend to overfit noisy PLL instances eventually, thereby leading to performance degradation.
The effect on different candidate generations. Also, we further analyze the effect of the pro- posed CLSP method on uniform, LD, and ID generations. Under uniform generation, PLL meth- ods generally have a distinct performance degradation as g increases. For example in Table on CIFAR- 10, the average reduced test accuracy of PLL methods from = 0.4 to q = 0.6 is about 3.6%, while the average reduced test accuracy of PLL methods using the pruned candidate label set from q = 0.4 to q = 0.6 is about 0.6%. This validates the effectiveness of the proposed CLSP method to eliminate uniformly generated candidate labels. Particularly, the improvement is more significant under LD and ID generations. This implies that the proposed CLSP method is superior against knotty candidate labels in LD and ID cases.
The effect on LT data distribution. As shown in Table 3, the performance of PLL methods has a serious performance degradation. This is because the imbalaneeed data affects label disam-
8

---

Published as a conference paper at ICLR 2024
Empriica 6g on CFAR110
Emptrical on CIFAR-100
Grepiical i an CIFA--10
Empitiaal CFAA-330
Empiical POF
Empirical POF
07
-45
Empirical
Unliom
Empiniaa
L0
-3
k-
k
Figure 3: Empirically calculated values of 8% and a under various settings and feature extractors.
Unfoorm
LD
CIFAR-10
CIFAR-100
CIFAR-11
CIFAR-100
200
0e
S00
280
100
150
280
2.
2.6
7.0
800
Figure 4: Various values of k and on CIFAR- 10 and CIFAR-100 under different PLL settings.
biguation greatly. We can see that PLL methods have significant performance improvement by training with the pruned PLL data.
The significant improvement the real-world dataset. Note that the PASCAL VOC dataset is real-world class-imbalanced PLL dataset (Hong et al., 2023). As shown in Table 2, the proposed CLSP method significantly improves the transductive and test accuracy of PLL methods. This shows the practical value of the proposed CLSP method in real-world tasks.
Parameter analysis. Since the values of & and Pa can be empirically calculated by d2 =
(E==1)) Ex I(u: = vi /)kkn and P = (Ei max Σ. T(y = (k) /kn, where Y? = (Mss) is the set of false candidate labels. Here, we show the empirical values of 6n and p on CIFAR- 10 and CIFAR-100. As shown in Figure 3, the overall trend of 6% and Pk decreasing as k increases. The better the feature extractor, the larger the value of 62, indicating high-quality representations. Figure 4 shows the results of various values of & and r on CIFAR-10 and CIFAR- 100 under different PI.I settings. From Figure 4, we can find that both the ID and LD candidate label generation processes lead to a larger value of Pt than the uniform case, which implies a higher label ambiguity. We employ BLIP-2 as the feature extractor with a large & and thus are able to combat the ID and LD cases. Based on the empirical observation, we further evaluate various values of k and r. In particular, we evaluate one parameter with another fixed. As shown in Figure 4, when the number of N-NN increases, the pruning error on uniform generation drops significantly at first and maintains stability under a large k. This phenomenon accords with the theoretical analysis that small & favors a large k. On the contrary, the pruning error under LD and ID candidate generations increases obviously as k increases. This means small l is enough under a large Pk- As for the parameter т. we can see that increasing the value of T consistently increases the pruning error under all cases. Fortunately, the pruning error growth under uniform candidate generation is relatively slow, and thus it is favorable to select a relatively large T.
CONCLUSION
In this paper. we pioneer a new PLL-related task called candidate label set pruning (CLSP) that aims to reduce the size of candidate label sets of PLL instances. To this end, we propose the first CLSP method that eliminates certain potential false candidate labels of PLL. instances based on a "down-vote" statistic from their k-NN instances in the representation space. Theoretically, we analyze the effects of the representation quality and label ambiguity against the upper bound of the pruning error. Empirically, extensive experiments on both benchmark-simulate and real-world PLL datasets validate the superiority of the proposed CLSP method to significantly improve state-of-the. art deep PLL methods. In the future, it is also interesting to develop more effective methods for CLSP. We hope our work will draw more attention of the PLL community from the carning-centrin perspective toward the data-centric perspective.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Lei Feng is supported by the National Natural Science Foundation of China (Grant No. 62106028) and the Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program. Guown Yang is supported by the National Natural Science Foundation of China (Grant No. 62172075). This paper is partially supported by the Sichuan Science and Technology Program (Grant No. 2022YFG0189).
REFERENCES
Ching-Hui Chen, Vishal M Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(7):1653-1677, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. Simclr: A simple frame- work for contrastive learning of visual representations. In Proceedings of the International Con- ference on Learning Representations, volume 2, 2020.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12(5):1501-1536 2011.
M. Everingham, S. M. A. Eslami, L.. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision, 111(1):9--33,, January 2015.
Lei Feng. Jiaqi Lv. Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial-label learning. Proceedings of the Advances Neural Information Processing Systems, 2020.
Xiuwen Gong, Dong Yuan, Wei Bao, and Fulin Luo. A unifying probabilistic framework for partially labeled data learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Shuo He, Lei Feng. Fengmao Lv, Wen Li, and Guowu Yang. Partial label learning with semantic label representations. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 545-553, 2022.
Shuo He, Lei Feng, and Guowu Yang. Partial-label learning with mixed closed-set and open-set out-of-candidate examples. In Proceedings of the ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, pp. 731-731, 2023a.
Shuo He, Guowu Yang, and Lei Feng. aandddteeawart selective disambiguation based malized entropy for instance- deeendeed partial- learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1792-1801, 2023b.
Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial label learning via dynamic rebalancing. In Proceedings of the International Conference on Learning Representations, 2023.
Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Advances in Intelligent Data Analysis, 10(5):419443)) 2006.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535-547, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technology Report, 2009.
Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C.H. Hoi. Lavis: A one-stop library for language-vising intelligence. In ACI. pp. 31-41, 2023a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Proceedings of the Advances in Neural Information Processing Systems, vol- ume 34, pp. 9694-9705, 2021.
10

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Proceedings of the Advances in Neural Information Processing Systems, volume 23, 2010.
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identifica- tion of true labels for partial-label learning. In Proceedings of the International Conference on Machine Learning, pp. 6500-6510, 2020.
Jiaqi Lv, Biao Liu, Lei Feng, Ning Xu, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. Gm-pll: graph matching based partial label learning. IEEE Transactions on Knowledge and Data Engineering, 2019.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Find- ing important examples early in training. In Proceedings of the Advances in Neural Information Processing Systems, volume 34, pp. 20596-2066 2021.
Congyu Qiao, Ning Xu, and Xin Geng. Decomposition-basee generation process for instance- dependent partial label learning. In Proceedings of the International Conference on Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proceedings of the International Conference on Learning Representations, 2018.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In Proceedings of the International Conference Learning Representations, 2018.
Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. In Proceedings of the Advances in Neural Information Processing Systems, 2022a.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng. Gang Niu, Gang Chen, and Junbo Zhao Pico:
Contrastive label disambiguation for partial label learning. In Proceedings of the International Conference on Learning Representations, 2022b.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico+: Contrastive label disambiguation for robust partial label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning, In Proceedings of the International Conference on Machine Learning, 2021.
Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning, In Proceedings of the International Conference Machine Learning, pp. 24212-24442 2-22222. PMLR, 2022.
Jiay Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.
S Xia, Jiaqi Lv, Ning Xu, and Xin Geng. mubbiguity-inducee contrastive learning for instance- dependent partial label learning. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3615-3611, 2022.
11

---

Published as a conference paper at ICLR 2024
Mingyu Xu, Zheng Lian, Lei Feng, Bin Liu, and Jianhua Tao. Alim: Adjusting label importance mechanism for noisy partial label learning. In Proceedings of the Proceedings of the Advances in Neural Information Processing Systems, 2023a.
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5557-5564, 2019.
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependen partial label learning. In Proceedings of the Advances Neural Information Processing Systems, volume 34, 2021.
Ning Xu, Biao Liu, Jiaqi Lv, Congyu Qiao, and Xin Geng. Progressive purification for instance- dependent partial label learning. In Proceedings of the International Conference Machine Learning, pp. 38551-38565. PMLR, 2023b.
Yan Yan and Yuhong Guo. Mutual partial label learning with competitive label noise. In Proceedings of the International Conference on Learning Representations, 2023a.
Yan Yan and Yuhong Guo. Partial label unsupervised domain adaptation with prostppo align- ment. In Proceedings of the International Conference on Learning Representations, 2023b.
Zinan Zeng, Shijie Xiao, Kui Jia, Tsung-Har Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 708 0088-15,220 2013.
Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Exploit- ing class activation value for partial-label learning. In Proceedings of the International Conference on Learning Representations, 2022.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based ap- proach. In Proceedings of the International Joint Conference Artificial Intelligence, 2015.
Min-Ling Zhang, Bin-Bin Zhou, and Xu- Xu-Ying Liu. Partial label learning via feature-aware disam- biguation. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, pp. 1335-1344 1344, 2016.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Proceedings of the International Conference on Learning Representations, 2021.
Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1): 44-53, 2018
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training a model to predict. In Proceedings of the International Conference Machine Learning, pp. 27412-27421 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
A
THE PSEUDO-CODE OF THE PROPOSED ALGORITHM
Algorithm 1: The proposed CLSP method
Input: PL.. dataset DD (a,, Y:) a feature extractor Φ(-), parameters k, T.
Output:
The pruned PI.L dataset D
1/ Feature Extract
Obtain feature representations of all instances the feature extractor by ((-);
// Pruning towards each instance
for i n do
// k-NN searching using faiss
Search k-NN instances [(=, (1) y(1)), (2, (e, (k)
on D for each instance (æ;, Yi);
// Down-voting from k-NN instances
Calculate a down-vote statistic O; towards each instance by Eq. (I);
// Counting
Count the number of eliminated candidate labels n towards each instancea, by Eq. (2);
// Pruning each candidate label set
Select the eliminated candidate labels Y, based the highest down-voting by Eq. (3);
// Obtain the pruned candidate label set
Obtain the pruned candidate label set towards the instance æ;: Y; YAY.
end
Return the pruned PLL dataset D {(x,, Yo)
As shown in Algorithm 1, the proposed method does not involve the complex and time-consuming training process. The most time-consuming step k-NN searching where we utilize the open-source library Faiss to accelerate the process. The pruned PLL dataset can be directly leveraged in existing deep PLL methods.
B THE PROOF OF THEOREM 1
Recall that in Section 3.3, there are one random variable Oiy B(k, 1 - L) and (Y7)) i.i.d. random
variables Osg' B(k, g). The number eliminated candidate labels 7, [1, [Y']]. The event of
incorrect pruning for the instance z; is [O(5) (5.)
y + and (*)) is the k-th ( Now, we derive an upper bound for the probability of
< Osyl where &
order statistic [0(0) (k))
1// getting incorrect pruning for the instance æ:
ig'.".. ....
by
&
P( < 0ky) =
Σ
P(O < 0ig = -j =)
j-0
(()
(5,)
-j- 1) =)
by
(| m
11 (1- (1-1 bōg (ko,j)
j=1 m=
r((n-) is the combination formula, 65, (k, j) = (1- ah- denotes the probability mass function of binomial distribution B(k, dg), and 1p, (k: 1,5) where Log jk 5h 1 f)-- dt is the regularized incomplete beta function.
where & = (DX)1 - (*) n!
C THE PROOF OF THEOREM 2
Based on the notations in the proof of Theorem 1, we further denote y! and x? by two numbers of eliminated candidate labels respectively satisfying 7 < o/ Then can derive upper bound for
13

---

Published as a conference paper at ICLR 2024
Table The values of parameters &t and used in the proposed method under various PLL settings.
Uniform C-100
LD
ID
LT
Setup
C-10
T-I
C-10
C-100
C-10
C-100
T-1
C-10
C-100
VOC
T
0.6
0.6
0.4
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.1
k
150
150
150
50
150
5
5
50
50
50
the extra pruning error
(5) < OGny)
-P(O
< 0igy)
Σ
P(O
(
<iylOOig P)i =]
==
P(O b
(C)
Oiy Ory P) = =
M11
(1"((1 (1- - -7)(11(1 (.(1x(--m) bsg (k;,
m=
(
17" 17" 77) (1x.1 1m) bs. (k,
m=
11" (011-))m1 -7) >:1-mb5 (k. (k,
j=
m=
where <! (DY - % + 1), ([Y)) %/ 1), and other defiintiionnn are the same as that in Theorem 1.
D EXPERIMENTAL CONFIGURATION
In this section, we further show the detailed experimental configuration of the proposed method and partial-label learning (PLL) methods. First, we present the values of parameters k and used in the proposed method on each dataset under different partial-labed learning (PLL) settings.
Candidate label generation. Formally, given an example (x,, tion models intrinsically consider the flipping probability of false label v/S y: n V/ e y being a candidate one. (1) Uniform Generation (Lv al., 2020): this case is naive simple generation model that assumes each domain label has the same flipping probability of being candidate. The uniform real-value flipping probability is P(ū e Yly) qi (2) Lebel-dependent Generation (Wang et al., 2022b): instead of treating all domain classes equally, semantic-similarr domain labels against the true label are more likely to be inside candidate label set. The label-wise flipping probability is P(ð, E Yly) g;: (3) mnstance-depender more realistic scenario, candidate labels are usually generated according the specific features of the instance. The sample- and label-wise flipping probabil- ity is P(0k E Y: a.,, yi) 4ij- Specifically, we consider a uniform probability 4 [0.4, 0.6] on CIFAR- 10, q [0.3, 0.5] on CIFAR-IO-LT, q = [0.05, 0.1] on CIFAR-100, q = [0.03, 0.05] on CIFAR-100-LT, and = 10.01, 0.05] on Tiny-ImageNet respectively. For the label-dependent generation, we consider candidate probability vector q [0.5, 0.4, 0.3, 0.2,0.1] for each label on CIFAR-10, and generate hierarchical candidate labels that belong to the same super-class with probability 0.5 on CIFAR-100. For the instance depenndet generation, following the previous work (Xu et al., 2021), we use the prediction of a neural network trained with original clean labels as the sample- and label-wise flipping probability Qi--
p( p( y). candidate genera-
Generation (Xu al., 2021): the above two cases do
not consider the instance itself.
Deep PLL methods. CC (Feng et al., 2020): classifier method that assumes a set- level uniform data generation process; PRODEN (Lv et al., 2020); self-training-basee method that progressively identifies the true labels using the output of the classifier itself:; LWS (Wen et al., trade-ofi off between losses 2021) I): a set of loss functions that weights the risk function by means motivated by Class Activa- on candidates and con-candidates; CAVL (Zhang al., 2022): a method tion Map (CAM) and using the corresponding class activation value for label disambiguation. PiCO (Wang et al., 2022b): contrastive marning-based method that establishes class prototypes for label
14

---

Published as a conference paper at ICLR 2024
disambiguation; CRPLL (Wu al., 2022): regularization based method that achieves state-of- the-art performance in instance- -indeeedddnne PLL; ABLE (Xia et al., 2022): an umbiguity-induce contrastive learning method that leverages label information into contrastive learning; IDGP (Qiao et al., 2023): a maximum a posterior (MAP) based probabilistic method that considers a compo- sitional generation process of candidates; SoLarr (Wang et al., 2022a): an optimal transport-basee framework that refines the disambiguated labels towards matching the marginal class prior distri- bution; RECORDS (Hong et al., 2023): dynamic rebalancing method that is benign to the label disambiguation process and theoretically converges to the oracle class prior; POP (Xu et al., 2023b): a theoretically grounded algorithm to filter out false candidate labels progressively in every epoch.
More details about the k-NN algorithm and feature extractors. We employ the visual encoder of BLIP-2 to extract 768-dimensional high-quality representations for all training PI.L instances. Then, we leverage faiss to conduct the fast k-NN searching for each instance based on the squared Euclidean Distance. Besides, we additionally train two visual-only ResNet-18-baal ResNet-SSL, ResNet-S) on each PLL dataset. Note that we did not them in the proposed method and just used them for comparing the effects of different types of feature extractors. Specifically, ResNet-S is trained with the original clean supervision using the cross-entropy loss (note that do not employ data augmentation techniques), while ResNet-SSL. is trained by the self-supervised learning method SimCLR without any supervision. The weak and strong data augmentations used in SimCLR follow the original configurations on the corresponding PIIL dataset. In addition, ResNet-T directly employed by loading the checkpoint "IMAGENETIK_VE" The dimension of these three models' output representations is 512.
models (i.e.,
Parameters and T in the proposed method. As shown in Table 5, based on our theoretic analysis for the effect of k and 7 (T) against the upper bound Eq. (4) in Theorem I, we commonly select large and J on CIFAR-10 and CIFAR-100 datasets under uniform generation due to the high-qualit extracted features on CIFAR-basee datasets and the low label ambiguity under uniform generation, while we choose relatively low l and t under generation due to the knotty label ambiguity, or on VOC due to the relatively low-quality extracted features of examples in the dataset. The effectiveness of the theoretic-inspired guidance is validated in the parameter analysis shown in Figure 4 where the appropriate values of parameters k and 7 effectively reduce the pruning error.
Training scheme of PLL methods. Besides, we further present the detailed training scheme of PLL methods including model architecture, learning rate, learning rate scheduler, weight decay, batch size, and data augmentation. On the whole, we employ a base training scheme: a ResNet- 18 model, learning rate is 1e-2, and weight decay is le-3. On CIFAR-I0 and CIFAR-100, CC, PRO- DEN, LWS, and CAVI. do not employ a learning rate scheduler and the data augmentation technique which is the same as the original implementation. But, more difficult datasets CIFAR-10-1E 10-IIT CIFAR-100-LY 100-LT Tiny-ImageNet, and VOC, they are equipped with a consistency regularization with augmented examples and nique that starts with very large learning rate and then aggressively decreases it to a value near 0 before increasing the learning rate again. Especially, on VOC, the epoch is set to 100 for all PLL methods to avoid overfiuting- Although there are different configurations for PLL methods, they both employ the same training scheme for the original and pruned candidate label sets, which ensures the performance improvement comes from only the effect of pruning.
scheduler which is scheduling tech-
Following the previous work (Hong et al., 2023), we construct the dataset where objects in images are cropped as instances and all objects appearing in the same original image are regarded as the labels of a candidate set. As for the characteristics of PASCAL VOC, the number of classes is 20, the number of training (test) instances is 11706 (4000), and the average number of candidate labels is 2.46. Particularly, the imbalance ratio is 118.8.
PASCAL VOC.
E MORE EXPERIMENTAL RESULTS
In this section, we present additional experiment results including the Fl score of using different feature extractors, transductive accuracy comparison, and training loss curves of PLL methods.
hugp://gttuh..cm/ssseeseee////
butp://gtthbbbomm/ffccooorrsesee http://gtthbbcmmmooogoeeeeeeeees
15

---

Published as a conference paper at ICLR 2024
Table 6: Fl score (%) of different feature extractors adopted in the proposed algorithm.
Dataset
ResNet-SSL ResNet-S
ResNet-I
CLIP
ALBEF
BLIP-2
0.4
91.71
79.47
89.55
90.82
91.14
91.61
0.6
90.05
76.98
87.87
89.03
89.42
89.97
CIFAR-10
LD
97.74
78.91
91.94
95.63
96.33
97.48
ID
70.41
63.73
68.03
69.62
69.85
70.50
0.05
89.16
71.28
87.42
88.07
88.39
89.08
0.1
87.87
67.11
85.55
86.35
86.86
87.69
CIFAR-100
H-0.5
96.00
69.93
86.43
89.07
90.70
93.89
ID
34.35
16,85
30.39
31.31
31.64
32.53
0.01
79.87
70.53
82.40
82.71
82.66
82.97
Imay-lm
0.05
71.25
62.04
73.58
74.05
74.00
74.57
ID
48.22
45.72
49.20
49.01
49.07
49.05
VOC
86.33
73.48
78.10
79.93
80.05
78.99
Table Transductive accuracy comparison on CIFAR-10 and CIFAR-100 datasets under uniform, label-dependent (LD), and instance-dependee (ID) PLL settings. The row in gray color indicates the PLL method using the pruned candidate label set. The better result highlighted in bold.
Dataset
CC
PRODEN
LWS
CAVI.
PiCO
CR
ABLE
IDGP
90.56
90.54
92.19
88.52
96.10
97.02
96.10 10
92.86
0.4
96.56
93.77
96.90
92.70
97.23
99.06
98.15
96,90
80.02
86.98
86.94
61.33
94.51
96.52
93.72
82.12
0.6
93.9%
92.66
94.99
91.04
96.56
98.53
97.16
95.57
C-10
95.87
93.87
96.55
92.23
97.42
99.49
98.41
96.43
L.D
97.83
94.88
99.09
94.42
97.84
99.54
99.35
97.38
76.66
81.98
80.13
71.78
91.56
88. 88.26
90.52
85.6 35.64
ID
86.57
89.20
91.02
86.75
94.95
96.26
94.81
92.81
91.81
76.22
90.34
67.54
93.23
97.49
95.57
86.12
0.05
93.97
78. 19
94.80
77.79
94.52
98.09
97.10
89.53
87.62
71.03
79.96
38.14
72.81
95.38
92.30
84.32
91.00
75.94
89.86
72.67
92.25
96.39
94.76
85.88
C-100
86.30
41.31
82.83
71.39
86.39
92,90
89.17
82.73
H-0.5
91.89
84.74
93.15
76.82
90.57
95.11
94.51
84.52
79.42
67.76
75.64
63.67
82.26
81.18
84.01
76.34
81.09
69.17
77.30
67.48
83.61
81.21
85.33
77.21
Different feature extractors. As shown in Table present the F1 score using different feature extractors including visual-only models ResNet-S, ResNet-SSL, and ResNet-I and vision-language pre-trained models CLIP, ALBEF, and BLIP-2. We mance. This is reasonable but impractical since it uses clean supervision. On the whole. feature extractors of CLIP, ALBEF, and BLIP-2, achieve better performance than visual-only ResNet-SSL and ResNet-I models. This shows the powerful visual representation ability of multi-modal models.
that ResNet-S achieves the best perfor-
Transductive accuracy comparison. As shown in Table CIFAR-JO and CIFAR-100 datasets and Table 8 on CIFAR-1O-LY and CIFAR-100- datasets and Table 9 on Tiny- ImageNe, we can see that the performance improved in almost all cases (147/149 7%). Moreover, the improve- ment in transductive accuracy is more significant than that in test accuracy. These observations defi- nitely validate that the proposed CL.SP method greatly boosts label disambiguation of PIJL methods. Besides, we further discover that naive PLL methods (e.g., CC, PRODEN, LWS, and CAVL) have more significant performance improvements than advanced PLL methods (c.g., PiCO, CRDPLL, ABLE, and IDGP). We think that this is because naive PLL methods have limited capability in label disambiguation compared with advanced PLL methods, hence the proposed CL.SP method has bigger effect promoting label disambiguation in naive PLL methods.
Training loss curves. As shown in Figure 5, we present training loss curves PLL methods on CIFAR-10 10, 100,R-100, Imny--mag,Net, and VOC datasets. Note that Bold (thin) curves indicate
16

---

Published as a conference paper at ICLR 2024
Table 8: Transductive accuracy comparison class-imbalanced CIFAR-10 and CIFAR-100 100 datasets under long-tailed PLL settings. The row in gray color indicates the PLL method using the pruned candidate label set. The better result is highlighted bold.
Dataset
6
CC
PRODEN
LWS
CAVL
CR
SoLar
RE
94.57
95.16
95.10
86.37
96.82
98.65
85.18
50
95.57
96.13
96.02
86.88
98.26
98.87
87.89
0.3
94.86
95.05
94.95
89.47
97.09
97.63
78.26
100
96.36
96.65
96.58
90.20
98.04
98.12
82.23
C-10
89.33
90.85
90.57
83.74
91.47
96.22
79.51
50
91.53
92.30
92.18
86.46
95.35
97.45
78.33
90.68
91.68
91.72
87.03
91.62
95.43
73.05
100
93.22
93.95
93.58
88.15
94.23
96.65
71.12
92.62
91.85
91.84
83.20
95.45
94.72
87.42
50
93.81
93.58
93.43
85.12
96.41
95,90
88.48
0.03
92.55
91.76
91.46
86.17
95.61
93.56
87.97
100
93.97
93.41
93.20
87.04
96.38
94.82
88.57
C-100
89.10
86.63
86.57
77.38
92.50
91.23
90.30
50
90.29
88.63
88.54
79.61
93.72
91.86
91.33
0.05
88.63
87.28
79.15
92.28
90.62
90.18
100
90.29
89.19
89.14
79.61
93.72
92.21
91.51
Table 9: Transductive accuracy comparison on Tiny-ImageNet datasets under uniform and instance- dependent PLL settings. The row in gray color indicates the PLL method using the pruned candidate label set. The better result is highlighted in bold.
Dataset
q
CC
PRODEN
LWS
CAVL
CRDPLL
96.93
96.83
97.08
96.57
97.49
0.01
97.66
97.62
97.75
97.45
98.03
89.56
89.21
89.96
53.14
90.70
Tiny-ImageNet
0.05
91.36
91.04
90.84
77.09
92.31
75.81
74.41
75.86
67.39
77.19
ID
77.50
75.93
77.36
70.65
78.43
the corresponding PLL method with the original (pruned) candidate label set. We can see that PLL methods generally have faster convergence by using the pruned candidate label set. This phenomenon also discloses another characteristic of the pruning that accelerates the convergence of PLL methods. Generally, the training loss values of PLL methods tend to be smaller after training with pruned data. However, we found an unusual phenomenon where the training loss values of many PLL methods become larger on PASCAL VOC after training with pruned PLL data. We reckon that there are two reasons. Firstly, there exists higher proportion of noisy PLL instances in PASCAL VOC (where the pruning error about 5.2%) than in CIFAR and Tiny-ImageNet (where the pruning emror is less than 1% in most cases). Secondly, label disambiguation of training PLL instances in PASCAL VOC is more challenging due to the complicated visual objects in PASCAL VOC. Hence, fitting noisy PLI instances in PASCAL VOC is more difficult, thereby leading larger training loss value. Notably, the performance the involved PL.L methods on PASCAL VOC still has significant improvements, which implies this phenomenon may be benign for PLL methods.
17

---

Published as a conference paper at ICLR 2024
GRAR
GFAR-0
CIEELL CRANN)
9pan
Pyan
09
Lose
Lone 2
110
PRCOMELIT sugett
U90 Tevinggerr
CWS
CREPUL
Pyeeh
Fynnh
Pyns
Figure 5: Training loss curves of PRODEN, CC, LWC, CAVL, and CRDPLL methods on CIFAR- 10, CIFAR-100, Tiny-ImageNet, and VOC datasets. Bold (thin) curves indicate the corresponding method with the original (pruned) candidate label set. Best viewed in color.
Table 10: Test accuracy comparison on Tiny-ImageNo dataset under uniform and instance- dependent PLL settings. The row in gray color indicates the methoo using the praned candidate label set. The better result is highlighted bold.
Dataset
q
CC
PRODEN
LWS
CAVL
CRDPLL
65.04
65.21
66.92
64.97
67.48
0.01
65.35
65.28
66.98
65.43
67.56
63.06
63.02
64.34
35.53
65.99
Imageema
0.05
63,42
63.55
65.61
52.29
66.21
61.06
59.12
61.56
53.52
63.70
ID
62.11
60.15
62.30
55.95
64.27
18


Reviews:
[{"review_id": "ZtyjDGPA8h", "reviewer": "Reviewer_kuyu", "summary": "The paper pioneers a data-centric study for the problem of partial-label learning (PLL) where each training instance is assigned with some additional false candidate labels along with its true label and proposes a new PLL related task named candidate label set pruning (CLSP) that aims to filter out false candidate labels of the training PLL data. To this end, the paper proposes the first kNN-based CLSP method that eliminates candidate labels of each training instance which have the high “down votes” from its kNN instances. Theoretically, the authors prove an upper bound of the pruning error and analyze the effect of representation quality and candidate generation against it. Empirically, after training with the pruned data, existing PLL algorithms have a significant performance improvement, which validates the effectiveness of the proposed method.", "questions": "-\tHow about pruning on noisy PLL data? I am curious about whether the proposed method could be used on noisy PLL data whose true label is outside the candidate label set.\n\n-\tI find some bad cases in the experiment (shown in Table 1 and Table 8) where the performance drops after pruning. How to explain this phenomenon? \n\n-\tCould you show the values of delta_k and pho_k on the real-world dataset VOC (which are not shown in Figure 3)?\n\n-\tA unified proportion tau for each training instance is used in Eq. (2). Are there other ways to adaptively control the number of eliminated candidate labels for each training instance?\n\nOverall, it is a good work on PLL, but there are still minor issues that can be further improved. I may consider increasing my score if the above listed weaknesses and questions can be clearly addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed task CLSP is very significant and novel in PLL. Instead of studying learning-centric training algorithms, the authors take a different path to study filtering out false candidate labels before the training of networks, which improves the labeling quality of training PLL data and boosts the performance of existing PLL algorithms.          \n\n-\tThe proposed CLSP method is simple but effective, achieving impressive empirical results on various PLL settings (random, LD, ID), benchmarks (CIFAR, Tiny-ImageNet, and VOC), and ten PLL algorithms.\n\n-\tThe theoretical analysis of the pruning error is very interesting. The findings in the numerical simulation experiment achieve a good guidance for the selection of parameters k and tau in the practical employment.", "weaknesses": "-\tMore empirical analysis in the experiment should be presented, such as which PLL algorithms are more sensitive to the pruning method. \n\n-\tMore explanations on bad cases are needed to show the limitation of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OOJhKk41cp", "reviewer": "Reviewer_zBDH", "summary": "The paper involves the partial label learning problems where each training example is equipped with multiple candidate labels instead of the only one ground-truth label provided in the conventional supervised learning setup. The paper proposes an innovative task to assistant partial label learning, i.e., candidate label set pruning, which targets at removing false candidate labels of each training example. The proposed pruning approach leverages the inconsistency of representation and label spaces to select certain candidate labels being abandoned beforehand.\n\nThe authors prove that the pruning error is upper bounded by the representation quality, the process of candidate label generation, and the pruning proportion. Moreover, they perform a numerical simulation experiment to empirically show how these factors affects the upper bound, which provides a practical guidance for selecting parameters k and $\\tau$ in the proposed approach. \n\nThe paper conducts comprehensive experiments on various datasets CIFAR-10, CIFAR-100, Tiny-ImageNet, and PASCAL VOC with different settings of partial label learning including uniform -, label-dependent -, and instance-dependent candidate label generation. Besides, ten state-of-the-art partial label learning algorithms are used to compare the performance improvement. The overall experiment results show that the pruning approach enables these algorithms have a great performance gain specially on more difficult settings.", "questions": "1.\tIt would be appreciated if a clearer explanation for Definition 2 is provided. Why is this Definition needed?\n\n2.\tHow are these values of k and $\\gamma_{i}$ calculated in Figure 1? \n\n3.\tWhy does the loss curve on VOC in Figure 5 have a rise (except PRODEN algorithm)? This phenomenon is different from other cases on CIFAR and Tiny-ImageNet datasets.\n\n4.\tHow is the PASCAL VOC dataset used for partial label learning algorithms?\n\n5.\tWhat is the potential limitation of the proposed approach? Discussing this point is also important to have a comprehensive understanding for the proposed approach.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tOriginality. The originality of the paper lies in the proposed task candidate label set pruning and the corresponding approach for the task. For what I can tell, it is the first work to propose the task for partial label learning. Moreover, the idea of leveraging the inconsistency of representation and label spaces to filter out candidate labels is also novel.\n\n2.\tQuality. The approach proposed in the paper makes both theoretical and technical contributions. The theoretical analysis of the upper bound is very interesting. The proposed approach is technically sound, which is validated by significant performance improvements in the experiment.\n\n3.\tClarity. The paper is well organized and easy to understand the motivation. The related work is introduced adequately. The proposed task candidate label set pruning has a formal clear definition (definition 1). \n\n4.\tSignificance. The paper brings a new data-centric view for the area of partial label learning, which is significant for the development of partial label learning. Perhaps, more attention of researchers could be shifted from designing complex training methods to studying efficient pruning methods.", "weaknesses": "1.\tThe numerical simulation experiment about the calculating of values and conclusions is not shown clearly enough.\n\n2.\tThe PASCAL VOC dataset used in the experiment is not introduced well, as the dataset is not originally for partial label learning.\n\n3.\tDetail of trained feature extractors (ResNet-SSL, ResNet-S) is shown unclearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0eniLWWQNM", "reviewer": "Reviewer_289c", "summary": "This paper focuses on partial label learning, a paradigm of weakly supervised learning, and proposes a training-free method that prunes candidate label sets based on the inconsistency between the representation space and the candidate label space. In particular, when examining each potential label associated with a training instance, if it is not among the candidate labels of the instance's closest neighbors in the feature space, there is a notable likelihood of it being an erroneous label. Theoretically, it provides an upper bound of the per-example pruning error rate and analyzes how the representation quality affects the proposed algorithm.", "questions": "1.What is the main difference between a data-centric method and a pre-processing method?\n\n2.Does the feature extractor in the KNN algorithm come from the classifier during the training process? If yes, it seems unreasonable to say that the method is training-free. If not, a pre-trained model should be introduced.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. Different from the previous learning-centric PLL methods, the method of this paper is proposed from a data-centric perspective, which is novel. \n2. The paper theoretically provides an upper bound of the per-example pruning error rate and analyzes how the representation quality affects the proposed algorithm, which is solid.\n3. The proposed method is easy to understand and implement.\n4. The paper conducts extensive experiments on various settings of PLL.", "weaknesses": "Weaknesses:\n1. My major concern is that the proposed method will transform a PLL problem into an UPLL problem, which is more challenging due to the existence of the correct label may not be guaranteed in the candidate label set. Although it provides an upper bound of the per-example pruning error rate, the negative impact of eliminating the correct label from the candidate label set is still unknown.\n2. The proposed method is dependent on the KNN algorithm, which should be given more details in the main body of the paper. For example, it could be found in the appendix that the KNN algorithm are implemented on the output of a feature extractor. However, what the feature extractor comes from is unknown.\n3. [1] also attempts to filter out the incorrect candidate labels, which is suggested to be considered in related works, and even experiments.\n\n\n[1] Xu, Ning, et al. \"Progressive purification for instance-dependent partial label learning.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8yefDy5J8Q", "reviewer": "Reviewer_i6zq", "summary": "This paper present a novel insight into PLL from the perspective of candidate label set pruning (CLSP), and propose the first CLSP method based on a \"down-vote\" kNN. The authors also theoretically analyze the effects of the feature quality and label ambiguity against the pruning error. Extensive experiments are conducted on various datasets to validate the superiority of CLSP.", "questions": "1. A small typo? I guess the author may forget to divide n in the formulation of \\beta in Definition 1, as \\beta does not equal to 1 but n in the optimal pruning case. Can author clarify that?\n2. It is vague to describe PLL in the abstract that \"Partial-label learning (PLL) allows each training example to be equipped with a\nset of candidate labels.\" Does the author intentionally ignore the assumption that \"only one is the ground-truth label\"? It is better to clarify as some PLL research wave the limitation to investigate a new PLL task (called Unreliable or Noisy PLL). \n3. Some state-of-art PLL methods are missing in the reference, such as, A Unifying Probabilistic Framework\nfor Partially Labeled Data Learning; Mutual Partial Label Learning with Competitive Label Noise.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is fresh to PLL community from a data prunning perspective, and propose the first candidate label set pruning (CLSP) method based on kNN. The theorectical analysis of the prunning error is reasonable and the comprehensive experimental results validate the effectiveness of the proposed method. The paper is well organized and the expressions are clear. This work is excellent, and will inspire many researchers in PLL community.", "weaknesses": "No big problem, only some minor issues, such as, typo, vague expression, and missing reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)