Mutual Partial Label Learning with Competitive Label Noise
Authors: Yan Yan
Yuhong Guo
Venue: ICLR 2023 poster (2023)
Keywords: Partial label learning
label noise
classification

Abstract: Partial label learning (PLL) is an important weakly supervised learning problem, where each training instance is associated with a set of candidate labels that include both the true label and additional noisy labels. Most existing PLL methods assume the candidate noisy labels are randomly chosen, which hardly holds in real-world learning scenarios. In this paper, we consider a more realistic PLL scenario with competitive label noise that is more difficult to distinguish from the true label than the random label noise. We propose a novel Mutual Learning based PLL approach named ML-PLL to address this challenging problem. ML-PLL learns a prediction network based classifier and a class-prototype based classifier cooperatively through interactive mutual learning and label correction. Moreover, we use a transformation network to model the association relationships between the true label and candidate labels, and learn it together with the prediction network to match the observed candidate labels in the training data and enhance label correction. Extensive experiments are conducted on several benchmark PLL datasets, and the proposed ML-PLL approach demonstrates state-of-the-art performance for partial label learning.

Published as a conference paper at ICLR 2023
MUTUAL PARTIAL LABEL LEARNING WITH COMPETITIVE LABEL NOISE
Yan Yan', Yuhong Guo
'Carleton University, Ottawa, Canada yanyane cunet. carieton..c.o ca, yuhong guo8carleton. ca
CIFAR Al Chair, Amii, Canada
ABSTRACT
Partial label learning (PLL) is an important weakly supervised learning problem, where each training instance is associated with a set of candidate labels that in- clude both the true label and additional noisy labels. Most existing PLL methods assume the candidate noisy labels are randomly chosen, which hardly holds real-world learning scenarios. In this paper, consider more realistic PLL sce- nario with competitive label noise that is more difficult to distinguish from the true label than the random label noise. We propose novel Mutual Learning based PLLL approach named MI.-PLL to address this challenging problem. MI.PIIII learns a prediction network based classifier and a class- pass-protooypp based classifier coop- eratively through interactive mutual learning and label correction. Moreover, we use a transformation network to model the association relationships between the true label and candidate labels, and learn it together with the prediction network to match the observed candidate labels in the training data and enhance label correc- tion. Extensive experiments are conducted on several benchmark PLL datasets, and the proposed ML-PLL approach demonstrates state-of-the-arr performance for partial label learning.
INTRODUCTION
As it is costly and difficult to annotate each instance with precise label, weakly supervised learn- ing (WSL) has been widely studied in recent years (Zhoul 2018), which includes, but not limited to, semi-supervised learning (Van Engelen & Hoos 2020 Ouali et al., 2020). noisy label learning (Natarajan et al.) 2013, Feng et 2021), positive-
positive-unlabeles unlabeled learning (Kiryo et al. [017] [hu et al.) 2020), and partial multi-label learning (Xie & Huang 2018 Yan & |uo Partial label learning (PLL) is typical WSL problem and aims to learn model from training samples with overcomplete labels; that is, each training sample is associated with a set of candidate labels that include both the true label and additional label noise-noisy labels. PLL has been widely applied in many real-world learning scenarios, including automatic face naming (Hüllermeier & Beringer] (0066] Zeng et al, 2013 web mining (Luo & Orabona 2010), and multimedia content analysis (Zeng et al. 2013).
Since the ground-truth label hidden in the candidate label set and not available to the learning algorithms, the main challenge of PLL lies in candidate label disambiguation. To address this chal- lenge, two main label disambiguation strategies have been proposed: average-based disambiguation strategy and identification--ase candidate label equally in the model training phase and averages all the modeling outputs from each candidate label in the testing phase (Cour et al.) [201]] Hüllermeier Beringer 2006) Zhang & Yu 2015). Although this strategy is simple and clear, it can make the true label overwhelmed by noisy labels without sufficient differentiation, and lead to poor prediction performance. The identification- based disambiguation strategy treats the ground-truth label as a latent variable and tries to identify the true label by deriving different confidence scores for the candidate labels (Feng & An 2018 2019 Yao et al. 2020b Yu & Zhang 2016 Xu et al. 202T). The identification-base
disambiguation strategy. Average-based disambiguation treats each
disambigua- tion approaches are able to achieve relatively better prediction performance than the average-bant based disambiguation approaches by handling the candidate labels with discrimination, but they still suffer from the potential drawback of accumulating label identification errors and severely disrupting the subsequent model training. In addition, these existing methods usually restricted standard

---

Published as a conference paper at ICLR 2023
machine learning frameworks with linear or kernel-based approaches, which have difficulties to deal with large-scale datasets.
Recently, deep learning has been widely used in addressing PLL problems (Feng et al. 2020b
Wu et al.) 2022). For example, the work in (Feng et al., (020b) proposes two PLL methods, a risk-consistent and classifier-consistent algorithms, based deep neural networks. The methods in (Wen et al.] 2021] Lv et al.] (020) try to progressively identify the ground-truth label during training procedures by employing self-training technique. A more recent work in (Wang et al. 2022b) tackles PLL by using contrastive representation learning with class-prototype based label disambiguation. Although achieves satisfactory prediction performance, the contrastive learning procedure is time-consuming and resource-dente -ew.urdddddddiing.... Moreover, all these methods have a common drawback: they automatically assume random noise in the label space; that is, they assume the candidate noisy labels are randomly sampled from a uniform generating procedure. However, during the noisy label creation process real-world scenarios, such as crowd-soureing, the noisy labels are typically associated with the true label and dependent on the input sample, which makes the label noise more difficult to distinguish from the ground-truth label than a random label.
In this paper, we consider more realistic and challenging PLL learning scenario, where the noisy labels are competitive and difficult to distinguish from the true label given the imput data sample. Intuitively, competitive label noise can demonstrate stronger association relationships with true label than a random label noise, and hence are more likely to be chosen as candidate labels. For example, for online image annotation, when the object contained in an image is an "alpaca", competitive label "camel" could have a large probability to be chosen by an annotator with limited expertise due to the similar appearances of "alpaca" and "camel", while labels such as "dog" or "duck" are less likely to be picked as part of the candidate label set due to their relatively weak association with the ground -truh "alpaca".
Motivated by the above consideration, we propose novel and effective Mutual Partial Label Learn- ing approach (ML-PLL) under the competitive label noise learning scenario, which learns pre- diction network based classifier and a class-prototype based classifier interactively through label correction and mutual learning. Specifically, ML-PP.L performs noisy label correction by integrat- ing the outputs of both the prediction network classifier and the class-prototype based classifier, while using the corrected pseudo-labels as the targets for the prediction network training and using the output of the prediction network as target for the class- pesstpprt based classifier training. In ad- dition, a transformation network is proposed to model the association relationships between the true label and the noisy candidate labels, which can further enhance the classifier training with respect to ground-truth label disambiguation. Extensive experiments are conducted on several benchmark PLL datasets, while the proposed approach, ML.PLL., demonstrates state-of-the-art performance.
2
RELATED WORK
2.1 STANDARD PARTIAL LABEL LEARNING
The main challenge for addressing the PLL problem lies in how to disambiguate the candidate la- bels. Average-basee based disambiguation strategy and identification-base disambiguation strategy two main strategies deployed in PLL. The werage--ased disambiguation strategy treats each candi- date label equally in the model induction and averages all the modeling outputs from all the candidate labels as the final prediction. For example, the works in (Cour et al., 2011 Zhang & Yu (015)) distin- guish the averaged candidate label prediction from the non-candidate ones. The identification
disambiguation strategy treats the ground-truth label as a latent variable and identifies the true label by deriving confidence scores for all the candidate labels (Feng An 2018: 2019 Yu & Zhang
2016). For example, the works in (Zhang et al.) 2016: Xu et al. [019 Wang et al. 2022a) try to ddentify the true label by employing iterative label refningg procedures and leveraging the topologi- cal information in the feature space. However, these methods may suffer from the cumulative errors induced in the error-prone label confidence estimation along the topological structure. In addition, number of PLL methods propose to employ the off-the-shelf learning techniques, such maxi- mum likelihood, k-nearest-nen neighb,, maximum margin, boosting, and error-correcting output codes (ECOC), to tackle PLL problems. For the maximum likelihood technique, the likelihood of each PL training sample is defined over its candidate label set instead of its implicit ground-truth label [Liu & Dietterich] 2012). For the k-nearest neighbor technique, the candidate labels from neighbor
2

---

Published as a conference paper at ICLR 2023
instances are aggregated to produce the final prediction on test instance (Hüllermeier & Beringer 2006 Gong et al. [017] Zhang & Yu 2015). For the maximum margin technique, the classification margin is defined over the predictive difference between the candidate labels and the non-candidate labels for each PL. training sample (Nguyen & Caruana 2008) Yu & Z/hang] 2016). For the boosting technique, the weight of each PI. training instance and the confidence value of each candidate label being the truth label are refined in each boosting round (Tang & Zhang] (017). For the ECOC tech-
nique, the binary ECOC classifiers are built based on the binary training sets filtered by the candidate labels (Zhang et al. 2017).
2.2 DEEP NEURAL NETWORK BASED PARTIAL LABEL LEARNING
Standard PLL methods are usually restricted to linear based models, which have difficulty to deal with large-scale datasets. Deep learning based PLL has recently started gaining attention from the research community due to its powerful capability of training on large-scale datasets. Yao backbones and exploit the temporal- ensembling technique for model training. Yao et al. (2020b) propose to train two networks a mutual learning manner, where both networks interact with each other and alleviate the error accu- mulation problem. [yy et al. (2020) attempt to progressively identify the true labels in a self-training manner, which can be compatibly trained with stochastic optimization. Feng et al.) (2020b) assume that the probability of each candidate label being chosen is uniform and present two provably con- sistent methods: a risk-consistent method and ctossseee method. Another work in (Wen et al. 2021) induces family of loss functions, which generalizes the uniform generation procedures. In addition, a recent work in (Wang et al. 2022b) handles PLL by conducting class prototype based label disambiguation following the idea of contrastive representation learning (He et al. 2020 Li
eet al.] (2020a) adopt deep convolutional neural networks
et al. 2020). which achieves impressive prediction performance.
Although the abovementioned approaches produce competitive performance, most of them assume the noisy labels in the candidate label set are randomly produced, which does not hold in many real-world learning scenarios. By contrast, we tackle more realistic and challenging PLL scenario, where the noisy candidate labels are competitive and difficult to distinguish from the true label.
PROPOSED APPROACH
In this section, we delineate the details of our proposed mutual partial label learning model (ML- PLL) in the learning scenario with competitive label noise. Given partial label training set D (X,Y) Y) = {(x, T- where x, E X' denotes the i-th instance expressed in the input space X, and y; E 10. denotes the candidate label indicator vector associated with x;. The multiple values in y; denote either the ground-truth label or irrelevant noisy labels, which form the candidate label set S, C y/ for x,, where y = {1, ., ,, L} denotes the multi-class label space. The task of PLL is to induce multi-class classifier from the PLL training set D. We assume a challenging realistic PLL scenario, where the noisy labels in the candidate label set are difficult to distinguish from the ground-truth truth label given the input instance.
The proposed ML-PLL model is illustrated in Figure [ which mainly comprises the follow- ing component networks: feature extractor G that extracts features from the input instance G(x) E Rm, a projection network H that maps the feature vector v into a low-dimensional em- bedding H(v) E IR:: a prediction network based classifier F; class- protooype based classifier; and a transformation network 7' that estimates the association relationships between the true label and candidate labels. The prediction network based classifier and the class-prototype based classifier are learned together through noisy label correction and interactive mutual learning, while the predic- tion network and the transformation network are learned together match the observed candidate labels in the training data and enhance label disambiguation. We elaborate these components and the proposed approach in the following subsections.
3.1 NOISY LABEL CORRECTION
The main challenge of PLL lies in that the ground-truth -ruth label coexists with the irreverent noisy labels in the candidate label set and is unknown to the learning algorithm. The key of PLL is to correct the candidate label vectors and identify the true label vectors. Given an interactive pair
3

---

This noisy label correction procedure can be applied on the n training instances to produce corrected soft pseudo-label vectors ....... J. With the progress of prediction network training and glass-prototype update (elaborated the next subsection), we expect each produced soft pseudo- Published as a conference paper at ICLR 2023
Lcon
R
Updating
Candidate labal sot
Prediction
*
Prediction
Prontypa- based prediction P
+11
A
G
Updesing
O
Embedding
Protolype C
Figure I: The proposed ML-PLL model. It learns the prediction network based classifier F and class-prototype based classifier in mutual learning manner, while the transformation network T is learned cooperatively with the prediction network to match the observed candidate labels.
of classifiers prediction network classifier F and a class-prototype based classifier, propose the following simple mutual label correction procedure. For any training instance X,, we correct its partial label indicator vector towards the true label indicator vector by exploiting the predicted pseudo-label vector. Specifically, we produce the soft pseudo-label vector qi E 1)4 for x; by taking weighted convex combination of the prediction network's softmax output probability vector F(v,) and the class-prototype based prediction probability vector p.
qi = AF(v:) + (1 A)P:
(1)
where À E (0, 1) is the combination weight parameter. Here the prediction network built above the feature extractor G, such as F(v,) F(G(x,)) while p, is produced by the similarity-base class-prototys classifier as follows:
exp(z; Ct/T)
Pik - exp(z; /T)
where r is a temperature parameter, T. is the number of class categories, z; Hx)(x;)) denotes the low-dimensional embedding of the features for instance and denotes the prototype vector for the k-th category, which will be elaborated in the next subsection. Instead of using Euclidean distance, use the dot-product operator "." to compute the inner product similarity between the instance and the class prototypes in the embedding space for class prediction probability calculation.
To maintain effective soft pseudo-label vector for each instance and eliminate the noise predic- tions outside of the candidate label set, we further restrict the soft pseudo-labels within the candidate label set and transform q; into q; with row normalization,
Gike
if Yik O;
Σ
7sk
otherwise.
label vector will be better aligned with the corresponding true label indicator vector, which can interactively provide a good target for the subsequent prediction network training and gradually mitigate the negative impact of the label noise during the training process.
4

---

Published as a conference paper at ICLR 2023
3.2 BATCH- WISE MOVING AVERAGE BASED PROTOTYPE UPDATE
The effectiveness of noisy label correction relies on accurate class probability predictions, which depend both the prediction network and the class-prototypes. Instead of training an entirely independent class-prototype based classifier, we propose to produce the class prototypes based on the output of the prediction network classifier. Specifically, we compute the class prototypes in the low-dimensional embedding space from the partial label training set D in the following way:
C₁ EXTED H(G(x.) 1(5x== 11)
if k arg max F(G(x;))
for You otherwise
kes,
(4)
2 1)
where is an indicator function; y, denotes predicted one-hot label vector for x,, which is deter- mined by the output of the prediction network F and restricted to the candidate label set Sq. In this way, we can automatically eliminate the prediction noise outside of the candidate labels.
With the prototype computing method above, we can update each class-prototype by recomputing during each training iteration. However, such an update process will be computationally intensive and lead to intolerable training latency, since it requires going through the whole training set. To ad- dress this problem, we propose to perform batch-wise moving average based prototype update; that is, we update the class-prototypes for the current mini-batch by consulting the prototypes computed from previous mini-batches in a moving average manner:
Normalize(rea + (1 (1-7)ck).
where dg is the mean embedding for the k-th class within the current mini-batci which can be computed with Eq. (4) by limiting D to the current mini-batch; E [0. 1] is momentum coeffi- cient; and Normalizo(c) (c) = c/||c/12- We expect such aatch-wise prototype update method can be computationally efficient and enhance the gradual adjustment and stability of the class prototypes.
3.3 PREDICTION NETWORK TRAINING WITH MUTUAL LEARNING
With the noisy label correction method above, the corrected soft pseudo-label vectors can be used target for prediction network training. Specifically,
learn the prediction network based probabilistic classifier on the training data D by minimizing the following classification loss:
C,(,,G,H)= 1
Σ Σ n XxNDk=1
Toelog( (G(x:)))
(6)
where Ti denotes the k-th entry of the corrected soft pseudo-label vector q. for the i-th instance, and (.)(.) denotes the k-th entry of the prediction network's softmax output, indicating the predicted probability of the instance belonging to the k-th class. We expect the prediction network based classifier F can be improved by training with the corrected soft pseudo-labels.
Meanwhile, we can learn an additional class caasotprtttype based classifier by using the output of the prediction network as prediction target and deploying the following mutual learning loss:
C G. H)
1 Σ
Σ - (G(xx)))oo(p))
X k=1
where piu is the predicted probability for the i-th instance belonging to the k-th class by the prototype based classifier using [q. This mutual learning loss is expected to enforce both classifiers to collaborate with each other and gradually lead to consistent predictions that can better align with the underlying true label indicator vectors.
3.4 TRANSFORMATION BASED LABEL ASSOCIATION LEARNING
As previously discussed, the noisy labels in the candidate label set typically have association lationships with the ground-trun -ruth label and can be competitive with the true label given the input instance in the real-worle learning scenarios. In this subsection, association relationships between the noisy candidate labels and the true label using a transformation network T, aiming to enhance the label disambiguation process.
propose learn the competitive
5

---

Published as a conference paper at ICLR 2023
We assume the transformation network T can be learned to produce transformation matrix R = [ray] EXL= such that R can capture the transformation relationships from a given ground-truth label to the associated noisy candidate labels and recover the candidate label set. The entry Ti of R reflects the probability of transforming the j-th label to a candidate label when the i-th label is the ground-truth. When rij is large, the j-th label has strong association relationship with the i-th ground-truth label and hence has a larger probability to be chosen as candidate label. Towards this goal, we propose to produce the transformation matrix R by applying the transformation network T on the observed candidate label matrix Y = Ty xx from the training set, such that R = T(Y), since the observed candidate label matrix contains both the true labels and the associated noisy can- didate labels. The transformation network 7 can then be learned through prediction based candidate label matrix reconstruction. Specifically, we propose to reconstruct the normalized candidate label matrix by transforming the prediction outputs of the prediction network based classifier F with the transformation matrix R. We use the symmetric Kullback-Leiblee (KL) divergence to match each re- constructed candidate label distribution with its corresponding observed candidate label distribution and form the following consistency loss:
Loon (1,
E (KL(F(G(())) (XL,) X,rD
where V, Y//1/ is row normalized candidate label matrix, expressing the candidate label distributions in the training set. It is however computationally intensive to compute the transforma- tion matrix R by taking the whole candidate label matrix Y as input. We further propose compute and update R on each mini-batch in a moving average style, such as
nR + (1 -)
where R' is the transformation matrix computed from the observed partial label matrix on the current and 1) E [0, 1] a momentum coefficient.
By exploiting the label association relationships between the true-label and the candidate labels, we expect the consistency loss in (q. can enhance noisy label correction and prediction network learning, thus consequently pushing the outputs of the prediction network towards the true labels.
3.5 LEARNING WITH ML-PLL MODEL
By integrating the classification loss in o6. the mutual learning loss in Eq. 7 and the consistency loss in Eq. (8) together, get the following overall training loss for the proposed ML-PLL model,
L Le + aLmi + BLeon
(10)
where and 3 are trade-off hyperparameters that control the relative importance of the mutual learning loss and the consistency loss, respectively. We perform training by minimizing this objec- tive with a mini-batch based stochastic gradient descent (SGD) algorithm.
4
EXPERIMENT
4.1
EXPERIMENTAL SETTING
Datasets We conducted experiments on four widely used benchmark image datasets: Fashion- MNIST (Xiao et al., 2017), Kuzushij-MNNIST (Clanuwat et al. 2018), CIFAR-IO and CIFAR-100 Krizhevsky et al. 2009). We transform these datasets into partial-labe datasets by using the follow- ing competitive and uniform label noise generation processes. First, to generate competitive label noise. we train a neural network on the original clean dataset to predict the probability of each cat- egory label being the true label for a given instance, and then randomly choose the irrelevant labels with high predicted probabilities as noisy candidate labels. Specifically, assume p. is the predicted label probability vector for an instance x; whose true label is class j. We first exclude the true label by setting P.j O, and then randomly choose labels from the top-K predictions in p.-i.e. the K irrelevant label categories with highest predicted probabilities as the noisy candidate labels. Intu- itively, these irrelevant labels with higher predicted probabilities demonstrate stronger association relationships and are more competitive with the true label. The chosen competitive noisy labels, to- gether with the true label j. form the candidate label set for instance x;. Following this competitive

---

Published as a conference paper at ICLR 2023
Table 1: Test accuracy (mean=std) comparison on Fashion- -NNIST, Mizzushji-MNNIS and CIFAR-
10 with competitive label noise at different label ambiguity levels. The best results are bold.
Data set
Method
Avg.#CL=33
Avg.#CL =4
Avg.#CL=
ML-PLL
0.06%
89.94 0.08%
89.37-00.09%
PICO
90.06=10.07%
89.75 10%
88.9%=0.00%%
Fashion-MNIST
LWS
89.51
88.52=0.11%
81.07=00.05%
CC
88.69==0.133%
88.23
88.05=00.36%
MSE
79.91
74.01=0.255%
67.11=
EXP
09.40:10.15%
89.15=0.1058
88.34=0.200%
ML-PLL
9..70=0..22%
0.07%
9..87=0.07%%
PICO
94.86=10.00%%
94.13=00.7%
93.95=00.05%
Kuzushij-MNIST
LWS
93.04=0.00%
9..50=0.05%
91.21 0.06%
CC
94.58
94.29
93. 0.00%
0.06%
MSE
64.99-1:0.25
63.98 0.27%
61.29 0.
EXP
0.
93.54=
92.04
ML-
84.95 0.05%
83.52
0.09%
PiCO
83.96%
82.04=0..8%%
76.42:10.11%
CIFAR-10
LWS
83.15
7..85=0.17%
49.65=#0.25%
CC
73.42++0:44%%
69.57 0.48%
62.
MSE
53.43
44.21
33.52
EXP
73.93
7.224
$1.96 0.28%
Table 2: Test accuracy (se) comparison CIFAR-100 with competitive label noise at dif- ferent label ambiguity levels. The best result in each setting is highlighted in bold.
Data
Method
Avg.#CL
Avg.#CL
Avg.#CL
ML-PLL
0.10%
57.56:F0.06%
53.53
PICO
09.86=0.100%
54.07-10.08%
49.22.10.05%
CIFAR-100
LWS
55.00=10.20%
48.78-L0.37%
36.99=10.40%
CC
07.29:10.26%
51.00:10.29%
05.00=
MSE
39.01
29.76=:0.11%
18.52
EXP
0..26
40.38:10.27%
30.44-0.229%
candidate label noise generation process, produce partial-label datasets by choosing competi- tive label noise from the top-6 predictions for Fashion-MNIST, Kuzushij-MNISST and CIFAR-10, while choosing from the top-20 predictions for CIFAR-100. 100. The average number of candidate la- bels (Avg.#CL. and the characteristics of each benchmark dataset corrupted by the competitive label the uniform generation procedure in (I.v al., 2020) to generate uniform label noise. It generates noisy candidate labels by flipping each incorrect label with a probability p, which indicates the label ambiguity level. When encountering a special case where no irrelevant label is chosen, we randomly select one irrelevant label as the additional candidate label to ensure the label sets for all training instances are corrupted.
noise are reported in Appendix A.2] Second,
Comparison Methods We compare the proposed ML-PLL method with the following deep neural network based PLL methods: PiCO (Wang et al.) 2022b) is contrastive learning based state-of-ther art PLL method, which disambiguates the candidate labels by refining the pseudo-labels a moving- average manner; LWS (Wen et al.) 2021) is a discriminative PLL method, which considers the loss trade-off between the candidate labels and non-candidate ones; CC (Feng et al. 2020b) is a classifier- consistent method, which leverages transition matrix induce empirical risk estimator; MSE and EXP (Feng et al.] 2020a) are two baselines that exploit mean square error and exponential loss, respectively, risks for inducing unbiased risk estimators. For each comparison method, the same backbone structure for feature extraction and adopt the same predictive model. Moreover, all the comparison methods are configured with the suggested parameters according to the source literature. For each experiment, we report the average test accuracy and standard deviation based on independent runs.
Implementation Details We use LeNet-5 as the backbone on the Fashion-MNIST and Kuzushiji- MNIST datasets, and use 18-layer ResNet as the backbone on CIFAR-I0 and CIFAR-100, for feature extraction. The projection network is a two-layer MLP that outputs 128 28-dimensional embeddings. The prediction network is linear classifier. The weighted combination parameter À in Eq. the temperature parameter 7 in (q. and the momentum coefficients in (5 and (q.(9) are set to 0.99, 1, 0.999, and 0.99 respectively. In all the experiments, utilize standard SGD optimizer
7

---

Published as a conference paper at ICLR 2023
Table 3: Test accuracy (mean=std) comparison on Fashion-MNIST. Kuzushiji-MNIS, and CIFAR-
10 with uniform label noise at different label ambiguity levels. The best results are in bold.
Data set
Method
p p=1.11
p
p 0.5.
ML-PLL
0.055
92.23=00.129
91.86-00.08%
PICO
9..90=0.00%%
93.11 0.08%
92.38=10.03%
Pushion-MNISS
LWS
91.44=+0.13%
91.85=0..14 0.114
90.59=
CC
92.26=t:0112%
91.75=00.04%
90.92=00.06%
MSE
07.4%
87.02
85.76=10.08%
EXP
91.02:+00.06%
89.88=0.069
89.21
ML-PLL
97.63=
97.45
96.69-0.005%
PICO
97.58-+0.06%
97.32=0.00%%
-..05%
Kuzushij-MNIST
LWS
96.22 10%%
96.15%0.24%
9.043=
CC
0..4%%
0.00%
95.62 10%
MSE
82.87
81.20
EXP
0..3%
94.21%0005%
93.56=
ML-
9.16410.11%
94.05=
92.60=10.21%
PiCO
9.
9..18
93.5
CIFAR-10
LWS
90.30=10.6099
88.99 1.3%%
86. 0.85%
CC
82.30
79.08 0.07%
74.05 0.335%
MSE
0..57%
EXP
79.23 0.110%
75.79
70.34 1.32%
Table 4: Test accuracy (mean-Estd) comparison on CIFAR- 100 with uniform label noise at different label ambiguity levels. The best result in each setting is highlighted in bold.
Data
Method
p 000.11
p 0.05
p 0.1
ML-PLL
0.15%
73.06:1:0.33%
72.51
PICO
73.09:10.34%
72.74-:0.30%
69.91-0.11%
CIFAR-100
LWS
65.78=+0.02%
59.56:1:0.33%
53.53=+0.08%
CC
49.76=10.45%
47.62.10.08%
35.72-10.47%
MSE
47.45 1.50%
45.05
33.27
EXP
49.17
16.02-11.822
10.4%%
with momentum of 0.9 and weight decay of 1e-3 for model training. The mini-batch size, learning rate and total training epochs are set to 128, 0.01 and 400 respectively. The parameter a and A are chosen from {0.5, 0.6, 0.7, 0.8, 0.9, and 21,2, 4, 5, 6), respectively, according the accuracy on validation dataset (10% of the training dataset).
4.2 EXPERIMENTAL. RESULTS WITH COMPETITIVE LABEL NOISE
We compared the proposed ML-PLL method with the five state-of-the- art comparison methods on the four datasets corrupted by competitive label noise. The comparison results on Fashion-MNISTT, KuzushijiMNISST and CIFAR-10 with different label ambiguity levels, indicated by the average number of candidate labels, are reported in Table [ From the table we can see that the proposed method consistently outperforms all the other comparison methods, and the performance gains yield by ML-PLL over the other methods are quite notable many cases. For example, ML-PLL outper- forms the best alternative comparison method by 1.48% and 2.83% on CIFAR- 10 with Avg.#CL= 4 and respectively. The experimental results on the more challenging dataset CIFAR- 1000 which has large label space (7. = 100), are reported in Table 2 From the table we can see that the pro- posed MI.PPLI again outperforms all the other comparison methods and demonstrates remarkable performance improvement on such complicated dataset. For example, ML-PLL improves the best comparison method, PiCO, by 1.53%, 3.49% and 4.31% respectively across the three label ambi- guity levels. Moreover, the comparison methods e.g, PiCO, LWS, and cC-perform well with the low label ambiguity level (Av2, but their performance degrades dramatically with the increase of the label ambiguity level. By contrast, the proposed ML-PLL method not only consis- tently outperforms the other methods with notable performance gains, but also produces larger gains with higher label ambiguity levels. All these results validate the efficacy of the proposed ML-PLL approach in addressing the competitive label noise.
4.3 EXPERIMENTAL RESULTS WITH UNIFORM LABEL NOISE
To further validate the effectiveness of the proposed method, we also conducted experiments on the four datasets (Fashion-MNISST, Kuzushij-MNISST, CIFAR-10 and CIFAR- 100) corrupted by the
8

---

Published as a conference paper at ICLR 2023
Table 5: Ablation results on Fashion-MNIST, Kuzushiji-MNIST, and CIFAR-10 with Avg.#CL = and on CIFAR- with Avg.#CL
Fashion M
KuzushijiMMNIST
CIFAR-I0
CIFAR-100
Ablation variant
(Avg.#CI.=
(Avg.#CI.
(Avg.#CI. 4)
(Avg.#CI, 6)
Full Model
89.94%
95.38%
83.52%
57.56%
ML-PLL-wlo-ml
86,49%
92.18%
80.17%
55.27%
ML-PLL-woo-col
84,93%
91.41%
79.08%
54.15%
ML-PLL-wWo-lc
79.94%
82.10%
51.64%
37.51%
CLS
75.90%
77.66%
47.06%
33.60%
uniform label noise. The label ambiguity level in this scenario is controlled by the p value- a larger probability for producing noisy candidate labels leads to larger candidate label sets. All the com- parison results reported in Table B and Table 4 From Table E we see that although the proposed method is not specially designed for uniform label noise, it still achieves competitive per- formance with the best method across all cases. In T the proposed method even demonstrates substantial superiority over the other methods and produces larger performance gains with higher label ambiguity levels. These results on the challenging CIFAR-100 dataset are consistent with the ones reported in the scenario with competitive label noise that is, the proposed ML-PLL method yields substamtial and increasing performance gains when the label ambiguity level increases. All these results demonstrate the efficacy of the proposed method in handing both competitive and uni- form label noise for PLL.
4.4 ABLATION STUDY
The objective of the proposed method contains three loss terms: the classification loss, the interactive mutual learning loss, and the consistency loss. In addition, noisy label correction is an indispens- able component in the proposed mutual learning model. To validate each part's contribution, we conduct an ablation study to compare the proposed MI.PPL method with the following ablation variants: (1) ML-PLL-w/o-ml, which drops the mutual learning loss from the full model; (2) ML- PLL-W/o-con, which drops the consistency loss; (3) ML-PLL--Wo-lc, which drops the noisy label correction procedure; and (4) CLS, which only
standard classification loss by dropping the mutual learning loss, the consistency loss and the noisy label correction procedure. The comparison results are reported in Table 5 From the table the four variants produced interior results, which suggests that all the components contribute to the full model in different degrees. Among the three variants of MI-PL.L-w/o-ml, ML-PL.L-W/o-com and ML-PLL-w/o-Ic, ML-PLL-w/o-lc produces the largest performance degradation, which demon- strates the importance of the noisy label correction procedure for label disambiguation and classifier learning. Meanwhile, ML-PLL-w/o-ml ml and ML-PLL-w/o-coE notable performance drops from the full model, which validates the contributions of both the mutual learning loss and the transformation network based consistency loss. The results suggest exploiting the association relationships between the competitive label noise and the ground-truth label can help PL... Moreover, the large performance gap between the full model and the baseline variant CLSS further validates the integrating power of the proposed method on combining all these three con- tributing components for PLL with competitive label noise: noisy label correction, mutual learning, and transformation network based label association exploitation.
see that compared with the full model, all
also produced consistent and very
CONCLUSION
This paper tackles more realistic PLL scenario with competitive label noise that can be more diffi- cult to distinguish from the ground-truie label than random noisy labels. We propose a novel mutual partial label learning approach called MI-PLL to address this challenging problem. ML-PLL learns a prediction network based classifier and a class-prototype based classifier cooperatively with label correction under a mutual learning framework. Moreover, we introduce transformation network to model the association relationships between the ground-truth label and the candidate noisy labels, and learn together with the prediction network through candidate label set reconstruction to en- hance label correction. The comprehensive experiments several benchmark datasets corrupted with both competitive and uniform label noise demonstrate that the proposed approach significantly outperforms the state-of-the-art PI.. methods.

---

Published as a conference paper at ICLR 2023
REFERENCES
Tarin Clanuwat, Mikel Bober-Irizar Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research (JMLR), 12: 1501-1135 2011.
Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In International Joint Conference on Artificial Intelligence (LJCAI), 2018.
Lei Feng and Bo An. Partial label learning with self- seff-guided retraining. In AAAI Conference Artificial Intelligence (AAAI), 2019.
Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with multiple complementary labels. In International Conference on Machine Learning (ICML), 2020a.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial-label learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li, and Bo An. Can cross entropy loss be robust to label noise? In International Conference on International Joint Conferences on Artificial Intelligence (ICCAI), 2021.
Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang, Jie Yang, and Dacheng Tao. A regularization approach for instance-basee based superset label learning. IEEE Transactions on Cybernetics, 48(3): 967-978, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis, 10(5):419-439, 2006.
Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.0799) 2020.
Liping Liu and Thomas Dietterich. A conditional multinomial mixture model for superset label
learning. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
Jiqqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identifi- cation of true labels for partial-label learning. In International Conference on Machine Learning (ICML), 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Nam Nguyen and Rich Caruana. Classification with partial labels. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2008
Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep supi-suue learning. arXiv preprint a 20Xx:2006.05278 2020.
10

---

Published as a conference paper at ICLR 2023
Senlin Shu, Zhuoyi Lin, Yan Yan, and Li. Learning from multi class positive and unlabeled data. In IEEE International Conference on Data Mining (ICDM), 2020.
Cai-Zhi Tang and Min-Ling Zhang. Confidence-rateed discriminative partial label learning. In AAAI
Conference on Artificial Intelligence (AAAI), 2017.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning, Machine Learn- ing, 109(():373-44)) 2020.
Deng- Bao Wang, Min-Ling Zhang, and Li Li. Adaptive graph guided disambiguation for partial label learning. IEFE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44 (12):8796-8811, 2022a.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: contrastive label disambiguation for partial label learning. arXiv preprint arXiv:2201.0895, 2022b.
Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning. International Conference on Machine Learning (ICML), 2021.
Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. International Conference on Machine Learning (ICML). 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (ee::170877777777 2017.
Ming-Kun Kun Xie and Sheng-Jun Jun Huang. Partial multi-label learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yan Yan and Yuhong Guo. Adversarial partial multi-label learning with label disambiguation. In AAAI Conference Artificial Intelligence (AAAI), 2021.
Yao Yao, Jiehui Deng, Xiuhua Chen, Chen Gong, Jianxin Wu, and Jian Yang. Deep discriminative cnn with temporal ensembling for ambiguously-laellee image classification. In AAAI Conference on Artificial Intelligence (AAAI), 2020a.
Yao Yao, Chen Gong, Jiehui Deng, and Jian Yang. Network cooperation with progressive disam- biguation for partial label learning. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2020b.
Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In Asian conference on machine learning (ACML), 2016.
Zinan Zeng, Shijie Xiao, Kui Jia, Tsung- Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based ap- proach. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
Min-Ling Zhang, Min-Bin Zhou, and Xu- YunYing Liu. Partial label learning via feature- disam- biguation. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions on Knowledge and Data Engineering, 29110):155-2116, 2017.
Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1): 44-53, 2018.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING ALGORITHM FOR ML-PLL
We present the mini-batch based training algorithm for ML-PLL in Algorithm !
Algorithm Training Algorithm for ML-PLL.
Input: PLL training set D; the initialized model; trade-off hyperparameters a and ,..
for s 1: max_mumm ______ummmpphte do
for t= 1: num iterations do
Sample a batch with m samples from D.
// Noisy label correction
Compute the class prototype- based prediction probability vectors (P:} via 2q.2)
Perform noisy label correction to get {@:} via Eq. and Eq.(3)
Batch-wise moving average based prototype update
Compute the class prototypes via Eq
the current batch B.
Update the class prototypes via Eq.
I/ Update of the transformation matrix
Compute and update the transformation matrix via Eq.
// Classification loss calculation
(F G, H) = 1 Σx Ex,=B B ExEr - (G(x:)))
Interactive mutual learning loss calculation
mm (F, G, H) = + Expr
B Ek= - (G(x)))log(pa)
11 Consistency loss calculation
Loo G, H, = Ex,rB (L((F(G())) KL, F(G()))))
Network updating
== Le + aLml + BLeoon
Update network F, and via stochastic gradient descent by minimizing L.
end for
end for
A.2 DATASET
We produce PLL datasets with different average number of candidate labels (Avg.#CL) from each benchmark dataset using the competitive label noise generation process described in subsection The average number of candidate labels (Avg##CL) and the characteristics of each benchmark dataset corrupted by the competitive label noise are recorded in Table 6
4.1
A.3
PARAMETER SENSITIVITY ANALYSIS
The trade-off parameters a and 3 control the relative importance of the mutual learning loss and the consistency loss, respectively. We conduct experiments on CIFAR-10 the label ambiguity level of Avg#CL = 4 to study the impacts of a and 3 on the performance of ML-PLL. We in- vestigate the impact of a by experimenting with different values from (0.5, 0.6, 0.7, 0.8, 0.9, 1} while setting 8 = 4. and investigate the impact of 3 by experimenting with different 3 values from 21, 4, 5, 6) while setting a 0.6. The results are presented Figure a and Figure 2(b) respectively. From Figure 2(a) we can see the best performance is achieved when
0.6, while the performance difference is very small for varying a within [0.5, 0.7]. But further increasing a towards I leads to performance degradation. The results in Figure 2(b) however demonstrate larger variations with different 8 values, suggesting the performance value. When 3 is very small, the performance of ML-PLL is relatively poor. With the increase of the 3 value, the performance improves as the consistency loss begins to work. But when / is too large, the performance degrades again since the consistency loss over-dominate.
ML-PLL is more sensitive to the
12

---

Published as a conference paper at ICLR 2023
Table 6: Characteristic of the datasets corrupted by the competitive label noise generation process.
Dataset
#Train
#Test
#Feature
Classes
Avg.#CL
Fashion-MNIST
60,000
10.000
784
10
3, 4, 5
KuzushijiMMNIST
60,000
10,000
784
10
3,4,5
CIFAR-10
50,000
10,000
3,072
10
3,4,5
CIFAR-100
50,000
10,000
3,072
100
2, 10
85
84-
classification accuracy
classification accuracy
84
822
83
82-
76
74
80
0.5
0.6
0.7
0.8
0.9
B
Vary the parameter
(b) Vary the parameter /
Figure 2: Test performance with different and values for the proposed ML-PLL on CIFAR-10.
OURS
LWS
CC
EXP
90p
80
classilication accuracy
40
20
Top-8
Top-6
Top-5
Top-4
Figure 3: Impact of the competitive noise level for ML-PLL on the CIFAR-10 10 dataset.
A.4 IMPACT OF COMPETITIVE NOISE LEVELS
As described in subsection we generate the competitive label noise by randomly selecting ir- relevant labels from the top-K predicted categories, where K controls the competitivity level of the chosen noisy candidate labels. A smaller K value forces the noisy candidate labels to be chosen from the fewer irrelevant labels with the highest few predicted probabilities of being the true label, and hence indicates a higher competitive noise level. Here we conduct experiments on CIFAR- 10 to investigate the impact of the competitive noise level by producing partial- training data using different K values from {4, 5, 6, 8} but the same average number of candidate labels, Avg#CL = 4. The comparison results are presented Figure We can see that with the decrease of the K value and consequently the increase of the competitive noise level, the performance of all the comparison methods degrades general. However, the proposed ML-PLL method consistently outperforms all the other comparison methods. Moreover, the performance of ML-PLL is relatively more stable across different K values, and its performance gains over the other methods become larger with the increase of the competitive noise level (the decrease of K), while the other methods encounter more dramatic performance degradation with smaller K values. These results again validate the effectiveness of ML-PLLL in addressing competitive label noise.
13

---

Published as a conference paper at ICLR 2023
Table 7: Accuracy (mean--sst)) CIFAR-10 with different label noise generation processes.
Method
ML-PLL
LWS
CC
EXP
MSE
Casc (I)
9..91=0.04%
90.82+0.03% 69.42:0.07%
90.57=0.11% 69.53=0..12%
76.43-00.08% 60.47=0.14%
65,8310, 19%
Case (2)
93.11=0.08%
40,03:10.20%
A.5 INFLUENCE OF NOISY LABEL GENERATION
To further investigate the impact of noisy label generation model performance, follow (Wen
eet al., 2021) to conduct additional experiments with alternative noisy label generation processes. Specifically, generate noisy labels CIFAR-I0 with the following two commonly used label flipping matrices in the form of [pg]]xx
(Wang et al.
2022b):
1
0.5
07
0.9
0.7
0.5
0.3
0.1
0
0
1
0.5
1
0.9
0.7
0.5
0.3
0.1
(1)
(2)
00.5
0
L0.9
0.7
0.5
0.3
0.1
Each entry Piy of the flipping matrix specifies the probability of the j-th label being selected as candidate label when the i-th label is the ground-truth label. We conduct experiments with the partial label data generated using these two flipping matrices and the comparison results are reported in T where case (1) and case (2) correspond using the partial label data generated with the flipping matrix (1) and (2), respectively. From the table, we can see that the proposed ML-PLL method outperforms all the other methods in both cases. As case (2) is more challenging than case (I), all the other comparison methods produce much inferior performance in case (2). However the proposed ML-PLL maintains similar good performance ML-PLL outperforms the best comparison method, LWS, by 4.09%, while in the more challenging case (2), ML-PLL outperforms the best comparison method, CC. by 23.58%. These additional experimental results further validate the effectiveness of the proposed method in addressing the partial label learning problem.
both cases. In the simple case (I),
14


Reviews:
[{"review_id": "NvzFHldV-Xk", "reviewer": "Reviewer_qLYR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a PLL approach named ML-PLL to address a more challenging setting–competitive label noise. It addresses the competitive noisy labels by following components: label correction, prediction network training with mutual learning, and transformation based label association learning, which makes couple classifiers learn and benefit from each other in a mutual learning framework thus enhancing the PLL process. Experiments validate the effectiveness of the proposed method in addressing the challenging competitive label noise.", "summary_of_the_review": "This paper explores a challenging problem and proposes a novel mutual learning framework which makes a contribution to PLL.", "strength_and_weaknesses": "Strengths:\n1. The exploration of the competitive label noise in PLL is novel and the learning process is complete.\n2. The paper exploits the transformation network and prediction network to match the observed candidate label distribution is interesting.\n3. The experiments are relatively sufficient.\n\nWeaknesses:\n1. The motivation of the paper lacks a more detailed explanation on competitive noisy labels (shouldn't the phrases be \"noisy labels\" and \"label noise\"? and \"noise\" in PLL and \"noise\" in noisy-label learning are easily confused).\n2. The instance-dependent PLL addresses the instance-dependent label noise, can you analyze the difference between the competitive noisy labels and instance-dependent noisy labels.\n3. It is better to improve Figure 1, the components and the symbols in the figure are too small.\n4. From the ablation study, we can see that dropping the noise label correction lead to largest performance degradation, can you explain more about it. For addressing the competitive noisy labels, the consistency loss would be more important to the full model. Moreover, from the parameter sensitivity analysis, we can also see that the consistency loss term needs large weight and important to the full model. So why dropping the noise label correction leads to more performance drop than dropping the consistency loss term.\n5. What’s the difference between varying the number of averaging candidate labels and competitive level k. Does increasing the averaging candidate labels can averaging candidate labels the competitive level? Similar, does increasing the competitive level can increase the averaging candidate labels?\n6. More SOTA baselins should be added, e.g., PiCo[Wang, et al., ICLR'22].", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5_AkvHGJ4N", "reviewer": "Reviewer_8WLg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a mutual learning framework called ML-PLL for partial label learning under competitive label noise. The proposed ML-PLL method disambiguates the irrelevant label noise with noise label correction and learns a couple classifiers (a prediction network based classifier and a class-prototype based classifier) simultaneously in the mutual learning framework. Experimental results validate the effectiveness of ML-PLL in addressing competitive label noise.", "summary_of_the_review": "This paper addresses a realistic learning scenario and proposes a novel approach which makes a good contribution to partial label learning.", "strength_and_weaknesses": "Strengths:\n\n1、This paper considers a more realistic PLL scenario with competitive noise labels that can be more difficult to distinguish from the ground-truth label than the random noise labels.\n\n2、The proposed method learns a standard classifier and additional class-prototype based classifier in a formalized mutual learning framework, which is interesting.\n\n3、The experiment results demonstrate the effectiveness of the proposed ML-PLL method in addressing competitive label noise when compared with existing PLL methods.\n\nWeaknesses:\n\n1、In this paper, the author learns competitive noise labels that different from the existing PLL learning scenarios. Can you give more explanation on the competitive label noise and give the difference between the ‘competitive label noise’ and ‘instance-dependent label noise’.\n\n2、The papers propose two classifiers can be used for prediction, but it is not clear how the final prediction achieved: use the prediction network based classifier, class-prototype based classifier, or average both classifiers’ prediction as the final results.\n\n3、How is the corrected label aligned with the ground-truth along the noise label correction? Since the ground-truth label is concealed in the candidate label set.\n\n4、For the Figure 3 in the appendix, can you explain more about how you vary the k value to increase the competitive level.\n\n5、In prediction network training with mutual learning, can you further explain how the two losses improve the both classifiers evaluation performance and the generalization performance.\n\n6、The author discusses the shortcomings of the existing PLL methods that cannot be used in addressing competitive label noise, but it is unclear how can we improve the state-of-arts methods and enhance their ability to competitive label noise.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5GMEbShCtoY", "reviewer": "Reviewer_CWMY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the author address the partial-label learning problem, a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label. In this scenario, the main challenge lies in candidate label disambiguation. The authors raise the problem of the selection of noise-labels (confounders), which is usually done at random. The authors argue that in a real scenario, the noise-labels are much more similar to the real one, therefore more difficult to be distinguished. Therefore, (**contrib-1**) they propose to use noise labels that are competitive and difficult to distinguish from the true label (labels with strong relationships with GT).\n\nThe second contribution (**contrib-2**) is about the partial label learning approach that is proposed, called e Mutual Partial Label Learning approach (ML-PLL). \n\n**ML-PLL is done by these main components**:  \n\n- They propose a noise label correction module and used the corrected pseudo-labels as the targets for the other components/networks of the proposed approach:\n- F: prediction network based classifier. \n- CP: Similarity based class-prototype classifier\n- T: transformation network\n\nML-PLL learns F and CP cooperatively with label correction under the mutual learning framework. The network T is then used to estimate the relationships between the true label and candidate labels.\n\nExperiments are performed on public datasets (Fashion-MNIST, Kuzushiji-MNIST, CIFAR-10/100) where the noise label set is hand-crafted wrt the **contrib-1**.\n", "summary_of_the_review": "I'm satisfied with the response of the authors. The different points I had raised were taken into account and my doubts were clarified. \n\n------------------------\nI understand the insight behind the competitive noise, and I believe it is also a key point to evaluate the effective effectiveness of the proposed method, but I have doubts about how this choice can change the task (making it simpler or harder). In general, noise labels act by confounding, that is a causal concept, and as such, cannot be described in terms of correlations or associations. If the confounder is chosen with a particular strategy, intuitively, there is no control over whether or not its presence will be that of confounding. \n\nTo demonstrate the competitive label noise hypothesis, a comparison with some public datasets (such as those chosen in the paper) is necessary while maintaining the same split (train / test set) and choice of ground truth of one or more reference papers (e.g., 5).\n\n[5] Wen, H., Cui, J., Hang, H., Liu, J., Wang, Y. and Lin, Z., 2021, July. Leveraged weighted loss for partial label learning. In International Conference on Machine Learning (pp. 11091-11100). PMLR.\n\nThe authors argue that in a real scenario the noisy labels are much more similar to the real one, therefore more difficult to be identified/distinguished. As the proposed approach appears to be designed to work in this setting, is it possible to empirically prove this assumption, for example on a public benchmark? Or, a dataset which represents, more closely, the characteristics of this real scenario.\n", "strength_and_weaknesses": "**Strength**:\n\nThe idea about the new challenging PLL scenario where the noise labels in the candidate label set are difficult to distinguish from the ground-truth label given the input instance [**to be proved**].\n\nThe authors argue that in a real scenario the noisy labels are much more similar to the real one, therefore more difficult to be identified/distinguished. However, the choice about the datasets in the paper, seem to be very standard and far from the alleged \"real scenario\" mentioned by the authors. As the proposed approach appears to be designed to work in this setting, is it possible to empirically prove this assumption, for example on a public benchmark? A dataset which represents, more closely, the characteristics of this real scenario.\n\n\n**Weaknesses**:\n\n\n1. **competitive label noise** and 2. **Experiments**.\n\nI understand the insight behind the competitive noise, but I have doubts about how this choice can change the task (simpler or harder). In general, a confounder is a variable whose presence affects the other variables studied so that the results do not reflect the actual relationship (e.g. in regression). If the confounder is chosen with very strong relationships with the GT, intuitively its presence will no longer be that of confounder. \n\n- Would it be possible to have an experiment in which the proposed method is tested on a random selection of noise labels or the same procedure used in [5]?\n\nTo demonstrate the competitive label noise hypothesis, a comparison with some public datasets (such as those chosen in the paper) is necessary while maintaining the same split (train / test set) and choice of ground truth.\n\nMy doubt is reinforced by the comparison of the LWS method in Table 2, with the results presented in the original paper [5] on the same data (with the difference in the strategy for creating labels)  \n\n[5] Wen, H., Cui, J., Hang, H., Liu, J., Wang, Y. and Lin, Z., 2021, July. Leveraged weighted loss for partial label learning. In International Conference on Machine Learning (pp. 11091-11100). PMLR.\n\n\n3. **difficult to read**\n\n\nX.**Minor**:\n\n- Equations: in general, the equations are defined as dependent on variables which then do not appear in the function definition.\n\nOne example that applies to many equations:\n\n**Eq. 6, sec 3.3**\n\nL_c ( F, G, H) = .... q Log(F(G(x))\n\nq is the corrected soft pseudo-label vector.\n\nOnly going through the previous sections and Fig 1, it is possible to get that q is related to p that is related to H(G(x)). Therefore, q = Func1( F and Func2( H(G(x) ) ). \n\n\n- prototype-based classifier introduced in sec 3.2, however, sec 3.1 is all about the prototype-based classifier output and prediction probability vector p_i (based on the class-prototype classifier). \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster