ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
Authors: Mingyu Xu
Zheng Lian
Lei Feng
Bin Liu
Jianhua Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: Partial label learning; Noisy label learning

Abstract: Noisy partial label learning (noisy PLL) is an important branch of weakly supervised learning. Unlike PLL where the ground-truth label must conceal in the candidate label set, noisy PLL relaxes this constraint and allows the ground-truth label may not be in the candidate label set. To address this challenging problem, most of the existing works attempt to detect noisy samples and estimate the ground-truth label for each noisy sample. However, detection errors are unavoidable. These errors can accumulate during training and continuously affect model optimization. To this end, we propose a novel framework for noisy PLL with theoretical interpretations, called ``Adjusting Label Importance Mechanism (ALIM)''. It aims to reduce the negative impact of detection errors by trading off the initial candidate set and model outputs. ALIM is a plug-in strategy that can be integrated with existing PLL approaches. Experimental results on multiple benchmark datasets demonstrate that our method can achieve state-of-the-art performance on noisy PLL. Our code is available at: https://github.com/zeroQiaoba/ALIM.

ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
Mingyu Xu 1.- Zheng Lian't Lei Feng", Bin Liu! Jianhua Ta0
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences School of Computer Science and Engineering, Nanyang Technological University DDepartment of Automation, Tsinghua University Beijing National Research Center for Information Science and Technology, Tsinghua University (xumingyu2021, lianzheng2016) 2ia ac cn
Abstract
Noisy partial label learning (noisy PLL) is an important branch of weakly su- pervised learning. Unlike PLL where the ground-truth label must conceal in the candidate label set, noisy PLL relaxes this constraint and allows the ground-truth label may not be in the candidate label set. To address this challenging problem, most of the existing works attempt detect noisy samples and estimate the ground- truth label for each noisy sample. However, detection errors are unavoidable, These errors accumulate during training and continuously affect model optimization. To this end, we propose novel framework for noisy PLL with theoretical interpre- tations, called "Adjusting Label Importance Mechanism (ALIM)", It aims to reduce the negative impact detection errors by trading off the initial candidate set and model outputs. ALIM is plug-in strategy that can be integrated with existing PLL approaches. Experimental results on multiple benchmark datasets demonstrate that our method can achieve state-of-the-art performance noisy PLL. Our code is available at: https//githbb.com/eenqiiaaabb ALLM..
Introduction
Partial label learning (PLL) [ 2] (also called ambiguous label learning 3 and superset label learning (5] (6) is typical type of weakly supervised Iearning. Unlike supervised learning where each sample is associated with ground-truth label, PLL needs to identify the ground-truth label from of candidate labels. Due the low annotation cost of partially labeled samples, PLL has attracted increasing attention from researchers and has been applied to many areas, such as object 4 web mining 77) and ecological informatics
recognition
The basic assumption of PLL that the ground-truth label must be the candidate label set [ However, this assumption may not be satisfied due to the unprofessional judgment of annotators [10] Recently, some researchers have relaxed this assumption and focused on a more practical setup called noisy PLL [T] In noisy PLL. the ground truth label may not be in the candidate label set. To deal with this task, Lv et al. samples. However, they cannot fully exploit the useful information in noisy data. To this end, Lian et al. [12 and Wang et al. samples. However, detection errors unavoidable. These emror will accumulate and continuously affect model optimization, thereby limiting their performance on noisy PLL.
[IT] utilized noise-tolerant loss functions to avoid overemphasizing noisy
[3] proposed to detect noisy samples and estimate pseudo labels for these
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Classification
Pseudo Label Generation
6x0
0.00
Image
630
820
(
600
Hemee
dleer
denley
9.00
Porte
deer
deniey
P(r)
{())
La)
1:0
330
140
0:40
user A: Horse
3.00
house
mula
deer
Montey
9.0
hore
male
denr
Bovter
user B: Mule
user Deer
P())
(Fte) +M0 Sta)))
Ground Truth: Donkey
Origin PLL
ALIM
The structure of ALIM. The network receives input æ and produces softmax prediction probabilities P(x). Different from traditional PLL that fully trusts the candidate set, our method can deal with noisy samples through weighting mechanism.
Figure 1:
To alleviate this problem, we propose a novel framework for noisy PLL called "Adjusting Label Importance Mechanism (ALIM)". Although may make mistakes in noisy sample detection, allows us to leverage the initial candidate set and restart correction. To reduce manual efforts in hyper- parameter tuning, we propose an adaptive strategy to determine the weighting coefficient for ALIM. To further improve noise tolerance, we equip ALIM with mixup training [14], a powerful technique in noisy label learning [15]. We also perform theoretical analysis from the perspective objective functions and EM algorithms and prove the feasibility of our method. To verify its effectiveness, we conduct experiments on multiple benchmark datasets. Experimental results demonstrate that ALIM outperforms currently advanced approaches, setting new state-of-the-arr records. The main contributions this paper can be summarized as follows:
We propose plug-in in framework for noisy PLL with theoretical interpretations. Our method can deal with noisy samples by trading off the initial candidate set and model outputs.
We propose an adaptive strategy to adjust the weighting coefficient for ALIM. Furthermore, we combine our method with the mixup training to improve noise robustness.
Experimental results on multiple datasets demonstrate the effectiveness of our method. ALIM is superior currently advanced approaches noisy PLL.
2 Methodology
2.1 Problem Definition
Let X' be the input space and y be the label space with c distinct categories. We consider a dataset D containing N partially labeled samples ((x)), Here, S(x) E {0, 1)c is the candidate set for the sample E X. We denote the tth element of S(x) as (2(),, which is equal to I if the label i is in candidate set, and otherwise. Meanwhile, we denote P(x) E IRS, w(x) E IRS, and y(x) E RR as the prediction probabilities, the pseudo label, and the ground-truth label of the sample The gth elements of P(x) and w(x) are represented as (x(z) and w, (z), respectively. this paper, all product operations between matrices Hadamard products.
The goal of noisy PLL to learn a classifier that can minimize the classification risk on D. Unlike PLL, noisy PLL allows the ground-truth label may not be in the candidate set. For fair comparison, we adopt the same data generation procedure as previous works [12]. Specifically, to generate candidate labels, we first flip incorrect labels into false positive labels with probability and aggregate the flipped ones with the ground-truth label. Then, we assume that each sample has a probability of being the noisy sample. For each noisy sample, we further select label from the non-candidate set, move into the candidate set, and move the ground-truth label out of the candidate set. We denote the probability g the ambiguity level and the probability 7 the noise level.

---

2.2 Motivation
First, let us review our own exam experience. When we are unfamiliar with test, we believe that the correct answer must be in the candidate set. Even if every option is wrong, we still choose the most likely answer. But as we become more familiar with the test, we learn to question the correctness of the candidate set. If we believe every option is wrong, we will consider answers outside the candidate set. In noisy PI..,, this strategy can also used to handle noisy samples.
2.3 ALIM Framework
Inspired by the above idea, we propose simple yet effective framework called ALIM. The overall structure shown in Figure I Specifically, we first predict the softmax probabilities P(x) for each sample æ. In traditional PLI,, we fully trust the candidate set and generate the pseudo label as follows:
w(x) = Normalize ((z))
(1)
where Normalize(-) is a normalization function that ensures Σ (;( = Different from traditional PLL, ALIM introduces coefficient control the reliability of the candidate set:
((x) = S(x) + A(1 - (1-5(x)),
(2)
w Normalize (5(z)
Here, = means that we fully trust the given candidate set S(x); À means that we don't believe the candidate set but trust our judgment P(z). According Eq. 2 the time complexity of this process is mere (c.N), where e is the number of categories and N is the number of samples.
In this paper, we discuss two choices of normalization functions: Onehot(-) and Scale(-). Specifically, Onehot(-) sets the maximum value and others to O; Scale(-) introduces scaling factor K and normalizes the probabilities as follows:
1/K 24
Scale(2)
(4)
Eg 24 i=1
where 2; is the 8th element of
2.4 Theoretical Analysis
To verify feasibility of our method, we conduct theoretical analysis from two perspectives: (1) manually-designe objective function; (2) the classic expectation expectation-maximiii maximization (EM) algorithm [16].
2.4.1
Interpretation from Objective Functions
Let P., Sü, and w; be abbreviations for Pz(x), Sa(r),, and (x(x), respectively. During training, we should optimize the following objectives:
Minimize the classification loss on w(x) and P(x).
(x) should small at non-candidate labels.
Entropy regularization on w<x) to avoid overconfidence of pseudo labels.
w(x) should satisfy 0 < w, < and Ei= w. = 1.
Then, the final objective function is calculated as follows:
Wh log P, + M
Σ W.S: -1 K
ws log w;
s.t.
1, W/ 2 0,
(5)
3

---

where M and K are penalty factors. By using Lagrange multipliers, we can observe that the penalty factor K different for two normalization functions: K for Onehot(-) and K for Scale The penalty factor M has strong correlation with the weighting coefficient À in Eq. [ i.e., = e Larger M (or smaller A) means that we have tighter constraints on (Σiz) W.sS; - T), and therefore we should trust the given candidate set more. It is identical to the meaning of À in our ALIM (see Section 2.3
2.4.2 Interpretation from EM Perspective
EM aims to maximize the likelihood of the corpus D. Following previous works make a mild assumption:
5 17 we first
Assumption In noisy PLL. the ground-truth label may not be in the candidate set S(x). We assume
that each candidate label (z) = } has an equal probability axx) of generating ((x) and each non-candidate label (i = 0} has an equal probability ((x) of generating (ax).
Besides the interpretation from objective functions, we further explain ALIM from the EM perspec- tivein Appendix [ We prove that the E-step aims to predict the ground-truth label for each sample and the M-step aims minimize the classification loss. Meanwhile, ALIM is simplified version of the results derived from EM. Specifically, EM uses an instance-dependent X(x) 3(z)/a(z) while ALIM uses a global A. In the future, we will explore the performance of the instance-dependent A(x)) for noisy PLL. Additionally, EM connects traditional PLL with noisy PLL. In traditional PLL, we assume that the ground-trute label must be in the candidate set, i.e., ((x) Then, our noisy PLL
approach ALIM will degenerate to the classic PLL method RC [8]
2.5 Optional Key Components
We further introduce several key components to make our method more effective, including the adaptively adjusted strategy and the mixup training.
2.5.1 Adaptively Adjusted Strategy
Appropriate À is important for ALIM. Too small À makes fully trust the candidate set, thus easily over-fitting on noise samples; too large makes us ignore the prior information the candidate set, thus limiting the classification performance. Therefore, we propose a manually adjusted strategy to find a proper in a predefined parameter space.
To further reduce manual efforts, we also propose an adaptively adjusted strategy. Specifically, we first estimate the noise level of the dataset. Intuitively, we can randomly select a subset from D, annotate the ground-truth -rutt labels by professional annotators, and estimate the noise level of the dataset. Alternatively, we can automatically estimate noise rate via the Gaussian mixture model [19 20 or
cross-validation [21] 22]. The estimated noise level is represented 17. Based on Appendix C we prove that the value of Eq. caan be viewed as a metric, and the r/-quantile of this value can treated as the adaptively adjusted X.
max; (,(x)
(6)
max; - (())) ZED
In Section 3.2 We further present results without noise rate estimation and manually adjust À as a hyper-parameter. Through experimental analysis, this approach can also achieve competitive performance. Therefore, the adaptively adjusted strategy is optional. Its main advantage is to reduce manual efforts in hyper-parameter tuning and realize a more automatic approach for noisy PI.I..
2.5.2 Mixup Training
Since the mixup training is powerful in noisy label learning [5, we further combine this strategy with ALIM for noisy PL... Consider pair of samples v; and æ, whose pseudo labels are denoted as w(x;) and w(zz)), respectively. Then, we create a virtual training sample by linear interpolation:
Fmis 0.2; (1 -a)æz
Wmix = aw)(z;) + - aa)
(8)

---

Algorithm 1: Pscudo-code of ALIM.
Input: Dataset D with the estimated noise level 7), predictive model f. warm-up epoch eo, number of epoch Emar weighting coefficient A, trade-off between losses Amis. Output: The optimized model f.
for 1, co do
Warm up by training f with Eyn;
end
for Co. Emox do
for (2%, S(x))) D de
Calculate the output of the predictive model f(x));
Obtain the pscudo label w(x;) by Eq. 3
Get the PLL loss Lpt between (2:) and ((re):
10
if mixup truining then
Sample Beta(C. () and sample my from Create the virtual sample (Amix, Wmix) by Eq Calculate the mixup loss Lmix between o mxxx and Wmixi
13
14
Get the final loss L = Lpu + Amax Cmixi
15
else end
16
Get the final loss - Cpui
17
18
end
19
Optimize f by minimizing L;
if adaptively adjusted À then 20
Create an empty list G; for (æ., ((x:)) }
23
do Calculate the output the predictive model f(x)):
24
25
Store the value in Eq. t G;
26
end
27
À - 1/ quantile of the list G;
end
29 end
a ~ Beta(C, S), where Ç is parameter in the beta distribution. We define the mixup objective Lmix the cross-entropy loss (f and Wmis- During training, combine the mixup loss Lmix and the PLL loss Epul into a joint objective function C = Lpll Amix where Amix controls the trade-off between two losses. The pseudo-code of ALIM summarized in Algorithm
3 Experiments
3.1 Experimental Setup
Corpus Description the main experiments, we evaluate the performance on two benchmark
datasets of noisy PLI., CIFAR-10 [4] and CIFAR-100 [26] We choose the noise level v) from (0.1, 0.2, 0.3). Since CIFAR-106 has more classes than CIFAR-10, we consider E (0.1,0.3, 0.5) for CIFAR-10 and q E (0.01, 0.03. 0.05) for CIFAR-100. In Section 3.2 also conduct experi- ments on fine-grained datasets (CUB-200 200 17] and SIFAR-100H 24 and consider severe noise.
Baselines To verify the effectiveness of our method, we implement the following state-of-the-ar methods baselines: 1) CC [18]: classifier-consistent PLL method under the uniform candidate label generation assumption; 2) RC [18]: a risk-consistent PLL method under the same assumption as CC: 3) LWC 255: a PLL method that considers the trade-off between losses on candidate and : PLL method that combines the weighted loss with the sigmoid a PLL method using contrastive learning; 6) CRDPLL (6) PLL method that exploits consistency regularization on the candidate set and supervised learning on the non-candidate set; 7) IRNet [1]: a noisy PLL method that progressively purifies noisy samples; 8)
non non-candidate sets; 4) LWS 255
activation function; 5) PiCO
PiCO+ [3]
noisy PLL method using semi-supervised contrastive framework.

---

Table 1: Performance of different methods. & denotes the models without mixup training, and ?
denotes the models with mixup training. By default, we combine ALIM with PiCO for noisy PLL.
CHEAR-I0
9
0.1
0.3
0.5
11 m
0.
1
0.2
17
0.3
12 = 0.1
1) = 0.2
7
0.3
7
0.1
2)
0.2
0.3
OCC
79.81 22
77.06a0.18
73.8720.31
74.09
71.43a0.56
68.08
09.872
59.35s0:22
48.93=0.52
ORC
80.87+0.30
78.22+0.23
75.24=0.17
79.69=0.37
75.69=0.63
71.01=0.54
72.46=1.51 51
59.72+0.42
49.74=0.70
OLWC
79.1340.55
75.1540.46
74.1720.48
77.4720.56
74.02a0.35
69.10=0.59
70.59 34
57.42=1.14
48.9340.37
OLWS
82.9740.24
79.4640.09
74.28+0.79
80.93=0.28
76.07
69.70=0.72
70.41=2668
58.26+0.28
39.4243.09
OPICO
90.78+0.24 24
87.27=0.11
84.96+0.12
89.71=0.18
85.78+0.23
82.25=0.32
88. 1
82.41+0.30
68.7542.22
CCRDLL
93,48:0.11
89.13+0.39
86. 19:048444
92.
86.96=0.21
83,40=0.14
91.10=0.07
82.30=0.46
73.78+0.55
OPCO+
93.6420.19
93.13:0.26
92.1820.38
92.3220.08
92.22=0.01
89.9520.19
91 01.0720.02
89.68%0.01
84.0840.42
ORRNe
93,44:0.21
92.57=0.25
92.38
92.81=0.19
92.1840.18
91.35=0.088
91.5110.05
90.76:00.10
19.
ALIM-Scale
94.15+0.14
93.41=0.04
93.28:0.08
93,40=0.03
92.6920.11
92.52=0.12 92.01=0.19
90.92:0.10
86.5120.21
ALLM Onnehol
94.1540.115
94.04+0.16
93.77=0.27
93.44
93.2540.08
92.42=0.17
92.67=0.12
91,83:0.08
89.80=0.38
VPCCO
04.5820.02
94, 13
99
94 04.0240.03
94.0520.01
92.9450 24
93. 03
92.6520.26 26
28.
ALLMM
95.71
05.5080.08
95.3120.16 95.35
94.7720.07
94. 94.3620.03
94.71=0.04
93.8240.13
90.6340.11 10
ALIM Onehot
95.8340.13
95.8640.15
95.75+0.19
95.52=0.15
95.41=0.13
94.67=0.21
95.19=0.24
93.89=0.21
92.26=+0.29
0.01
4 0.03
0.05
CIFAR-100
17
1/
0.2
1 0.5
0.1
7 =
7 0.3
0
0.2
O0C
53.6340.46
48.8420.19
45.50=0.25
51,85=0.18
47,4820.30
30
43.3750.42
50.64:0.40
45.8750.36
40.8720.47
ORC
52.73+1.05
48.59+ 1.04
45.77+0.31
12.15=0.19
48.25+0.38
43.92=0.37
46.62=0.34
45.46+0:21
40.31+0.55
OLWC
53.1640.87
48.6440.33
45.51=0.28
51.69=0.28
47 97.60=0044
43.39=0.118 18
50.55=0.34
35.0520.28
39 83483+0.00 30
OLWS
56.05:0.20
50.66:10.9
45.7120.45
53,5920.45
48 48.28:0.44
42.20=0.49
45 :6260.0044
39.6320.80
33.6010.66 64
OPIO
68.2740.08
62.24
58.97=0.09
67.38:20.99
58.64=0.28 62.01
67.52=0.43
61.52+0.28
18.1640.65
CRDPLLL
68.1240.13
65.32+0.34
62.94+0.28
67.53+0.07
64.29=0.27
61.79=0.11
67.1740.04
64.11+0.42
61.03+0.43
PICO
71.42+0.25
70, 70,22+0.33
66 14+0.32
70.89=0.13
69.05=0.01
64.22=0.23
00,40==011
66.67+0.1 18
62 (2.24+0.35 38
IRNNe
71.17=0.14
70. 10.10:2.22
68.77:0.28
11.01
70.15=0.17
68.18=0.30
70.73=0.09
69.33+0.51
68.0920.11 12
ALIM-Scale
74.35+0.11 18
73.
13.50=0.13
74.33+0.11
72.60=0.19
71.69=0.39
73.77=0.05
72.39:0.04
7166+0.14
ALIM-Onchot
72.26+0.23
71,98+0.29
71.0420.31
71.43=0.21
70.79=0.43
70.14=0.25 25
72.28=0.28
70.60:0.44
70.05=0043
VPICO
75.04+0.18 18
74.31+0.22
71.79+0.17
14.68=0.19
73.65+0.23
69.97-0.01
73.06+0..0 16
71.37+0.16 16
67.56+0011 17
Seale
77.37+0.32
76.81=0.05
76.45=0.30
77. 07.68
76.63=0.19
75.92=0.14
76.86=0.23
76.44=0.12
75.67=0.17
76.5240.19
76.55a0.24
76.0920.23
27,2720,23
76.2920.41
75.29=0.57
76.87=0.20
75.2320.42
74.4920.61
Implementation Details There are mainly three user-specific parameters in ALIM: X, Amix, and eo- Among them, controls the trade-off between the initial candidate set and model outputs. This paper proposes two selection strategies for X, i.e., manually and adaptively adjusted strategies. For the first one, we treat À as a hyper-parameter and select from {0.1, 0.2, 0.3, 0.4, 0.5, 0.7). For the second one, we automatically determine À using the estimated noise level. Amix controls the trade-off between the PL.L loss and the mixup loss, and we set Amix = 1.0 as the default parameter. eo is the start epoch of ALIM, and select it from {20, 40, 80, 100, 140). Following the standard experimental setup in PLL 25), we split a clean validation set from the training set to determine Then, we transform the validation set back to its original PLL form and incorporate it into the training set to accomplish model optimization. To optimize all trainable parameters, we choose the SGD optimizer with momentum of 0.9 and set the weight decay to 0.001. We set the initial learning rate to 0.01 and adjust it using the cosine scheduler. To eliminate the randomness of the results, we run each experiment three times and report the average result and standard deviation the test set. All experiments are implemented with Py PyTorch 27 and carried out with NVIDIA Tesla V100 GPU.
3.2 Experimental Results and Discussion
Main Results For a fair comparison, we reproduce all baselines using the same data generation strategy as previous works [1]. In Table we observe that a large portion of improvement due to mixup training rather than model innovation. To this end, we compare different approaches under the same mixup strategy. Experimental results demonstrate that our method succeeds over all baselines under varying ambiguity levels and noise levels. The main reason lies in two folds. On the one hand, existing PLL methods are mainly designed for clean samples but ignore the presence of noisy samples. Our method can deal with noisy samples by trading off the initial candidate set and model outputs. On the other hand, existing noisy PLL methods generally need to detect noisy samples, but detection errors are unavoidable. These errors can accumulate and continuously affect the training process. Differently, ALIM can deal with this problem by taking advantage of the initial candidate set and restarting correction. These results prove the effectiveness of ALIM in noisy PI.L.
Since ALIM is a plug-in strategy, we integrate it with existing PLL methods and report results in Table We observe that ALIM always brings performance improvement under noisy conditions, verifying the effectiveness and compatibiliyy of our method. Mcanwhile,
Compatibility of ALIM

---

Table 2: Compatibility of ALIM on different PLL methods.
CIFAR-10 10
PL.
ALIM
9 0.3
17 0.1
71 0.2
1 = 0.3
1) 0.1
1) 0.2
1) 0.3
1) = 0.1
7) = 0.2
0.3
ORC
80.87=0.30
78.22+0.23
75.24+0.17
79.6940.37
75.6940.63
71.01+0.54
72.46=1.51
59.72+0.42
49.74+0.70
ORC
88.81=0.17
17.1000.20
85.53+0.05
86.21+0.11
83.64+0.07
79.8++0.43
77.40=0.31
69.13+0.71
56.75+1.58
PPIO
90.78=0.24
87.27=0.11
84.96+0.11
89.71+0.18
85.7840.23
82.2540.32
88. 18.11
82.41=0.30
68.75+2.22
PICO
94.15=0.15
94.04+0.16
93.77+0.27
93,4440.1
93.2540.08
92,42+0.17
92.67=0.12
91.83+0.00 08
89 80+0000000 38
CRDPLL
93.4820.17
89. 19.13=0.39
86. 1920.48
92.7340. 19
86.9640.21
83.4020.11
91.10=0.07
82.30=0.46
73 75.78800.55
OCRDPLL
96.0320.23
95.01a0.32
93.3620.10
95.32a0.13
93.27a0.29
91.20a0.06
93.82a0.05
90.20e0.04
84.24x0.22
CIFAR-100
PLL
ALIM
9= 0.01
0.03
9
== 0.1
1) 0.2
1 = 0.3
1) ===00 0.1
1)
0.2
y= 0.3
1 0.1
0.2
0.3
ORC
52.73=1.05
48.59=1.04
45.7750.31
52.1540.11
48.2540.35
45.92=0.77
40.6250.34
45.4660.21
40.31=0.55
ORC
V
61.4620.26
60.10=0.23
55.67:0.28
57.43=6.20
52.9810.27
48.7420.37
56.40=0.60
51.91=0.12
46.87+0.74
OPICO
x
68.27=0.08
62.24+0.31
58 18.97=0.09
67 38.03.9
62.01+0.33
58.6440.25
67 67.52=0.43
61.52=0.25 28
58. 18.1800.66
OPICO
72.2620.23
71.98 11.98:0.29
71.04:0.31
7144340.21
70.7940.45
72.28=0.28 70.1440.25
70.60=0.44
70.05+0.43
SCRDPLL
68.1220.13
65.3220.34
62. 94.90.282
67 53,0.07
64 64.29.0.27
61.79g0.11
67.1720.04
64.1120.42
61.0320043
OCRDPLL
69.98s0.30
68.58=0.16
66.90+0.16
69.60+0.20
67.6740.22
15.0.20
68.75=0.06
67.07+0.29
64.69+0.23
Table 3: Performance fine- fine-grained datasets.
Table 4: Performance with severe noise.
Method
(g
0.05,
CUB-200
0.2) (g = 0.5, 0.2)
CIFAR-IOOH
Method
(g
CIFAR-100
0.05, 7 0.4) (g
CIFAR-100
0.05, q) = 0.5)
OCC
26.98%1.16
34.3720.99
RRC
44.74+2.47
48.03=0.47
ORC
33.64220.2
26.91210.83
OLWS
18.65+2.15
12.18=6.12
PICO
44,17+0.00
15,
OGCE
5.13+38.65
33.21=2.03
CCRDPLL
57.10=0.24
52.1040.36
MSE
20.92+1.20
35,20=1.03
A Scale
70.37=00.6
64.7420.11
CRRDLLL
19+0.72
53.72=0.21
OALIM-Onndot
68 08.8220.37
63.3940.82
PICO
53.05+2.03
59,81=0.25
V/CCC+
66.41#0.50
60.50>0.99
PPCC
60.6520.79
68.3120.47
AALIM Scale
68.3820.47
73.42:0.18
ALLM--caale
74.98:0.16
72.26:0.25
VA MAL.IM-Oneho
63.91+0.35
72.36=0.20
ALLM--Onebot
71.7610.56
69.59a0.62
integrating ALIM with PiCO generally achieves better performance under severe noise (1) 0.3). In noisy label learning, self-training suffers from accumulated errors caused by sample-selection bias 28
To address this issue, researchers propose co-training [29], which contains multiple branches and cross-updates these branches. Among all PLL methods, PiCO is natural co-training network with two branches: one for classification and one for clustering [17). The output of the classification branch guides the model to update the clustering prototypes; the output of the clustering branch affects the update of the pseudo label for classification. These results verify the effectiveness of co-training under noisy conditions. Therefore, combine ALIM with PiCO in the following experiments.
Fine-Grained Classification Performance Considering that similar categories are more likely to be added to the candidate set, we conduct experiments on more challenging fine-grained datasets, CIFAR- 100H [17] and CUB-200 124). Different from previous works [12] that generate candidate labels from the entire label space, this section considers the case of generating candidate labels belonging to the same superclass. Experimental results in Table demonstrate that ALIM outperforms existing methods on fine-grained classification tasks, verifying the effectiveness of our method.
Robustness with Severe Noise severe noise conditions. In this section, we compare ALIM with the most competitive baselines (RC, PiCO, CRDPLL, and PiCO+) under 1) E {0.4, 0.5). Experimental results in Table 4 show that ALIM succeeds over all baselines under severe noise. Taking the results f 0.5 as example. our method outperforms the currently advanced approaches by 11.76%. These results demonstrate that our ALIM is more robust to noise than existing PLL and noisy PLL methods.
To demonstrate noise robustness, we conduct experiments under
Discussion on Normalization Functions In this section, we compare the performance of two normalization functions. Table [ we observe that Onehot(-) performs slightly better on CIFAR-10 10 and Scale(-) performs slightly better on CIFAR-106 In Table Scale(-) generally achieves better performance fine-grained datasets and severe noise conditions. The main difference between these normalization functions is that Onehot(-) compresses estimated probabilities into a specific class, while Scale(-) preserves prediction information for all categories. For simple datasets like CIFAR-10. the class with the highest probability is likely to be correct. Therefore, the compression operation can
7

---

(a) epoeh
(b) 200 spech
(c) 400 epoch
(d) epach
Figure 2: Distribution of the value in Eq. 6 for clean and noise subsets with increasing training
iterations. We conduct experiments on CIFAR-10 (g = 0.3. 0.3) with 80.
86-46
74.8
58.77
52.44
68.35
72.55
28.31
80.22
(wil mixup)
(wf mixup)
(c) CIFAR-10U (wfo mixup))
(d) CIFAR-100 mixup)
Figure 3: Classification performance of manually adjusted À and adaptively adjusted A. We conduct
experiments on CIFAR-1O (q = 0.3, 7j = 0.3) and CIFAR-I00 (q = 0.05, r) = 0.3). We mark the results of the adaptively adjusted strategy with red lines.
$3.99
71.06
73.12
(a) Auix on CIEAR-1O (eg
80)
co en CIFAR 10 Dagix
11
(c) Anix on CIFAR 100 Ceo
80)
(d) co on CIBAR- 100 OA 31
Figure 4: Parameter sensitivity analysis CIFAR- 10 (7 0.3) and CIFAR-100 (q 0.05) with mixup training. The noise level of these datasets fixed to r = 0.3.
reduce the negative impact of other categories and achieve better performance on noisy PLL. For challenging datasets, the class with the highest predicted value has low probability of being correct. Therefore, this compression operation may lead to severe information loss. More importantly, ALIM consistently outperforms existing methods regardless of the normalization function. Therefore, the main improvement comes from our ALIM rather than normalization functions.
Rationality Adaptively Adjusted Strategy this section, we further explain the rationality of our adaptively adjusted strategy. Figure visualizes visualizs the distribution of the value in Eq. 6 for clean and noise subsets with increasing training iterations. We observe that this value an effective indicator for distinguishing clean samples from noisy samples. At the same time, our adaptively adjusted À serves as a suitable boundary for clean and noisy subsets.
Role of Adaptively Adjusted Strategy important to select proper in ALIM. In this paper, we propose two selection strategies: adaptively and manually adjusted strategies. Figure presents classification performance of these strategies. Experimental results show that different strategies can achieve similar performance. Therefore, the main advantage of this adaptively adjusted strategy to reduce manual efforts hyper-parameter tuning. The performance improvement still comes from ALIM, which exploits coefficient to control the reliability of the candidate set.

---

Parameter Sensitivity Analysis In this section, we perform parameter sensitivity analysis on two key hyper-parameters: the warm-up epoch eg. and the trade-off between the PLL loss and the mixup loss Amix- In Figure 4 we observe that choosing an appropriate Amix is important for AL.M, and Amix 1.0 generally achieves competitive results. Meanwhile, with the gradual increase of co. the test accuracy first increases and then decreases. This phenomenon indicates that ALIM needs warm-up training. But too large eg can also cause the model to overfit noise samples.
Related Work
In PLL, the ground-truth label is concealed in the candidate set. To deal with this task, the core is to disambiguate the candidate labels. In this section, we first introduce two typical disambiguation strategies, i.e., average-bar based methods and identification- based methods. Then, we briefly review some recent works on noisy PLL.
Average-based PL.. The most intuitive solution the average-based approach, which assumes that each candidate label has equal probability of being the ground-truth label. For example, Hüllermeier et al. [30) utilized k-nearest neighbors for label disambiguation. For each sample, they treated all candidate labels of its neighborhood equally and predicted the ground-truth label through the majority vote. Differently, Cour et al. [3T] maximized the average output candidate labels and non-candidate labels in parametric models. However, these average-based methods can be severely affected by false positive labels [9].
To address the shorteomings of the above methods, researchers have focused on identification- based methods. Different from average-base methods that treat all candi- date labels equally, identification-baat bbadde methods treat the ground-truth label latent variable and maximize its estimated probability by maximum margin 332 or maximum likelihood (5] criteria.
Identification-based PLL
Recently, deep learning has been applied to identification-bri based methods and achieved promising results. For example, Lv et al. [33] proposed self- trafninr strategy to disambiguate candidate labels. Feng et al. [18] introduced classifier- and risk-consistent algorithms under the uniform candidate label generation assumption. Wen et al. [25] relaxed this assumption and proposed a family of loss functions for label disambiguation. To learn more discriminative representations, Wang et al. [17]
exploited contrastive learning to deal with partially labeled samples. More recently, Wu et al. [26 used consistency regularization the candidate set and supervised learning the non-candidate
set, achieving promising results under varying ambiguity levels. The above methods rely on a basic assumption that the ground-truth label must be in the candidate But this assumption may not be satisfied due to the unprofessional judgment of annotators.
Noisy PLL Recently, noisy PLL has attracted increasing attention from researchers due to its more practical setup. Its core challenge is how to deal with noisy samples. For example, Lv et al. [II utilized noise-tolerant loss functions to avoid overemphasizing noisy samples during training. Lian et al. [2] proposed an iterative refinement network to purify noisy samples and reduce the noise level of the dataset. Wang al. 133 detected clean samples through a distance-based sample selection mechanism and dealt with noisy samples via semi-supervised contrastive framework. These noisy PLL methods generally need detect noisy samples, but detection errors are unavoidable. These eπrors can accumulate and continuously affect model training. To reduce the negative impact of prediction errors, we propose a novel framework for noisy PLL. Experimental results on multiple datasets demonstrate the effectiveness of our method under noisy conditions.
5 Conclusion
This paper introduces novel framework for noisy PLL called ALIM. To deal with noisy samples, it exploits the weighting mechanism adjust the reliability of the initial candidate set and model outputs. To verify its effectiveness, we conduct experiments on multiple benchmark datasets under varying ambiguity levels and noise levels. Experimental results demonstrate that our ALIM achieves state-of-the-art classification performance noisy PLL, especially in severe noise conditions and fine-grained datasets. Meanwhile, ALIM a low-computation plug-in strategy that can be easily integrated with existing PLL frameworks. Furthermore, we interpret the rationality and effectiveness

---

of the adaptively adjusted strategy. We also conduct parameter sensitivity analysis and reveal the impact of different papeeepaaameters. is worth noting that this paper leverages a global À to measure the reliability of the candidate set. In the future, we will explore the instance-dependent X(x) during training. At the same time, we will investigate the performance in more experimental settings, such as class-imbalanced conditions.
Acknowledge
This work is supported by the National Natural Science Foundation of China (NSFC) (No.61831022. No.62276259, No.62201572, No.U21B2010, No. No.62271083) Beijing Municipal Technology Commission of Zhongguancun Science Park No.221 100004821013, Open Research Projects of Zhejiang Lab (NO. 2021 KHOAB00), CCF-Baidu Open Fund NF0022222,,5), the National Natural Science Foundation of China (Grant No. 62106028), Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program, Chongqing Artifi- cial Intelligence Innovation Center, CAAl-Huawei MindSpore Open Fund, and Openl Community (https:/opemi..cc..cccm
10

---

References
[1] Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2107-2113, 2018.
[2] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the AAAI Conference on Artificial Intelligence, pages
2020.
[3] Yi-Chen Chen, Vishal M Patel, Rama Chellappa, and P Jonathon Phillips. Ambiguously labeled learning using dictionaries. IEEE Transactions on Information Forensics and Security, 9(12):0066-2888 2014.
[4] Ching Hui Chen, Vishal M Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(():1653- 1667, 2017.
[5] Li-Ping Liu and Thomas G Dietterich. A conditional multinomial mixture model for superset label learning. In Proceedings of the 25th International Conference on Neural Information Processing Systems, pages 548-556, 2012.
[6] Liping Liu and Thomas Dietterich. Learnability of the superset label learning problem. In Proceedings of the International Conference on Machine Learning, pages 1629-1637, 2014.
[7] Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, pages 39-43, 2008.
[8] Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich, Rank-loss support instance machines for miml instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 534-542, 2012.
[9] Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Proceedings of the 15th International Conference on Neural Information Processing Systems, pages 921-928, 2002.
[10] Jesús Cid-Sueiro. Proper losses for learning from partial labels. In Proceedings of the 25th International Conference on Neural Information Processing Systems, pages 1565-1573, 2012.
[11] Jigqi Lv, Lei Feng, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. arXiv preprint 1XXiv:2106.06152.. 2021.
[12] Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Arnet: Automatic refinement network for noisy partial label learning. arXiv preprint arXiv:221 1.04774, 2022.
[13] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico+: Contrastive label disambiguation for robust partial label learning. arXiv preprint arXiv:2201.0888 089904 2022.
[14] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations, pages 1-13, 2018.
[15] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In Proceedings of the International Conference on Learning Representations, pages 1-14, 2020.
[16] Arthur Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological)) 39(1):1-22, 1977.
[17] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In Proceedings of the International Conference Learning Representations, pages 1-18, 2022.

---

[18] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Proceedings of the Advances in Neural Information Processing Systems, pages 10960, 2020.
[19] Eric Arazo, Diego Ortego, Paul Albert, Noel O' and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In Proceedings of the International Conference on Machine Learning, pages 312-321, 2019.
[20] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- lae-Gil Lee. Robust learning by self-transition for handling noisy labels. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1490-1500, 2021.
[21] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):44--40) 2015.
[22] Hwanjun Song, Minseok Kim, and Jae- he--ii Lee. Selfie: Refurbishing unclean samples for robust deep learning. In Proceedings of the International Conference on Machine Learning, pages 5907-5915, 2019.
[23] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[24] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
[25] Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning. In Proceedings of the International Conference on Machine Learning, pages 11091-11 1 100, 2021.
[26] Dong-Dong Wu, Deng-Bao Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. Proceedings of the International Conference on Machine Learning, pages 24212 222225, 2022.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 8026-8037, 2019.
[28] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. Proceedings of 2018.
the International Conference on Machine Learning, pages
[29] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pages 92-100, 1998.
[30] Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelli- gent Data Analysis, (0(5):419443) 2006.
[31] Timothee Cour, Benjamin Sapp. Chris Jordan, and Ben Taskar. Learning from ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 919-926, 2009.
[32] Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In Proceedings of the Asian Conference on Muchine Learning, pages 96-111, 2016.
[33] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the International Conference on Machine Learning, pages 6500-6510, 2020.
12

---

A Interpretation from Objective Functions
In this section, we provide proofs of the Onehot(-) normalization function and the Scale(+) normal- ization function from the perspective of objective functions.
A.1 Proof for Onehot Normalization
For K 0, we choose the following objective function during training:
Σ W.S -
max
Wi log P.
+
i=1
s.t.
W2 1, W/s 2
(9)
Introduce Lagrange multipliers 6,, i [1 and into Eq. have:
C = Σ wi log P. + M Σ W;S: -1 1) + イ
Wi
+
(10)
Combined with the Karush- Karush-Kuhn-- Tucher (KKT) conditions, the optimal point should satisfy:
log P. MS; - 7 + d; =0
(11)
1, 0 2 0, w. 2 0, 0,6,u; 0.
(12)
1=1
Since S, E (0, 1}. we have MS, = log(e. S + (1 (1-S,)). The equivalent equation of Eq. 11 is:
6 = - log(e M 5; + P.)
(13)
Combined with 5, 2 0 in Eq. 12 we have:
2 max (log(e S + - P.)
(14)
5, 0 is true if > max, (log(e M S; + (1 - P.)) According to 8,w; 0, we always have W4: 0, which conflicts with Di 1 W/ = 1. Therefore, we get:
max (log(e M S, (1)))))))
(15)
We assume that only one in [1, c] reaches the maximum 7, then we have 0, [i,c//io. Combined with Eis wt = 1, we get wio Therefore, w(x) should satisfy:
w (x) = Onehot (log(e 5(x) + (1 - ((=))))))()))
(16)
We mark À -M and convert Eq. 16 its equivalent version:
w x
Onehot ((S(z) + X(1 ((z))
(17)
A.2 Proof for Scale Normalization
For K 0, log w; ensures that w, must be positive. Therefore, the constraint w, can be excluded. Then, the objective function can be converted to:
wi log P, M
Σ WiS: -1
K Ws log w;
s.t.
= 1.
(18)
13

---

Introduce the Lagrange multiplier in Eq. 18 we have:
wi log my +2 11 (1__") wi mz)
L = w log P. + M
w;::: 1 K
(19)
1=1
Since the optimal point should satisfy V 0, we have:
log P. + MS; - K (1 + lo - - =0 0.
(20)
Since S, E {0,1), we have MS; = log(e M S + (1 - -. The equivalent equation of Eq.
20 is:
log(e" M S; + (1 - S:) P. (K + 7) - K log wi 0,
(21)
w (c S + (1 -S)) P.
(22)
ek
We mark X = e-M Then, we have:
W. ((S. A(1 X(1
S.)) cl{(T-M)/K
(23)
Since E: Wh have:
((S: + A(1 - P.))
=1,
(24)
(1/K-M)/K
1 11 M)/K
((S) + A(1 S()
(25)
Combine Eq. 23 and Eq. 25 and we have:
((S, + X(1 P.)) W!; Eis + X(1 P.))
(26)
Combined with the definition of Scale(-) in Eq. this equation can be converted to:
w (x) Scale ((S(x) A(1 ((z)))
(27)
B EM Perspective of ALIM
EM aims to maximize the likelihood of the dataset DD:
max
log P (x, S(x); = max 6
log
P (x, ((c, = i;0)
8
RED
DED
P (sr, ((x), y(x) = 1;
max
log
Wi(x)
wx)x)
aED
1=1
((x), 1:
max
(x( log
(28)
wy wx(x)
RED
where 0 is the trainable parameter. The last step Eq. 28 utilizze Jensen's inequality, Since the log(-) function is strictly concave, the equal sign takes when P(x,S(r)) v(x) = i; ()) is some constant C, i.e.,
wi (x) = - P(z, i;;).
(29)
14

---

Considering that Ei=1 (i((x) = 1, we can further get:
C=
P(z. v()
(30)
Then, we have:
(x(x) P(x, (()) = =;;))
P(a,S(22) v(x) = -;;0) P(y(x) i|z, S(x):)). P(x, S(x);
Plr,
i;0)
(31)
In the EM algorithm [16], the E-step aims to calculate w. (x)x) and the M-step aims to maximize the lower bound of Eq. 28
(x(x) log P(x, S(x), v(z) = i:
argmax
w;(xx)
ZED
argmax
wx(x)) log P(x, S(x), 1;0).
(32)
E-Step. In this step, we aim to predict the ground-truth label for each sample:
(x(x) P(y(x) S(x): P(S()))((=)= = ((v(z) i|2; 0)
P(S(=)|a;
= -( = -(r;;)
(33)
-| 0)
According to Assumption
we have:
[a(x), S(or))) =1
(z))
P(S(z)lv(z),z) (z))) (z)
(34)
It can be converted to:
P(S(s)lu(a) 2) (z)) + B(x) (x(o)(z)))
(35)
Then, we get the equivalent equation of Eq. 33
w,(x) = Σ ( =|=;
+
(36)
i| 0)
We mark A(x) ((x)/a(x) and P.(x)) P
= i|.; -
Then, we get:
(x(x) Ei (S.(z) + x(x S()))
(37)
connects traditional PLL and noisy PLL. In traditional PLL, we assume that the ground -ruuh label must be in the candidate set, i.e., (xx) 0. Since A(z) (z))/a(z) 0, Eq. 37 degenerates to:
(.(x)
wi (a) Σ (.())
(38)
which is identical to the classic PLL method, RC 18]
M-Step. The objective function of this step is:
argmax
(x(x:) log P(x, S(x) = i;
RED
arrgmax Σ
(x(x:) log ((x: 0) P(y(2) = i|.: (1)()))))))))))
= i,
(39)
RED
15

---

Considering that P 8) P(x) and P(S(z)lv(x)= = -, = ((S(z)l((z) = i, i,)), the equiva- lent version of Eq.
argmax
ma(x) log P(y(2) = =1=;; 0).
(40)
iE
Therefore, the essence of the M-step is to minimize the classification loss.
C
Adaptively Adjusted À
Since 7) controls the noise level of the dataset, we have:
P(S (z))
r).
(41)
After the warm-up training, we assume that the predicted label generated by ALIM j(x) arg maxigisa w(x) is accurate, i.e., j(x) = v(z). Then we have:
(() (x) = 0) =
(42)
To estimate the value of X, we first study the equivalent meaning of (z(=)(z) = 0:
max (5.(z) (1 (()))) (.(x) > (=(=)=0
max (S.(x) (A1 St(x)) P.(x).
(43)
S((=)=
We simplify the left and right sides of Eq 43 as follows:
max (S,(2) + A(1 (.x)))))P((2
S((x)=1
max (S((z) + A(1 S(())) S,(z)=0
max (,)))
8(())=1
max
(())
9,(2)=0
= max (z)
(45)
max A(1 - (.))))
(44)
Then, we have:
max A(1 S((x)))((x) 2 max Pz()),
(46)
max; (z())
(47)
max; (1- (())
Therefore, (Z) = 0) 77 can be converted to:
max; (,(x)P((z)
max; (1 (xxx))),(xx)
= 7/.
(48)
It means that À is the quaunile of
max; S,(z)
(49)
max: (.())) ZED
16

---

D Effect of Scaling Factor
Larger K produces smoother pseudo labels. To investigate its impact, we perform parameter sensitivity analysis in 5 We observe that suitable K is between 0 and 1. Too large K will produce over-smoothed pseudo labels and leads to performance degradation. In this paper, we choose K = as the default parameter for the Scale(-) normalization function.
Table 5: Parameter sensitivity analysis on the scaling factor.
CIFAR-10
CIFAR-100
(g
0.3, 7) = 0.3)
(g = 0.05, 1] = 0.3)
0.1
94.87=0.08
75.29#0.44
0,2
94.72+0.18
75.60+0.41
0.5
94.69+0.15
75.47=0.03
1.0
94.3620.03
75.67=0.17
1.5
93.0420.65
72.82+0.23
17


Reviews:
[{"review_id": "6pJATYAfOe", "reviewer": "Reviewer_7tpZ", "summary": "This paper focused on the problem of noisy partial label learning (noisy PLL), which means the candidate label set in the partial label learning does not always contain the ground-truth label. Compared with the existing studies which attempted to detect noisy samples yet lead to the accumulation of detection errors, this work proposed to reduce the impact of the detection errors by training off the initial candidate set and model outputs. The proposed method, named \"Adjusting Label Importance Mechanism (ALIM)\", can adaptively adjust the weighting coefficient between the initial candidate set and the model predictions (i.e., pseudo-labels output by the model). The theoretical analysis showed that the proposed method can be well interpreted from two perspectives of heuristic design and classic Expectation-maximization (EM) algorithm. Combined with Mixup training strategies, the proposed approach can be further plugged into the existing noisy PPL baselines and bring consistent improvements. Extensive experiments demonstrated the effectiveness of the proposed framework.", "questions": "- In Line 140, what is the definition of $\\mathcal{L}_{pll}$? It suddenly appeared here without any pre-defined formation. Which kind of PLL loss did it indicate, the objective in Eq.(5)?\n- I recommend the authors reformulate the optimization objective in Eq. (5) as a $\\min$ form, which will be better compatible with the form of the loss function. \n- Compared to the results reported on the CIFAR10 dataset, the reviewer found that ALIM-Scale consistently performed better than ALIM-Onehot on other datasets like CIFAR100 and CUB200. Is it a necessity that scaling performs better than one shot on datasets with larger class numbers, or a coincidence? Could the author provide more analyses and explanations about this point?\n- Some potential typos:\n    - Line 95, Appendix ??\n    - Line 107, Appendix ??\n    - Line 126, Appendix ??", "limitations": "No", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear. The analysis of the issues of the existing methods motivated the proposed method in a straightforward manner.\n-  The proposed objective can be interpreted under the EM framework, which provided a convincing mathematical explanation.\n- The proposed method, ALIM, is also compatible with other related approaches. It can work as a plug-in in the existing noisy PLL algorithms, which enhanced the flexibility of ALIM.\n- The experimental part conducted empirical comparisons/ablation studies with respect to different factors, e.g., the different noise levels and ambiguity levels, and the different choices of the key components within the proposed framework.", "weaknesses": "- Some technical details were not clear to the reviewers. See the next part for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 7tpZ\nWe thank the reviewer's appreciation of clear writing and rich experimental results. We try to address your concerns as follows:\n\n**Q1**: In Line 140, what is the definition of $L_{\\text{pll}}$? It suddenly appeared here without any pre-defined formation. Which kind of PLL loss did it indicate, the objective in Eq. 5?\n\n**A1**: We apologize for this unclear expression. ALIM is a plug-in strategy that can be integrated with existing PLL approaches. In this paper, $L_{\\text{pll}}$ means the loss function of PLL approaches. Since different PLL methods use distinct formulations of $L_{\\text{pll}}$, we do not provide the specific formulation of $L_{\\text{pll}}$. We will clarify it in the revised paper.\n\n\n**Q2**: I recommend the authors reformulate the optimization objective in Eq. (5) as a min form, which will be better compatible with the form of the loss function.\n\n**A2**: Thank you very much for your valuable comments. In the revised version, we reformulate the objective function as follows:\n\\begin{align}\n&\\min\\ -\\sum_{i=1}^cw_i\\log P_i - M \\left(\\sum_{i=1}^cw_iS_i-1\\right) + K \\sum_{i=1}^cw_i \\log w_i \\nonumber\\\\\n&s.t. \\sum_{i}^c w_i =1, w_i \\geq 0, \\nonumber\n\\end{align}\n\n\n**Q3**: Compared to the results reported on the CIFAR10 dataset, the reviewer found that ALIM-Scale consistently performed better than ALIM-Onehot on other datasets like CIFAR100 and CUB200. Is it a necessity that scaling performs better than one shot on datasets with larger class numbers, or a coincidence? Could the author provide more analyses and explanations about this point?\n\n**A3**: Good question! The main difference between these normalization functions is that $\\text{Onehot}(\\cdot)$ compresses estimated probabilities into a specific class, while $\\text{Scale}(\\cdot)$ preserves prediction information for all categories. For simple datasets like CIFAR-10, the class with the highest probability is likely to be correct. Therefore, the compression operation can reduce the negative impact of other categories and achieve better performance on noisy PLL. For challenging datasets, the class with the highest predicted value has a low probability of being correct. Therefore, this compression operation may lead to severe information loss. More importantly, ALIM consistently outperforms existing methods regardless of the normalization function (see Table 1). Therefore, the main improvement comes from our ALIM rather than normalization functions. We will add these discussions to the revised paper.\n\n\n**Q4**: Some potential typos.\n\n**A4**: We apologize for these typos. In Lines 95, 107, and 126, these symbols indicate Appendix A, Appendix B, and Appendix C, respectively. We will correct these typos in the revised version.\n"}, {"review_id": "U6Ei80bdmv", "reviewer": "Reviewer_7LXj", "summary": "The paper considers noisy partial label learning when the provided label is not in the candidate label set. The paper proposes \"Adjusting Label Importance Mechanism\" to trade off the initial candidate set and model outputs. Some theoretical analysis is done and good empirical results are shown against selected baselines. ", "questions": "1. When the label is not in the candidate set, it’s also likely that it's not in 1-S(x). Actually, not being in the label space might even be the main reason that the label is not in the candidate set. How to handle this case? It seems there is an assumption that the label is for sure in the given label space, and it might not be in the provided candidate which is a subset of the label space. This should be made clear. \n\n2. In Equation 5, why does M correspond to the lambda in equation 2? In section 2.4.2, it is also unclear what is the connection to EM. \n\n3. Abstract mentions theoretical guarantees, what are the guarantees? It is not mentioned in the paper anywhere. \n\n4. From the theoretical analysis, it seems there is an equivalence between the proposed method to an (existing?) objective function. Then, why would one need the proposed method?", "limitations": "Limitations are not discussed. Authors should discuss when the proposed method doesn't work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The considered problem (i.e. the provided label is not in the candidate label set) might be a relatively new problem and is less considered in existing work.\n\n2. Some theoretical analysis is done.\n\n3. A few tricks are proposed to make the method more effective.\n\n4. Good experiment results are shown against baselines. ", "weaknesses": "1. Writing can be improved. Currently, it seems the manuscript is prepared in a hurry. For example, there are errors when referring to appendices: “Appendix ??”. Overall, I had a hard time following. \n\n2. The core contribution is not clear. The motivation in Section 2.2 is hard to follow. From Section 2.2, it seems the paper is trying to consider the case when the ground-truth label is not in the provided candidate set of labels. It should be made clear why this is an important problem and how the proposed method addresses the problem. Also, there are concerns on the core technique. see Q1 and Q4. \n\n3. The theoretical analysis is not clear see Q2. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 7LXj\nWe sincerely thank the reviewer for your detailed comments and suggestions. We try to address each comment as satisfactorily as possible:\n\n**Q1**: Writing can be improved. Currently, it seems the manuscript is prepared in a hurry. For example, there are errors when referring to appendices: “Appendix ??”. Overall, I had a hard time following.\n\n**A1**: We apologize for these typos. In Lines 95, 107, and 126, these symbols indicate Appendix A, Appendix B, and Appendix C, respectively. We will correct these typos in the revised paper. Meanwhile, we are trying our best to improve the quality of writing.\n\n**Q2**: The core contribution is not clear. The motivation in Section 2.2 is hard to follow. From Section 2.2, it seems the paper is trying to consider the case when the ground-truth label is not in the provided candidate set of labels. It should be made clear why this is an important problem and how the proposed method addresses the problem.\n\n**A2**: Thank you very much for your valuable comments. In the revised version, we clarify the importance of this task and present our approach more clearly.\n\n*Importance of the task*: PLL is a typical type of weakly supervised learning. Due to the low annotation cost of partially labeled samples, PLL has attracted increasing attention from researchers and has been applied to many areas. The basic assumption of traditional PLL is that the ground-truth label must be in the candidate label set. However, this assumption may not be satisfied due to the unprofessional judgment of annotators. Recently, noisy PLL has attracted increasing attention from researchers due to its more practical setup. In noisy PLL, the ground-truth label may not be in the candidate label set. *Reviewer q7G6 also highlights the importance of noisy PLL.*\n\n*Methodology*: In this paper, we propose a novel framework for noisy PLL with theoretical guarantees, called ALIM. It exploits the weighting mechanism to adjust the reliability of the initial candidate set and model outputs. To reduce manual efforts in hyper-parameter tuning, we propose an adaptive strategy to determine the weighting coefficient. To further improve noise tolerance, we equip ALIM with mixup training. We also perform theoretical analysis from the perspective of objective functions and EM algorithms and prove the feasibility of our method.\n\n**Q3**: When the label is not in the candidate set, it’s also likely that it's not in $1-S(x)$. Actually, not being in the label space might even be the main reason that the label is not in the candidate set. How to handle this case? It seems there is an assumption that the label is for sure in the given label space, and it might not be in the provided candidate which is a subset of the label space. This should be made clear.\n\n**A3**: Good question! Current works on noisy PLL [1, 2, 3] assume that the ground-truth label must be in the given label space. For a fair comparison, we also conduct experiments under this assumption. Your comment points a new direction for noisy PLL. In the future, we will try to combine noisy PLL with out-of-distribution (OOD) detection strategies to address this challenging task.\n\n[1] Lv, Jiaqi, Biao Liu, Lei Feng, Ning Xu, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. \"On the robustness of average losses for partial-label learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).\n  \n[2] Lian, Zheng, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. \"Arnet: Automatic refinement network for noisy partial label learning.\" arXiv preprint arXiv:2211.04774 (2022).\n\n[3] Wang, Haobo, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. \"PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning.\" arXiv preprint arXiv:2201.08984 (2022).\n\n\n**Q4**: In Equation 5, why does $M$ correspond to the lambda in equation 2? In section 2.4.2, it is also unclear what is the connection to EM.\n\n**A4**: We thank the reviewer’s comments. We have provided proofs in Section 2.4 and Appendix.\n\nIn Appendix A, we prove the relationship between $\\lambda$ and $M$, i.e., $\\lambda = e^{-M}$.\n\nIn Appendix B, we explain ALIM from an EM perspective. We prove that the E-step aims to predict the ground-truth label for each sample and the M-step aims to minimize the classification loss.\n\n\n**Q5**: Abstract mentions theoretical guarantees, what are the guarantees? It is not mentioned in the paper anywhere.\n\n**A5**: We have provided theoretical guarantees in Section 2.4 and Appendix. To verify the feasibility of ALIM, we conduct theoretical analysis from the perspective of objective functions (see Appendix A) and EM algorithms (see Appendix B).\n\n\n**Q6**: From the theoretical analysis, it seems there is an equivalence between the proposed method to an (existing?) objective function. Then, why would one need the proposed method?\n\n**A6**: In this paper, we propose a novel framework for noisy PLL with theoretical guarantees. The necessity of our method can be illustrated in two aspects:\n\n(1) Some influential works in PLL [1, 2] are also derived from the loss function. The core lies in how to properly design the loss function and make the assumption to obtain an easy-to-implement formula. This process requires a lot of efforts.\n\n(2) Noisy PLL is an emerging direction of machine learning. This objective function is specifically designed for noisy PLL, which does not exist in the previous work.\n\n[1] Feng, Lei, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. \"Provably consistent partial-label learning.\" Advances in neural information processing systems 33 (2020): 10948-10960.\n\n[2] Wen, Hongwei, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. \"Leveraged weighted loss for partial label learning.\" In International Conference on Machine Learning, pp. 11091-11100. PMLR, 2021."}, {"review_id": "DAY36xxOR7", "reviewer": "Reviewer_BGEz", "summary": "This paper focuses on a relaxed version of partial label learning, called noisy PLL. Different from PLL, noisy PLL allows the ground-truth label may not be in the candidate set. To deal with this problem, this paper proposes a framework called ALIM, which is a plug-in strategy with theoretical guarantees. Experimental results on CIFAR-10, CIFAR-100, and fine-grained datasets prove the effectiveness of ALIM on noisy PLL. This paper is well-written and easy to follow.", "questions": "Can the authors explain the impact of the scaling factor $K$ on the performance and how it was chosen?", "limitations": "Please refer to the weakness and question.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper presents a plug-and-play framework for the challenging noisy PLL task.\n2.\tThis paper conducts theoretical analysis from objective functions and EM algorithms.\n3.\tALIM shows competitive performance compared with the currently advanced strategies.\n", "weaknesses": "$Scale(\\cdot)$ normalization function introduces a scaling factor $K$. But in the following experiments, this paper does not specify the selection process of this factor. Please explain its impact on the performance and how it was chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n# Response to Reviewer BGEz\nWe thank the reviewer's appreciation of the writing and sufficient experimental results. We try to address your concerns as follows:\n\n**Q1**: $\\text{Scale}(\\cdot)$ normalization function introduces a scaling factor $K$. But in the following experiments, this paper does not specify the selection process of this factor. Please explain its impact on the performance and how it was chosen.\n\n**A1**: Thanks for your valuable comments. The scaling factor $K$ affects the smoothness of pseudo labels. Larger $K$ produces smoother pseudo labels. To investigate its impact, we perform a parameter sensitivity analysis in Table 1. We observe that a suitable $K$ is between 0 and 1. Too large $K$ will produce over-smoothed pseudo labels and leads to performance degradation. In this paper, we choose $K=1$ as the default parameter for $\\text{Scale}(\\cdot)$ normalization function.\n\n| $K$  | CIFAR-10  ($q=0.3, \\eta=0.3$) | CIFAR-100 ($q=0.05, \\eta=0.3$) |\n| ---- | ----------------------------- | ------------------------------ |\n| 0.1  |   94.87±0.08                        |          75.29±0.44                      |\n|  0.2     |    94.72±0.18                           |     75.60±0.41                           |\n|  0.5    |       94.69±0.15                        |         75.47±0.03                       |\n|1.0|94.36±0.03|75.67±0.17|\n|1.5|93.04±0.65|72.82±0.23|\n\n\n"}, {"review_id": "rZc8IXyKtG", "reviewer": "Reviewer_q7G6", "summary": "This paper addresses the noise partial label learning problem, i.e., the ground-true may also be in the non-candidate set. The proposed method adjusts the importance of the labels by considering the output of the candidate label set model in an integrated manner. The proposed method in this paper can be viewed as a post-processing means for prediction models, and the proposed method achieves by superior performance compared to existing methods on multiple benchmark datasets.", "questions": "1.\tIn line 91, the authors describe that w(x) should keep very small data, so how can the true labels be extracted from the set of non-candidate labels?\n2.\tHow is the conclusion derived from the different values of K in the paper, which correspond to different normalizations?\n3.\tWhat is the principle of Mixup Training in noise partial label learning against noise?\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThe noise PLL issue is important and the authors have a good summary of the unresolved issues in that paper from existing work.\n2.\tRich experimental results.\n", "weaknesses": "1. This paper mainly uses the candidate label set and non-candidate label set information to post-process the prediction results, and the innovation is weak.\n2. The framework of the method has a stronger assumption that P(x) can also achieve better results in noise partial label learning problem, otherwise it is difficult to inform the subsequent processing. From Figure 1, we can see that for noise partial label learning, P(x) has been able to calculate the confidence of each label well, and the confidence of the true label is higher compared to ALIM.\n3. While the way of adjusting label importance proposed in the paper avoids errors when detecting noise directly, the method used in this paper will also assign a certain confidence level to all non-candidate labels, which will also introduce noise.\n4. One of the key points of the method proposed in this paper is the setting of the hyper-parameter lambda. An adaptive method is proposed in this paper, but the adaptive parameter is not analyzed in the experimental results to know at what value the setting is reasonable.\n5. For Mixup Training, the paper does not explain why it can counteract the effects of noise in PLL.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer q7G6\nWe thank the reviewer's appreciation of rich experimental results. We try to address each comment as satisfactorily as possible:\n\n**Q1**: This paper mainly uses the candidate label set and non-candidate label set information to post-process the prediction results, and the innovation is weak.\n\n**A1**: Thanks for your valuable comments, and we would like to present our contribution clearly. This paper proposes ALIM, a plug-in framework for noisy PLL. We perform theoretical analysis from the perspective of objective functions and EM algorithms, demonstrating the feasibility of our method. Experimental results show that ALIM outperforms currently advanced approaches, setting new state-of-the-art records (see Tables 1$\\sim$4). \n\nWe would like to argue that although our ALIM is easy to implement, it does not mean that our innovation is weak. In contrast, our method has strong motivation and sufficient theoretical guarantees. Such a simple strategy can bring performance improvements to varying PLL approaches, which fully validates its effectiveness.\n\n\n**Q2**: The framework of the method has a stronger assumption that $P(x)$ can also achieve better results in noise partial label learning problem, otherwise it is difficult to inform the subsequent processing. From Figure 1, we can see that for noise partial label learning, $P(x)$ has been able to calculate the confidence of each label well, and the confidence of the true label is higher compared to ALIM.\n\n**A2**: Thanks very much for your valuable comments. In noisy label learning, previous works [1, 2] have demonstrated that when there is a mixture of clean and noisy samples, networks tend to fit the former before the latter. In noisy PLL, researchers [3] also find the same phenomenon. At some intermediate epochs, $P(x)$ can achieve relatively reliable predictions in noisy PLL so that it can be treated as a confidence measure. But if we do not apply ALIM in subsequent training but fully trust $P(x)$, the classification performance will eventually degrade (see Table 2 and Figure 3).\n\n[1] Jiang, Lu, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. \"Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.\" In International conference on machine learning, pp. 2304-2313. PMLR, 2018.\n\n[2] Arazo, Eric, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin McGuinness. \"Unsupervised label noise modeling and loss correction.\" In International conference on machine learning, pp. 312-321. PMLR, 2019.\n\n[3] Lian, Zheng, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. \"Arnet: Automatic refinement network for noisy partial label learning.\" arXiv preprint arXiv:2211.04774 (2022).\n\n\n**Q3**: While the way of adjusting label importance proposed in the paper avoids errors when detecting noise directly, the method used in this paper will also assign a certain confidence level to all non-candidate labels, which will also introduce noise.\n\n**A3**: Good question! As you pointed out, our method assigns certain confidence to non-candidate labels. For clean samples, this process introduces noise. But for noisy samples, this process allows us to figure out the ground-truth label concealed in the non-candidate set. Therefore, the key is to find the trade-off between them. Experimental results also verify the effectiveness of our strategy in noisy PLL.\n\n\n**Q4**: One of the key points of the method proposed in this paper is the setting of the hyper-parameter $\\lambda$. An adaptive method is proposed in this paper, but the adaptive parameter is not analyzed in the experimental results to know at what value the setting is reasonable.\n\n**A4**: Since $\\lambda$ changes adaptively during training, we cannot provide its specific value. But according to Figure 2, we prove the rationality of this adaptive parameter. In this figure, we observe that this adaptively adjusted $\\lambda$ serves as a suitable boundary for clean and noisy subsets.\n\n**Q5**: For Mixup Training, the paper does not explain why it can counteract the effects of noise in PLL. What is the principle of Mixup Training in noise partial label learning against noise?\n\n**A5**: Our motivation comes from noisy label learning. Since previous works have demonstrated that mixup training is very effective in noisy label learning [1, 2], we wonder whether this strategy can also bring performance improvement for noisy PLL. Experimental results in Table 1 verify our conjecture.\n\n[1] Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. \"mixup: Beyond Empirical Risk Minimization.\" In International Conference on Learning Representations. 2018. \n\n[2] Li, Junnan, Richard Socher, and Steven CH Hoi. \"DivideMix: Learning with Noisy Labels as Semi-supervised Learning.\" In International Conference on Learning Representations. 2019.\n\n\n**Q6**: In line 91, the authors describe that $w(x)$ should keep very small data, so how can the true labels be extracted from the set of non-candidate labels?\n\n**A6**: We apologize for this unclear description. For clean samples, $w(x)$ should be small at the non-candidate set. But for noisy samples, $w(x)$ should be large at the non-candidate set. Therefore, the key is to find the trade-off between them. In this paper, we do not strictly restrict this objective but use a penalty factor $M$ (see Eq. 5). Larger $M$ means that we have tighter constraints on $\\left(\\sum_{i=1}^cw_iS_i-1\\right)$.\n\n**Q7**: How is the conclusion derived from the different values of $K$ in the paper, which correspond to different normalizations?\n\n**A7**: We thank the reviewer’s valuable comments. According to our proof in Appendix A, the penalty factor $K$ is different for two normalization functions: $K=0$ for $\\text{Onehot}(\\cdot)$ and $K>0$ for $\\text{Scale}(\\cdot)$. The main reason is that, for $K>0$, $\\log w_i$ ensures that $w_i$ must be positive. Therefore, we can exclude the constraint $w_i \\geq 0$ in Eq. 5."}]

Author rebuttal:

# General Response
Dear Reviewers, Area Chairs, and Program Chairs:

We would like to express our gratitude to all reviewers for taking their valuable time to review our paper. We sincerely appreciate the reviewers for noting that our motivation is "clear" (7tpZ), the mathematical explanation is "convincing" (7tpZ), and the proposed method is "flexible" (7tpZ) and "effective" (7LXj). We also thank the reviewers for noting that our paper is "well-written and easy to follow" (BGEz), the noisy PLL problem is "important" (q7G6), and our experimental results are "good" (7LXj), "competitive" (BGEz), and "rich" (q7G6). Meanwhile, we appreciate the reviewers for pointing out the shortcomings. Your valuable comments help us improve this paper. We try to address each comment as satisfactorily as possible. Please find the responses to each reviewer’s comments below.

**Motivation and Main Contribution**

Noisy PLL is an important branch of weakly supervised learning. Unlike PLL where the ground-truth label must be concealed in the candidate label set, noisy PLL relaxes this constraint and allows the ground-truth label may not be in the candidate label set. Most of the existing works attempt to detect noisy samples and estimate the ground-truth label for each noisy sample. However, detection errors are unavoidable. These errors can accumulate during training and continuously affect model optimization. To alleviate this issue, we propose a *plug-in* framework for noisy PLL with *theoretical guarantees*. Due to its compatibility, it can be *easily integrated with existing PLL methods* and achieves performance improvement under noisy conditions. Its *unprecedented success* at varying ambiguity levels and noise levels has paved a new way for noisy PLL and can inspire more relevant research.

**Summary of Reviews**

Reviewer 7tpZ and BGEz vote for acceptance, while Reviewer 7LXj and q7G6 give negative comments. Therefore, we try to address the main concerns of Reviewer 7LXj and q7G6:

Reviewer 7LXj gives negative comments on the writing and contribution. We deeply appreciate the reviewer's valuable comments. We are trying our best to improve the quality of writing. It is also worth noting that the evaluation of writing can be subjective, varying from individual to individual. For example, Reviewer 7tpZ and BGEz give high scores for our presentation, soundness, and contribution. In the revised version, we clarify the importance of this task and present our approach more clearly. This paper focuses on an important task in machine learning, noisy PLL. To deal with this problem, we propose a plug-in framework with theoretical guarantees. Due to its compatibility, it can be easily integrated with existing PLL methods. Experimental results on multiple datasets demonstrate its effectiveness.

Reviewer q7G6 raises concerns about our innovation because ALIM looks like a post-processing strategy for prediction results. We would like to argue that although our method is easy to implement, it does not mean that our innovation is weak. In contrast, ALIM has strong motivation and sufficient theoretical guarantees. Such a simple strategy can bring performance improvements to varying PLL approaches, which fully validates its effectiveness. Meanwhile, the reviewer raises concerns about some modules. In the response, we try to address these concerns as satisfactory as possible.

We kindly ask the reviewers to take the above clarifications into account when considering score adjustments. We welcome any further discussion with the reviewers.

Best regards,

Paper4978 Authors

Meta-review: This paper proposes Adjusting Label Importance Mechanism (ALIM), a method for noisy partial label learning. The setting is noisy in the sense that the training set of labels for an example might not actually contain the true label. ALIM takes a pseudolabeling approach, in which the predictions of the model being trained are weighed against the training partial labels. Essentially, confident predictions of the model can overrule noisy labels. Experiments show that ALIM can outperform state of the art methods on CIFAR 10, CIFAR 100, CUB 200 and CIFAR 100H.

In general, the reviewers liked the extensive experimental evaluation and the fact that ALIM can be plugged into many existing approaches. Some of the reviewers had doubts about the formulation and explanation of ALIM that were resolved during the discussion period. The authors are encouraged to add these clarifications to the final version.

One weakness that the reviewers identified is ambiguity in the claim of theoretical "guarantees." The theoretical analysis given is an interpretation, not a guarantee. No guarantee is stated about the performance of the algorithm in any sense. The analysis only shows that the behavior of ALIM can be interpreted in terms of well known frameworks like EM. The authors are strongly encouraged to clarify this in their final version.
Decision: Accept (poster)