Title: ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Partial label learning; Noisy label learning
Abstract: Noisy partial label learning (noisy PLL) is an important branch of weakly supervised learning. Unlike PLL where the ground-truth label must conceal in the candidate label set, noisy PLL relaxes this constraint and allows the ground-truth label may not be in the candidate label set. To address this challenging problem, most of the existing works attempt to detect noisy samples and estimate the ground-truth label for each noisy sample. However, detection errors are unavoidable. These errors can accumulate during training and continuously affect model optimization. To this end, we propose a novel framework for noisy PLL with theoretical interpretations, called ``Adjusting Label Importance Mechanism (ALIM)''. It aims to reduce the negative impact of detection errors by trading off the initial candidate set and model outputs. ALIM is a plug-in strategy that can be integrated with existing PLL approaches. Experimental results on multiple benchmark datasets demonstrate that our method can achieve state-of-the-art performance on noisy PLL. Our code is available at: https://github.com/zeroQiaoba/ALIM.

ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning

Abstract

Noisy partial label learning (noisy PLL) is an important branch of weakly supervised learning. Unlike PLL where the ground-truth label must conceal in the candidate label set, noisy PLL relaxes this constraint and allows the ground-truth label may not be in the candidate label set. To address this challenging problem, most of the existing works attempt to detect noisy samples and estimate the groundtruth label for each noisy sample. However, detection errors are unavoidable. These errors can accumulate during training and continuously affect model optimization. To this end, we propose a novel framework for noisy PLL with theoretical interpretations, called 'Adjusting Label Importance Mechanism (ALIM)'. It aims to reduce the negative impact of detection errors by trading off the initial candidate set and model outputs. ALIM is a plug-in strategy that can be integrated with existing PLL approaches. Experimental results on multiple benchmark datasets demonstrate that our method can achieve state-of-the-art performance on noisy PLL. Our code is available at: https://github.com/zeroQiaoba/ALIM.

1 Introduction

Partial label learning (PLL) [1, 2] (also called ambiguous label learning [3, 4] and superset label learning [5, 6]) is a typical type of weakly supervised learning. Unlike supervised learning where each sample is associated with a ground-truth label, PLL needs to identify the ground-truth label from a set of candidate labels. Due to the low annotation cost of partially labeled samples, PLL has attracted increasing attention from researchers and has been applied to many areas, such as object recognition [4], web mining [7], and ecological informatics [8].

The basic assumption of PLL is that the ground-truth label must be in the candidate label set [9]. However, this assumption may not be satisfied due to the unprofessional judgment of annotators [10]. Recently, some researchers have relaxed this assumption and focused on a more practical setup called noisy PLL [11]. In noisy PLL, the ground-truth label may not be in the candidate label set. To deal with this task, Lv et al. [11] utilized noise-tolerant loss functions to avoid overemphasizing noisy samples. However, they cannot fully exploit the useful information in noisy data. To this end, Lian et al. [12] and Wang et al. [13] proposed to detect noisy samples and estimate pseudo labels for these samples. However, detection errors are unavoidable. These errors will accumulate and continuously affect model optimization, thereby limiting their performance on noisy PLL.

user A:

Horse

Ground Truth:

Donkey

user C:

Deer

user B:

Mule

Image

Origin PLL

ALIM

Classification

P(c)

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00

horse

mule

P(x)

deer

donkey

horse

mule

S(x)

deer

donkey

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00

horse

mule

deer

donkey

horse

mule

deer

donkey

P(x)S(x)

P(x)(S(x)+x(1-S(x)))

To alleviate this problem, we propose a novel framework for noisy PLL called 'Adjusting Label Importance Mechanism (ALIM)'. Although we may make mistakes in noisy sample detection, it allows us to leverage the initial candidate set and restart correction. To reduce manual efforts in hyperparameter tuning, we propose an adaptive strategy to determine the weighting coefficient for ALIM. To further improve noise tolerance, we equip ALIM with mixup training [14], a powerful technique in noisy label learning [15]. We also perform theoretical analysis from the perspective of objective functions and EM algorithms and prove the feasibility of our method. To verify its effectiveness, we conduct experiments on multiple benchmark datasets. Experimental results demonstrate that ALIM outperforms currently advanced approaches, setting new state-of-the-art records. The main contributions of this paper can be summarized as follows:

We propose a plug-in framework for noisy PLL with theoretical interpretations. Our method can deal with noisy samples by trading off the initial candidate set and model outputs.

We propose an adaptive strategy to adjust the weighting coefficient for ALIM. Furthermore, we combine our method with the mixup training to improve noise robustness.

Experimental results on multiple datasets demonstrate the effectiveness of our method. ALIM is superior to currently advanced approaches on noisy PLL.

2 Methodology

2.1 Problem Definition

Let X be the input space and Y be the label space with c distinct categories. We consider a dataset D containing N partially labeled samples { ( x, S ( x )) } . Here, S ( x ) ∈ { 0 , 1 } c is the candidate set for the sample x ∈ X . We denote the i th element of S ( x ) as S i ( x ) , which is equal to 1 if the label i is in the candidate set, and 0 otherwise. Meanwhile, we denote P ( x ) ∈ R c , w ( x ) ∈ R c , and y ( x ) ∈ R 1 as the prediction probabilities, the pseudo label, and the ground-truth label of the sample x . The i th elements of P ( x ) and w ( x ) are represented as P i ( x ) and w i ( x ) , respectively. In this paper, all product operations between matrices are Hadamard products.

The goal of noisy PLL is to learn a classifier that can minimize the classification risk on D . Unlike PLL, noisy PLL allows the ground-truth label may not be in the candidate set. For a fair comparison, we adopt the same data generation procedure as previous works [12]. Specifically, to generate candidate labels, we first flip incorrect labels into false positive labels with a probability q and aggregate the flipped ones with the ground-truth label. Then, we assume that each sample has a probability η of being the noisy sample. For each noisy sample, we further select a label from the non-candidate set, move it into the candidate set, and move the ground-truth label out of the candidate set. We denote the probability q as the ambiguity level and the probability η as the noise level.

2.2 Motivation

First, let us review our own exam experience. When we are unfamiliar with a test, we believe that the correct answer must be in the candidate set. Even if every option is wrong, we still choose the most likely answer. But as we become more familiar with the test, we learn to question the correctness of the candidate set. If we believe every option is wrong, we will consider answers outside the candidate set. In noisy PLL, this strategy can also be used to handle noisy samples.

2.3 ALIM Framework

Inspired by the above idea, we propose a simple yet effective framework called ALIM. The overall structure is shown in Figure 1. Specifically, we first predict the softmax probabilities P ( x ) for each sample x . In traditional PLL, we fully trust the candidate set and generate the pseudo label as follows:

where Normalize ( · ) is a normalization function that ensures ∑ c i =1 w i ( x ) = 1 . Different from traditional PLL, ALIM introduces a coefficient λ to control the reliability of the candidate set:

Here, λ = 0 means that we fully trust the given candidate set S ( x ) ; λ = 1 means that we don't believe the candidate set but trust our own judgment P ( x ) . According to Eq. 2, the time complexity of this process is mere O ( cN ) , where c is the number of categories and N is the number of samples.

In this paper, we discuss two choices of normalization functions: Onehot ( · ) and Scale ( · ) . Specifically, Onehot ( · ) sets the maximum value to 1 and others to 0; Scale ( · ) introduces a scaling factor K > 0 and normalizes the probabilities as follows:

where z i is the i th element of z .

2.4 Theoretical Analysis

To verify the feasibility of our method, we conduct theoretical analysis from two perspectives: (1) the manually-designed objective function; (2) the classic expectation-maximization (EM) algorithm [16].

2.4.1 Interpretation from Objective Functions

Let P i , S i , and w i be abbreviations for P i ( x ) , S i ( x ) , and w i ( x ) , respectively. During training, we should optimize the following objectives:

Minimize the classification loss on w ( x ) and P ( x ) .

w ( x ) should be small at non-candidate labels.

Entropy regularization on w ( x ) to avoid overconfidence of pseudo labels.

w ( x ) should satisfy 0 ≤ w i ≤ 1 and ∑ c i =1 w i = 1 .

Then, the final objective function is calculated as follows:

where M and K are penalty factors. By using Lagrange multipliers, we can observe that the penalty factor K is different for two normalization functions: K = 0 for Onehot ( · ) and K > 0 for Scale ( · ) . The penalty factor M has a strong correlation with the weighting coefficient λ in Eq. 2, i.e., λ = e -M . Larger M (or smaller λ ) means that we have tighter constraints on ( ∑ c i =1 w i S i -1) , and therefore we should trust the given candidate set more. It is identical to the meaning of λ in our ALIM (see Section 2.3).

2.4.2 Interpretation from EM Perspective

EM aims to maximize the likelihood of the corpus D . Following previous works [5, 17], we first make a mild assumption:

Assumption 1 In noisy PLL, the ground-truth label may not be in the candidate set S ( x ) . We assume that each candidate label { i | S i ( x ) = 1 } has an equal probability α ( x ) of generating S ( x ) and each non-candidate label { i | S i ( x ) = 0 } has an equal probability β ( x ) of generating S ( x ) .

Besides the interpretation from objective functions, we further explain ALIM from the EM perspectivein Appendix B. We prove that the E-step aims to predict the ground-truth label for each sample and the M-step aims to minimize the classification loss. Meanwhile, ALIM is a simplified version of the results derived from EM. Specifically, EM uses an instance-dependent λ ( x ) = β ( x ) /α ( x ) , while ALIM uses a global λ . In the future, we will explore the performance of the instance-dependent λ ( x ) for noisy PLL. Additionally, EM connects traditional PLL with noisy PLL. In traditional PLL, we assume that the ground-truth label must be in the candidate set, i.e., β ( x ) = 0 . Then, our noisy PLL approach ALIM will degenerate to the classic PLL method RC [18].

2.5 Optional Key Components

We further introduce several key components to make our method more effective, including the adaptively adjusted strategy and the mixup training.

2.5.1 Adaptively Adjusted Strategy

Appropriate λ is important for ALIM. Too small λ makes us fully trust the candidate set, thus easily over-fitting on noise samples; too large λ makes us ignore the prior information in the candidate set, thus limiting the classification performance. Therefore, we propose a manually adjusted strategy to find a proper λ in a predefined parameter space.

To further reduce manual efforts, we also propose an adaptively adjusted strategy. Specifically, we first estimate the noise level of the dataset. Intuitively, we can randomly select a subset from D , annotate the ground-truth labels by professional annotators, and estimate the noise level of the dataset. Alternatively, we can automatically estimate noise rate via the Gaussian mixture model [19, 20] or cross-validation [21, 22]. The estimated noise level is represented as η . Based on Appendix C, we prove that the value of Eq. 6 can be viewed as a metric, and the η -quantile of this value can be treated as the adaptively adjusted λ .

In Section 3.2, We further present results without noise rate estimation and manually adjust λ as a hyper-parameter. Through experimental analysis, this approach can also achieve competitive performance. Therefore, the adaptively adjusted strategy is optional. Its main advantage is to reduce manual efforts in hyper-parameter tuning and realize a more automatic approach for noisy PLL.

2.5.2 Mixup Training

Since the mixup training is powerful in noisy label learning [14, 15], we further combine this strategy with ALIM for noisy PLL. Consider a pair of samples x i and x j whose pseudo labels are denoted as w ( x i ) and w ( x j ) , respectively. Then, we create a virtual training sample by linear interpolation:

Algorithm 1: Pseudo-code of ALIM.

α ∼ Beta ( ζ, ζ ) , where ζ is a parameter in the beta distribution. We define the mixup objective L mix as the cross-entropy loss on P ( x mix ) and w mix. During training, we combine the mixup loss L mix and the PLL loss L pll into a joint objective function L = L pll + λ mix L mix, where λ mix controls the trade-off between two losses. The pseudo-code of ALIM is summarized in Algorithm 1.

3 Experiments

3.1 Experimental Setup

Corpus Description In the main experiments, we evaluate the performance on two benchmark datasets of noisy PLL, CIFAR-10 [23] and CIFAR-100 [23]. We choose the noise level η from { 0 . 1 , 0 . 2 , 0 . 3 } . Since CIFAR-100 has more classes than CIFAR-10, we consider q ∈ { 0 . 1 , 0 . 3 , 0 . 5 } for CIFAR-10 and q ∈ { 0 . 01 , 0 . 03 , 0 . 05 } for CIFAR-100. In Section 3.2, we also conduct experiments on fine-grained datasets (CUB-200 [17] and CIFAR-100H [24]) and consider severe noise.

Baselines To verify the effectiveness of our method, we implement the following state-of-the-art methods as baselines: 1) CC [18]: a classifier-consistent PLL method under the uniform candidate label generation assumption; 2) RC [18]: a risk-consistent PLL method under the same assumption as CC; 3) LWC [25]: a PLL method that considers the trade-off between losses on candidate and non-candidate sets; 4) LWS [25]: a PLL method that combines the weighted loss with the sigmoid activation function; 5) PiCO [17]: a PLL method using contrastive learning; 6) CRDPLL [26]: a PLL method that exploits consistency regularization on the candidate set and supervised learning on the non-candidate set; 7) IRNet [12]: a noisy PLL method that progressively purifies noisy samples; 8) PiCO+ [13]: a noisy PLL method using a semi-supervised contrastive framework.

Implementation Details There are mainly three user-specific parameters in ALIM: λ , λ mix, and e 0 . Among them, λ controls the trade-off between the initial candidate set and model outputs. This paper proposes two selection strategies for λ , i.e., manually and adaptively adjusted strategies. For the first one, we treat λ as a hyper-parameter and select it from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 7 } . For the second one, we automatically determine λ using the estimated noise level. λ mix controls the trade-off between the PLL loss and the mixup loss, and we set λ mix = 1 . 0 as the default parameter. e 0 is the start epoch of ALIM, and we select it from { 20 , 40 , 80 , 100 , 140 } . Following the standard experimental setup in PLL [17, 25], we split a clean validation set from the training set to determine hyper-parameters. Then, we transform the validation set back to its original PLL form and incorporate it into the training set to accomplish model optimization. To optimize all trainable parameters, we choose the SGD optimizer with a momentum of 0.9 and set the weight decay to 0.001. We set the initial learning rate to 0.01 and adjust it using the cosine scheduler. To eliminate the randomness of the results, we run each experiment three times and report the average result and standard deviation on the test set. All experiments are implemented with PyTorch [27] and carried out with NVIDIA Tesla V100 GPU.

3.2 Experimental Results and Discussion

Main Results For a fair comparison, we reproduce all baselines using the same data generation strategy as previous works [12]. In Table 1, we observe that a large portion of improvement is due to mixup training rather than model innovation. To this end, we compare different approaches under the same mixup strategy. Experimental results demonstrate that our method succeeds over all baselines under varying ambiguity levels and noise levels. The main reason lies in two folds. On the one hand, existing PLL methods are mainly designed for clean samples but ignore the presence of noisy samples. Our method can deal with noisy samples by trading off the initial candidate set and model outputs. On the other hand, existing noisy PLL methods generally need to detect noisy samples, but detection errors are unavoidable. These errors can accumulate and continuously affect the training process. Differently, ALIM can deal with this problem by taking advantage of the initial candidate set and restarting correction. These results prove the effectiveness of ALIM in noisy PLL.

Compatibility of ALIM Since ALIM is a plug-in strategy, we integrate it with existing PLL methods and report results in Table 2. We observe that ALIM always brings performance improvement under noisy conditions, verifying the effectiveness and compatibility of our method. Meanwhile,

integrating ALIM with PiCO generally achieves better performance under severe noise ( η = 0 . 3 ). In noisy label learning, self-training suffers from accumulated errors caused by sample-selection bias [28]. To address this issue, researchers propose co-training [29], which contains multiple branches and cross-updates these branches. Among all PLL methods, PiCO is a natural co-training network with two branches: one for classification and one for clustering [17]. The output of the classification branch guides the model to update the clustering prototypes; the output of the clustering branch affects the update of the pseudo label for classification. These results verify the effectiveness of co-training under noisy conditions. Therefore, we combine ALIM with PiCO in the following experiments.

Fine-Grained Classification Performance Considering that similar categories are more likely to be added to the candidate set, we conduct experiments on more challenging fine-grained datasets, CIFAR100H [17] and CUB-200 [24]. Different from previous works [12] that generate candidate labels from the entire label space, this section considers the case of generating candidate labels belonging to the same superclass. Experimental results in Table 3 demonstrate that ALIM outperforms existing methods on fine-grained classification tasks, verifying the effectiveness of our method.

Robustness with Severe Noise To demonstrate noise robustness, we conduct experiments under severe noise conditions. In this section, we compare ALIM with the most competitive baselines (RC, PiCO, CRDPLL, and PiCO+) under η ∈ { 0 . 4 , 0 . 5 } . Experimental results in Table 4 show that ALIM succeeds over all baselines under severe noise. Taking the results on η = 0 . 5 as an example, our method outperforms the currently advanced approaches by 11.76%. These results demonstrate that our ALIM is more robust to noise than existing PLL and noisy PLL methods.

Discussion on Normalization Functions In this section, we compare the performance of two normalization functions. In Table 1, we observe that Onehot ( · ) performs slightly better on CIFAR-10 and Scale ( · ) performs slightly better on CIFAR-100. In Table 3 ∼ 4, Scale ( · ) generally achieves better performance in fine-grained datasets and severe noise conditions. The main difference between these normalization functions is that Onehot ( · ) compresses estimated probabilities into a specific class, while Scale ( · ) preserves prediction information for all categories. For simple datasets like CIFAR-10, the class with the highest probability is likely to be correct. Therefore, the compression operation can

(a)

100 epoch

(b)

200 epoch

(c)

400 epoch

(d)

800 epoch

4000

0009

0009

4000

clean

3500

noisy

3500

3500

3500

3000

adaptive

3000

3000

3000

frequency

2500

frequency

frequency

2500

frequency

2500

2000

2000

2000

2000

1500

1500

1500

1500

1000

1000

1000

1000

500

500

500

500

0t

10-1

100

102

102

102

10-1

109

101

102

102

101

100

101

102

10-2

10-1

10#

201

102

metric

metric

metric

metric

(a)

CIFAR-10 (w/o mixup)

(b)

CIFAR-10 (w/ mixup)

(c)

CIFAR-100 (w/o mixup)

(d)

CIFAR-100 (w/ mixup)

92.03

92.42

94.51

94.67

70.90

70.05

74.98

74.49

accuracy

90.07

accuracy

92.44

accuracy

68.36

accuracy

72.05

88.12

90.36

65.81

69.13

86.16

test

88.29

test

63.27

test

66.20

84.21

86.21

60.72

63.28

82.25

84.14

58.18

60.35

80.29

0.0

0.1

0.2

0.3

0.4

0.5

0.7

82.07

0.0

0.1

0.2

0.3

0.4

0.5

0.7

55.64

0.0

0.1

0.2

0.3

0.4

0.5

0.7

57.42

0.0

0.1

0.2

0.3

0.4

0.5

0.7

入

入

入

入

(a)

λ

mix

on CIFAR-10 (

e

0

= 80

)

(b)

e

0

on CIFAR-10 (

λ

mix

= 1

)

(c)

λ

mix

on CIFAR-100 (

e

0

= 80

)

(d)

e

0

on CIFAR-100 (

λ

mix

= 1

)

94.91

94.67

73.88

74.49

94.41

94.41

73.11

73.65

accuracy

93.91

accuracy

94.14

accuracy

72.35

72.81

93.42

test

93.88

test

71.58

test

71.96

92.92

93.61

70.82

71.12

92.42

93.35

70.05

70.28

91.92

0.0

0.1

0.5

入mix

1.0

2.0

5.0

8.0

93.09

20

40

80

100140300400500

eo

69.28

0.0

0.1

0.5

Amix

1.0

2.0

5.0

8.0

69.44

20

40

80100140300400500

eo

reduce the negative impact of other categories and achieve better performance on noisy PLL. For challenging datasets, the class with the highest predicted value has a low probability of being correct. Therefore, this compression operation may lead to severe information loss. More importantly, ALIM consistently outperforms existing methods regardless of the normalization function. Therefore, the main improvement comes from our ALIM rather than normalization functions.

Rationality of Adaptively Adjusted Strategy In this section, we further explain the rationality of our adaptively adjusted strategy. Figure 2 visualizes the distribution of the value in Eq. 6 for clean and noise subsets with increasing training iterations. We observe that this value is an effective indicator for distinguishing clean samples from noisy samples. At the same time, our adaptively adjusted λ serves as a suitable boundary for clean and noisy subsets.

Role of Adaptively Adjusted Strategy It is important to select a proper λ in ALIM. In this paper, we propose two selection strategies: adaptively and manually adjusted strategies. Figure 3 presents the classification performance of these strategies. Experimental results show that different strategies can achieve similar performance. Therefore, the main advantage of this adaptively adjusted strategy is to reduce manual efforts in hyper-parameter tuning. The performance improvement still comes from ALIM, which exploits a coefficient to control the reliability of the candidate set.

Parameter Sensitivity Analysis In this section, we perform parameter sensitivity analysis on two key hyper-parameters: the warm-up epoch e 0 , and the trade-off between the PLL loss and the mixup loss λ mix. In Figure 4, we observe that choosing an appropriate λ mix is important for ALIM, and λ mix = 1 . 0 generally achieves competitive results. Meanwhile, with the gradual increase of e 0 , the test accuracy first increases and then decreases. This phenomenon indicates that ALIM needs warm-up training. But too large e 0 can also cause the model to overfit noise samples.

4 Related Work

In PLL, the ground-truth label is concealed in the candidate set. To deal with this task, the core is to disambiguate the candidate labels. In this section, we first introduce two typical disambiguation strategies, i.e., average-based methods and identification-based methods. Then, we briefly review some recent works on noisy PLL.

Average-based PLL The most intuitive solution is the average-based approach, which assumes that each candidate label has an equal probability of being the ground-truth label. For example, Hüllermeier et al. [30] utilized k-nearest neighbors for label disambiguation. For each sample, they treated all candidate labels of its neighborhood equally and predicted the ground-truth label through the majority vote. Differently, Cour et al. [31] maximized the average output of candidate labels and non-candidate labels in parametric models. However, these average-based methods can be severely affected by false positive labels [9].

Identification-based PLL To address the shortcomings of the above methods, researchers have focused on identification-based methods. Different from average-based methods that treat all candidate labels equally, identification-based methods treat the ground-truth label as a latent variable and maximize its estimated probability by maximum margin [32] or maximum likelihood [5] criteria.

Recently, deep learning has been applied to identification-based methods and achieved promising results. For example, Lv et al. [33] proposed a self-training strategy to disambiguate candidate labels. Feng et al. [18] introduced classifier- and risk-consistent algorithms under the uniform candidate label generation assumption. Wen et al. [25] relaxed this assumption and proposed a family of loss functions for label disambiguation. To learn more discriminative representations, Wang et al. [17] exploited contrastive learning to deal with partially labeled samples. More recently, Wu et al. [26] used consistency regularization on the candidate set and supervised learning on the non-candidate set, achieving promising results under varying ambiguity levels. The above methods rely on a basic assumption that the ground-truth label must be in the candidate set. But this assumption may not be satisfied due to the unprofessional judgment of annotators.

Noisy PLL Recently, noisy PLL has attracted increasing attention from researchers due to its more practical setup. Its core challenge is how to deal with noisy samples. For example, Lv et al. [11] utilized noise-tolerant loss functions to avoid overemphasizing noisy samples during training. Lian et al. [12] proposed an iterative refinement network to purify noisy samples and reduce the noise level of the dataset. Wang et al. [13] detected clean samples through a distance-based sample selection mechanism and dealt with noisy samples via a semi-supervised contrastive framework. These noisy PLL methods generally need to detect noisy samples, but detection errors are unavoidable. These errors can accumulate and continuously affect model training. To reduce the negative impact of prediction errors, we propose a novel framework for noisy PLL. Experimental results on multiple datasets demonstrate the effectiveness of our method under noisy conditions.

5 Conclusion

This paper introduces a novel framework for noisy PLL called ALIM. To deal with noisy samples, it exploits the weighting mechanism to adjust the reliability of the initial candidate set and model outputs. To verify its effectiveness, we conduct experiments on multiple benchmark datasets under varying ambiguity levels and noise levels. Experimental results demonstrate that our ALIM achieves state-of-the-art classification performance on noisy PLL, especially in severe noise conditions and fine-grained datasets. Meanwhile, ALIM is a low-computation plug-in strategy that can be easily integrated with existing PLL frameworks. Furthermore, we interpret the rationality and effectiveness

of the adaptively adjusted strategy. We also conduct parameter sensitivity analysis and reveal the impact of different hyper-parameters. It is worth noting that this paper leverages a global λ to measure the reliability of the candidate set. In the future, we will explore the instance-dependent λ ( x ) during training. At the same time, we will investigate the performance in more experimental settings, such as class-imbalanced conditions.

References

Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In Proceedings of the International Joint Conference on Artificial Intelligence , pages 2107-2113, 2018.

Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6575-6582, 2020.

Yi-Chen Chen, Vishal M Patel, Rama Chellappa, and P Jonathon Phillips. Ambiguously labeled learning using dictionaries. IEEE Transactions on Information Forensics and Security , 9(12):2076-2088, 2014.

Ching-Hui Chen, Vishal M Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(7):1653-1667, 2017.

Li-Ping Liu and Thomas G Dietterich. A conditional multinomial mixture model for superset label learning. In Proceedings of the 25th International Conference on Neural Information Processing Systems , pages 548-556, 2012.

Liping Liu and Thomas Dietterich. Learnability of the superset label learning problem. In Proceedings of the International Conference on Machine Learning , pages 1629-1637, 2014.

Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval , pages 39-43, 2008.

Forrest Briggs, Xiaoli Z Fern, and Raviv Raich. Rank-loss support instance machines for miml instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 534-542, 2012.

Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In Proceedings of the 15th International Conference on Neural Information Processing Systems , pages 921-928, 2002.

Jesús Cid-Sueiro. Proper losses for learning from partial labels. In Proceedings of the 25th International Conference on Neural Information Processing Systems , pages 1565-1573, 2012.

Jiaqi Lv, Lei Feng, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. arXiv preprint arXiv:2106.06152 , 2021.

Zheng Lian, Mingyu Xu, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Arnet: Automatic refinement network for noisy partial label learning. arXiv preprint arXiv:2211.04774 , 2022.

Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico+: Contrastive label disambiguation for robust partial label learning. arXiv preprint arXiv:2201.08984 , 2022.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations , pages 1-13, 2018.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In Proceedings of the International Conference on Learning Representations , pages 1-14, 2020.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, 1977.

Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In Proceedings of the International Conference on Learning Representations , pages 1-18, 2022.

Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Proceedings of the Advances in Neural Information Processing Systems , pages 10948-10960, 2020.

Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In Proceedings of the International Conference on Machine Learning , pages 312-321, 2019.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Robust learning by self-transition for handling noisy labels. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1490-1500, 2021.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(3):447-461, 2015.

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In Proceedings of the International Conference on Machine Learning , pages 5907-5915, 2019.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. In Proceedings of the International Conference on Machine Learning , pages 11091-11100, 2021.

Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In Proceedings of the International Conference on Machine Learning , pages 24212-24225, 2022.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pages 8026-8037, 2019.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Proceedings of the International Conference on Machine Learning , pages 2304-2313, 2018.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory , pages 92-100, 1998.

Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis , 10(5):419-439, 2006.

Timothee Cour, Benjamin Sapp, Chris Jordan, and Ben Taskar. Learning from ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 919-926, 2009.

Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In Proceedings of the Asian Conference on Machine Learning , pages 96-111, 2016.

Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the International Conference on Machine Learning , pages 6500-6510, 2020.

A Interpretation from Objective Functions

In this section, we provide proofs of the Onehot ( · ) normalization function and the Scale ( · ) normalization function from the perspective of objective functions.

A.1 Proof for Onehot Normalization

For K = 0 , we choose the following objective function during training:

Introduce Lagrange multipliers δ i , i ∈ [1 , c ] and γ into Eq. 9, we have:

Combined with the Karush-Kuhn-Tucker (KKT) conditions, the optimal point should satisfy:

Since S i ∈ { 0 , 1 } , we have MS i = log( e M S i +(1 -S i )) . The equivalent equation of Eq. 11 is:

Combined with δ i ≥ 0 in Eq. 12, we have:

δ i > 0 is true if γ > max i ( log( e M S i +(1 -S i )) P i ) . According to δ i w i = 0 , we always have w i = 0 , which conflicts with ∑ c i =1 w i = 1 . Therefore, we get:

We assume that only one i 0 ∈ [1 , c ] reaches the maximum γ , then we have w i = 0 , i ∈ [1 , c ] /i 0 . Combined with ∑ c i =1 w i = 1 , we get w i 0 = 1 . Therefore, w ( x ) should satisfy:

We mark λ = e -M and convert Eq. 16 to its equivalent version:

A.2 Proof for Scale Normalization

For K > 0 , log w i ensures that w i must be positive. Therefore, the constraint w i ≥ 0 can be excluded. Then, the objective function can be converted to:

Introduce the Lagrange multiplier γ in Eq. 18, we have:

Since the optimal point should satisfy ∇ w L = 0 , we have:

Since S i ∈ { 0 , 1 } , we have MS i = log( e M S i +(1 -S i )) . The equivalent equation of Eq. 20 is:

We mark λ = e -M . Then, we have:

Since ∑ c i w i = 1 , we have:

Combine Eq. 23 and Eq. 25 and we have:

Combined with the definition of Scale ( · ) in Eq. 4, this equation can be converted to:

B EMPerspective of ALIM

EM aims to maximize the likelihood of the dataset D :

where θ is the trainable parameter. The last step of Eq. 28 utilizes Jensen's inequality. Since the log( · ) function is strictly concave, the equal sign takes when P ( x, S ( x ) , y ( x ) = i ; θ ) /w i ( x ) is some constant C , i.e.,

Considering that ∑ c i =1 w i ( x ) = 1 , we can further get:

Then, we have:

In the EM algorithm [16], the E-step aims to calculate w i ( x ) and the M-step aims to maximize the lower bound of Eq. 28:

E-Step. In this step, we aim to predict the ground-truth label for each sample:

According to Assumption 1, we have:

It can be converted to:

Then, we get the equivalent equation of Eq. 33:

We mark λ ( x ) = β ( x ) /α ( x ) and P i ( x ) = P ( y ( x ) = i | x ; θ ) . Then, we get:

It connects traditional PLL and noisy PLL. In traditional PLL, we assume that the ground-truth label must be in the candidate set, i.e., β ( x ) = 0 . Since λ ( x ) = β ( x ) /α ( x ) = 0 , Eq. 37 degenerates to:

which is identical to the classic PLL method, RC [18].

M-Step. The objective function of this step is:

Considering that P ( x ; θ ) = P ( x ) and P ( S ( x ) | y ( x ) = i, x ; θ ) = P ( S ( x ) | y ( x ) = i, x ) , the equivalent version of Eq. 39 is:

Therefore, the essence of the M-step is to minimize the classification loss.

C Adaptively Adjusted λ

Since η controls the noise level of the dataset, we have:

After the warm-up training, we assume that the predicted label generated by ALIM ˆ y ( x ) = arg max 1 ≤ i ≤ c w ( x ) is accurate, i.e., ˆ y ( x ) = y ( x ) . Then we have:

To estimate the value of λ , we first study the equivalent meaning of S ˆ y ( x ) ( x ) = 0 :

We simplify the left and right sides of Eq.43 as follows:

Then, we have:

Therefore, P ( S ˆ y ( x ) ( x ) = 0) = η can be converted to:

It means that λ is the η -quantile of

D Effect of Scaling Factor

Larger K produces smoother pseudo labels. To investigate its impact, we perform a parameter sensitivity analysis in Table 5. We observe that a suitable K is between 0 and 1. Too large K will produce over-smoothed pseudo labels and leads to performance degradation. In this paper, we choose K = 1 as the default parameter for the Scale ( · ) normalization function.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the noise partial label learning problem, i.e., the ground-true may also be in the non-candidate set. The proposed method adjusts the importance of the labels by considering the output of the candidate label set model in an integrated manner. The proposed method in this paper can be viewed as a post-processing means for prediction models, and the proposed method achieves by superior performance compared to existing methods on multiple benchmark datasets.\n\nSTRENGTHS:\n1.\tThe noise PLL issue is important and the authors have a good summary of the unresolved issues in that paper from existing work.\n2.\tRich experimental results.\n\nWEAKNESSES:\n1. This paper mainly uses the candidate label set and non-candidate label set information to post-process the prediction results, and the innovation is weak.\n2. The framework of the method has a stronger assumption that P(x) can also achieve better results in noise partial label learning problem, otherwise it is difficult to inform the subsequent processing. From Figure 1, we can see that for noise partial label learning, P(x) has been able to calculate the confidence of each label well, and the confidence of the true label is higher compared to ALIM.\n3. While the way of adjusting label importance proposed in the paper avoids errors when detecting noise directly, the method used in this paper will also assign a certain confidence level to all non-candidate labels, which will also introduce noise.\n4. One of the key points of the method proposed in this paper is the setting of the hyper-parameter lambda. An adaptive method is proposed in this paper, but the adaptive parameter is not analyzed in the experimental results to know at what value the setting is reasonable.\n5. For Mixup Training, the paper does not explain why it can counteract the effects of noise in PLL.\n\nQUESTIONS:\n1.\tIn line 91, the authors describe that w(x) should keep very small data, so how can the true labels be extracted from the set of non-candidate labels?\n2.\tHow is the conclusion derived from the different values of K in the paper, which correspond to different normalizations?\n3.\tWhat is the principle of Mixup Training in noise partial label learning against noise?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper focuses on a relaxed version of partial label learning, called noisy PLL. Different from PLL, noisy PLL allows the ground-truth label may not be in the candidate set. To deal with this problem, this paper proposes a framework called ALIM, which is a plug-in strategy with theoretical guarantees. Experimental results on CIFAR-10, CIFAR-100, and fine-grained datasets prove the effectiveness of ALIM on noisy PLL. This paper is well-written and easy to follow.

STRENGTHS:
1.	This paper presents a plug-and-play framework for the challenging noisy PLL task.
2.	This paper conducts theoretical analysis from objective functions and EM algorithms.
3.	ALIM shows competitive performance compared with the currently advanced strategies.

WEAKNESSES:
$Scale(\cdot)$ normalization function introduces a scaling factor $K$. But in the following experiments, this paper does not specify the selection process of this factor. Please explain its impact on the performance and how it was chosen.

QUESTIONS:
Can the authors explain the impact of the scaling factor $K$ on the performance and how it was chosen?

LIMITATIONS:
Please refer to the weakness and question.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers noisy partial label learning when the provided label is not in the candidate label set. The paper proposes \"Adjusting Label Importance Mechanism\" to trade off the initial candidate set and model outputs. Some theoretical analysis is done and good empirical results are shown against selected baselines.\n\nSTRENGTHS:\n1. The considered problem (i.e. the provided label is not in the candidate label set) might be a relatively new problem and is less considered in existing work.\n\n2. Some theoretical analysis is done.\n\n3. A few tricks are proposed to make the method more effective.\n\n4. Good experiment results are shown against baselines.\n\nWEAKNESSES:\n1. Writing can be improved. Currently, it seems the manuscript is prepared in a hurry. For example, there are errors when referring to appendices: “Appendix ??”. Overall, I had a hard time following. \n\n2. The core contribution is not clear. The motivation in Section 2.2 is hard to follow. From Section 2.2, it seems the paper is trying to consider the case when the ground-truth label is not in the provided candidate set of labels. It should be made clear why this is an important problem and how the proposed method addresses the problem. Also, there are concerns on the core technique. see Q1 and Q4. \n\n3. The theoretical analysis is not clear see Q2.\n\nQUESTIONS:\n1. When the label is not in the candidate set, it’s also likely that it's not in 1-S(x). Actually, not being in the label space might even be the main reason that the label is not in the candidate set. How to handle this case? It seems there is an assumption that the label is for sure in the given label space, and it might not be in the provided candidate which is a subset of the label space. This should be made clear. \n\n2. In Equation 5, why does M correspond to the lambda in equation 2? In section 2.4.2, it is also unclear what is the connection to EM. \n\n3. Abstract mentions theoretical guarantees, what are the guarantees? It is not mentioned in the paper anywhere. \n\n4. From the theoretical analysis, it seems there is an equivalence between the proposed method to an (existing?) objective function. Then, why would one need the proposed method?\n\nLIMITATIONS:\nLimitations are not discussed. Authors should discuss when the proposed method doesn't work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focused on the problem of noisy partial label learning (noisy PLL), which means the candidate label set in the partial label learning does not always contain the ground-truth label. Compared with the existing studies which attempted to detect noisy samples yet lead to the accumulation of detection errors, this work proposed to reduce the impact of the detection errors by training off the initial candidate set and model outputs. The proposed method, named \"Adjusting Label Importance Mechanism (ALIM)\", can adaptively adjust the weighting coefficient between the initial candidate set and the model predictions (i.e., pseudo-labels output by the model). The theoretical analysis showed that the proposed method can be well interpreted from two perspectives of heuristic design and classic Expectation-maximization (EM) algorithm. Combined with Mixup training strategies, the proposed approach can be further plugged into the existing noisy PPL baselines and bring consistent improvements. Extensive experiments demonstrated the effectiveness of the proposed framework.\n\nSTRENGTHS:\n- The motivation is clear. The analysis of the issues of the existing methods motivated the proposed method in a straightforward manner.\n-  The proposed objective can be interpreted under the EM framework, which provided a convincing mathematical explanation.\n- The proposed method, ALIM, is also compatible with other related approaches. It can work as a plug-in in the existing noisy PLL algorithms, which enhanced the flexibility of ALIM.\n- The experimental part conducted empirical comparisons/ablation studies with respect to different factors, e.g., the different noise levels and ambiguity levels, and the different choices of the key components within the proposed framework.\n\nWEAKNESSES:\n- Some technical details were not clear to the reviewers. See the next part for more details.\n\nQUESTIONS:\n- In Line 140, what is the definition of $\\mathcal{L}_{pll}$? It suddenly appeared here without any pre-defined formation. Which kind of PLL loss did it indicate, the objective in Eq.(5)?\n- I recommend the authors reformulate the optimization objective in Eq. (5) as a $\\min$ form, which will be better compatible with the form of the loss function. \n- Compared to the results reported on the CIFAR10 dataset, the reviewer found that ALIM-Scale consistently performed better than ALIM-Onehot on other datasets like CIFAR100 and CUB200. Is it a necessity that scaling performs better than one shot on datasets with larger class numbers, or a coincidence? Could the author provide more analyses and explanations about this point?\n- Some potential typos:\n    - Line 95, Appendix ??\n    - Line 107, Appendix ??\n    - Line 126, Appendix ??\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}