Exploiting Class Activation Value for Partial-Label Learning
Authors: Fei Zhang
Lei Feng
Bo Han
Tongliang Liu
Gang Niu
Tao Qin
Masashi Sugiyama
Venue: ICLR 2022 Poster (2022)
Keywords: Partial-label Learning
Class Activation Map

Abstract: Partial-label learning (PLL) solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include the true label. Recent advances showed that PLL can be compatible with deep neural networks, which achieved state-of-the-art performance. However, most of the existing deep PLL methods focus on designing proper training objectives under various assumptions on the collected data, which may limit their performance when the collected data cannot satisfy the adopted assumptions. In this paper, we propose to exploit the learned intrinsic representation of the model to identify the true label in the training process, which does not rely on any assumptions on the collected data. We make two key contributions. As the first contribution, we empirically show that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in images, could surprisingly be utilized to make accurate predictions on selecting the true label from candidate labels. Unfortunately, as CAM is confined to image inputs with convolutional neural networks, we are yet unable to directly leverage CAM to address the PLL problem with general inputs and models. Thus, as the second contribution, we propose the class activation value (CAV), which owns similar properties of CAM, while CAV is versatile in various types of inputs and models. Building upon CAV, we propose a novel method named CAV Learning (CAVL) that selects the true label by the class with the maximum CAV for model training. Extensive experiments on various datasets demonstrate that our proposed CAVL method achieves state-of-the-art performance.

Published as a conference paper at ICLR 2022
EXPLOITING CLASS ACTIVATION PARTIAL-LABEL LEARNING
VALUE FOR
Fei Zhang 1.2
Lei Feng") 3.1 Bo Han
Tongliang Liu
Gang Niu
Tao Qin
Masashi Sugiyama
Shanghai Jiao Tong University Hong Kong Baptist University
*Chongqing University
"The University of Sydney 'RIKEN Microsoft Research Asia
The University of Tokyo
ferenas @ sjtu.edu..cn wenggecquu...cccc en, bhanml@ comp.hkbu.edeeie hww..puuuu...ukk.. tongliang. wiw.g..luueee @eee@edydyee..aaa gang.niu niu @ @riken.jp, taoqin caciii@@miccccooooooc sugi @ Soeiikkku.kkooa....
ABSTRACT
Partial-label learning (PLL) solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include the true label. Recent advances showed that PLL can be compatible with deep neural net- works, which achieved state-of-the-art performance. However, most of the exist- ing deep PLL methods focus on designing proper training objectives under various assumptions on the collected data, which may limit their performance when the collected data cannot satisfy the adopted assumptions, In this paper, we propose exploit the learned intrinsic representation of the model identify the true label in the training process, which does not rely on any assumptions on the collected data. We make two key contributions. As the first contribution, we empirically show that the class activation map (CAM), simple technique for discriminat- ing the learning patterns of each class in images, could surprisingly utilized make accurate predictions on selecting the true label from candidate labels. Unfortumately, as CAM is confined to image imputs with convolutional neural net- works,
yet unable to directly leverage CAM address the PLL problem with general inputs and models. Thus, as the second contribution, we propose the class activation value (CAV), which owns similar properties of CAM, while CAV is versatile in various types of inputs and models. Building upon CAV, we propose novel method named CAV Learning (CAVL) that selects the true label by the class with the maximum CAV for model training. Extensive experiments on various datasets demonstrate that our proposed CAVL method achieves state- of-the-art performance.
INTRODUCTION
To liberate humans from exhaustive label annotation work, numerous researchers have dedicated themselves to investigating various weakly supervised learning (WSL) (Zhou, 2017) problems, including but not limited to noisy-label learning (Liu & Tao, 2015; Xia al., 2020; Han et al., 2020), semi-supervised learning (Zhu & Goldberg, 2009; Miyato et al., 2018; 8; Luo et al., 2018), and multiple-instance learning (Zhou al., 2012). This paper focuses on another popular WSL problem called partial-label learning (PLL) (Jin & Ghahramani, 2002; Cour et al., 201 which aims to learn model from training examples equipped with set of candidate labels that include the true label. Due to the cost and difficulty of annotating every example exactly with the true label in huge datasets, PLL has been widely applied to various tasks such as multimedia context analysis (Zeng et al., 2013) and web mining (Luo & Orabona, 2010).
To solve the PLL problem, there are two mainstream strategies to discriminate the unknown true label from candidate labels, including the average-bused strategy (ABS) and the identification-burel strategy (IBS). ABS always treats each candidate label equally and averages the model outputs of
Cooresponndence Lei Feng (Ifeng@ (cien@@cqu.edu.cn) and Bo Han (bhanml (bhanml@comp..ll www.ooo......ccc.o.oocoom

---

Published as a conference paper at ICLR 2022
all candidate labels for prediction (Hüllermeier & Beringer, 2006; Cour et al., 2011; Feng et al., 2020; Yao et al., 2020; Wen et al., 2021). IBS concentrates on iteratively selecting one label from the candidate label set as the true label to exclude the uncertainty. forming PLL into an ordinary classification problem (Jin & Ghahramani, 2002; Liu & Dietterich, 2014; Zhang & Yu, 2015; Zhang al., 2016). Among these methods, only several of them (Yao et al., 2020; Wen et at., 2021) can be compatible with deep neural networks, which achieved state-of-the-ann performance.
Most of the existing PLL methods aim at designing proper training objectives under various assump- tions on the collected data. For example, the consistent methods proposed by Feng et al. (2020) are based on specific data generation assumption. The derivation of the PRODEN method (Lv al., 2020) stems from the assumption that the true label achieves the minimal loss among the candidate labels. Yan Guo (2021) proposed the MGPLI, method based on the assumption of non-random label noise. Although these methods have achieved generally great empirical performance, their performance could be degraded when the collected data cannot meet the adopted assumptions.
In this paper, we aim to investigate novel PLL method by exploiting the learned intrinsic repre- sentation of the model to identify the ground truth in the training process without relying on any assumptions on the collected data. We focus on the class activation mup (CAM) (Zhou et al., 2016), which is a prevailing tool for visualizing the representation information of convolutional neural ner- work (CNN)-buee -based models and could be easily obtained by the weighted linear sum of the feature maps. As CAM is able to discriminate the learning patterns of each class for the model, we conjec- ture that CAM could be used to guide the model spot the true label. To verify this assumption, we conducted a pilot experiment on CIFAR-10 10 (Krizhevsky et al., 2009), where the candidate label sets were generated in two different ways. As shown in Figure 1 (please refer Section 2.2), is surprising to find that CAM can potentially help the model recognize the true labels, and such capacity of CAM synchronously changes with the classifier performance during the whole training phase. Based on the experimental results, it can be conjectured that such intrinsic representations can be useful to distinguish the true label from candidate labels for PI.I..
Motivated by the above observations, we naturally consider leveraging CAM to solve the PLL prob- lem. However, CAM was only proposed for addressing image datasets with using deep models built on CNN, revealing two limitations its application. Firstly, CAM cannot be adopted to the classi- fier based on shallow models such as the linear model and multilayer perceptron (MLP). Secondly, CAM is unable to deal with any inputs other than the image. To overcome these shortcomings, for the first time propose a simple but effective tool-the the class activation value (CAV) to capture the learned representation information in a more general way, which is essentially the weighted output of the classifier. Experimental results in Figure 2 and Appendix B.7 also shows that our CAV works similarly as CAM during the training phase.
Building upon CAV, we propose a simple yet effective PLL method called CAV Learning (CAVL) that guides the model to differentiate the true label from the candidate set during the training pro- cess. Specifically, we first train the model by treating all candidate labels equally and obtain CAVs of given training examples, and then regard the class with the maximum CAV as the true label for each instance during the training process. In this way. CAVL transforms PLL into supervised learning, thereby making the model reliable to recognize the true label with learned representation informa- tion by CAV. Extensive experiments on benchmark-simun proposed CAVL method achieves statc-of-the thet-art performance.
and real-world datasets show that our
2
DISCOVERING CLASS ACTIVATION MAP FOR PARTIAL-LABEL LEARNING
In this section, we provide a detailed discussion about our motivation and empirically show that the class activation map could be helpful for addressing PLL.
2.1 CLASS ACTIVATION MAP
The class activation map (CAM) (Zhou et al., 2016) is a popular and elementary mechanism in computer vision to represent the discriminative part of an input image captured by the classifier to identify each class. In other words, CAM manifests the learning patterns of the classifier, denoted as f. to the specific class in the image. Let us denote an input image as x E dx (with c channels, height h and width w) and CAM of E Rd=oxhxw as Rkxhx where k is the
2

---

Published as a conference paper at ICLR 2022
0.5
Dynamic
0.5
Dynamic
0.4
Accuaracy
0.4
Accuaracy
Power
0.3
Power
0.2
Test Accuracy (IM)
Test Accuracy (IM)
0.1
Accuracy (CAM)
0.2
Test Accuracy (CAM)
Train Accuracy (CAM)
Train Accuracy (CAM)
0.0
0.10
2
6 Training Epoch
4
8
10
2
6 Training Epoch
4
8
10
(a) ResNet + USS
(b) DenseNet +
Methods (a) are implemented by ResNet with using USS generate the candidate set. Methods (b) are conducted by DenseNet with the candidate label sets produced by FPS. For three lines representing accuracy performance, the results drawn by the blue dashed line and red line ("CAM" line) are obtained by using the "CAM label" selection strategy, namely training the model with regarding the class with the most number of activated seeds in its CAM the true label. Specifically, the red line depicts the training accuracy measured by selecting
CAM with the maximum activated seeds from the candidate set blue line simply depicts the test accuracy of the classifier. The yellow dashed line ("IM" line) depicts the test accuracy of the classifier trained by IM. The paired circle indicates the similar accuracy fluctuation between the classifier and CAM, which shows the dynamic attribute. The double arrow marks the performance gap between "CAM" label selection strategy and IM, which represents the power r attribute.
true label,
Figure 1: Comparison of accuracy performance of two different label selection strategies.
number of classes. We note that could be generated by training specific classification model fer which normally comprises of an encoder function X + IR xhxw for feature map extraction, Global Average-Poolingg layer gap R xhxw -> R x and linear layer with weight R where of is the number of channels in the feature maps. Therefore, CAM of a related to the j-th class could be obtained by
m = ΣE1
C1 (2))) kk)
(I)
where 03 E RRIxC is the corresponding linear weight related to the j-th object. CAM is weighted linear sum of feature maps and the linear weights. The weakness of CAM is the limited applications, since not all classification network architectures follow the model mentioned above. To improve its generalization for any CNN-based model f. Grudient-weighted CAM (Grad-CAM) (Selvaraju al., 2017) was proposed implement such an internll representation by reasonably leveraging class-specific gradient information. Concretely, let us denote by g" ddd==xhxe the Grad-CAM of related the j-th object, and g¹ could be expressed
h
0f((x)
g¹
n=1 de (x=( e² (xx).
X W
Intuitively, the derivative of logit f(x) with respect to feature map e(x) is used as the weights for calculating g. Note that both Grad-CAM and CAM generated by weighted sums of the feature maps, and Grad-CAM is equivalent to CAM when f follows the same architecture as fe. Based on Grad-CAM, Grad-CAM++ (Chattopadhay et al., 2018) aims provide better visual explanations of CNN and occurrences multiple foreground objects. Thanks to such a simple technique, numer- ous problems in computer vision such as interpretation (Xu al., 2019a) and weakly supervised semantic segmentation (Wei et al., 2016) have achieved marvelous progress. In this paper, we aim extract and improve the useful knowledge from CAM to address the PLL problem, leading the classifier to identify the ground truth.
2.2 PILOT EXPERIMENT ON CAM
Generally, Grad-CAM or CAM treated as the internal representation of We believe that such mechanism could guide of to differentiate the true label from the candidate set because it is con- structed by taking advantage internal elements in f. To validate our conjecture, we conducted pilot experiment by adopting two different label selection methods. The first one is the intuitive
3

---

Published as a conference paper at ICLR 2022
method (IM), which simply regards all the labels from the candidate sets the true labels with using cross entropy loss. The second one is the "CAM label" selection strategy, which discriminates the true label from the candidate label set by using CAM. During each training epoch, we calculated the number of the foreground seeds of CAM from each candidate label set. Specifically, to calculate counted the elements with positive values in m Note
the number of foreground seeds of m that each æ could possess k CAMs and the true label is always in the candidate set, Thus we se- lected the label from the candidate set, whose CAM owns most foreeround seeds as the true label. The label candidate label sets were generated by two different approaches:(1) Uniformly Sampling Strategy (USS). Uniformly sampling the candidate label set for each training instance from all the possible candidate label sets (Feng et al., 2020). (II) Flipping Probability Strategy (FPS). By setting a flipping probability y to any false label, the false label could be selected as a candidate label with probability q (Feng & An, 2019a; Yan & Guo, 2020; Lv et al., 2020; Wen et al., 2021). Here the detailed training settings of the experiments are similar to the experiment part (please refer to Section 4.1.1 for details). ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) were chosen as the backbones to train a classifier on CIFAR-10 10 (Krizhevsky et al., 2009).
Figure 1 shows the average results of 5-time trials and we find that CAM demonstrates two helpful features during the training phase. Firstly, it is clear that the performance of the classifier trained by "CAM label" selection strategy is better than IM (the gap between the blue dashed line and the yellow dashed line), which shows that CAM could potentially guide the classifier toward the true label from the candidate set. Here we name this attribute of CAM as power. Secondly, it is found that the accuracy of identifying true labels by CAM is synchronously changed with that by the classifier. Several fluctuations (marked by the paired circle) of the classifier performance (test accuracy) and CAM accuracy are similar to large extent, resulting in continuous improve- ment to the model itself. Here we name this observed feature of CAM as dynamic. The power and dynamic attributes disclose the fact that CAM may show more than we thought, which is an inspiring finding to address PLL. The power attribute motivates us to consider that the classifier may learn more accurate true labels if it is forced to approximate the label distribution recognized by CAM. The dynamic attribute may guarantee that such guidance would be constantly effective during the whole training phase since CAM would synchronously be self-improved as the classifier becomes stronger. Hence it would be reasonable and meaningful to explore such internal sign as guidance for PLL.
3 PROPOSING CLASS ACTIVATION VALUE FOR PARTIAL-LABEL LEARNING
3.1 CLASS ACTIVATION VALUE-SIMPLE AND GENERAL REPRESENTATION
In this paper, we aim to propose a CAM-like tool as guidance for ff to identify the true label from can- didate labels. CAM or Grad-CAM is specially designed for convolutional neural network (CNN)- based models, which mainly deals with image datasets with retaining spatial information. However, to the best of our knowledge, most PLL methods are implemented in datasets with various forms, including the image datasets such as MNIST (LeCun et al., 1998), and feature vectors such as Bird- Song (Briggs al., 2012). The networks on these datasets could be directly adapted to elementary models such as the linear or multilayer perceptron (MIP) for f to deal with the inputting flattened- dimension vectors. Without spatial information or CNN-based architectures, CAM-based tools invalid to represent the internal learning pattern. To make such a concept versatile, we aim to pro- pose another novel CAM-like tool with simple implementation that could be adopted in a wider range of backbones and problems. We note that the (softmax) outputs can replace the feature map e(x:) since the encoder e could be directly regarded as f. Thus, it is fundamental to formulate an internal component for value-based based framework. For each candidate label, apart from its class out- put, its gradient of training loss with respect to the model output can also capture the importance degree of the candidate label to some extent. Inspired by the formulation of Grad-CAN in Eq. (2) that regards the gradient information flow to the last convolution layer as the importance of cach neuron to the j-th class, we can intuitively derive the importance of each candidate label as follows:
O(- log( (f(z)))) 3 (f(x)) 1 (f(x)) - 1/ v (f(x)),
(f)
where e) R² + R is the activation function imposed on the logit vector f(x) to approximate the class-condittonal probability P(y (z),, and sa is mostly implemented by the SoftMax function. The
4

---

Published as a conference paper at ICLR 2022
Algorithm CAVL Algorithm
Input: Model epoch maxx iteration Zmaxx partial labeled training set ID) {( S:))
for t 1,2,... ...,, max do
Shuffle 13;
for 1, max do
Sample mini-batch D2 = S1)lž= from D; t then
Update / by minimizing the cross entropy loss function R2 in Eq. (6) with treating all the candidate labels from S, equally;
else
Obtain the CAV for each 2; by Eq. (4);
Generate v/ for 2 by selecting the maximum v from S, by Eq. (7); Update f by minimizing the cross entropy loss function Reavt in Eq. (8) with treating v/} as the sole true label for each Zy;
end if end for end for Output: f.
derivation process of the last inequality of Eq. (3) is provided in Appendix A. From Eq. (3), we can observe that the importance e of each candidate label j depends on the softmax confidence score did (f(x)). However, we argue that such calculation of o is actually suboptimal, because yid (f())) conveys less information of model output than the logit f fixx)) as 1 (f())) could be considered as condensed version of 1 (xx). Therefore, instead of directly adopting Eq. (3), we propose to replace vid (a())) in Eq. (3) by f (ar), and the importance of each candidate label would be calculated by the following simple way:
v -1)
(4)
We call the importance o calculated by Eq. (4) as the class activation value (CAV) of the j-th class, as it could be regarded as value-based "CAM". Compared with CAM, we directly treat f fxx as the "feature map" so that can be obtained by weight imposed the class output of the model. In summary, both CAV and CAM are the weighted operation to the feature maps (or outputs in the classifier). Differently from CAM, CAV is essentially a value so that it could be easily obtained and applied to different backbones. Besides, our CAV could also be generated with less time complexity since there is no need to process the weighted sum of the feature maps.
3.2 PARTIAL-LABEL LEARNING WITH CLASS ACTIVATION VALUE
In this section, we introduce how to effectively address PLL by using CAV in detail. Firstly, we give a formal introduction to PLL. For a k-class classification problem, let X E Rd be the feature (input) space and (ar; 1 E X be the input data, where m refers to the number of training examples. Let Y = {1, .... k} be the label space. Let us denote C = {29) 0 Y as the candidate label space where 2 is the power set of Y, and |C| = 2 shows that the candidate label set not supposed to be the empty set nor the whole label set. For each training instance I;, let ys E Y be the ground truth label and S, E C be the candidate label set. We denote P(æ, y) and P(x,, S) as the probability densities of fully labeled examples and partially labeled examples respectively. Based on the crucial assumption of PLL that the candidate label set of each instance must include the correct label, we have y S,, ie.,
(5) PL. aims to learn a classifier f X -> R with training examples independently and identically sampled from P(x, S) to make correct predictions for test examples. To the best of our knowledge, most methods focus on designing the loss by considering all candidate labels or iteratively extracting the true label from the theoretical aspect. Differently from them, concentrate on how to solve PLL by efficiently utilizing the learned representation by CAV, which could be potential and effective guidance for helping ff recognize the true label.
P(y E S.ly = yi, x = mx) = 1, E Y.AS E C.
Note that the learned representation could be directly obtained by using CAV after firstly training f in one epoch. For training f in the first epoch, here directly utilize the IM strategy, i.e., treating
5

---

Published as a conference paper at ICLR 2022
every candidate label equally by cross entropy loss [(f(z), S. Hence the empirical risk function R1 for the first epoch could be defined as
E
jes; ((f(z.),
(6)
770
After training ʃ one epoch, the CAV v of every candidate label could be gained for each a by Eq. (4). To keep the classifier away from the negative effect by the false positive labels from the candidate set, we use the knowledge by CAV to guide the following learning process after the first training epoch, and the potential true label for æ; is selected by
y = arg max V/ jES,
(7)
where y is the selected true label from the candidate label for .;.- In this way, 3/ could be straightly treated as the ground truth so that PLL could be transferred to classical supervised learn- ing. Finally, the risk function Reart for the following epochs could be expressed as
(c, - - 1-1 F(S(x)))))) v()
(8)
As I gradually learns to approximate P[y' CAV would also be updated identify the true label. Based on such a progressive label selection on the candidate label set, our Class Activation Value Learning (CAVL) is supposed to progressively training an accurate model with correctly selected labels. The pseudo-code of CAVL is given in Algorithm 1.
4 EXPERIMENTS
In this section, extensive experiments on various datasets are implemented to verify the effectiveness and rightness of our proposed CAV and CAVL method.
4.1 CAVL PERFORMANCE
4.1.1 BENCHMARK DATASET COMPARISONS
Datasets and backbones. We use four popular benchmark datasets to test the performance of our CAVL, which are MNIST (LeCun al.. 1998), Fashion-MNIST (Xiao et al., 2017), Kuzushiji- MNIST (Clanuwat et al., 2018) and CIFAR-10 (Krizhevsky et al., 2009). Note that it is necessary to manually generate the candidate label sets since they are supposed to be used for single-classffictionn problems. Recall that we introduce two different candidate label generation process (please refer to Section 2.2 for details) (Feng& & An, 2019a;b; Feng et al., 2020; Lv et al., 2020; Wen et al., 2021), ie., USS and FPS. The former one is implemented by uniformly sampling label set from all the partial label space C for each instance, and the latter one sets a flipping probability to any irrelevant label so that it could become we set E {0.3, O.5, 0.7] to represent different ambiguity degrees. For MNIST, Kuzushij-MNIIST and Fashion-MNIST, we adopt 3-layer MLP and 5-layer LeNet the backbones. For CIFAR-10, we choose 34-layer ResNet (He et al., 2016) and 22 22-layer DenseNet (Huang et al., 2017) as the base models.
item in the candidate label set with probability q. In this section
Compared methods and training settings. We compare our CAVL with four state-of-the-art approaches including PRODEN (Lv et al., 2020), Leveraged Weighted (LW) Loss (Wen et al., 2021). RC and CC (Feng et al., 2020). For all methods we search the initial learning rate from (0.0001, 0.001, 0.01, 0.05, 0.5). and weight decay from {10-6 10 ..., 10 10-'). We take a mini-batch size of (32, 256} images and train all the methods using Adam (Kingma & Ba, 2015) optimizer for 250 epochs. Hyper-parameters are searched to maximize the accuracy on a validation set containing 10% of the training samples annotated by true labels. All the implemented methods are trained on RTX3090 GPU with 24 GB memory. To guarantee the fairness of comparisons, repeatedly conduct all experiments 5 times and record the average accuracy with the standard deviation following the conventions in Feng et al. (2020); Lv et al. (2020); Wen et al. (2021). This work is partially supported by Huawei MindSpore (Huawei, 2020).
Experiment results. As shown in Table and 2, our CAVL its superiority on four benchmark datasets with various backbones. Additionally, our CAVL. outperforms nearly all faate-of-the-an

---

Published as a conference paper at ICLR 2022
Table Test performance of CAVL and other methods on benchmark datasets using data generation by USS. The best results among all methods with the same backbone are marked in bold.
Dataset (Backhones)
Method
Accuracy
CC
97.77 = 0.12% / 98. 0.06%
RC
98.05
0.15% /99.03 1 0.04%
MNIST (MLP/L.Nee)
LWW
97.99 0.06% /98.74 0.05%
PRODEN
97.02 0.08% /98881 0.04%
CAVL
98.06 0.05%
99.14 0.03%
CC
88.87 0.32% /93.83 0.20%
RC
89,36 0.30% / 94.01 1 0.15% %
Kuzushiji-MNIST (MLP/LeNct)
LW
87.98
0.39% 91.52 = 0.65%
PRODEN
88.75 0.29% / 93.94 = 0.18%
CAVL
88.45 0.22% / 93.25
0.21%
CC
87.90 0.27% / 88.96 0.14%
RC
88.40 0.13% / 89.51 0.11%
Fashion-MNIST (MLP/LeNct)
LW
88.16 0.12% / 88.28 = 0.33%
PRODEN
88.82 0.15%
89.23 1 0.12%
CAVI
88.93 0.16% / 89.99 0.10%
CC
75.74 0.19%
76.78 0.33%
RC
77.98 0.46%
788.56
0.37%
CIFAR-10 (ResNee/DenseNet)
LW
76.82
0.21% / 78.08 0.66%
PRODEN
77.62
0.34%
78.72 0.48%
CAVI
78.05 0.32% 79.10 0.25%
Table 2: Test performance of CAVL and other methods on benchmark datasets using data generation by FPS. The best results among all methods with the same backbone are marked in bold.
Dataset (Backbones)
Method
Q00.3
q=0.5
q=0.7
CC
98,87 : 0.15%
98.49:1 0.07%
98,17 : 0.12%
RC
98.88 + 0.07%
98.531 0.11%
98.12 + 0,05%
MNIST (LeNet)
I.W
98.53 + 0.12%
98.68+ 0.06%
97.35 + 0.13%
PRODEN
98.72 + 0.13%
98.62+ 0.09%
98.08 + 0.04%
CAVL
98.90 + 0.12%
98.71 0.04%
98.20 + 0.11 %
CC
93.11 + 0.08%
90.87 = 0.06%
89.98 +: 0.14%
RC
93.21 + 0.17%
91.19++ 0.22%
90.15. 0.04%
Kuzushiji-MNIST (LeNet)
LW
92.65 + 0.18%
91.28-t 1.1%%
90.55 + 0.17%
PRODEN
93.51 + 0.20%
91.23.1 0.17%
90.07 1 0.08%
CAVL
93.82 + 0.21 %
91.57:1 0.11%
86.05:1 0.25%
CC
89.41 + 0.17%
88.72 + 0.066
85.87 + 0.25%
RC
89.53 - 0.07%
88.8411 0.14%
85.41 1 0.18%
Fushion-MNIST (LeNet)
LW
89.19 - 0.08%
87.19.1 0.23%
85.92 1 0.13%
PRODEN
89.63 + 0.05%
88.78:t 0.15%
85.81 + 0.16%
CAVL
89.77 + 0.04%
88.92.1: 0.11%
86,25 1 0.18%
CC
77.32 1 0.14%
76.42 = 0.13%
66.17 1 0.25%
RC
78.14 + 0.12%
77.42 0.16%
70.21 + 0.15%
CIFAR-10 (DenseNet)
LW
80,95 + 0.17%
78.72 = 0.17%
71.26 + 0.16%
PRODEN
79.05 + 0.11%
77.521 0.18%
70.35 + 0.18%
CAVL
81,58 + 0.22%
79.69
0.17%
65.86 + 0.21%
methods using the candidate labels generated by both USS and FPS, which shows that our CAVL does not rely on any data generation assumption. Specifically, it worth mentioning that CAVL is able to show competitive performance with RC, which is theoretically proved to be possess optimal performance in data distribution generated by USS. Therefore, the results reasonably verify the generalization and effectiveness of our CAV and CAVI.
4.1.2 REAL-WORLD DATASET COMPARISONS
Datasets and backbones. We select five real-world datasets including Lost (Cour et al., 201 1), MSRCv2 (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng et al., 2013) and Yahoo (Guillaumin et al., 2010). Note that these real-world datasets have their candidate labels. As common practice in (Feng al., 2020; Ly et al., 2020; Wen al., 2021), we select

---

Published as a conference paper at ICLR 2022
Table 3: Test performance of CAVL and other methods uisng linear model on real-world datasets. The best and second best results among all methods are marked in bold and underline.
Real-world datasets
Method
Lost
MSRCv2
Birdsong
SoccerPlayer
YahooNews
IPAL
71.16 + 2.56%
50.64 3.85%
70.32 + 4.85%
55.42 + 0.92%
66.43 1.32%
CLPL
75.01 + 4.39%
36.72 + 4.61%
64.35 + 1.28%
37.01 + 1.02%
45.21 0.82%
PLSVM
48.91+ 3.33%
35.95 + 3.96%
48.99 + 1.98%
45.90 + 0.98%
57.02 1.02%
PLKNN
37.7341 2.85%
41.28 +: 2.25%
64.32 = 1.05%
48.21 = 1.21%
40.67 + 1.58%
PLECOC
50.95 + 7.81%
43.25 = 3.61%
70.51 = 4.31%
55.29 = 1.95%
61.23 + 1.52%
SURE
75.56 # 5.62%
46.72 + 4.21%
55.42 = 1.52%
48.61 = 0.84%
55.17 + 1.05%
LW
79.74 + 3.81%
45.65 + 3.12%
72.01 = 2.31%
57.12 = 3.25%
67.94 + 1.64%
RC
80.45 + 3.85%
46.85 + 2.52%
72.15 + 2.14%
$7.05 + 2.15%
68. + 0.67%
CC
73.78 + 4.34%
45.13 + 2.67%
71.86 = 1.34%
56,54 + 0.74%
67.00 :: 0.35%
PRODEN
80.12 4.52%
45.32 + 2.38%
71.90 = 2.34%
56.12 + 3.12%
67.92 0.48%
CAVL
82.07 + 3.18%
48,39 +: 2.12%
71.92 0.73%
57.32 + 2.24%
68.89 0.55%
Train accuracy (CAV)
Train accuracy (CAM)
Test accuracy (CAVL)
MNIST-Linear
MNISI-MUI
KMNESI -Linear
KMNIST-MLP
0.95
0.55
1.00
0.94
0.95
bocuarary
0.00
9,000
0.85
Acouaracy
0.00
0.80
0.00
0.00
0.80
Training Epoch
000
Training Epoch
100
250
0.75
Training Epoch
100
100
200
250
0.7%
Traning Epoch
100
150
FMNIST-
FMNIST-MLP
CIFAR10 ResNet
Restet
DeARi0-DenseNet
0.08
6.88
0.80
Accasnay
0.90
0:00
0.94
0.650
0.50
0:00
0.00
0.75
0.80
0.75
0.70
0.84
0,52
0.00
0.7g
50
100
150
250
0.00g
0.700
0.60
Training Epoch
Training Epoch
Training Epoch
Figure/ 2: Study of CAV attributes various backbones and datasets. Note that here CAM could
only been obtained in deep CNN architectures, namely ResNet and DenseNet.
1-layer linear model and 3-layer MLP as the backbones to validate the performance of our CAVL in these five datasets.
Compared methods and training settings. Apart from the four methods mentioned in Sec- tion 4.1.1, I. we add SURE (Feng An, 2019b), CLPL (Cour al., 2011), IPAL (Zhang Yu, 2015), PLSVM (Elkan & Noto, 2008), PLECOC (Zhang et al., 2017) and PLKNN (Hüllermeier & Beringer, 2006) as extra methods for comparison, where the parameters are searched maximize the accuracy a validation set (10% of the training set). We repeatedly run all experiments 10 times, and report the average accuracy with the standard deviation for each method. Other settings similar to Section 1.1. The comparison results using MLP is recorded in Appendix B.2.
Experiment results. In Table 3, our proposed CAVL shows the best performance in nearly all five real datasets. Besides, CAM is unable to be extracted with the word embedding input such as BirdSong, lacking spatial information. Therefore, the achievement of CAVL on these real datasets shows the generalization and superiority of our CAVL to shallow model, where CAV could serve as the CAM to lead the classifier to learn the true label.
4.2 CAV ATTRIBUTES
In this part, we conduct several experiments to present how these attributes of CAV guide the learn- ing process. We implement CAVL with four backbones including linear, MLP, ResNet and DenseNet on four benchmark datasets, whose candidate label sets are generated by USS. The training settings similar to the ones in Section 1. Figure 2 illustrates the averaging performance of CAV, CAM (for deep networks) and the model itself during the 5-time training process. The red (blue) line in Figure 2 depicts the training accuracy measured by treating the maximum CAV (CAM) from the
8

---

Published as a conference paper at ICLR 2022
candidate set the true label, and the eyan line depicts the test accuracy of the model. We note that here CAM is simply extracted from the classifier for comparison, which is different from its usage in Section 2.2. As shown in Figure it is clear that the classifier learns to approximate the better data distribution by CAV, and the accuracy trend recognized by CAV is consistent with the classifier performance in the test datasets, which validates the Power and Dynami attribute of CAV.
RELATED WORK
In this section, we give brief introduction to the two mainstream strategies for partial-label learn- ing (PLL), i.e., the averaged- strategy (ABS) and the idemtification-bssed strategy (IBS).
ABS treats all candidate labels in an equal manner and then averages the model outputs of all can- didate labels for prediction. Some non-parametric methods (Hüllermeier & Beringer, 2006; Gong al.. 2017) concentrated on predicting the label by leveraging the outputs of its neighbors. Fur- thermore, some methods (Cour et al., 2009; Zhang et al., 2016; Yao et al., 2020) aimed to subtly leverage the labels outside the candidate set so as to discriminate the potential true label. Yao et al. (2020) designed an entropy- -ased regularizer to minimize the entropy of each label to maximize the margin between the potential true label and the unlikely labels. A recent study (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021; Lv et al., 2021) focused on the data generation process and proposed a classifeer-onnsisentt method based on a transition matrix. (Chen et al., 2020) proposed a new problem General Partial Label Learning and leveraged graph neural network to tackle the label ambiguity.
IBS aims to constantly identify the most possible true label from the candidate label set to eliminate the label ambiguity. Early works treated the potential truth label as a latent variable and optimized the objective function by the maximum likelihood criterion (Jin & Ghahramani, 2002; Liu & Di- etterich, 2014) or the maximum margin criterion (Nguyen & Caruana, 2008; Yu & Zhang, 2016). Later, taking advantage of topological information of feature space generate the score of each candidate label (Zhang & Yu, 2015; Zhang et al., 2016; Feng & An, 2018; Wang et al., 2019) at- tracted much attention from researchers. These methods commonly aimed to iteratively update the confidence of each candidate label based on the widely used assumption that similar instance are supposed to possess the same label. Xu et al. (2019b) proposed to model the generalized label distribution by leveraging the topological information of the feature space. Lyu et al. (2021) refor- mulated PLL into a matching selection problem and proposed Learning framework to solve the problem. It is worth noting that IBS is commonly susceptible to the false positive labels that co-occur with the true label in the candidate label set. Thus our CAVL, which belongs to IBS, also owns this inevitable shortcoming.
Graph-Matching based Partial Label
6 CONCLUSION
In this paper, we exploited the learned intrinsic representation of the model for partial-label learn- ing (PLL). We made two key contributions. Firstly, we found that the class activation map (CAM), simple technique for discriminating the learning patterns of each class in image, could sur- prisingly be used to differentiate the true label from candidate labels. Unfortunately, we are yet unable to directly use CAM for PLL, as CAM is subject image inputs with convolutional neural networks. Thus, our second contribution is to propose the class activation value (CAV), which not only owns similar properties of CAM but also is versatile in various types of inputs and models. Based on CAV, we proposed a simple yet effective PLL method that selects the true label by the class with the maximum CAV for model training. Comprehensive experimental results on various datasets demonstrated that our CAVL achieved tht performance.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
FZ and BH were supported by NSFC Young Scientists Fund No. 62006202, RGC Early Career Scheme No. 22200720 and HKBU CSD Departmental Incentive Grant. BH was also supported by MSRA StarTrack Program. LF was supported by the National Natural Science Foundation of China (Grant No. 62106028) and CAAl-Huawei MindSpore Open Fund. TLL was supported by Australian Research Council Projects DE- DE-199011173 and DP-220102121. MS was supported by JST CREST Grant Number JPMJCR18A2
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, el. A closer look at memorization in deep networks. In International Conference on Machine Learning (ICML), pp. 233-242. PMLR, 2017.
Forrest Briggs, Xiaoli Fern, and Raviv Raich. Rank-loss support instance machines for miml instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference Knowledge Discovery and Data Mining (KDD), pp. 3344-52,225 2012.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-
cam++: Generalized gradient- based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018.
Brian Y. Chen, Bo Wu, Alireza Zareian, Hanwang Zhang, and Shih-Fu Chang, General partial label learning via dual bipartite graph autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:/8/2.01718,, 2018.
Timothee Cour, Benjamin Sapp, Chris Jordan, and Ben Taskar. Learning from ambiguously labeled images. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 919-926. IEEE, 2009.
Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research (JMLR), 12:1501-1536, 2011.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference Knowledge Discovery and Data Mining (KDD), pp. 213-220, 2008
Lei Feng and Bo An. Leveraging latent label distributions for partial label learning. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 2107-2113, 2018
Lei Feng and Bo An. Partial label learning by semantic difference maximization. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 2294-2300, 2019a.
Lei Feng and Bo An. Partial label learning with self gelddd retraining. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 3542-3549 2019b.
Lei Feng. Jiaqi Lv. Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial-label learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang. Jie Yang, and Dacheng Tao. A regularization approach for instance-based superset label learning. IEEE Transactions on Cybernetics, 48(3): 967-978, 2017.
Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multiple instance metric learning from automatically labeled bags of faces. European Conference on Computer Vision (ECCV), pp. 634-647, Springer, 2010.
10

---

Published as a conference paper at ICLR 2022
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning (ICML), pp. 4006 4016. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4700, 2017.
Huawei. Mindspore, 2020. URL https:////// / /www /www.miiddpoor....eem./////////e cn/.
Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intelligent Data Analysis, 10(5):419 439, 2006.
Rong Jin and Zoubin Ghahramani. Learning with multiple labels. Advances in Neural Information Processing Systems (NeurIPS), volume 2. pp. 897-904. Citeseer, 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, (1)2)22-223222 1998.
Liping Liu and Thomas Dietterich. Learnability of the superset label learning problem. Interna- tional Conference on Machine Learning (ICML), pp. 1629-1637. PMLR, 2014.
Liping Liu and Thomas G Dietterich. A conditional multinomial mixture model for superset la- bel learning. Advances Neural Information Processing Systems (NeurfPS), pp. 548-556. Citeseer, 2012.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 38(3):447-461, 2015.
Jie Luo and Francesco Orabona. Learning from candidate labeling sets. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1896-8905 2018
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identifi- cation of true labels for partial-label learning. In International Conference on Machine Learning (ICML), pp. 6500-651 6510. PMLR, 2020.
Jiaqi Lv, Lei Feng, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robust- ness of average losses for partial-label learning. arXiv preprint arXiv:2106.06152 2021.
Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. Gm-pll: Graph matching based partial label learning. IEEE Transactions on Knowledge and Data Engineering (TKDE), 33(2):521-535, 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-super learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 187999999 2018
Nam Nguyen and Rich Caruana. Classification with partial labels. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD), pp- 551-559, 2008.
11

---

Published as a conference paper at ICLR 2022
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-base local- ization. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pp- 618-626, 2017.
Deng-Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining (KDD), pp. 83-91, 2019.
Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Jiashi Feng, Yao semantic
Zhao, and Shuicheng Yan. Stc: A simple complex framework for segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 39(11); 2314-2320, 2016.
Hongwei Wen, Jingyi Cui, Hanyuan Hang. Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pp. 11091-11100
PMLR, 2021.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems (NeurIPS), 33:7597-7610, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (tt:::/708007777 2017.
Kaidi Xu, Sijia Liu, Pu Zhao, Pu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus, Yanzhi Wang, and Xue Lin. Structured adversarial attack: Towards general implementation and better interpretability. In International Conference on Learning Representations (ICLR), 2019a.
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 5557-5564, 2019b.
Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In Proceedings of the AAAI Conference Artificial Intelligence (AAAT), volume 34, pp. 6575-6582, 2020.
Yan Yan and Yuhong Guo. Multi-level generative models for partial label learning with non-random label noise. In Proceedings of the International Joint Conference on Artificial Intelligence (IJ-
CAI), pp. 3264 1270, 2021.
Yao Yao. Jichui Deng, Xiuhua Chen, Chen Gong, Jianxin Wu, and Jian Yang. Deep discriminative cnn with temporal ensembling for ambiguousiy-labete image classification. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), volume 34, pp. 12669-12676, 2020.
Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In Asian Conference on Machine Learning (ACML), pp. 96-111. PMLR, 2016.
Zinan Zeng, Shijie Xiao, Kui Jia, Tsung-Han Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 708-715, 2013.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance- based ap- proach. In fwenty-fourth International Joint Conference on Artificial Intelligence (IJCAI). 2015.
Min-Ling Zhang, Bin-Bin Zhou, and Xu- Yu-Ying Liu. Partial label learning via feature-aware disam- biguation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1335-1344, 2016.
Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-fre partial label learning. IEEE Transactions Knowledge and Data Engineering (TKDE), 2910:22155-211 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929 2016.
12

---

Published as a conference paper at ICLR 2022
Zhi Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1): 44-53, 08 2017.
Zhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, and Yu-Feng Li. learning. Artificial Intelligence, 176(1):2291-233 2012.
multi-label
Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures Artificial Intelligence and Machine Learning, 3(1):1-130, 2009.
A DERIVATION PROCESS OF EQUATION. (3)
The SoftMax function is vج (z) exp(x,)/
exp(z,). where æ E refers the. Then Σÿ= /
can obtain the derivation of - log((b> with respect to the xy by
a( (1-log(//)) (x))) 0( log( and (x))) (x)
0z,
Orly (x)
Ōr,
(x)
(x)) dzy
Σ e (E21)2
(x))
(Ez)(
(9)
e²
t
)1
(a)
Ei (e=1)2
Σ (et))2
(ix) (1) 2 (x:)
(x)
B DETAILED SUPPLEMENTARY FOR EXPERIMENTS
B. BENCHMARK DATASETS
In Section 4.1.1, four widely-used benchmark datasets, i.c. MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017). Kuzushij-MMNIST (Clanuwat et al., 2018) and CIFAR-10 (Krizhevsky al., 2009). Table 4 lists the characteristics of these datasets. We respectively de- scribe these datasets as follows.
MNIST: It is a 10-class dataset of handwritten digits. Each data is a 28 x 28 grayscale image.
Rashion-MNISTT It is also 10-class dataset. Each instance fashion item from one of the 10 classes, which are T-shirt/top, trouser, pullover, dress, sandal, coat, shirt, sneaker, bag, and ankle boot. Moreover, each image a 28 28 grayscale image.
Kuzushiji-MNIST. Each instance a 28 28 grayscale image associated with one label of 10-class cursive Japanese ("Kuzushiji") characters.
32 colored image in RGB format. It is ten-class dataset of objects including airplane, bird, automobile, cat, deer, frog, dog, horse, ship, and truck.
CIFAR-IO: Each instance
B.2 REAL DATASETS
In Section 4.1.2, We select five real-world datasets including Lost (Cour et al., 2011), MSRCv2 (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng al., 2013) and Yews (Guillaumin et al., 2010). Here we make a comprehensive descriptions about them shown follows.
13

---

Published as a conference paper at ICLR 2022
Lost, Soccer Player and Yahoo! News: They crop faces in images or video frames in- stances, and the names appearing on the corresponding captions or subtitles are considered candidate labels.
MSRCv2: Each image segment is treated as sample, and objects appearing in the same image are regarded as candidate labels.
BirdSong:
The singing syllables of birds are regarded instances and bird species who are jointly singing during any ten seconds are represented as candidate labels
Table 4: Characteristics of benchmark datasets
Datasets
#Train
#Test
#Features
#Classes
MNIST
60,000
10,000
784
10
Fashion-MNee MNIIT 60,000
10,000
784
10
Kuzushiji-MNN MNIST
60,000
10,000
784
10
CIFAR-10 50,000
10,000
3072
10
Table 5: Characteristics of weal-world datasets
Datasets
Application Domain
#Examples
#Features
#Classes
Avg #CLs
Lost
Automatic face naming
1,122
108
16
2.23
MSRCv2
Object classification
1,758
48
23
3.16
BirdSong
Bird song classification
4,998
38
13
2.18
Soccer Player
Automatic face naming
17,472
279
171
2.09
Yahoo! News
Automatic face naming
22,991
163
219
1.91
Table 6: Test performance of the CAVL and other methods uisng MLP real-world datasets. The best and second best results among all methods are marked in bold and underline.
Real- world datasets
Method
Lost
MSRCv2
Birdsong
SuccerPlayer
YahooNews
LW
67.53 + 3.21%
50,541 3.23%
71.74 1.35%
53.29 + 1.95%
65. 0.92%
RC
75.89 = 5.10%
51.13 + 2.67%
10.14- + 1.54%
53.98 + 0.85%
67.56++ 2.53%
CC
70.10 + 5.20%
50.57 + 3.47%
70.51 + 1.41%
54.87 = 0.25 %
67.75:1 1.12% %
CAVL
71.82 3.24%
51.55 + 3.31%
69.25 1.44%
54.83
0.74%
65.32+ 1.04%
Table 5 shows the average number of candidate labels (Avg. # CLs) per instance. Table 6 presents the performance comparison between other methods and our CAVL. method based on MI.P model, the results of which also validate the effectiveness our CAVL.
B.3 ABLATION STUDIES ON THE STARTING EPOCH IN CAVI
In Section 3.2, we propose the CAVL method to address PLL. Specifically, we use the IM to train the classifier in the first epoch, and then the potential true labels could be obtained with CAV after the following training epochs. Here we set the epoch number for starting using the CAVL method, i.c. the starting epoch, one. To further validate the effectiveness of our CAVL, we explore the effects of the starting epoch on our method. Specifically, we implement CAVL with two backbones including LeNet and ResNet on four benchmark datasets. We repeatedly run the experiments 5 times by using candidate labels generated by USS. Following similar training settings in Section 4.1.1, we set the total training epoch as 250, and select the starting epoch from [10, 50, 100, 150). IM used before the selected starting epoch and CAVI. is implemented after the selected epoch.
As shown in Table the accuracy of the classifier becomes lower as the increase of the starting epoch in CAVL. Let us take the results in KMINIST as an example, the CAVL shows its superiority with setting the starting epoch as achieving 93.25 + 0.21% accuracy performance. As the starting epoch increases, the classifier is more negatively affected by the false positive labels in the candidate sets and dragged away from the true labels by IM. Thus it only achieves 88.54 + 0.26% with the starting epoch as 150. This phenomenon is reasonable because the DNN learn patterns first, which suggests that deep networks can gradually memorize the data. moving from regular data to irregular data such outliers and mislabeled data (Arpit al., 2017). Therefore, our CAVL could not
14

---

Published as a conference paper at ICLR 2022
Table The performance of our CAVL with different settings of the starting epoch.
Dataset (Backbones)
Starting Epoch
MNIST (LeNet)
KMNIST (LeNet)
FMNIST (LeNet)
CIFAR-I0 10 (ResNet)
99.14 0.03%
93.25 0.21
89.99 + 0.10%
78.05 + 0.32% %
10
97.82 0.04%
92.84 0.21%
88.75 + 0.14%
75.27 1 0.42%
50
97.25 + 0.03%
91.23 + 0.25%
87.98 +: 0.15%
72.17 = 0.31%
100
96.44 0.07%
89.66
0.42%
86.92 0.16%
71.55 +: 0.18%
150
96.38 0.07%
88.54 0.26%
86.07 + 0.34%
69.87 = 0.12%
sufficienlly help the classifier to deal with this memorization effect during the rest of the training epochs. In conclusion, the effectiveness of our CAVL would be weakened extending the starting epoch.
B.4 GENERATION CANDIDATE LABELS
In Section 4.1.1 we introduce two different generation ways for the candidate label sets, i.e. USS, Uniformly sampling label set from all the partial label space for each instance. FPS, Setting flipping probability q to any irrelevant label which could possibly become one item in the candidate label set with probability
For USS, each partially labeled example (æ, S) is independently drawn from probability distribu- tion with the following density:
P(x, - == (( = = -
2--1-1 i
(10)
i S.
The generation process assumes that the candidate label set S is independent of the instance æ. There are totally 2 possible candidate label sets that contain the specific true label y. Therefore, Eq. (10) illustrates the candidate label set for each instance is uniformly sampled.
For FPS, we set flipping probability g to any irrelevant label that possibly entries the candidate label set. Here we introduce the class transition matrix (denoted by T) for partially labeled data, where Tij refers to the probability of the label j being a candidate label given the true label i for each instance. Note that T, = 1 always holds since the true label always belongs to the candidate label. Tiş = q, f j.j holds for other elements. The matrix representation of T expressed as:
1
9
4
q
q
9
9
q
9
q
9
4
9
9
q
q
q
q
q
q
1
q
4
q
9
q
1
9
q
9
9
q
9
4
Q
9
9
9
4
9
4
B.5 COMPARED METHONS
The compared PLL methods are listed as follows.
SURE (Feng & An, 2019b): It iteratively enlarges the confidence of the candidate label with the highest probability to be the correct label.
CLPL (Cour et al., 201 I): It uses a convex formulation by using the one- vereuveess-lal all strategy in the multi class loss function.
15

---

Published as a conference paper at ICLR 2022
IPAL (Zhang & Yu, 2015): is non non-parametric method that applies the label propagation strategy to iteratively update the confidence of each candidate label.
a maximum margin-based method that differentiates candidate labels from non-candidate labels by maximizing the margin between them.
PLSVM (Elkan Noto, 2008):
PLECOC (Zhang et al., 2017): It adapts the Error-Crrretinn Output Codes method to deal with partially labeled examples in a disambiguation -fee manner.
PLKNN (Hüllermeier & Beringer, 2006): It adapts the widely- k-nearest neighbors method to make predictions for partially labeled examples.
CC RC (Feng et al., 2020): The former method a novel risk-consistent partial label based on the generation model. LW (Wen et al., 2021): The method proposes family of loss function for the first time, where introduces the leverage parameter 33 to consider the trade-off between losses on partial labels and non-partial labels.
learning method and the latter one is
For all the above methods, their parameters are specified or searched according to the suggested parameter settings by respective papers maximize the accuracy performance on validation set, which is made by the 10% of the training set.
B.6 DETAILS OF THE BACKBONES
In Section 4 we select five network architectures as the backbones for modelling our classifier, which are linear, 3-layer MLP, 5-layer LeNet, 34-layer ResNet (He et al., 2016) and 22-layer DenseNet (Huang et al., 2017). The linear model is a linear-in-input model:d k. MLP refers to 3-layer fully connected networks with ReLU the activation function, and the architecture is 500 k. LeNet is comprised of Convolution Layer and fully connected layer. 34-layer ResNet and 22-layer DenseNet strictly follow the implementation by He al. (2016) and Huang et al. (2017).
B.7 PILOT EXPERIMENTS IN
DATATSETS
In Section 2.2 we present the pilot experiments on CIFARIO, which shows that CAM possess two attributes, i.e., power and dynamic. Here provide more results five real-world datasets to prove the rightness of our proposed CAV. The real-world datasets already have their candidate labels. Following similar training settings in Section 2.2, we select 1-layer linear as the backbone for the classifier and respectively use the IM and CAV to train in 10 epochs. Intuitively, using CAV to train the model equal to CAVL. Note that CAM could not be obtained in the linear module. Figure 3 shows the average results of 10-time trials and is easily found that CAV also shows power and dynamic attributes in real-world datasets, validating the generalization and effectiveness of our CAV.
C FURTHER EXPLANATIONS ON CLASS ACTIVATION MAP
Here we provide a detailed explanation of CAM. For a well-trained CNN module for addressing the classification problem, the Class Activation Map (CAM) is simply weighted linear sum of the presence of the visual patterns at different spatial locations for the input images. CAM aims to mine out what the network focuses on the input images related true labels during the training phase. known that CNN-based module is a "black box" and the meaning behind CAM is the exploration of the inner principles inside such technology. Based on CAM, it is convenient to visualize the learning patterns of the CNN-based modules. Figure 4 shows several CAM results selected from (Zhou et al., 2016). Here we take one example of the Teapot for illustration, the CAM of the image related the teapot shows the most discriminative regions in the head part, which manifests that the classification classifies the image to the teapot since finds and concentrates on the head part of the teapot. Thanks to CAM, could conveniently and effectively investigate the inner principles of CNN-based modules by simply utilizing the components from the network itself. This is also the reason why we name it "internal representation", which is essentially form of intermediate outputs.
16

---

Published as a conference paper at ICLR 2022
Train accuracy (CAM)
accuracy (CNV
Test accuracy (IM4)
Dynamic
Power
Lost-Linear
MSRCv2-Linear
0.8
0.8
0.7
0.6
0.6
Accuarany
Accuatai
0.5
0.4
0.4
0.3
0.3
0.1
0.1
0.0g
0.0
Training Epoch
Training Epoch
Soccer Player-Linear
BirdSong-Linear
0.8
0.7
0.6
0.6
Accuaracy
Acouaracy
0.5
0.4
0.3
0.3
0.2
02
0.1
0.0g
0.0
2
4
10
Training Epoch
Training Epoch
Vahoo/News-Lue
0.7
0.6
Accueracy
0.5
0.4
0.3
0.2
0.1
0.0
Training Epoch
Figure 3: Comparison of accuracy performance on five real-world datatsets obtained by IM and CAVL. Note that the training method is IM.
Mushroom
Penguin
Teapot
Figure 4: Some CAM samples selected from (Zhou et al., 2016).
17


Reviews:
[{"review_id": "zu3x2tEuNCA", "reviewer": "Reviewer_sUUe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThis paper introduces the class activation map (CAM) for handling identification in PLL.\n2.\tThis paper proposes the class activation value (CAV) for identification via capturing the learned representation information in a more general way instead of CAM as CAM is confined for image datasets and CNN model.\n\nWeakness:\n1.\tThe authors claim that “We believe that such a mechanism (Grad-CAM or CAM) could guide f to differentiate the true label from the candidate set because it is constructed by taking advantage of internal elements in f” and conducted a pilot experiment to check whether CAM with the most foreground seeds belongs to the true label. However, the pilot experiment cannot empirically validate this claim, as the experiment is only conducted on the CIFAR-10 with artificial candidate labels generated by two simple strategies but not conducted on real-world PLL datasets or adopts some realistic candidate labels generation strategies such as class-dependent strategy. \n2.\tThe proposed method CAVL adopts the strategy that selects the “true” label from the candidate label set and only considers the loss on the “true” label after the first training epoch. However, the selected “true” label may be false-positive and the authors also claim that the potential true label would be updated. In this case, the training model could be affected by false-positive labels since CAVL only considers the “true” label.  [1] proves that models trained by this strategy cannot achieve good performance. \n3.     There is no theoretical result provided to validate the effectiveness of CAV for PLL.\n4.\tThe authors adopt the PLL experimental settings in [1], but they do not compare the method in [1].\n5.\tThe authors claim that the hyper-parameters are selected to exploit the best performance on the validation set containing 10% of the training examples. They should show the train/ validation/ test split. RC, CC and CAVL are suggested to have the same learning rate, weight decay and mini-batch size as they adopt the same model for fair comparisons.\n\n\nSuggestion:\nThe authors are suggested to provide source code as supplementary material so that the reviewers could check the reproducibility of the experimental results.\n\n[1] Progressive Identification of True Labels for Partial-Label Learning, ICML 2020.", "summary_of_the_paper": "This paper focuses on the problem of partial-label learning (PLL), where each training instance is assigned a set of candidate labels that include the true label.  This paper shows that class activation map (CAM) could identify the true label from candidate labels for PLL. In addition, the authors propose the class activation value (CAV) for identification via capturing the learned representation information in a more general way instead of CAM as CAM is confined for image datasets and CNN model. CAV Learning that selects the true label by the class with the maximum CAV for model training is proposed. Experiments on various datasets demonstrate that the proposed method achieves state-of-the-art performance.", "summary_of_the_review": "This paper introduces the class activation map (CAM) for handling identification in PLL. But the effectiveness of CAM for PLL is not validated. Several of the paper’s claims are incorrect or not well-supported. The contributions are only marginally novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rkhZnvP4az-", "reviewer": "Reviewer_1wbR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It is innovative to use CAM to select true labels from candidates. The paper provides a new perspective to transfer PLL problems to supervised learning, which is worth deeper research in future work.\n2. The proposed CAVL method is effective and efficient.\n3. The experiments are sufficient, including various types of data and backbone. The ablation study on partial label generation methods strongly supports the intuition that CAM does not rely on any assumption on the collected data as previous work does.\n\n\nweaknesses：\n1. The main contribution is to utilize CAM to solve PLL problems and make some modifications on CAM to adapt to more data types and backbone. The original contributions do not seem to be enough and the novelty is a little weak.\n2. The explanation of the power and dynamic attributes is not clear. Referring to section 2.2, how does these two attributes show that CAM can act as the guidance for PLL problems.\n3. In the CAVL method, referring to section 3.2, the potential true label is selected based on CAV after training the model for only one epoch instead of several epochs. How do you decide when to select the potential true label?  What if the wrong labels were chosen, I did not see experiments or explanations in this case.\n4. The paper needs to also provide the results and notice that realistic dataset instead of using generated visual partial labels only.  \ne.g. the datasets and references should be considered in the related work survey or experiments:  \nPartial Label Learning via Label Enhancement AAAI 19  \nGeneral Partial Label Learning via Dual Bipartite Graph Autoencoder AAAI 20  \nGM-PLL: Graph Matching based Partial Label Learning TKDE 21  \n\nquestions:\nThe CAV is similar to adding attention values, what are the most important novelty and differences of the method with other closely related work.\n", "summary_of_the_paper": "This paper intends to exploit the learned representation of the model to tackle partial-label learning tasks. The paper begins by a PILOT experiment to show that the class activation map is better at selecting the true label from the candidate set than the model output itself. To overcome the limitation that CAM cannot be applied to linear model and non-image data, they propose class activation value (CAV) to replace CAM and show similar properties. ", "summary_of_the_review": "The paper proposes a novel CAV-based method to solve PLL problems and shows a promising performance than several state-of-the-art methods. Based on CAV, they propose a CAV learning method to select a potential true label for training and thus transfer PLL to supervised learning. Extensive and solid experiments prove the effectiveness of the proposed method. The paper provides a new way to tackle the PLL problem. The  main concern is the CAV is similar to the attention mechanism, and some of paragraphs are still need to explain.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UawypdqEwV", "reviewer": "Reviewer_Dgv3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) Interesting empirical observation. CAM is well known as a simple technique for discriminating the learning patterns of classes. This paper shows that it can be also used for selecting the true label from candidate labels by calculating the number of positive elements in the CAM of candidate labels.\n\n(2) Novel methodological contribution. This paper proposes Class Activation Value (CAV, which is versatile in models and inputs) and uses CAV to select the true label from candidate labels. In this way, a novel partial label learning method is obtained. This method is novel while keeping its simplicity.\n\n(3) Experiments are extensive and validate the effectiveness of the proposed method.\n\nCons:\n\n(1) I understand that CAM can be used for selecting the true label from candidate labels by calculating the number of positive elements in the CAM of candidate labels. However, I feel that it took me a long time to understand this operation. There is only one sentence about that. So the authors are encouraged to give more descriptions on how we use CAM to select the true label. This will make it easier for readers to understand.\n\nQuestions:\n\n(1) I feel that this paper actually focuses on the pseudo labeling task in a weakly supervised learning problem. So I would like to how can the proposed method be used in another weakly supervised learning problem like semi-supervised learning. Can the authors provide some discussions on that?\n\n(2) The authors mentioned the average-based strategy and the identification-based strategy for partial-label learning. I think the proposed method in this paper is an identification-based strategy, and the authors are encouraged to clearly state that in this paper.\n\n", "summary_of_the_paper": "This paper studies an interesting weakly supervised learning problem called partial-label learning, which solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include a set of candidate labels that include the true label. This paper empirically shows that the Class Activation Map (CAM), a simple technique for discriminating the learning patterns of each class in images, is better at making accurate predictions than the model itself on selecting the true label from candidate labels. Based on this motivation, this paper proposes a versatile version of CAM called Class Activation Value (CAV) and proposes a novel method to selects the true label by the maximum CAV for partial-label learning. Experiments validate the effectiveness of the proposed method.", "summary_of_the_review": "This paper investigates partial-label learning. This paper makes two key contributions including an interesting empirical observation and a novel methodological contribution. To the best of my knowledge, the empirical observation is disclosed by this paper for the first time and the proposed method is simple yet effective. Comprehensive experimental results support what the authors claimed in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the so-called partial-label learning problem and proposes a class activation map that is better at making accurate predictions than the model itself on selecting the true label from candidate labels. The authors investigate the approach in experimental results on four benchmark image datasets. 

The reviewers appreciated the simplicity of the approach and its effectiveness in practice. The reviewers raised questions how to apply the approach to another weakly supervised learning problem such as semi-supervised learning and whether the approach is an identification-based strategy. The reviewers also raised several questions asking for more details.

The authors submitted responses to the reviewers' comments. After reading the response, updating the reviews, and discussion, the reviewers who took part in the discussion considered that their  “questions have been well addressed” and that the “authors’ responses basically provided the answers to the questions”.  

The feedback provided was already fruitful and the final version should be already improved. 

Accept. Poster.
Decision: Accept (Poster)