PARTIAL-LABEL LEARNING
Year: 2022

Abstract:
Partial-label learning (PLL) solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include the true label. Recent advances showed that PLL can be compatible with deep neural networks, which achieved state-of-the-art performance. However, most of the existing deep PLL methods focus on designing proper training objectives under various assumptions on the collected data, which may limit their performance when the collected data cannot satisfy the adopted assumptions. In this paper, we propose to exploit the learned intrinsic representation of the model to identify the true label in the training process, which does not rely on any assumptions on the collected data. We make two key contributions. As the first contribution, we empirically show that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in images, could surprisingly be utilized to make accurate predictions on selecting the true label from candidate labels. Unfortunately, as CAM is confined to image inputs with convolutional neural networks, we are yet unable to directly leverage CAM to address the PLL problem with general inputs and models. Thus, as the second contribution, we propose the class activation value (CAV), which owns similar properties of CAM, while CAV is versatile in various types of inputs and models. Building upon CAV, we propose a novel method named CAV Learning (CAVL) that selects the true label by the class with the maximum CAV for model training. Extensive experiments on various datasets demonstrate that our proposed CAVL method achieves stateof-the-art performance.

heading: 1 INTRODUCTION
text: To liberate humans from exhaustive label annotation work, numerous researchers have dedicated themselves to investigating various weakly supervised learning (WSL) (Zhou, 2017) problems, including but not limited to noisy-label learning (Liu & Tao, 2015; Xia et al., 2020; Han et al., 2020), semi-supervised learning (Zhu & Goldberg, 2009; Miyato et al., 2018; Luo et al., 2018), and multiple-instance learning (Zhou et al., 2012). This paper focuses on another popular WSL problem called partial-label learning (PLL) (Jin & Ghahramani, 2002; Cour et al., 2011), which aims to learn a model from training examples equipped with a set of candidate labels that include the true
†Correspondence to Lei Feng (lfeng@cqu.edu.cn) and Bo Han (bhanml@comp.hkbu.edu.hk).
label. Due to the cost and difficulty of annotating every example exactly with the true label in huge datasets, PLL has been widely applied to various tasks such as multimedia context analysis (Zeng et al., 2013) and web mining (Luo & Orabona, 2010).
To solve the PLL problem, there are two mainstream strategies to discriminate the unknown true label from candidate labels, including the average-based strategy (ABS) and the identification-based strategy (IBS). ABS always treats each candidate label equally and averages the model outputs of all candidate labels for prediction (Hüllermeier & Beringer, 2006; Cour et al., 2011; Feng et al., 2020; Yao et al., 2020; Wen et al., 2021). IBS concentrates on iteratively selecting one label from the candidate label set as the true label to exclude the uncertainty, forming PLL into an ordinary classification problem (Jin & Ghahramani, 2002; Liu & Dietterich, 2014; Zhang & Yu, 2015; Zhang et al., 2016). Among these methods, only several of them (Yao et al., 2020; Wen et al., 2021) can be compatible with deep neural networks, which achieved state-of-the-art performance.
Most of the existing PLL methods aim at designing proper training objectives under various assumptions on the collected data. For example, the consistent methods proposed by Feng et al. (2020) are based on a specific data generation assumption. The derivation of the PRODEN method (Lv et al., 2020) stems from the assumption that the true label achieves the minimal loss among the candidate labels. Yan & Guo (2021) proposed the MGPLL method based on the assumption of non-random label noise. Although these methods have achieved generally great empirical performance, their performance could be degraded when the collected data cannot meet the adopted assumptions.
In this paper, we aim to investigate a novel PLL method by exploiting the learned intrinsic representation of the model to identify the ground truth in the training process without relying on any assumptions on the collected data. We focus on the class activation map (CAM) (Zhou et al., 2016), which is a prevailing tool for visualizing the representation information of convolutional neural network (CNN)-based models and could be easily obtained by the weighted linear sum of the feature maps. As CAM is able to discriminate the learning patterns of each class for the model, we conjecture that CAM could be used to guide the model to spot the true label. To verify this assumption, we conducted a pilot experiment on CIFAR-10 (Krizhevsky et al., 2009), where the candidate label sets were generated in two different ways. As shown in Figure 1 (please refer to Section 2.2), it is surprising to find that CAM can potentially help the model recognize the true labels, and such capacity of CAM synchronously changes with the classifier performance during the whole training phase. Based on the experimental results, it can be conjectured that such intrinsic representations can be useful to distinguish the true label from candidate labels for PLL.
Motivated by the above observations, we naturally consider leveraging CAM to solve the PLL problem. However, CAM was only proposed for addressing image datasets with using deep models built on CNN, revealing two limitations on its application. Firstly, CAM cannot be adopted to the classifier based on shallow models such as the linear model and multilayer perceptron (MLP). Secondly, CAM is unable to deal with any inputs other than the image. To overcome these shortcomings, we for the first time propose a simple but effective tool—the class activation value (CAV) to capture the learned representation information in a more general way, which is essentially the weighted output of the classifier. Experimental results in Figure 2 and Appendix B.7 also shows that our CAV works similarly as CAM during the training phase.
Building upon CAV, we propose a simple yet effective PLL method called CAV Learning (CAVL) that guides the model to differentiate the true label from the candidate set during the training process. Specifically, we first train the model by treating all candidate labels equally and obtain CAVs of given training examples, and then regard the class with the maximum CAV as the true label for each instance during the training process. In this way, CAVL transforms PLL into supervised learning, thereby making the model reliable to recognize the true label with learned representation information by CAV. Extensive experiments on benchmark-simulated and real-world datasets show that our proposed CAVL method achieves state-of-the-art performance.

heading: 2 DISCOVERING CLASS ACTIVATION MAP FOR PARTIAL-LABEL LEARNING
text: In this section, we provide a detailed discussion about our motivation and empirically show that the class activation map could be helpful for addressing PLL.

heading: 2.1 CLASS ACTIVATION MAP
text: The class activation map (CAM) (Zhou et al., 2016) is a popular and elementary mechanism in computer vision to represent the discriminative part of an input image captured by the classifier to identify each class. In other words, CAM manifests the learning patterns of the classifier, denoted as f , to the specific class in the image. Let us denote an input image as x ∈ Rd=c×h×w (with c channels, height h and width w) and CAM of x ∈ Rd=c×h×w as m ∈ Rk×h×w where k is the number of classes. We note thatm could be generated by training a specific classification model fc, which normally comprises of an encoder function e : X → Rcf×h×w for feature map extraction, a Global Average-Pooling layer gap : Rcf×h×w → Rcf×1, and a linear layer with weight θ ∈ Rk×Cf , where cf is the number of channels in the feature maps. Therefore, CAM of x related to the j-th class could be obtained by
mj = ∑Cf
i=1 (θj)ie i(x), j ∈ {1, . . . , k}, (1)
where θj ∈ R1×Cf is the corresponding linear weight related to the j-th object. CAM is a weighted linear sum of feature maps and the linear weights. The weakness of CAM is the limited applications, since not all classification network architectures follow the model fc mentioned above. To improve its generalization for any CNN-based model f , Gradient-weighted CAM (Grad-CAM) (Selvaraju et al., 2017) was proposed to implement such an internal representation by reasonably leveraging class-specific gradient information. Concretely, let us denote by gj ∈ Rd=c×h×w the Grad-CAM of x related to the j-th object, and gj could be expressed as
gj = 1 h× w ∑cf z=1 ∑h m=1 ∑w n=1 ∂f j(x) ∂ez(x)m,n ez(x). (2)
Intuitively, the derivative of logit f(x) with respect to feature map e(x) is used as the weights for calculating g. Note that both Grad-CAM and CAM are generated by weighted sums of the feature maps, and Grad-CAM is equivalent to CAM when f follows the same architecture as fc. Based on Grad-CAM, Grad-CAM++ (Chattopadhay et al., 2018) aims to provide better visual explanations of CNN and occurrences of multiple foreground objects. Thanks to such a simple technique, numerous problems in computer vision such as interpretation (Xu et al., 2019a) and weakly supervised semantic segmentation (Wei et al., 2016) have achieved marvelous progress. In this paper, we aim to extract and improve the useful knowledge from CAM to address the PLL problem, leading the classifier to identify the ground truth.

heading: 2.2 PILOT EXPERIMENT ON CAM
text: Generally, Grad-CAM or CAM is treated as the internal representation of f . We believe that such a mechanism could guide f to differentiate the true label from the candidate set because it is constructed by taking advantage of internal elements in f . To validate our conjecture, we conducted a pilot experiment by adopting two different label selection methods. The first one is the intuitive method (IM), which simply regards all the labels from the candidate sets as the true labels with using cross entropy loss. The second one is the “CAM label” selection strategy, which discriminates the true label from the candidate label set by using CAM. During each training epoch, we calculated the number of the foreground seeds of CAM from each candidate label set. Specifically, to calculate the number of foreground seeds of mj , we counted the elements with positive values in mj . Note that each x could possess k CAMs and the true label is always in the candidate set. Thus we selected the label from the candidate set, whose CAM owns most foreground seeds as the true label. The label candidate label sets were generated by two different approaches:(I) Uniformly Sampling Strategy (USS). Uniformly sampling the candidate label set for each training instance from all the possible candidate label sets (Feng et al., 2020). (II) Flipping Probability Strategy (FPS). By setting a flipping probability q to any false label, the false label could be selected as a candidate label with a probability q (Feng & An, 2019a; Yan & Guo, 2020; Lv et al., 2020; Wen et al., 2021). Here the detailed training settings of the experiments are similar to the experiment part (please refer to Section 4.1.1 for details). ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) were chosen as the backbones to train a classifier on CIFAR-10 (Krizhevsky et al., 2009).
Figure 1 shows the average results of 5-time trials and we can find that CAM demonstrates two helpful features during the training phase. Firstly, it is clear that the performance of the classifier
(a) ResNet + USS (b) DenseNet + FPS Methods in (a) are implemented by ResNet with using USS to generate the candidate set. Methods in (b) are conducted by DenseNet with the candidate label sets produced by FPS. For three lines representing accuracy performance, the results drawn by the blue dashed line and red line (“CAM” line) are obtained by using the “CAM label” selection strategy, namely training the model with regarding the class with the most number of activated seeds in its CAM as the true label. Specifically, the red line depicts the training accuracy measured by selecting the CAM with the maximum activated seeds from the candidate set as the true label, and the blue line simply depicts the test accuracy of the classifier. The yellow dashed line (“IM” line) depicts the test accuracy of the classifier trained by IM. The paired circle indicates the similar accuracy fluctuation between the classifier and CAM, which shows the dynamic attribute. The double arrow marks the performance gap between “CAM” label selection strategy and IM, which represents the power attribute.
Figure 1: Comparison of accuracy performance of two different label selection strategies.
trained by “CAM label” selection strategy is better than IM (the gap between the blue dashed line and the yellow dashed line), which shows that CAM could potentially guide the classifier toward the true label from the candidate set. Here we name this attribute of CAM as power. Secondly, it is found that the accuracy of identifying true labels by CAM is synchronously changed with that by the classifier. Several fluctuations (marked by the paired circle) of the classifier performance (test accuracy) and CAM accuracy are similar to a large extent, resulting in continuous improvement to the model itself. Here we name this observed feature of CAM as dynamic. The power and dynamic attributes disclose the fact that CAM may show more than we thought, which is an inspiring finding to address PLL. The power attribute motivates us to consider that the classifier may learn more accurate true labels if it is forced to approximate the label distribution recognized by CAM. The dynamic attribute may guarantee that such guidance would be constantly effective during the whole training phase since CAM would synchronously be self-improved as the classifier becomes stronger. Hence it would be reasonable and meaningful to explore such internal sign as guidance for PLL.

heading: 3 PROPOSING CLASS ACTIVATION VALUE FOR PARTIAL-LABEL LEARNING

heading: 3.1 CLASS ACTIVATION VALUE—SIMPLE AND GENERAL REPRESENTATION
text: In this paper, we aim to propose a CAM-like tool as guidance for f to identify the true label from candidate labels. CAM or Grad-CAM is specially designed for convolutional neural network (CNN)based models, which mainly deals with image datasets with retaining spatial information. However, to the best of our knowledge, most PLL methods are implemented in datasets with various forms, including the image datasets such as MNIST (LeCun et al., 1998), and feature vectors such as BirdSong (Briggs et al., 2012). The networks on these datasets could be directly adapted to elementary models such as the linear or multilayer perceptron (MLP) for f to deal with the inputting flatteneddimension vectors. Without spatial information or CNN-based architectures, CAM-based tools are invalid to represent the internal learning pattern. To make such a concept versatile, we aim to propose another novel CAM-like tool with simple implementation that could be adopted in a wider range of backbones and problems. We note that the (softmax) outputs can replace the feature map e(x) since the encoder e could be directly regarded as f . Thus, it is fundamental to formulate an internal component for a value-based framework. For each candidate label, apart from its class output, its gradient of training loss with respect to the model output can also capture the importance degree of the candidate label to some extent. Inspired by the formulation of Grad-CAM in Eq. (2)
Algorithm 1 CAVL Algorithm Input: Model f , epoch Tmax, iteration Zmax, partial labeled training set D = {(xi, Si)}mi=1.
for t = 1, 2, ..., Tmax do Shuffle D; for z = 1, 2, ..., Imax do
Sample mini-batch Dz = {(xj , Sj)}uj=1 from D; if t = 1 then
Update f by minimizing the cross entropy loss function R̂1 in Eq. (6) with treating all the candidate labels from Sj equally; else Obtain the CAV v for each xj by Eq. (4); Generate y′j for xj by selecting the maximum v from Sj by Eq. (7); Update f by minimizing the cross entropy loss function R̂cavl in Eq. (8) with treating y′j as the sole true label for each xj ;
end if end for
end for Output: f .
that regards the gradient information flow to the last convolution layer as the importance of each neuron to the j-th class, we can intuitively derive the importance of each candidate label as follows:
vj = ∣∣∣∣∂(− log(ψj(f(x))))∂f j(x) ∣∣∣∣ψj(f(x)) = ∣∣ψj(f(x))− 1∣∣ψj(f(x)), (3)
where ψ : Rk → Rk is the activation function imposed on the logit vector f(x) to approximate the class-conditional probability P (y|x), and ψ is mostly implemented by the SoftMax function. The derivation process of the last inequality of Eq. (3) is provided in Appendix A. From Eq. (3), we can observe that the importance vj of each candidate label j depends on the softmax confidence score ψj(f(x)). However, we argue that such a calculation of vj is actually suboptimal, becauseψj(f(x)) conveys less information of model output than the logit f j(x) as ψj(f(x)) could be considered as a condensed version of f j(x). Therefore, instead of directly adopting Eq. (3), we propose to replace ψj(f(x)) in Eq. (3) by f j(x), and the importance of each candidate label would be calculated by the following simple way:
vj = |f j(x)− 1|f j(x). (4)
We call the importance vj calculated by Eq. (4) as the class activation value (CAV) of the j-th class, as it could be regarded as a value-based “CAM”. Compared with CAM, we directly treat f(x) as the “feature map” so that v can be obtained by a weight imposed on the class output of the model. In summary, both CAV and CAM are the weighted operation to the feature maps (or outputs in the classifier). Differently from CAM, CAV is essentially a value so that it could be easily obtained and applied to different backbones. Besides, our CAV could also be generated with less time complexity since there is no need to process the weighted sum of the feature maps.

heading: 3.2 PARTIAL-LABEL LEARNING WITH CLASS ACTIVATION VALUE
text: In this section, we introduce how to effectively address PLL by using CAV in detail. Firstly, we give a formal introduction to PLL. For a k-class classification problem, let X ∈ Rd be the feature (input) space and {xi}mi=1 ∈ X be the input data, where m refers to the number of training examples. Let Y = {1, 2, ..., k} be the label space. Let us denote C = {2Y\∅\Y} as the candidate label space where 2Y is the power set of Y, and |C| = 2k − 2 shows that the candidate label set is not supposed to be the empty set nor the whole label set. For each training instance xi, let yi ∈ Y be the ground truth label and Si ∈ C be the candidate label set. We denote P (x, y) and P (x, S) as the probability densities of fully labeled examples and partially labeled examples respectively. Based on the crucial assumption of PLL that the candidate label set of each instance must include the correct label, we have yi ∈ Si, i.e., P (yi ∈ Si|y = yi,x = xi) = 1,∀yi ∈ Y,∀Si ∈ C. (5) PLL aims to learn a classifier f : X → Rk with training examples independently and identically sampled from P (x, S) to make correct predictions for test examples. To the best of our knowledge,
most methods focus on designing the loss by considering all candidate labels or iteratively extracting the true label from the theoretical aspect. Differently from them, we concentrate on how to solve PLL by efficiently utilizing the learned representation by CAV, which could be potential and effective guidance for helping f recognize the true label.
Note that the learned representation could be directly obtained by using CAV after firstly training f in one epoch. For training f in the first epoch, here we directly utilize the IM strategy, i.e., treating every candidate label equally by cross entropy loss L(f(x), s), s ∈ S. Hence the empirical risk function R̂1 for the first epoch could be defined as
R̂1(L, f) = 1
m ∑m i=1 ∑ j∈Si L(f(xi), sj). (6)
After training f in one epoch, the CAV v of every candidate label could be gained for each x by Eq. (4). To keep the classifier away from the negative effect by the false positive labels from the candidate set, we use the knowledge by CAV to guide the following learning process after the first training epoch, and the potential true label for xi is selected by
y′i = argmax j∈Si vji , (7)
where y′i is the selected true label from the candidate label set for xi. In this way, y ′ i could be straightly treated as the ground truth so that PLL could be transferred to classical supervised learning. Finally, the risk function R̂cavl for the following epochs could be expressed as
R̂cavl(L, f) = 1
m ∑m i=1 L(f(xi), y′i). (8)
As f gradually learns to approximate P (y′|x), CAV would also be updated to identify the true label. Based on such a progressive label selection on the candidate label set, our Class Activation Value Learning (CAVL) is supposed to progressively training an accurate model with correctly selected labels. The pseudo-code of CAVL is given in Algorithm 1.

heading: 4 EXPERIMENTS
text: In this section, extensive experiments on various datasets are implemented to verify the effectiveness and rightness of our proposed CAV and CAVL method.

heading: 4.1 CAVL PERFORMANCE

heading: 4.1.1 BENCHMARK DATASET COMPARISONS
text: Datasets and backbones. We use four popular benchmark datasets to test the performance of our CAVL, which are MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), KuzushijiMNIST (Clanuwat et al., 2018) and CIFAR-10 (Krizhevsky et al., 2009). Note that it is necessary to manually generate the candidate label sets since they are supposed to be used for single-classification problems. Recall that we introduce two different candidate label generation process (please refer to Section 2.2 for details) (Feng & An, 2019a;b; Feng et al., 2020; Lv et al., 2020; Wen et al., 2021), i.e., USS and FPS. The former one is implemented by uniformly sampling a label set from all the partial label space C for each instance, and the latter one sets a flipping probability q to any irrelevant label so that it could become one item in the candidate label set with probability q. In this section we set q ∈ {0.3, 0.5, 0.7} to represent different ambiguity degrees. For MNIST, Kuzushiji-MNIST and Fashion-MNIST, we adopt 3-layer MLP and 5-layer LeNet as the backbones. For CIFAR-10, we choose 34-layer ResNet (He et al., 2016) and 22-layer DenseNet (Huang et al., 2017) as the base models.
Compared methods and training settings. We compare our CAVL with four state-of-the-art approaches including PRODEN (Lv et al., 2020), Leveraged Weighted (LW) Loss (Wen et al., 2021), RC and CC (Feng et al., 2020). For all methods we search the initial learning rate from {0.0001, 0.001, 0.01, 0.05, 0.1, 0.5}, and weight decay from {10−6, 10−5, ..., 10−1}. We take a mini-batch size of {32, 256} images and train all the methods using Adam (Kingma & Ba, 2015) optimizer for 250 epochs. Hyper-parameters are searched to maximize the accuracy on a validation
set containing 10% of the training samples annotated by true labels. All the implemented methods are trained on 1 RTX3090 GPU with 24 GB memory. To guarantee the fairness of comparisons, we repeatedly conduct all experiments 5 times and record the average accuracy with the standard deviation following the conventions in Feng et al. (2020); Lv et al. (2020); Wen et al. (2021). This work is partially supported by Huawei MindSpore (Huawei, 2020).
Experiment results. As shown in Table 1 and 2, our CAVL shows its superiority on four benchmark datasets with various backbones. Additionally, our CAVL outperforms nearly all state-of-the-art methods using the candidate labels generated by both USS and FPS, which shows that our CAVL does not rely on any data generation assumption. Specifically, it is worth mentioning that CAVL is able to show competitive performance with RC, which is theoretically proved to be possess optimal
performance in data distribution generated by USS. Therefore, the results reasonably verify the generalization and effectiveness of our CAV and CAVL.

heading: 4.1.2 REAL-WORLD DATASET COMPARISONS
text: Datasets and backbones. We select five real-world datasets including Lost (Cour et al., 2011), MSRCv2 (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng et al., 2013) and Yahoo!News (Guillaumin et al., 2010). Note that these real-world datasets have their candidate labels. As a common practice in (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021), we select 1-layer linear model and 3-layer MLP as the backbones to validate the performance of our CAVL in these five datasets.
Compared methods and training settings. Apart from the four methods mentioned in Section 4.1.1, we add SURE (Feng & An, 2019b), CLPL (Cour et al., 2011), IPAL (Zhang & Yu, 2015), PLSVM (Elkan & Noto, 2008), PLECOC (Zhang et al., 2017) and PLKNN (Hüllermeier & Beringer, 2006) as extra methods for comparison, where the parameters are searched to maximize the accuracy on a validation set (10% of the training set). We repeatedly run all experiments 10 times, and report the average accuracy with the standard deviation for each method. Other settings are similar to Section 4.1.1. The comparison results using MLP is recorded in Appendix B.2.
Experiment results. In Table 3, our proposed CAVL shows the best performance in nearly all five real datasets. Besides, CAM is unable to be extracted with the word embedding input such as BirdSong, lacking spatial information. Therefore, the achievement of CAVL on these real datasets shows the generalization and superiority of our CAVL to shallow model, where CAV could serve as the CAM to lead the classifier to learn the true label.

heading: 4.2 CAV ATTRIBUTES
text: In this part, we conduct several experiments to present how these attributes of CAV guide the learning process. We implement CAVL with four backbones including linear, MLP, ResNet and DenseNet on four benchmark datasets, whose candidate label sets are generated by USS. The training settings are similar to the ones in Section 4.1.1. Figure 2 illustrates the averaging performance of CAV, CAM (for deep networks) and the model itself during the 5-time training process. The red (blue) line in Figure 2 depicts the training accuracy measured by treating the maximum CAV (CAM) from the candidate set as the true label, and the cyan line depicts the test accuracy of the model. We note that here CAM is simply extracted from the classifier for comparison, which is different from its usage in Section 2.2. As shown in Figure 2, it is clear that the classifier learns to approximate the better data distribution by CAV, and the accuracy trend recognized by CAV is consistent with the classifier performance in the test datasets, which validates the Power and Dynamic attribute of CAV.

heading: 5 RELATED WORK
text: In this section, we give a brief introduction to the two mainstream strategies for partial-label learning (PLL), i.e., the averaged-based strategy (ABS) and the identification-based strategy (IBS).
ABS treats all candidate labels in an equal manner and then averages the model outputs of all candidate labels for prediction. Some non-parametric methods (Hüllermeier & Beringer, 2006; Gong et al., 2017) concentrated on predicting the label by leveraging the outputs of its neighbors. Furthermore, some methods (Cour et al., 2009; Zhang et al., 2016; Yao et al., 2020) aimed to subtly leverage the labels outside the candidate set so as to discriminate the potential true label. Yao et al. (2020) designed an entropy-based regularizer to minimize the entropy of each label to maximize the margin between the potential true label and the unlikely labels. A recent study (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021; Lv et al., 2021) focused on the data generation process and proposed a classifier-consistent method based on a transition matrix. (Chen et al., 2020) proposed a new problem General Partial Label Learning and leveraged graph neural network to tackle the label ambiguity.
IBS aims to constantly identify the most possible true label from the candidate label set to eliminate the label ambiguity. Early works treated the potential truth label as a latent variable and optimized the objective function by the maximum likelihood criterion (Jin & Ghahramani, 2002; Liu & Dietterich, 2014) or the maximum margin criterion (Nguyen & Caruana, 2008; Yu & Zhang, 2016). Later, taking advantage of topological information of feature space to generate the score of each candidate label (Zhang & Yu, 2015; Zhang et al., 2016; Feng & An, 2018; Wang et al., 2019) attracted much attention from researchers. These methods commonly aimed to iteratively update the confidence of each candidate label based on the widely used assumption that similar instance are supposed to possess the same label. Xu et al. (2019b) proposed to model the generalized label distribution by leveraging the topological information of the feature space. Lyu et al. (2021) reformulated PLL into a matching selection problem and proposed a Graph-Matching based Partial Label Learning framework to solve the problem. It is worth noting that IBS is commonly susceptible to the false positive labels that co-occur with the true label in the candidate label set. Thus our CAVL, which belongs to IBS, also owns this inevitable shortcoming.

heading: 6 CONCLUSION
text: In this paper, we exploited the learned intrinsic representation of the model for partial-label learning (PLL). We made two key contributions. Firstly, we found that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in an image, could surprisingly be used to differentiate the true label from candidate labels. Unfortunately, we are yet unable to directly use CAM for PLL, as CAM is subject to image inputs with convolutional neural networks. Thus, our second contribution is to propose the class activation value (CAV), which not only owns similar properties of CAM but also is versatile in various types of inputs and models. Based on CAV, we proposed a simple yet effective PLL method that selects the true label by the class with the maximum CAV for model training. Comprehensive experimental results on various datasets demonstrated that our CAVL achieved state-of-the-art performance.

heading: ACKNOWLEDGMENT
text: FZ and BH were supported by NSFC Young Scientists Fund No. 62006202, RGC Early Career Scheme No. 22200720 and HKBU CSD Departmental Incentive Grant. BH was also supported by MSRA StarTrack Program. LF was supported by the National Natural Science Foundation of China (Grant No. 62106028) and CAAI-Huawei MindSpore Open Fund. TLL was supported by Australian Research Council Projects DE-190101473 and DP-220102121. MS was supported by JST CREST Grant Number JPMJCR18A2.

heading: A DERIVATION PROCESS OF EQUATION. (3)
text: The SoftMax function is ψj(x) = exp(xj)/ ∑k
j=1 exp(xj), where x ∈ Rd refers to the. Then we can obtain the derivation of − log(ψj(x)) with respect to the xj by
∂(− log(ψj(x))) ∂xj = ∂(− log(ψj(x))) ∂ψj(x) ∂ψj(x) ∂xj
=− 1 ψj(x)
∂ψj(x)
∂xj
=− 1 ψj(x)
∑d i=1 e
xjexi − (exi)2∑d i=1(e xi)2
=− 1 ψj(x) ( exj∑d
i=1(e xi)2
)(1− e xj∑d
i=1(e xi)2
)
=− 1 ψj(x) ψj(x)(1− ψj(x))
= ψj(x)− 1.
(9)

heading: B DETAILED SUPPLEMENTARY FOR EXPERIMENTS
text: B.1 BENCHMARK DATASETS
In Section 4.1.1, we use four widely-used benchmark datasets, i.e. MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), Kuzushiji-MNIST (Clanuwat et al., 2018) and CIFAR-10 (Krizhevsky et al., 2009). Table 4 lists the characteristics of these datasets. We respectively describe these datasets as follows.
• MNIST: It is a 10-class dataset of handwritten digits. Each data is a 28 × 28 grayscale image.
• Fashion-MNIST: It is also a 10-class dataset. Each instance is a fashion item from one of the 10 classes, which are T-shirt/top, trouser, pullover, dress, sandal, coat, shirt, sneaker, bag, and ankle boot. Moreover, each image is a 28 × 28 grayscale image.
• Kuzushiji-MNIST: Each instance is a 28 × 28 grayscale image associated with one label of 10-class cursive Japanese (“Kuzushiji”) characters.
• CIFAR-10: Each instance is a 32 × 32 × 3 colored image in RGB format. It is a ten-class dataset of objects including airplane, bird, automobile, cat, deer, frog, dog, horse, ship, and truck.
B.2 REAL DATASETS
In Section 4.1.2, We select five real-world datasets including Lost (Cour et al., 2011), MSRCv2 (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng et al., 2013) and Yahoo!News (Guillaumin et al., 2010). Here we make a comprehensive descriptions about them shown as follows.
• Lost, Soccer Player and Yahoo! News: They crop faces in images or video frames as instances, and the names appearing on the corresponding captions or subtitles are considered as candidate labels.
• MSRCv2: Each image segment is treated as a sample, and objects appearing in the same image are regarded as candidate labels.
• BirdSong: The singing syllables of birds are regarded as instances and bird species who are jointly singing during any ten seconds are represented as candidate labels
Table 5 shows the average number of candidate labels (Avg. # CLs) per instance. Table 6 presents the performance comparison between other methods and our CAVL method based on MLP model, the results of which also validate the effectiveness our CAVL.
B.3 ABLATION STUDIES ON THE STARTING EPOCH IN CAVL
In Section 3.2, we propose the CAVL method to address PLL. Specifically, we use the IM to train the classifier in the first epoch, and then the potential true labels could be obtained with CAV after the following training epochs. Here we set the epoch number for starting using the CAVL method, i.e. the starting epoch, as one. To further validate the effectiveness of our CAVL, we explore the effects of the starting epoch on our method. Specifically, we implement CAVL with two backbones including LeNet and ResNet on four benchmark datasets. We repeatedly run the experiments 5 times by using candidate labels generated by USS. Following similar training settings in Section 4.1.1, we set the total training epoch as 250, and select the starting epoch from [10, 50, 100, 150]. IM is used before the selected starting epoch and CAVL is implemented after the selected epoch.
As shown in Table 7, the accuracy of the classifier becomes lower as the increase of the starting epoch in CAVL. Let us take the results in KMINIST as an example, the CAVL shows its superiority with setting the starting epoch as 1, achieving 93.25± 0.21% accuracy performance. As the starting epoch increases, the classifier is more negatively affected by the false positive labels in the candidate sets and dragged away from the true labels by IM. Thus it only achieves 88.54 ± 0.26% with the starting epoch as 150. This phenomenon is reasonable because the DNN learn patterns first, which suggests that deep networks can gradually memorize the data, moving from regular data to irregular data such as outliers and mislabeled data (Arpit et al., 2017). Therefore, our CAVL could not
sufficiently help the classifier to deal with this memorization effect during the rest of the training epochs. In conclusion, the effectiveness of our CAVL would be weakened as extending the starting epoch.
B.4 GENERATION OF CANDIDATE LABELS
In Section 4.1.1 we introduce two different generation ways for the candidate label sets, i.e, USS, Uniformly sampling a label set from all the partial label space C for each instance. FPS, Setting a flipping probability q to any irrelevant label which could possibly become one item in the candidate label set with probability q.
For USS, each partially labeled example (x, S) is independently drawn from a probability distribution with the following density:
P̃ (x, S) = ∑k
i=1 P (S|y = i)P (x, y = i), P (S|y = i) =
{ 1
2k−1−1 i ∈ S, 0 i /∈ S.
(10)
The generation process assumes that the candidate label set S is independent of the instance x. There are totally 2k−1 possible candidate label sets that contain the specific true label y. Therefore, Eq. (10) illustrates the candidate label set for each instance is uniformly sampled.
For FPS, we set a flipping probability q to any irrelevant label that possibly entries the candidate label set. Here we introduce the class transition matrix (denoted by T ) for partially labeled data, where Tij refers to the probability of the label j being a candidate label given the true label i for each instance. Note that Tii = 1 always holds since the true label always belongs to the candidate label. Tij = q, i 6= j holds for other elements. The matrix representation of T is expressed as: 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1 q q q q q q q q q q 1  B.5 COMPARED METHODS
The compared PLL methods are listed as follows.
• SURE (Feng & An, 2019b): It iteratively enlarges the confidence of the candidate label with the highest probability to be the correct label.
• CLPL (Cour et al., 2011): It uses a convex formulation by using the one-versus-all strategy in the multi class loss function.
• IPAL (Zhang & Yu, 2015): It is a non-parametric method that applies the label propagation strategy to iteratively update the confidence of each candidate label.
• PLSVM (Elkan & Noto, 2008): It is a maximum margin-based method that differentiates candidate labels from non-candidate labels by maximizing the margin between them.
• PLECOC (Zhang et al., 2017): It adapts the Error-Correcting Output Codes method to deal with partially labeled examples in a disambiguation-free manner.
• PLKNN (Hüllermeier & Beringer, 2006): It adapts the widely-used k-nearest neighbors method to make predictions for partially labeled examples.
• CC & RC (Feng et al., 2020): The former method is a novel risk-consistent partial label learning method and the latter one is classifier-consistent based on the generation model.
• LW (Wen et al., 2021): The method proposes a family of loss function for the first time, where introduces the leverage parameter β to consider the trade-off between losses on partial labels and non-partial labels.
For all the above methods, their parameters are specified or searched according to the suggested parameter settings by respective papers to maximize the accuracy performance on a validation set, which is made by the 10% of the training set.
B.6 DETAILS OF THE BACKBONES
In Section 4 we select five network architectures as the backbones for modelling our classifier, which are linear, 3-layer MLP, 5-layer LeNet, 34-layer ResNet (He et al., 2016) and 22-layer DenseNet (Huang et al., 2017). The linear model is a linear-in-input model:d − k. MLP refers to a 3-layer fully connected networks with ReLU as the activation function, and the architecture is d − 500 − k. LeNet is comprised of 2 Convolution Layer and 3 fully connected layer. 34-layer ResNet and 22-layer DenseNet strictly follow the implementation by He et al. (2016) and Huang et al. (2017).
B.7 PILOT EXPERIMENTS IN REAL-WORLD DATATSETS
In Section 2.2 we present the pilot experiments on CIFAR10, which shows that CAM possess two attributes, i.e., power and dynamic. Here we provide more results on five real-world datasets to prove the rightness of our proposed CAV. The real-world datasets already have their candidate labels. Following similar training settings in Section 2.2, we select 1-layer linear as the backbone for the classifier and respectively use the IM and CAV to train it in 10 epochs. Intuitively, using CAV to train the model is equal to CAVL. Note that CAM could not be obtained in the linear module. Figure 3 shows the average results of 10-time trials and it is easily found that CAV also shows power and dynamic attributes in real-world datasets, validating the generalization and effectiveness of our CAV.

heading: C FURTHER EXPLANATIONS ON CLASS ACTIVATION MAP
text: Here we provide a detailed explanation of CAM. For a well-trained CNN module for addressing the classification problem, the Class Activation Map (CAM) is simply a weighted linear sum of the presence of the visual patterns at different spatial locations for the input images. CAM aims to mine out what the network focuses on the input images related to true labels during the training phase. It is known that a CNN-based module is a ”black box” and the meaning behind CAM is the exploration of the inner principles inside such technology. Based on CAM, it is convenient to visualize the learning patterns of the CNN-based modules. Figure 4 shows several CAM results selected from (Zhou et al., 2016). Here we take one example of the Teapot for illustration, the CAM of the image related to the teapot shows the most discriminative regions in the head part, which manifests that the classification classifies the image to the teapot since it finds and concentrates on the head part of the teapot. Thanks to CAM, we could conveniently and effectively investigate the inner principles of CNN-based modules by simply utilizing the components from the network itself. This is also the reason why we name it ”internal representation”, which is essentially a form of intermediate outputs.

Review summaries:
This paper intends to exploit the learned representation of the model to tackle partial-label learning tasks. The paper begins by a PILOT experiment to show that the class activation map is better at selecting the true label from the candidate set than the model output itself. To overcome the limitation that CAM cannot be applied to linear model and non-image data, they propose class activation value (CAV) to replace CAM and show similar properties. 

This paper studies an interesting weakly supervised learning problem called partial-label learning, which solves the multi-class classification problem, where each training instance is assigned a set of candidate labels that include a set of candidate labels that include the true label. This paper empirically shows that the Class Activation Map (CAM), a simple technique for discriminating the learning patterns of each class in images, is better at making accurate predictions than the model itself on selecting the true label from candidate labels. Based on this motivation, this paper proposes a versatile version of CAM called Class Activation Value (CAV) and proposes a novel method to selects the true label by the maximum CAV for partial-label learning. Experiments validate the effectiveness of the proposed method.

This paper focuses on the problem of partial-label learning (PLL), where each training instance is assigned a set of candidate labels that include the true label.  This paper shows that class activation map (CAM) could identify the true label from candidate labels for PLL. In addition, the authors propose the class activation value (CAV) for identification via capturing the learned representation information in a more general way instead of CAM as CAM is confined for image datasets and CNN model. CAV Learning that selects the true label by the class with the maximum CAV for model training is proposed. Experiments on various datasets demonstrate that the proposed method achieves state-of-the-art performance.