Exploiting Class Activation Value for Partial-Label Learning

1 INTRODUCTION . To liberate humans from exhaustive label annotation work , numerous researchers have dedicated themselves to investigating various weakly supervised learning ( WSL ) ( Zhou , 2017 ) problems , including but not limited to noisy-label learning ( Liu & Tao , 2015 ; Xia et al. , 2020 ; Han et al. , 2020 ) , semi-supervised learning ( Zhu & Goldberg , 2009 ; Miyato et al. , 2018 ; Luo et al. , 2018 ) , and multiple-instance learning ( Zhou et al. , 2012 ) . This paper focuses on another popular WSL problem called partial-label learning ( PLL ) ( Jin & Ghahramani , 2002 ; Cour et al. , 2011 ) , which aims to learn a model from training examples equipped with a set of candidate labels that include the true †Correspondence to Lei Feng ( lfeng @ cqu.edu.cn ) and Bo Han ( bhanml @ comp.hkbu.edu.hk ) . label . Due to the cost and difficulty of annotating every example exactly with the true label in huge datasets , PLL has been widely applied to various tasks such as multimedia context analysis ( Zeng et al. , 2013 ) and web mining ( Luo & Orabona , 2010 ) . To solve the PLL problem , there are two mainstream strategies to discriminate the unknown true label from candidate labels , including the average-based strategy ( ABS ) and the identification-based strategy ( IBS ) . ABS always treats each candidate label equally and averages the model outputs of all candidate labels for prediction ( Hüllermeier & Beringer , 2006 ; Cour et al. , 2011 ; Feng et al. , 2020 ; Yao et al. , 2020 ; Wen et al. , 2021 ) . IBS concentrates on iteratively selecting one label from the candidate label set as the true label to exclude the uncertainty , forming PLL into an ordinary classification problem ( Jin & Ghahramani , 2002 ; Liu & Dietterich , 2014 ; Zhang & Yu , 2015 ; Zhang et al. , 2016 ) . Among these methods , only several of them ( Yao et al. , 2020 ; Wen et al. , 2021 ) can be compatible with deep neural networks , which achieved state-of-the-art performance . Most of the existing PLL methods aim at designing proper training objectives under various assumptions on the collected data . For example , the consistent methods proposed by Feng et al . ( 2020 ) are based on a specific data generation assumption . The derivation of the PRODEN method ( Lv et al. , 2020 ) stems from the assumption that the true label achieves the minimal loss among the candidate labels . Yan & Guo ( 2021 ) proposed the MGPLL method based on the assumption of non-random label noise . Although these methods have achieved generally great empirical performance , their performance could be degraded when the collected data can not meet the adopted assumptions . In this paper , we aim to investigate a novel PLL method by exploiting the learned intrinsic representation of the model to identify the ground truth in the training process without relying on any assumptions on the collected data . We focus on the class activation map ( CAM ) ( Zhou et al. , 2016 ) , which is a prevailing tool for visualizing the representation information of convolutional neural network ( CNN ) -based models and could be easily obtained by the weighted linear sum of the feature maps . As CAM is able to discriminate the learning patterns of each class for the model , we conjecture that CAM could be used to guide the model to spot the true label . To verify this assumption , we conducted a pilot experiment on CIFAR-10 ( Krizhevsky et al. , 2009 ) , where the candidate label sets were generated in two different ways . As shown in Figure 1 ( please refer to Section 2.2 ) , it is surprising to find that CAM can potentially help the model recognize the true labels , and such capacity of CAM synchronously changes with the classifier performance during the whole training phase . Based on the experimental results , it can be conjectured that such intrinsic representations can be useful to distinguish the true label from candidate labels for PLL . Motivated by the above observations , we naturally consider leveraging CAM to solve the PLL problem . However , CAM was only proposed for addressing image datasets with using deep models built on CNN , revealing two limitations on its application . Firstly , CAM can not be adopted to the classifier based on shallow models such as the linear model and multilayer perceptron ( MLP ) . Secondly , CAM is unable to deal with any inputs other than the image . To overcome these shortcomings , we for the first time propose a simple but effective tool—the class activation value ( CAV ) to capture the learned representation information in a more general way , which is essentially the weighted output of the classifier . Experimental results in Figure 2 and Appendix B.7 also shows that our CAV works similarly as CAM during the training phase . Building upon CAV , we propose a simple yet effective PLL method called CAV Learning ( CAVL ) that guides the model to differentiate the true label from the candidate set during the training process . Specifically , we first train the model by treating all candidate labels equally and obtain CAVs of given training examples , and then regard the class with the maximum CAV as the true label for each instance during the training process . In this way , CAVL transforms PLL into supervised learning , thereby making the model reliable to recognize the true label with learned representation information by CAV . Extensive experiments on benchmark-simulated and real-world datasets show that our proposed CAVL method achieves state-of-the-art performance . 2 DISCOVERING CLASS ACTIVATION MAP FOR PARTIAL-LABEL LEARNING . In this section , we provide a detailed discussion about our motivation and empirically show that the class activation map could be helpful for addressing PLL . 2.1 CLASS ACTIVATION MAP . The class activation map ( CAM ) ( Zhou et al. , 2016 ) is a popular and elementary mechanism in computer vision to represent the discriminative part of an input image captured by the classifier to identify each class . In other words , CAM manifests the learning patterns of the classifier , denoted as f , to the specific class in the image . Let us denote an input image as x ∈ Rd=c×h×w ( with c channels , height h and width w ) and CAM of x ∈ Rd=c×h×w as m ∈ Rk×h×w where k is the number of classes . We note thatm could be generated by training a specific classification model fc , which normally comprises of an encoder function e : X → Rcf×h×w for feature map extraction , a Global Average-Pooling layer gap : Rcf×h×w → Rcf×1 , and a linear layer with weight θ ∈ Rk×Cf , where cf is the number of channels in the feature maps . Therefore , CAM of x related to the j-th class could be obtained by mj = ∑Cf i=1 ( θj ) ie i ( x ) , j ∈ { 1 , . . . , k } , ( 1 ) where θj ∈ R1×Cf is the corresponding linear weight related to the j-th object . CAM is a weighted linear sum of feature maps and the linear weights . The weakness of CAM is the limited applications , since not all classification network architectures follow the model fc mentioned above . To improve its generalization for any CNN-based model f , Gradient-weighted CAM ( Grad-CAM ) ( Selvaraju et al. , 2017 ) was proposed to implement such an internal representation by reasonably leveraging class-specific gradient information . Concretely , let us denote by gj ∈ Rd=c×h×w the Grad-CAM of x related to the j-th object , and gj could be expressed as gj = 1 h× w ∑cf z=1 ∑h m=1 ∑w n=1 ∂f j ( x ) ∂ez ( x ) m , n ez ( x ) . ( 2 ) Intuitively , the derivative of logit f ( x ) with respect to feature map e ( x ) is used as the weights for calculating g. Note that both Grad-CAM and CAM are generated by weighted sums of the feature maps , and Grad-CAM is equivalent to CAM when f follows the same architecture as fc . Based on Grad-CAM , Grad-CAM++ ( Chattopadhay et al. , 2018 ) aims to provide better visual explanations of CNN and occurrences of multiple foreground objects . Thanks to such a simple technique , numerous problems in computer vision such as interpretation ( Xu et al. , 2019a ) and weakly supervised semantic segmentation ( Wei et al. , 2016 ) have achieved marvelous progress . In this paper , we aim to extract and improve the useful knowledge from CAM to address the PLL problem , leading the classifier to identify the ground truth . 2.2 PILOT EXPERIMENT ON CAM . Generally , Grad-CAM or CAM is treated as the internal representation of f . We believe that such a mechanism could guide f to differentiate the true label from the candidate set because it is constructed by taking advantage of internal elements in f . To validate our conjecture , we conducted a pilot experiment by adopting two different label selection methods . The first one is the intuitive method ( IM ) , which simply regards all the labels from the candidate sets as the true labels with using cross entropy loss . The second one is the “ CAM label ” selection strategy , which discriminates the true label from the candidate label set by using CAM . During each training epoch , we calculated the number of the foreground seeds of CAM from each candidate label set . Specifically , to calculate the number of foreground seeds of mj , we counted the elements with positive values in mj . Note that each x could possess k CAMs and the true label is always in the candidate set . Thus we selected the label from the candidate set , whose CAM owns most foreground seeds as the true label . The label candidate label sets were generated by two different approaches : ( I ) Uniformly Sampling Strategy ( USS ) . Uniformly sampling the candidate label set for each training instance from all the possible candidate label sets ( Feng et al. , 2020 ) . ( II ) Flipping Probability Strategy ( FPS ) . By setting a flipping probability q to any false label , the false label could be selected as a candidate label with a probability q ( Feng & An , 2019a ; Yan & Guo , 2020 ; Lv et al. , 2020 ; Wen et al. , 2021 ) . Here the detailed training settings of the experiments are similar to the experiment part ( please refer to Section 4.1.1 for details ) . ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) were chosen as the backbones to train a classifier on CIFAR-10 ( Krizhevsky et al. , 2009 ) . Figure 1 shows the average results of 5-time trials and we can find that CAM demonstrates two helpful features during the training phase . Firstly , it is clear that the performance of the classifier ( a ) ResNet + USS ( b ) DenseNet + FPS Methods in ( a ) are implemented by ResNet with using USS to generate the candidate set . Methods in ( b ) are conducted by DenseNet with the candidate label sets produced by FPS . For three lines representing accuracy performance , the results drawn by the blue dashed line and red line ( “ CAM ” line ) are obtained by using the “ CAM label ” selection strategy , namely training the model with regarding the class with the most number of activated seeds in its CAM as the true label . Specifically , the red line depicts the training accuracy measured by selecting the CAM with the maximum activated seeds from the candidate set as the true label , and the blue line simply depicts the test accuracy of the classifier . The yellow dashed line ( “ IM ” line ) depicts the test accuracy of the classifier trained by IM . The paired circle indicates the similar accuracy fluctuation between the classifier and CAM , which shows the dynamic attribute . The double arrow marks the performance gap between “ CAM ” label selection strategy and IM , which represents the power attribute . Figure 1 : Comparison of accuracy performance of two different label selection strategies . trained by “ CAM label ” selection strategy is better than IM ( the gap between the blue dashed line and the yellow dashed line ) , which shows that CAM could potentially guide the classifier toward the true label from the candidate set . Here we name this attribute of CAM as power . Secondly , it is found that the accuracy of identifying true labels by CAM is synchronously changed with that by the classifier . Several fluctuations ( marked by the paired circle ) of the classifier performance ( test accuracy ) and CAM accuracy are similar to a large extent , resulting in continuous improvement to the model itself . Here we name this observed feature of CAM as dynamic . The power and dynamic attributes disclose the fact that CAM may show more than we thought , which is an inspiring finding to address PLL . The power attribute motivates us to consider that the classifier may learn more accurate true labels if it is forced to approximate the label distribution recognized by CAM . The dynamic attribute may guarantee that such guidance would be constantly effective during the whole training phase since CAM would synchronously be self-improved as the classifier becomes stronger . Hence it would be reasonable and meaningful to explore such internal sign as guidance for PLL .

Summary: This paper intends to exploit the learned representation of the model to tackle partial-label learning tasks. The paper begins by a PILOT experiment to show that the class activation map is better at selecting the true label from the candidate set than the model output itself. To overcome the limitation that CAM cannot be applied to linear model and non-image data, they propose class activation value (CAV) to replace CAM and show similar properties.