Understanding Partial Multi-Label Learning via Mutual Information

1 Introduction . Partial multi-label learning ( PML ) [ 1 , 2 ] is a weakly supervised learning problem , where each instance is associated with a set of candidate labels , but only a part of them are the ground-truth labels while others are false positive labels . In recent years , many real-world applications are arising due to the growing demand for identifying ground-truth labels from partially labeled data , which are easier and less costly to obtain . For example , in crowdsourcing image annotation , a web image might be annotated online by a potential unreliable annotator with many specific labels , but only some of them are accurate . PML aims to train a classifier from partially labeled data so as to predict the ground-truth labels for an unseen instance automatically . The main challenge is how to deal with the ambiguities caused by false positive labels in candidate label set . One straightforward way is to simply treat all candidate labels equally as the ground-truth labels , and then solve the PML problem by standard multi-label classification methods [ 3 , 4 ] . However , these methods can be easily misled by the noisy false-positive labels in the candidate set , and fail to generalize well in testing . To deal with the problem , the state-of-the-art PML research attempts to identify the ground-truth labels directly from the candidate label set , which becomes a popular and effective disambiguation strategy . However , there are two fundamental issues , “ Can the ground-truth labels be identified precisely ? '' . If yes , “ How can the ground-truth labels be found ? '' . Moreover , the state-of-the-art PML methods are constructed by hand-designed heuristic modeling under considerations like feature/label correlations with no theoretical interpretation . In this paper , instead of hand-made heuristic modeling , we propose a novel MILI-PML model , which is derived from a clear probabilistic formulation and can be easily interpreted theoretically by information theory . Besides , MILI-PML naturally incorporates the feature/label relevancy considerations rather than that of heuristic modeling . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To answer the first question “ CAN '' , we define an -identifiable score function to evaluate whether a PML method is workable for identifying the ground-truth labels precisely . Moreover , we demonstrate that the derived MILI-PML is -identifiable under certain assumptions and conditions theoretically . To answer the second question “ HOW '' , we design an efficient optimization algorithm called MILI Algorithm to optimize the objective function . Extensive experiments are conducted on six synthetic datasets as well as four real-world datasets . The experimental results demonstrate that the proposed MILI-PML consistently outperforms the state-of-the-art methods in terms of five widely-used multi-label metrics including ranking loss , hamming loss , one error , coverage , and average precision . 2 Related Work . Partial multi-label learning ( PML ) [ 5 , 6 ] is different from multi-label learning [ 7 , 8 , 9 ] or partial label learning [ 10 , 11 , 12 , 13 ] . In PML , each instance is associated with a set of candidate labels , which contains multiple ground-truth labels and the others are false positive labels . The state-of-the-art PML studies attempt to identify the ground-truth labels directly from the candidate label set . [ 14 ] propose PARTICLE to extract credible labels with high confidence via propagation matrix and use the identified labels to train multi-label classifiers . [ 15 ] propose DRAMA to get the reliable labels with high confidence by employing the feature manifold , and then use the identified labels to train the multi-label classifier . Moreover , [ 1 ] propose PML-lc and PML-fp to optimize the label ranking confidence matrix in training classifiers which considers the label correlations and the feature prototype respectively . [ 16 ] develop fPML to optimize the label confidentce matrix by considering feature and label correlations . [ 17 ] propose PML-LRS to get label ranking which utilizes the low-rank and sparse decomposition to train classifiers while considering the feature and label interdependencies . [ 18 ] develop MUSER to train classifiers by decreasing the feature noise and label redundancy via mapping with orthogonality constraint and graph Laplacian regularization , which considers the feature correlation and label correlation simultaneously . However , existing methods have two common issues . First , models are hand-made heuristic with no theoretical interpretation . Second , existing partial multi-label learning ( PML ) methods assume that the ground-truth labels exist in the candidate label set and identify them directly . We then have the question : “ Can the ground-truth labels be identified precisely ? '' . If yes , “ How can the ground-truth labels be found ? `` . 3 The Model Derivation . In this section , we derive our method , i.e. , MILI-PML , from the following probabilistic formulation . Assume S = { ( xi , yi ) : i = 1 , . . . , n } are drawn i.i.d . n times from a probability distribution P . Each observation is a pairwise example ( x , y ) , consisting of a d-dimensional feature vector x = [ x1 , . . . , xd ] T , and a q-dimensional candidate label vector y = [ y1 , . . . , yq ] T , drawn from the underlying random variables X = [ X1 , . . . , Xd ] T and Y = [ Y1 , . . . , Yq ] T . We adopt a q-dimensional binary vector v : a 1 indicating the label is selected , a 0 indicating it is discarded . Notation yv indicates the vector of selected labels , that is , the full vector y projected onto the dimensions specified by v. Notation yv̄ is the complement , i.e. , the vector of remaining candidate labels . The full label vector y can therefore be expressed as y = { yv , yv̄ } , which means y is the concatenation of yv and yv̄ . In PML , yv is selected from y . Assume that Q is a hypothetical predictive model with two parameters : v representing which labels are selected and w representing parameter used for prediction . In order to identify the ground-truth labels from the candidate label set for PML , we can maximize the conditional likelihood of training dataset with respect to parameters { v , w } . The conditional likelihood given all training examples S = { ( xi , yi ) : i = 1 , . . . , n } can be expressed as follows : L ( v , w|S ) = n∏ i=1 Q ( yiv|xi , w ) ( 1 ) For convenience , we use the conditional log-likelihood corresponding to Eq . ( 1 ) as follows : ` ( v , w|S ) = 1 n n∑ i=1 logQ ( yiv|xi , w ) ( 2 ) By multiplying and dividing classifier Q by the distribution of identified ground-truth labels given features , i.e. , P ( yv|x ) , we can re-write the above as follows : ` ( v , w|S ) = 1 n n∑ i=1 log Q ( yiv|xi , w ) P ( yiv|xi ) + 1 n n∑ i=1 logP ( yiv|xi ) ( 3 ) Similarly , we can expand the second term in Eq . ( 3 ) by multiplying and dividing the probability P ( y|x ) , and get the following formulation : ` ( v , w|S ) = 1 n n∑ i=1 log Q ( yiv|xi , w ) P ( yiv|xi ) + 1 n n∑ i=1 log P ( yiv|xi ) P ( yi|xi ) + 1 n n∑ i=1 logP ( yi|xi ) ( 4 ) Furthermore , we use E ( X , Y ) to denote the expectation of the random variables ( X , Y ) . When n→ +∞ , Eq . ( 4 ) can be reformated as follows : ` ( v , w|S ) =E ( X , Y ) { log Q ( yv|x , w ) P ( yv|x ) } +E ( X , Y ) { log P ( yv|x ) P ( y|x ) } + E ( X , Y ) { logP ( y|x ) } ( 5 ) The first term of Eq . ( 5 ) equals to the following KL-divergence : E ( X , Y ) { log Q ( yv|x , w ) P ( yv|x ) } = −DKL ( P ( Yv|X ) ||Q ( Yv|X , w ) ) ( 6 ) As y = { yv , yv̄ } , the second term of Eq . ( 5 ) can be derived into the following formulation : E ( X , Y ) { log P ( yv|x ) P ( y|x ) } = −E ( X , Y ) { log P ( y|x ) P ( yv|x ) } =− ∑ ( x , y ) P ( x , y ) log P ( yv̄ , x|yv ) P ( x|yv ) =− ∑ ( x , y ) P ( x , y ) log P ( yv̄ , x|yv ) P ( x|yv ) P ( yv̄|yv ) − ∑ ( yv̄ , yv ) P ( yv̄ , yv ) logP ( yv̄|yv ) = −I ( Yv̄ , X|Yv ) +H ( Yv̄|Yv ) = −I ( Yv̄ , X|Yv ) ( 7 ) In Eq . ( 7 ) , I ( Yv̄ , X|Yv ) is the conditional mutual information and H ( Yv̄|Yv ) is the conditional entropy . The last equality holds as a result of the conditional entropy H ( Yv̄|Yv ) = 0 . This is because in PML setting , Yv is selected from Y , once Yv is known , then Y is of course known in advance , meanwhile , Yv̄ is complementary to Yv , thus , the uncertainty remaining in Yv̄ is zero , i.e. , H ( Yv̄|Yv ) = 0 . The detailed derivation of Eq . ( 7 ) can be found in the supplementary materials . Moreover , the third term of Eq . ( 5 ) equals to the conditional entropy −H ( Y|X ) , i.e. , 1 n n∑ i=1 logP ( yi|xi ) = −H ( Y|X ) ( 8 ) Combining together , we get the derived objective function as follows : ` ( v , w|S ) =−DKL ( P ( Yv|X ) ||Q ( Yv|X , w ) ) − I ( Yv̄ , X|Yv ) −H ( Y|X ) ( 9 ) The first term is a KL-divergence between the true and predicted ground-truth label distributions given features . The value of this term depends on how well the model Q can approximate P . The second term is the conditional mutual information between the remaining candidate labels and the features , given the identified ground-truth labels . The value of this term depends solely on the choice of ground-truth labels , and will decrease as the selected ground-truth label set Yv explains more about features X , until eventually becoming zero when the remaining candidate labels Yv̄ are all false positive labels , which contain no additional information about X . The last term is the conditional entropy of all candidate labels given features . This term quantifies the uncertainty remaining in the candidate labels after we know all features of training data , which is independent of parameters v and w as a constant .

Summary: This paper presents two fundamental questions about the assumption of existing PML research. The authors derive a novel PML method from a probabilistic formulation which meets the $\epsilon$-identifiable definition. They provide theoretical analysis and algorithms to verify the proposed two questions. Thorough comparative experiments validate the superiorities of the proposed method.