Multi-Label Learning with Pairwise Relevance Ordering

1 Introduction . Multi-label learning ( MLL ) solves problems where each object is assigned with multiple class labels simultaneously [ Zhang and Zhou , 2013 ] . For example , an image may be annotated with labels building , street and person . The goal of multi-label learning is to train a classification model that can predict all the relevant labels for unseen instances . A large number of recent works have witnessed the great successes that MLL has achieved in many real-world applications , e.g. , image annotation [ Chen et al. , 2019 ] , human attribute recognition [ Li et al. , 2016 ] , user profiling [ Liu et al. , 2021 ] , and protein function prediction [ Elisseeff and Weston , 2002 ] . Traditional multi-label learning studies assume that each instance has been precisely annotated with all of its relevant labels . However , in many real-world scenarios , it is difficult and costly to collect the precise annotations . Instead , each instance may be provided with the relative order of label pairs , where each label pair y y′ ( or y ≺ y′ ) indicates that label y is more relevant ( or irrelevant ) than label y′ to instance x , i.e. , p ( y = 1|x ) > p ( y′ = 1|x ) ( or p ( y = 1|x ) < p ( y′ = 1|x ) ) . Generally , deciding the relative order of label pairs would be much easier than collecting the precise annotations and thus less costly . For example , in medical image analysis , only experts with rich experiences can accurately identify the disease for a patient based on the medical image . In contrast , if the question is to decide which of two given diseases is more likely suffered by the patient , then even medical students with basic knowledge may easily provide the answer . While the annotation cost is significantly reduced with pairwise relevance ordering , the learning task becomes more challenging , since the supervised information of pairwise relevance ordering is much less than exact labels . We formalize this learning problem as a new framework called multi-label learning with pairwise relevance ordering ( PRO ) . Specifically , PRO attempts to learn a classification model from multi-label ∗Correspondence to : Sheng-Jun Huang ( huangsj @ nuaa.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . examples with the relative order of label pairs , where each label pair y y′ indicates three possible cases : 1 ) both y and y′ are relevant to x , i.e. , y = 1 , y′ = 1 ; 2 ) y is relevant to x while y′ is not , i.e. , y = 1 , y′ = 0 ; 3 ) both y and y′ are irrelevant to x , i.e. , y = 0 , y′ = 0 . PRO is a novel learning framework with significant difference from exiting settings . For example , semi-supervised multi-label learning ( SSMLL ) learns a classifier by exploiting a few of labeled examples as well as a large number of unlabeled examples [ Liu et al. , 2006 ] ; multi-label learning with missing labels ( MLML ) assumes that only a subset of labels are available for each instance [ Sun et al. , 2010 , Yu et al. , 2014 ] ; partial multi-label learning ( PML ) assigns each instance with a candidate label set [ Xie and Huang , 2018 , Zhang and Fang , 2020 ] ; multi-label learning with noisy labels assumes that multiple class labels may be flipped simultaneously with their respective probabilities [ Xie and Huang , 2021a ] . However , these frameworks do not consider multi-label examples with pairwise relevance ordering , and can not be employed to solve PRO problems . To deal with multi-label data with pairwise relevance ordering , we propose a cost-sensitive loss function for learning a multi-label classifier with empirical risk minimization . Theoretically , we show that the unbiased estimator of classification risk can be derived from only PRO examples if the surrogate loss function satisfies a mild condition , i.e. , the symmetric condition . The estimation error bound is established for the unbiased estimator , showing that learning with PRO examples can be multi-label consistent to the commonly used ranking loss . Extensive experimental results on multiple datasets and evaluation metrics demonstrate the practical usefulness of the proposed method . 2 Related Works . There are plenty of literature on multi-label learning . As one of the earliest representative methods , Binary Relevance simply decomposes the multi-label learning task into a set of binary classification problems [ Zhang and Zhou , 2013 ] . Nevertheless , such method neglects the label correlations , which are regarded as an essential information for multi-label classification . Therefore , there are many studies trying to learn a multi-label classifier by exploiting the label correlations [ Read et al. , 2011 ] . Some of these works focus on the pairwise correlation [ Elisseeff and Weston , 2002 , Li et al. , 2017 ] , while some others consider the high order correlation among all labels [ Chen et al. , 2019 ] . To solve SSMLL problems , some works attempt to learn a multi-label classifier based on the graph models [ Kong et al. , 2011 ] while some others utilize the low-rank assumption [ Jing et al. , 2015 ] . In addition to these methods , the co-training strategy [ Zhan and Zhang , 2017 ] and matrix factorization [ Liu et al. , 2006 ] are also employed to solve SSMLL problems . The pioneering MLML study [ Sun et al. , 2010 ] tries to construct a similarity graph for each label and the manifold regularization term is added to recover the missing labels . A linear classifier with the low-rank constraint is proposed to deal with large scale data with missing labels [ Yu et al. , 2014 ] . In [ Kanehira and Harada , 2016 ] , authors solve the MLML problem by viewing it as a positive-unlabeled learning task . Some other techniques also employ the robust loss [ Xu et al. , 2019 ] or the group lasso regularizer [ Bucak et al. , 2011 ] to solve MLML problems . In order to deal with partial-labeled data , the most commonly used strategy is disambiguation [ Cour et al. , 2011 ] , which recovers ground-truth labeling information for candidate labels . Some methods perform the disambiguation strategy by estimating a confidence for each candidate label [ Xie and Huang , 2018 , Yu et al. , 2018 , 2020 ] . Other methods utilize the decomposition scheme [ Sun et al. , 2019 ] or adversarial training [ Yan and Guo , 2021 ] . In [ Xie and Huang , 2021b ] , authors first solve PML problems by considering the generation process of noisy labels . A recent work utilizes the meta disambiguation strategy to deal with partial-labeled data [ Xie et al. , 2021 ] . Although it allows noisy labels hidden in the candidate set ( e.g. , PML ) or some labels are missed ( e.g. , SSMLL and MLML ) , the aforementioned frameworks consider the supervised information of the label-level , i.e. , each of labels is relevant or not , which can be still costly . Instead our proposed PRO framework considers pairwise relevance ordering , which can be much easier for annotators and thus less costly . In [ Huang et al. , 2015 ] , authors propose a multi-label active learning framework called AURO which queries the relevance ordering between two labels in every iteration . However , different from the proposed PRO framework , AURO directly asks annotators to provide each label pair with one of three possible cases that have been discussed in Section 1 , which can be regarded as a stronger supervised information with higher cost . AURO can not be used to solve the PRO problem . Furthermore , authors consider the pairwise supervision of instances and performing binary classification based on similar paired data and unlabeled examples [ Bao et al. , 2018 , Shimada et al. , 2021 , Bao et al. , 2020 ] . Another recent work trains the binary classifier under the supervision of pairwise confidence comparisons [ Feng et al. , 2021 ] . These methods focus on binary classification and can not be directly applied to solve the PRO problem . 3 Preliminaries . In this section , before deriving our main results for solving the PRO problem , we introduce some notations and provide the necessary preliminaries . In traditional multi-label learning task , let x ∈ X be a feature vector and y ⊆ Y its corresponding labels , where X = Rd is the feature space and Y = { 0 , 1 } q is the target space with q possible class labels . Here , yj = 1 indicates the j-th label is relevant to the instance ; yj = 0 , otherwise . Let D = { xi , yi } ni=1 be the given training examples , where each example is drawn i.i.d . according to the joint distribution p ( x , y ) . In multi-label learning , many loss functions have been proposed to measure the performance of learning algorithms , such as ranking loss , hamming loss , coverage and average precision [ Zhang and Zhou , 2013 ] . Among them , the ranking loss concerns about label pairs that are ordered reversely for an instance , which naturally considers the pairwise label correlation . Given the decision function f : X → Rd , the ranking loss can be defined as follows : L ( f ( x ) , y ) = ∑ 1≤j < k≤q I ( yj = 1 , yk = 0 ) ` ( fk , fj ) + I ( yj = 0 , yk = 1 ) ` ( fj , fk ) , ( 1 ) where ` ( fj , fk ) = I ( fj > fk ) + 1 2 I ( fj = fk ) . ( 2 ) Here , fj is the j-th component of f ( x ) and I ( · ) is the indicator function , which outputs 1 if the condition holds while outputs 0 otherwise . The goal of multi-label learning tasks is to learn the optimal classifier f by minimizing the following expected classification risk : R ( f ) = Ep ( x , y ) [ L ( f ( x ) , y ) ) ] = ∑ y∈Y p ( y ) Ep ( x|y ) [ L ( f ( x ) , y ) ) ] , ( 3 ) = ∑ y∈Y p ( y ) ∑ 1≤j < k≤q Ep ( x|yj=1 , yk=0 ) [ ` ( fk , fj ) ] + Ep ( x|yj=0 , yk=1 ) [ ` ( fj , fk ) ] , = ∑ 1≤j < k≤q π10jkEp10jk ( x ) [ ` ( fk , fj ) ] + π 01 jkEp01jk ( x ) [ ` ( fj , fk ) ] , where π10jk = p ( yj = 1 , yk = 0 ) ( or π 01 jk = p ( yj = 0 , yk = 1 ) ) denotes the positive-negative ( or negative-positive ) label pair prior probability , and p10jk ( x ) = p ( x|yj = 1 , yk = 0 ) ( or p01jk ( x ) = p ( x|yj = 0 , yk = 1 ) ) denotes the class-conditional probability density of data given the positivenegative ( negative-positive ) label pair . Accordingly , we define the minimal risk ( also called the Bayes risk ) as R∗ = inff R ( f ) . However , the loss function L is highly discontinuous and computationally NP-hard , which often makes the corresponding optimization problem hard to solve [ Gao and Zhou , 2013 ] . In practice , a feasible solution is to consider alternatively a surrogate loss function L which can be solved efficiently . Accordingly , the L-risk with respect to p ( x , y ) can be defined as : RL ( f ) = Ep ( x , y ) [ L ( f ( x ) , y ) ) ] = ∑ 1≤j < k≤q π10jkEp10jk ( x ) [ φ ( fj−fk ) ] +π 01 jkEp01jk ( x ) [ φ ( fk−fj ) ] , ( 4 ) where φ is a surrogate loss function . A common choice is hinge loss φ ( t ) = max ( 0 , 1 − t ) in [ Elisseeff and Weston , 2002 ] . Accordingly , we define the minimal L-risk ( also called the Bayes L-risk ) as R∗L = inff RL ( f ) .

Summary: This paper attempts to solve the multi-label learning problem with only pairwise relevance ordering. Authors argue that deciding the relative order of label pairs is obviously less laborious than collecting exact labels. A method is also presented to learn multi-label model with only pairwise relevance ordering and experiments shows its superiority.