Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization

√ c ) as shown in prior work . This may shed light on the superior performance of pairwise methods in practice , where real datasets are usually highly imbalanced . Moreover , we present an inconsistent reweighted univariate loss-based algorithm that enjoys an error bound of O ( √ c ) for promising performance as well as the computational efficiency of univariate losses . Finally , experimental results confirm our theoretical findings . 1 Introduction . Multi-Label Classification ( MLC ) [ 1 ] is an important task , in which each instance is associated with multiple labels simultaneously . It has been widely applied to text categorization [ 2 ] , bioinformatics [ 3 ] , multimedia annotation [ 4 ] , information retrieval [ 5 ] . Various measures [ 6 , 7 ] have been developed to evaluate MLC ’ s performance from diverse aspects owing to its complexity . Among them , the ranking loss ( RL ) ( or partial ranking loss , PRL ) [ 2 , 8 ] is a commonly used measure in practice ( or in theory ) . Formally , the RL calculates the fraction of pairs that a positive label does not precede a negative label according to the rank given by a score function ( or predictor ) . Minimizing such a loss is usually referred to as Multi-Label Ranking ( MLR ) [ 9 ] , which is the consideration in this paper . Since RL is non-convex and discontinuous , existing methods [ 6 ] seek to optimize certain convex surrogate losses for computational efficiency . These surrogate losses can be divided into two main categories : pairwise ones [ 8 ] and univariate ones [ 9 ] , which have their own advantages and limitations in terms of computational costs , theory and empirical performance . ∗Equal contribution . G. Wu is now at School of Software , Shandong University and C. Li is now at Gaoling School of AI , Renmin University of China . The work was done when they were at Tsinghua University . †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Computationally , the pairwise losses , defined over pairs of positive and negative labels , have a complexity depending on O ( c2 ) ( c is the number of labels ) , while the univariate ones enjoy a complexity depending on O ( c ) . The superiority of the latter is significant when c is large . Theoretically , the pairwise losses are not ( Fisher ) consistent w.r.t . the RL or the PRL [ 8 ] , while , remarkably , certain univariate ones are consistent w.r.t . the PRL [ 9 , 8 ] . Empirically , however , the consistent univariate losses usually have no significant superiority compared to the inconsistent pairwise losses [ 9 ] . In fact , we observed that the former under-perform the latter on 10 MLR benchmarks ( see results in Table 3 ) . Towards filling the gap between the existing theory and practice , this paper attempts to rigorously answer the following questions : Why inconsistent pairwise losses usually achieve better performance than consistent univariate losses in practice ? How to improve the univariate loss , which is preferable due to its computational efficiency ? A natural explanation of this gap is that although the ( Fisher ) consistency [ 10 , 11 ] provides valuable insights in the asymptotic cases , it can not fully characterize the behaviour of a surrogate loss when the number of training samples is not sufficiently large and the hypothesis space is not realizable . Therefore , this paper presents a systematic study in a complementary perspective of generalization error bounds [ 12 ] besides the consistency . We theoretically find two key factors of the distribution ( or dataset ) that affect the learning guarantees of algorithms : the instance-wise class imbalance and the label number c. Given extremely imbalanced data ( i.e. , the worst case ) , we prove that the consistent univariate losses based algorithms lead to an error bound ofO ( c ) while the pairwise losses based ones enjoy an error bound depending on O ( √ c ) [ 13 ] , which explains the empirical behaviour better on real datasets which are usually highly imbalanced ( see Table 3 ) . Further , we present two reweighted surrogate univariate losses that employ carefully designed penalties for positive and negative labels . Then , we analyze their consistency and generalization bounds of the corresponding algorithms . Surprisingly , though not consistent , one of them enjoys an error bound depending on O ( √ c ) in the worst case , which is nearly the same as the pairwise losses , and retains the computational efficiency . For balanced data ( i.e. , the best case ) , we also find that all these surrogate univariate losses share the same learning guarantees , with no dependence on c which is the same as the surrogate pairwise ones . Notably , in this case , these bounds are different from the classical probably approximately correct ( PAC ) ones [ 14 , 12 , 15 ] which hold for all distributions ( i.e . the worst case ) . Our main theoretical results are summarized in Table 1 . Experimental results also confirm our theoretical findings . 2 Preliminaries . In this section , we first introduce the problem setting of MLC and MLR . Then , we present the evaluation measures , risk , and regret of MLR . Notations . Let boldface lower and capital case letters denote vectors ( e.g. , a ) and matrices ( e.g. , A ) respectively . For a matrix A , ai , aj and aij denote its i-th row , j-th column , and ( i , j ) -th element respectively . For a vector a , ai denote its i-th element . For a set , | · | denotes the cardinality . [ [ π ] ] denotes the indicator function , i.e. , [ [ π ] ] = 1 if the proposition π holds and 0 otherwise . 2.1 Problem Setting . Let x ∈ X ⊂ Rd and y ∈ Y ⊂ { −1 , +1 } c denote the input and output respectively , where d is the feature dimension , c is the label size . yj = 1 ( or−1 ) indicates that the associated j-th label is relevant ( or irrelevant ) . Given a training set S = { ( xi , yi ) } ni=1 of n samples i.i.d . drawn from a distribution P over X × Y , the original goal of MLC is to learn a multi-label classifier H : Rd → { −1 , +1 } c. To solve MLC , a common approach is to first learn a vector-based score function ( or predictor ) f = [ f1 , ... , fc ] : Rd → Rc and then get the classifier H by a thresholding function . Multi-Label Ranking ( MLR ) aims to learn the best predictor from the finite training data in terms of some ranking-based measures , which is our consideration in this paper . 2.2 Evaluation Measures . To evaluate the performance of different approaches for MLR , many measures have been developed . Here we focus on two widely-used measures in practice ( or theory ) , which are defined below.3 Ranking Loss ( RL ) : L0/1r ( f ( x ) , y ) = 1 |S+y ||S−y | ∑ ( p , q ) ∈S+y ×S−y [ [ fp ( x ) ≤ fq ( x ) ] ] , ( 1 ) where S+y ( or S − y ) denotes the relevant ( or irrelevant ) label index set induced by y . Partial Ranking Loss ( PRL ) :4 L0/1pr ( f ( x ) , y ) = 1 |S+y ||S−y | ∑ ( p , q ) ∈S+y ×S−y [ [ [ fp ( x ) < fq ( x ) ] ] + 1 2 [ [ fp ( x ) = fq ( x ) ] ] ] . ( 2 ) Note that the only difference between these two measures is the penalty when fp ( x ) = fq ( x ) holds . Besides , it is easy to verify that RL upper bounds PRL , i.e . L0/1pr ( f ( x ) , y ) ≤ L0/1r ( f ( x ) , y ) . Although these two measures are almost the same in practice for the evaluation of one algorithm , they have different consistency properties for some surrogate losses theoretically [ 8 ] . 2.3 Risk and Regret . Since RL is non-convex and discontinuous , often leading to NP-hard problems [ 16 ] , it is optimized with convex surrogates in practice for computational efficiency . Define a surrogate loss Lφ : Rc × { −1 , +1 } c → R+ , where φ indicates the specific surrogate loss and will be detailed in the next section . Besides , define a vector-based predictor class F = { f : X → Rc } . For a predictor f ∈ F , its true ( 0/1 ) expected risk , surrogate expected risk , and surrogate empirical risk are defined as : R0/1 ( f ) = E ( x , y ) ∼P [ L0/1 ( f ( x ) , y ) ] , Rφ ( f ) = E ( x , y ) ∼P [ Lφ ( f ( x ) , y ) ] , R̂S ( f ) = 1 n n∑ i=1 Lφ ( f ( xi ) , yi ) . Besides , we use a superscript ( i.e. , pr or r ) to distinguish the risks for specific measures . Further , for convenience , we denote the expected risk conditioned on an instance x ( i.e. , the conditional risk ) as : R ( f |x ) = E y∼P ( y|x ) [ L ( f ( x ) , y ) |x ] = ∑ y L ( f ( x ) , y ) P ( y|x ) , ( 3 ) where L denotes the true or surrogate loss . Thus , the expected risk of f is R ( f ) = Ex∼P ( x ) [ R ( f |x ) ] . For each x , given the conditional distribution P ( y|x ) , we can get its optimal predictions as follows:5 f∗ ( x ) = arg min a∈Rc ∑ y L ( a , y ) P ( y|x ) , ( 4 ) where f∗ is called the Bayes predictor w.r.t . the loss L. Besides , the expected risk of f∗ ( i.e. , R ( f∗ ) ) is called the Bayes risk , which is the minimal expected risk w.r.t . the loss L and denoted by R∗ for 3Our definition is over one sample and can be averaged over multiple samples . 4Minimizing the PRL is equivalent to maximize the instance-AUC . 5Notably , the optimal predictions can be not just one value but a set with many elements that share the same minimal conditional risk . convenience . Then , we can define the regret ( a.k.a . excess risk ) of a predictor f w.r.t . the true and surrogate loss as follows . Reg0/1 ( f ) = R0/1 ( f ) −R∗0/1 , Regφ ( f ) = Rφ ( f ) −R ∗ φ . ( 5 ) Besides , we also use a superscript ( i.e. , pr or r ) to distinguish the regrets for specific measures . Moreover , let f̂n denote the learned predictor from finite training data S. Note that our goal is to find a predictor f̂n that achieves the minimal true regret ( i.e . Reg0/1 ( f̂n ) ) as possible as it can .

Summary: This paper analyzes theoretically the consistency and the generalization error of multilabel ranking algorithms. More specifically, some generalization error bounds are obtained for different loss functions, that are surrogates of the ranking loss or the partial ranking loss.   The analysis is restricted to two extreme cases: (1) instance-wise class balanced distribution (when the number of positive and negative classes is the same for all samples), and (2) extremely imbalanced distribution (when only one class is positive or only one class is negative, for all samples).  It turns out that, while the generalization error bounds for the balanced case are independent on the number of classes, $c$, the  bounds for the imbalanced case are $O(c)$ for some univariate losses (like $L_{u_2}$)  while it is $O(\sqrt{c})$ for pairwise losses (like $L_{pa}$).   Since, in practice, real datasets are highly imbalanced, the theoretical results might explain why $L_{pa}$ shows a better performance than $L_{pa}$ in the experiments with finite datasets, even thought only the latter is consistent.  Based on these findings, the paper proposes some re-weighting versions of the univariate losses with $O(\sqrt{c})$ generalization bounds. The consistency is lost, but some of the reweighting version of $L_{u_2}$ (in particular, $L_{u_3}$) outperform $L_{u_2}$  in the experiments.