Improving Tail Label Prediction for Extreme Multi-label Learning

Extreme multi-label learning ( XML ) works to annotate objects with relevant labels from an extremely large label set . Many previous methods treat labels uniformly such that the learned model tends to perform better on head labels , while the performance is severely deteriorated for tail labels . However , it is often desirable to predict more tail labels in many real-world applications . To alleviate this problem , in this work , we show theoretical and experimental evidence for the inferior performance of representative XML methods on tail labels . Our finding is that the norm of label classifier weights typically follows a long-tailed distribution similar to the label frequency , which results in the over-suppression of tail labels . Base on this new finding , we present two new modules : ( 1 ) RANKNET learns to re-rank the predictions by optimizing a population-aware loss , which predicts tail labels with high rank ; ( 2 ) TAUG augments tail labels via a decoupled learning scheme , which can yield more balanced classification boundary . We conduct experiments on commonly used XML benchmarks with hundreds of thousands of labels , showing that the proposed methods improve the performance of many state-of-the-art XML models by a considerable margin ( 6 % performance gain with respect to PSP @ 1 on average ) . 1 INTRODUCTION . Extreme multi-label learning ( XML ) aims to annotate objects with relevant labels from an extremely large candidate label set . Recently , XML has demonstrated its broad applications . For example , in webpage categorization Partalas et al . ( 2015 ) , millions of labels ( categories ) are collected in Wikipedia and one wishes to annotate new webpages with relevant labels from a huge candidate set ; in recommender systems McAuley et al . ( 2015 ) , one hopes to make informative personalized recommendations from millions of items . Because of the high dimensionality of label space , classic multi-label learning algorithms , such as Zhang & Zhou ( 2007 ) ; Tsoumakas & Vlahavas ( 2007 ) , become infeasible . To this end , a number of computational efficient XML approaches are proposed Weston et al . ( 2011 ) ; Agrawal et al . ( 2013 ) ; Bi & Kwok ( 2013 ) ; Yu et al . ( 2014 ) ; Bhatia et al . ( 2015 ) ; E.-H . Yen et al . ( 2016 ) ; Yeh et al . ( 2017 ) ; Yen et al . ( 2017 ) ; Tagami ( 2017 ) . In XML , one important statistical characteristic is that labels follow a long-tailed distribution as illustrated in Figure 4 ( left ) . Most labels occur only a few times in the dataset . Infrequently occurring labels ( referred to as tail label ) possess limited training samples and are harder to predict than frequently occurring ones ( referred to as head label ) . Many existing XML approaches treat labels with equal importance , such as Prabhu & Varma ( 2014 ) ; Babbar & Schölkopf ( 2017 ) ; Khandagale et al . ( 2019 ) , while Wei & Li ( 2018 ) demonstrates that most predictions of well-established methods are heads labels . However , in many real-world applications , it is still desirable to predict more tail labels which are more rewarding and informative , such as recommender systems Jain et al . ( 2016 ) ; Babbar & Schölkopf ( 2019 ) ; Wei & Li ( 2018 ) ; Wei et al . ( 2019 ) . To improve the performance for tail labels , existing solutions typically involve optimizing loss functions that are suitable for tail labels Jain et al . ( 2016 ) ; Babbar & Schölkopf ( 2019 ) , leveraging the sparsity of tail labels in the annotated label matrix Xu et al . ( 2016 ) , and transferring knowledge from data-rich head labels to data-scarce tail labels K. Dahiya ( 2019 ) . These methods typically achieve better performance on tail labels than standard XML methods which treat labels equally , while they usually involve high computational costs . Moreover , previous studies do not explicitly explain the underlying cause of the inferior performance of many standard XML methods for tail labels . In this work , we disclose theoretical and experimental evidence for the inferior performance of previous XML methods on tail labels . Our finding is that the norm of label classifier weights follows a long-tailed distribution similar to the label frequency as shown in Figure 4 ( middle ) , and the prediction score of tail labels thereby is underrated . To alleviate this problem , we propose to rectify the classifier ’ s outputs and training data distribution such that the prediction of tail labels is enhanced . We present two general modules suitable for any well-established XML methods : ( 1 ) RANKNET learns to re-rank the predictions by optimizing a population-aware loss function , which predicts tail labels with high rank ; ( 2 ) TAUG augments tail labels via a decoupled learning scheme , which reduces the skewness of training data and yields more balanced classification boundary . We conduct experiments to verify the effectiveness of the aforementioned instantiations . From our extensive studies across four benchmark datasets , we make the following intriguing contributions : • We show that from both theoretical and experimental perspectives , the norm of label classifier weights follow a long-tailed distribution , i.e. , the norms of head label classifier weights are considerably larger than that of tail label classifiers , which is a key cause of the inferior performance of many XML methods on tail labels . • We propose two general modules : RANKNET for prediction score re-ranking by optimizing a new population-aware loss , and TAUG for decoupled tail label augmentation . Both methods can be paired with any XML model without changing the model . • Experiments verify that our proposed modules achieve significant improvements ( 6 % w.r.t . PSP @ 1 on average ) for well-established XML methods on benchmark datasets . • We provide an ablation study to highlight the effectiveness of each individual factor . 2 PREVIOUS EFFORTS . Existing work on XML can be roughly categorized as three directions : One-vs-all methods . This branch of work trains classifiers for each label separately . Due to the huge size of label set , parallelization Babbar & Schölkopf ( 2017 ) , label partitioning Khandagale et al . ( 2019 ) , and label filter Niculescu-Mizil & Abbasnejad ( 2017 ) techniques are used to facilitate efficient training and testing . To alleviate memory overhead , recent works restrict the model capacity by imposing sparse constraints E.-H . Yen et al . ( 2016 ) or removing spurious parameters Babbar & Schölkopf ( 2017 ) . One criticism of one-vs-all methods is that it fails to capture label correlations . Embedding-based methods . Along this direction , researchers have proposed to embed the feature space and label space onto a joint low-dimensional space , then model the correlation between features and labels in hidden space Tai & Lin ( 2012 ) ; Chen & Lin ( 2012 ) ; Yu et al . ( 2014 ) ; Bhatia et al . ( 2015 ) ; Tagami ( 2017 ) ; Evron et al . ( 2018 ) . This method can dramatically reduce the model parameters compared with the one-vs-all methods , but involves solving complex optimization problems . Tree-based methods . In comparison to other types of approaches , tree-based methods greatly reduce inference time , which generally scales logarithmically in the number of labels . There are typically two types of trees including instance trees Prabhu & Varma ( 2014 ) ; Siblini et al . ( 2018 ) and label trees Daume III et al . ( 2016 ) ; You et al . ( 2018 ) , depending whether instance or label is partitioned in tree nodes . Tree-based methods usually suffer from low prediction accuracy affected by the cascading effect , where the prediction error at the top can not be corrected at a lower level . These methods can readily scale up to problems with hundreds of thousands of labels . However , Wei & Li ( 2018 ; 2019 ) claims that head labels make a significantly higher contribution to the performance than tail labels . Therefore , many work are conducted to improve the performance for tail labels . Optimization . Jain et al . ( 2016 ) proposes propensity scored loss functions that promote the prediction of tail label with high ranks . Xu et al . ( 2016 ) decomposes the label matrix into a low-rank matrix and a sparse matrix . The low-rank matrix is expected to capture label correlations , and the sparse matrix is used to capture tail labels . Babbar & Schölkopf ( 2019 ) views tail label from an adversarial perspective and optimizes hamming loss to yield a robust model . Knowledge transfer . K. Dahiya ( 2019 ) trains two deep models on head labels and tail labels . The semantic representations learned from head labels are transferred to the tail label model . These methods achieve better performance on tail labels than standard XML methods which treat labels equally , while they do not explicitly explain the underlying cause of the inferior performance of many standard XML methods for tail labels . In this work , we find that the classification boundary of existing XML methods is skewed to head labels , causing the inferior performance . 3 METHODOLOGY . In XML , as we possess fewer data about tail labels , models learned on long-tailed datasets tend to exhibit inferior performance on tail labels Wei & Li ( 2018 ) . However in practice , it is more informative and rewarding to accurately predict tail labels than head labels Jain et al . ( 2016 ) . In this work , we attempt to alleviate this problem from the perspective of the classification boundary . We make an observation that the norm of label classifier weights follow a long-tailed distribution similar to the label frequency , which means that the prediction of tail labels is over-suppressed . This finding provides an evidence for us to improve the prediction of tail labels . We present ways of rectifying the classifier ’ s outputs and data distribution via re-ranking and tail label augmentation , respectively . Notations . We first describe notations used through the paper . Let X = { xi } Ni=1 , Y = { yi } Ni=1 be a training set of size N , where yi is the label vector for data point xi . Formally , XML is the task of learning a function f that maps an input ( or instance ) x ∈ RD to its target y ∈ { 0 , 1 } L. We denote nj = ∑N i=1 yij as the frequency of the j-th label . Without loss of generality , we assume that the labels are sorted by cardinality in non-increasing order , i.e. , if j < k , then nj ≥ nk , where 1 ≤ j , k ≤ L. In our setting , we have n1 nL . According to the label frequency , we can split the label set into head labels and tail labels by a threshold τ ∈ ( 0 , 1 ) . We denote head label set H = { 1 , . . . , bτLc } and tail label set T = { bτLc+ 1 , . . . , L } . τ is a user-specified parameter . 3.1 THE LONG-TAILED DISTRIBUTION OF CLASSIFIER WEIGHTS NORM . We present a different perspective regarding XML model , showing its inferior performance on tail labels is due to the imbalanced classification boundary . In Figure 4 ( middle ) , we empirically observe that the norm of label classifier weights follows a similar long-tailed distribution as the label frequency . The results are produced on EUR-Lex dataset using a representative one-vs-all method Bonsai Khandagale et al . ( 2019 ) . A similar observation on Wiki10-31K dataset is presented in the supplementary material . Since the norm of tail label classifier weights is considerably smaller than that of head label classifier weights , the predicted score of tail labels are typically underestimated in inference . We further support our finding theoretically and demonstrate the fact that the small norm of tail label classifier weights is the root cause of inferior performance . We make the following mild assumption on the data : every input x is sampled from feature space completely at random , and there exists a constant threshold t > 0 for the input x , such that the top-k prediction for x is made as β ( k ) = { yl | P̂ ( yl | x ) ≥ t , 1 ≤ l ≤ L } , where P̂ ( yl | x ) denotes the estimated label distribution . We assume W = { wj } Lj=1 be the weight matrix of a standard XML method . In particular , for binary relevance and tree-based classifier , W can be obtained by optimizing Eq . ( 1 ) , where L denotes the loss function , e.g. , squared hinge loss , and constant λ is a trade-off parameter . Note that for some tree-based methods , such as Bonsai Khandagale et al . ( 2019 ) and Parabel Prabhu et al . ( 2018 ) , we consider W be the label classifier weights in leaf nodes , i.e. , excluding meta-labels of internal tree nodes . min wj ‖wj‖22 + λ N∑ i=1 L ( Yi , j , wTj xi ) , ∀1 ≤ j ≤ L ( 1 ) For deep learning methods , we denote W be the weights of the last linear layer for classification by optimizing Eq . ( 2 ) , where σ is the softmax function , fθ is the feature extractor parameterized by θ , and L denotes the selected loss function , e.g. , binary cross entropy . Note that this interpretation can also be adapted to typical embedding-based methods , such as Yu et al . ( 2014 ) , where fθ is linear and σ is the identity function . min W N∑ i=1 L ( yi , σ ( W > fθ ( xi ) ) ) ( 2 ) With the above setup , we summarize our findings in Theorem 1 . Theorem 1 . Let D = { ( xi , yi ) } Ni=1 be a sample set and W , which can be decomposed as { wj } Lj=1 , be the label classifier weights learned on D by optimizing Eq . ( 1 ) and Eq . ( 2 ) . For an uniformly sampled point x which is i.i.d . with points in D , we have ||wj || ∝ E [ yj ∈ β ( k ) ] , ∀1 ≤ j ≤ L , where β ( k ) denotes the k top-ranked indices of predicted labels in P̂ ( y | x ) . This theorem shows that the need for re-balancing the classifier weights to improve the performance on tail labels . Motivated by our finding , in the following we propose two new modules and discuss their effectiveness on tail labels . Proof of this theorem can be found in the supplementary material .

Summary: In prediction problems with millions of labels also known as Extreme Multi-label Learning (XML) problems, e.g., recommender systems, the model predictions are not as good for the tail (rarer) labels. This paper proposes two models for this problem. The first model is re-ranking-based, that is, it reranks the prediction scores of a standard XML model. The second model tries to augment the rarer labels to reduce the skew in data. Results shown on several real-world datasets highlight the superior predictive ability of the proposed reranking model for tail labels compared to a host of competitive baselines.