Venue: NAACL 2019

# A Submodular Feature-Aware Framework for Label Subset Selection in Extreme Classification Problems

Elham J. Barezi1,2,3, Ian D. Wood 1, Pascale Fung 1,2,4, Hamid R. Rabiee 3

###### Abstract

Extreme classification is a classification task on an extremely large number of labels (tags). User generated labels for any type of online data can be sparing per individual user but intractably large among all users. It would be useful to automatically select a smaller, standard set of labels to represent the whole label set. We can then solve efficiently the problem of multi-label learning with an intracatably large number of interdependent labels, such as automatic tagging of Wikipedia pages. We propose a submodular maximization framework with linear cost to find informative labels which are most relevant to other labels yet least redundant with each other. A simple prediction model can then be trained on this label subset. Our framework includes both label-label and label-feature dependencies, which aims to find the labels with the most representation and prediction ability. In addition, to avoid information loss, we extract and predict outlier labels with weak dependency on other labels. We apply our model to four standard natural language data sets including Bibsonomy entries with users assigned tags, web pages with user assigned tags, legal texts with EUROVOC descriptors(A topic hierarchy with almost 4000 categories regarding different aspects of European law) and Wikipedia pages with tags from social bookmarking as well as news videos for automated label detection from a lexicon of semantic concepts. Experimental results show that our proposed approach improves label prediction quality, in terms of precision and nDCG, by 3% to 5% in three of the 5 tasks and is competitive in the others, even with a simple linear prediction model. An ablation study shows how different data sets benefit from different aspects of our model, with all aspects contributing substantially to at least one data set.

+
Footnote †: _Proceedings of NAACL-HLT 2019_, pages 1009–1018 Minneapolis, Minnesota, June 2 - June 7, 2019. ©2019 Association for Computational Linguistics

## 1 Introduction

Multi-label learning has recently attracted attention in the research community due to an increase in applications such as semantic labeling of images and videos, bio-informatics, genetic functions, and music categorization. In addition, multi-label learning can address machine learning problems in web data mining, including recommender systems, multimedia sharing websites, and ranking Zhang and Zhang (2010).

An important application of extreme multi-label learning is automatic tagging and social tagging of large information collections such as Wikipedia or the Web. A user can add their own keywords to a text, as if they were the keywords they would use to look for the article in a search engine. Since tags use an open vocabulary, the number of tags is increasing continually in order to adjust to the needs of new information. Moreover, different users can assign different tags to the same resource, resulting in a great diversity of tags for that resource.

The biggest challenge of extreme multi-label learning is the dimension of the output space. As the number of output labels increases, the number of output states increases exponentially. In order to overcome this exponential growth, it is necessary to use label dependencies to simplify the problem Zhang and Zhang (2010); Tsoumakas et al. (2010).

We propose a submodular maximization approach with a linear cost to find an informative set of labels. In contrast to the other similar approaches Balasubramanian and Lebanon (2012); Bi and Kwok (2013) which consider only label-label dependencies, we also consider label-feature dependencies and outlier labels that are highly independent of other labels. Solving the problem using the selected (smaller number of) labels leads to minimizing both representation and training error.



Representation ability is equivalent to the power of the selected subset to reconstruct the remaining labels, and prediction ability is equivalent to training accuracy leading to less error propagation from predicted label subset to the remaining labels during reconstruction.

Submodular maximization approaches have proved very effective in many applications, such as finding the most influential nodes in social networks to maximize the spread of information (for applications such as advertising and marketing Kempe et al. (2003); Ohsaka et al. (2014)) and video and image collection summarization Gygli et al. (2015); Tschiatschek et al. (2014). There are many effective algorithms such as Mirzasoleiman et al. (2015) to make submodular optimization approaches much faster or do them in a distributed way Mirrokni and Zadimoghaddam (2015) to perform faster parallel processing for very large scale datasets.

## 2 Related Work

Many of the early proposed multi-label learning approaches struggle with large-scale applications, as they learn each label separately or investigate the label dependencies in a way that leads to a costly and complicated model Tsoumakas et al. (2010).

The other research trends is to transform the label space to a smaller space and map back the predicted results in the compressed space to the original space. Hsu et al. (2009) presented the first approach targeting label space compression based on compressed sensing, which assumes sparsity of the label space. An expensive optimisation problem has to be solved in the prediction step. Tai and Lin (2012); Chen and Lin (2012); Yu et al. (2014), and Lin et al. (2014) used orthogonal projections and low-rank assumptions to extract a label matrix decomposition and find a low-dimensional embedding space. In Bhatia et al. (2015b), the authors perform local embedding of the label vectors. To achieve stronger locality, they cluster the data into smaller regions, which is unstable and costly for high-dimensional spaces and one needs an ensemble of the learners to overcome this instability and achieve a good prediction accuracy.

Although the previously proposed approaches make the embedding space smaller and more tractable, they may lead to loss of information as a result of transforming the label space to lower-dimensional spaces. Many of these approaches rely on low-rank assumptions which transform the sparse label space to a new dense embedding space resulting in even lower accuracy, with a higher prediction cost in the new complicated space Bhatia et al. (2015a).

Balasubramanian and Lebanon (2012) and Bi and Kwok (2013) proposed to select a subset of the labels, and solve the problem in the original label space, based on structure sparsity optimization and SVD decomposition, correspondingly. However, these methods are not tractable for large scale data and not compatible for the real application data. In addition, they have ignored the training error in the label selection step which can lead to selection of the labels that are hard to predict resulting in training error propagation through the next steps.

Another recent thread of research includes the methods that partition the data into smaller groups: In the framework proposed by Barezi et al. (2017), the label space is divided into smaller independent groups, while Agrawal et al. (2013); Prabhu and Varma (2014); Prabhu et al. (2018) propose tree-based methods which partition the data into tree-structured hierarchical groups. These partitioning-based approaches avoid information loss from dimension reduction. However, finding a partitioning tree is a very complicated and time-consuming problem and these approaches require solving a complicated optimization problem to perform partitioning at each node, which is expensive and needs many training samples. In addition, the tree-based approaches suffer from error propagation through the hierarchy and need many training samples to avoid under-fitting in the lower levels of the partitioning tree Liu et al. (2005).

Instead of making the structural assumption on the relation between the labels, Yen et al. (2016) assume the label space is highly sparse and has a strong correlation with the feature space. They ignore the label space correlation information. Yen et al. (2017) proposed the parallel version of Yen et al. (2016).

## 3 Methodology

In this paper, we propose a landmark selection framework for selecting the most informative labels and to solve the multi-label learning problem with these labels. As an example, consider predicting the commercial impact of a new event on 

some global organizations (equivalent to the labels in our problem) given a history of the impact of previous events (equivalent to the features and training data in our problem). Instead of predicting the impact on each organization individually, we predict only the impact on a small number of organizations which are both easier to predict and analyze according to available data as well as being more indicative of the economy and the other organizations. Being indicative means that if we know the impact of the new event on these organizations, it can help us to predict the reaction of the other organizations. More formally, we optimize the above set function \(f(S)\) in Equation 1.

The proposed method includes both label-label and label-feature dependencies in order to minimize both representation and training error. Previous similar methods ignore label-feature dependencies in the subset selection step, allowing the training error for the selected subset of the labels to be propagated to the reconstructed labels and affecting the final predictions. In addition, to avoid information loss, we also extract and predict outlier labels with weak dependency on other labels and treat them separately.

Our construction results in a monotone submodular function of label sets allowing us to use a maximization framework that benefits from a good theoretical bound by a fast greedy approach with linear cost Nemhauser et al. (1978). We use a method based on Alternating Direction Method of Multipliers (ADMM) Boyd (2011) optimization to learn a linear mapping back to original label space. Therefore, during training, we can select and learn the most informative label subset using a submodular maximization framework of linear cost. During the prediction time, we can use the selected subset to represent the remaining labels using a linear equation with a linear cost in number of the labels.

### Overview of the Submodular Maximization Theorem

Submodular functions have a natural diminishing property which makes them suitable for many applications. A submodular function is a set function with the property that as the size of the selected subset increases, the incremental value of the function by adding a new element to the selected subset does not increase.

The formal definition of a submodular function is as follows:

**Definition 1**.: _For a set function \(f(S):2^{V}\to R\) defined for a finite ground set \(V=1,2,...n\), the marginal gain of adding each new member can be computed as \(\Delta_{f}(e|S)=f(e\cup S)-f(S)\). The function \(f(.)\) is submodular, if for each \(A\subseteq B\subseteq V\), \(e\in V\backslash A\cap V\backslash B\), then \(\Delta_{f}(e|A)\geq\Delta_{f}(e|B)\). Equivalently, the function \(f(S):2^{V}\to R\) is submodular if for any two sets \(A,~{}B\in V\), \(f(A\cup B)+f(A\cap B)\leq f(A)+f(B)\)._

Monotony of sunmodular functions is a useful property which means that the value of the function would not decrease by adding each new member to the input set, and can be defined as following.

**Definition 2**.: _A submodular function \(f(.)\) is monotone (non-decreasing) if for every \(T\subseteq S\), we have that \(f(T)\leq f(S)\)._

A simple example of a submodular function is the setup cost in a factory. Suppose that a factory is capable of making any one of a large finite set \(V\) of products. In order to produce product \(e\in V\), it is necessary to set up the machines needed to manufacture \(e\), and this costs money. The setup cost is non-linear, and it depends on which other products you choose to produce. For example, if you are already producing iPhones, then the setup cost for also producing iPads is small, but if you are not producing iPhones, the setup cost for producing iPads is large.

We can find a good approximation of the optimum answer for a monotone submodular maximization problems by using the greedy approach and considering the selected subset size constraint. More formally:

**Theorem 3**.: _Nemhauser et al. (1978) For a nonnegative, monotone submodular function \(f\), let \(S\) be a set of size \(k\) obtained by the greedy strategy similar to Algorithm 1. Then, \(f(S)\geq(1-1/e)f(S^{*})\), where \(S^{*}\) is the optimum solution, and \(e\) is Euler's constant approximately equal to \(2.71828\)_

### Submodularity for Label Subset Selection

We propose two submodular functions, aiming to select the most informative subset of the labels. The first function is a penalized version of the graph cut function. It scores label sets with correlation to the other labels and penalizes their similarity to the previously selected labels (\(f_{pen}\) in 

Equation 3). The graph is constructed using the labels as nodes and label correlations as weights for the graph edges. The second function scores the predictability of labels with respect to problem input features (\(f_{score}\) in Equation 5). Our final function for identifying the optimal subset of labels is a weighted sum of these (Equation 6).

We consider the label correlations as graph weights \(w\). The graph cut function \(f_{cut}(.)\) aims to find a subset of the graph nodes (labels) with the highest weights (strongest dependencies) to the remaining nodes (labels). This captures strong correlation of a label set to the other labels and thus its ability to reconstruct the other labels. The penalised version \(f_{pen}(.)\) adds one more term to increase the diversity of the selected labels and avoid choosing similar labels.

\[f_{cut}(S)=\sum_{i\in V\setminus S}\sum_{j\in S}w_{i,j} \tag{2}\]

\[f_{pen}(S)=f_{cut}(S)-\lambda\sum_{\begin{subarray}{c}i,j\in S\\ i\neq j\end{subarray}}w_{i,j}~{},~{}\lambda\geq 0 \tag{3}\]

**Theorem 4**.: \(f_{cut}(S)\) _is a submodular function and it is monotone for non-large values of \(|S|\)Nemhauser et al. (1978)._

**Theorem 5**.: \(f_{pen}(S)\) _is a submodular function and it is monotone for non-large values of \(\lambda\)Lin et al. (2009)._

The proofs for Theorem 4 and 5 is provided in supplementary Section.

It is important also to consider predictability, which is the training error for the selected subset of the labels, in order to avoid the prediction error of labels with high training error being propagated to the whole label space.

As an estimate of predictability we use either a \(G^{2}\) or \(\chi^{2}\) independence test for the discrete data, and Fishers \(Z\) or \(t\) test for the continuous data in order to reject or accept the null hypothesis of independence Tsamardinos and Borboudakis (2010). Since, this measures include an implicit normalization, the frequency of the classes in training data does not affect the sampling step.

A higher dependency score for each label and the input feature space means a stronger correlation of the label with the feature space and higher predictability. Given label predictability scores \(f_{ij}\) for label \(i\) and input feature \(j\) and \(D\) input features, we calculate dependency scores \(f_{i}\) of the \(i\)-th label and the input features:

\[f_{i}=\sum_{j=1}^{D}f_{ij} \tag{4}\]

Note that \(f_{i}\geq 0\). We then define the following set function, which also is monotone and submodular (Theorem 6):

\[f_{score}(S)=\sum_{i\in S}f_{i} \tag{5}\]

**Theorem 6**.: \(f_{score}(S)\) _is a submodular monotone function._

Proof.: For \(w_{i}=\) the sum over the dependency scores of the \(i\)-th label and the feature space, \(f(S)=\sum_{i\in S}w_{i}\) is a linear function with \(w_{i}\geq 0\). Any linear function of the form \(f(S)=\sum_{i\in S}w_{i}\) is a submodular function. If \(S\subset R,~{}\Delta_{f}(k|S)-\Delta_{f}(k|R)=0\Rightarrow~{}\Delta_{f}(k|S) \geq\Delta_{f}(k|R)\). Additionally, if \(\forall i\,w_{i}\geq 0\), then \(f\) is monotone, because \(f(S\cup k)-f(S)=w_{k},~{}w_{k}\geq 0\).

\(max_{|S|=k}f(S)=max\sum_{i\in S}w_{i}\). Therefore \(f(S)\) is a monotone submodular function. 

Since, any sum of submodular functions with positive coefficients is a submodular function, we can combine \(f_{pen}(.),~{}\text{and}~{}f_{score}(.)\) by positi 

weights, which results in a new submodular function that includes both representation ability and prediction ability of the selected labels. We choose a model parameter \(\gamma>0\) giving us our final sub-modular function:

\[f(S)=f_{pen}(S)+\gamma.f_{score}(S) \tag{6}\]

### Landmark Information Propagation

The main step of our proposed framework is to propagate the predicted value for the selected label subset to the full set of labels in order to recover the original space. Therefore, we aim to find a linear relation including the dependency of the selected labels and all the other labels. In the prediction step, this linear function obtains the full label set by combining the subset (\(Y_{s}\)) and outlier predictions (\(E\)) predicted by the regression functions discussed in next section 3.4. Given 1-hot representations \(Y_{s}\) over the reduced set of labels and \(Y\) the full set of labels, we seek matrices \(Z\) and \(E\) that recover the original labels:

\[Y=Y_{s}Z+E \tag{7}\]

To find optimal \(Z\) and \(E\), we solve the optimization problem Equation 8, where \(Y\) and \(Y_{s}\) are matrices populated with our training data. Note that \(\|E\|_{2,1}\) is the \(L_{1}\) norm of the \(L_{2}\) norms of the columns of \(E\).

\[\underset{Z,E}{\text{argmin}} (\|Z\|_{1}+\alpha\|E\|_{2,1})\] s.t. \[Y=Y_{s}Z+E\]

The sparse matrix \(Z\) is a \(k\times L\) matrix which includes a few representative labels (due to the sparsity constraint \(\|Z\|_{1}\)) for each label (\(Y=Y_{k}Z\)). The \(Z\) matrix includes the dependency information and performs propagation of the predicted label subset to the full label set, while nonzero columns of matrix \(E\) show the outlier and tail labels set \(O\), which cannot be computed perfectly through their relation to the other labels. The index set of the nonzero columns of matrix \(E\) indicates the outlier labels. \(\alpha\) is a model parameter.

The alternating direction method of multipliers (ADMM) method Boyd (2011); Nesterov (2004); Beck and Teboulle (2009) provides an efficient algorithm for solving this problem, achieving a convergence rate of \(O(1/T^{2})\) (where \(T\) is the number of iterations). ADMM solves the problem with more than one unknown variable, (\(Z\) and \(E\) in our case), by alternating between optimizing each variable using augmented Lagrangian. Please see the supplementary materials for more detail on the ADMM method and how it is applied in this case.

### Prediction and Mapping Back to the Original Label Space

We now train a linear classifier to predict labels in the reduced label set \(S\cup O\) and map back to the full label set. Given features of the training data \(X\), corresponding labels from selected and outlier labels \(Y_{S}\) and \(Y_{O}\), we learn linear regression parameters \(w_{s},~{}b_{s}\) for the selected labels and \(w_{e},~{}b_{e}\) for the outlier labels:

\[\underset{w_{s},b_{s}}{\text{argmin}} \|Y_{s}-(X*w_{s}+b_{s})\|+\tfrac{\lambda_{1}}{2}\|w_{s}\|^{2}\] \[\underset{w_{e},b_{e}}{\text{argmin}} \|Y_{o}-(X*w_{e}+b_{e})\|+\tfrac{\lambda_{2}}{2}\|w_{e}\|^{2} \tag{9}\]

Since all these training tasks are independent of each other, this step is highly parallelizable. The final values for the labels are computed by propagating the selected label subset through the linear relation 7:

\[\hat{Y}_{s} = X*w_{s}+b_{s}\] \[\hat{E} = X*w_{e}+b_{e} \tag{10}\] \[\hat{Y} = \hat{Y}_{s}Z+\hat{E} \tag{11}\]

An overview of steps for training and prediction are shown in Algorithms 2 and 3.

**Input**: Training Data \(X\) and \(Y\).

```
1: Find the best label subset by submodular optimization over function 6;
2: Find the linear propagation equation through ADMM optimization over problem 8.
3: Find the linear regression models over small subset of labels and outliers by Equation 9 Output**: Label subset, outliers, propagation and regression models.
```

**Algorithm 2** Training Algorithm.

## 4 Experiments

### Datasets

We used six different datasets in the experiments. The "Bibtex" dataset is a text dataset extracted from the Bibsonomy website Katakis et al. (2008) 

contains metadata for the bibtex items like the title of the paper, the authors, etc and extracts the features according to the term frequency. The "Mediamill" dataset is extracted from the Mediamill contest datasets, which include low-level multimedia features (visual and textual features) extracted from 85 hours of international news videos from the TRECVID 2005/2006 benchmark datasets Snoek et al. (2006) labeled using a lexicon of 101 semantic concepts, like commercials, nature, and baseball.

The "Eurlex" dataset includes 19,348 legal documents from European nations, containing several different types of documents, including treaties, legislation, case-law and legislative proposals, classified according to the EUROVOC descriptor using 3993 different classes, and 5000 features extracted using common TF-IDF term weighting Mencia and Furnkranz (2008). The "Delicious" dataset is a text dataset extracted from the del.icio.us social bookmarking site on the 1st of April 2007 and contains textual data of web pages along with their user defined tags Tsoumakas et al. (2008). The content of web pages was represented using the Boolean bag-of-words model. "Wiki10-31K" is a collection of social tags for given Wikipedia pages with TF-IDF features Zubiaga (2012). The statistics of these datasets are provided in Table 1.

### Experimental Setup

For the small datasets, "Bibtex", "Mediamill", "Delicious", and "Eurlex", the reported results are the average of 10 different experiments for random partitions of each dataset. For the larger dataset, "Wiki10-31K", we did one experiment with the training and testing partition reported in Table 1.

For all experiments we chose a label subset size of \(100\), except for Mediamill where we chose \(30\) since \(100\) would represent all labels. Model tuning is done in two phases: first we tune \(\alpha\) for group sparsity (Equation 8), and \(\gamma\) for weighting of the submodular functions (Equation 6), then we tune for \(\lambda_{1}\) and \(\lambda_{2}\), the regression parameters for mapping back to the original label set (Equation 9) with \(\alpha\) and \(\gamma\) fixed. All parameters were chosen by measuring the precision of 10-fold cross validation and using a grid search over the values \(\{0,10^{-3,...,+3}\}\) for each dataset.

The proposed method was compared with several state-of-the-art methods with diverse approaches. LEML Yu et al. (2014), CPLST Chen Lin (2012), CS Hsu et al. (2009) and SLEEC Bhatia et al. (2015b) which are embedding based approaches with a low-rank or sparse assumption in the label space. ML-SSP Bi and Kwok (2013) which solves the problem in the original label space which ignores the training error in the subset selection step. FastXML Prabhu and Varma (2014), and PD-sparse Yen et al. (2016) which do not use an embedding transformation and aim to solve the problem without using compression or sampling. We have used the reported results, if available, and otherwise tuned the parameters for the baseline algorithms by means of 10-fold cross validation.

## 5 Results and Discussion

Table 2 shows the average and standard deviation of Precision@k for the four small-scale datasets, "Bitex", "Medianill", "Delicious", and "Eurlex", and the large-scale dataset "Wiki10-31k". For "Wiki10-31k", results are reported only for those baselines that were tractable. The results for nDCG@k are included in supplementary Material, Table 5. Since the SLEEC and FastXML methods are ensemble-based, using multiple nonlinear models, it is not fair to compare them with the single model methods such as our own. These methods partition the sample space into smaller tractable clusters and obtain separate classifiers for each partition. We compare our method with these in Table 3.

The proposed approach in most cases has significantly better results than other methods on both measures. The embedding based approaches suffer from accumulation of the embedding and training error Balasubramanian and Lebanon (2012), however in the proposed approach, we have removed the embedding step and considered the training error minimization at the label subset selection step. On the other hand, the non-embedding approaches such as PD-sparse Yen et al. (2016) ignore the label space inter 

dependency information which can be useful to improve the prediction accuracy for the labels which are not easy to predict only from input features.

ML-CSSP (Bi and Kwok, 2013) and the work of Balasubramanian and Lebanon (2012) attempt, like us, to find the most informative labels in order to perform label subset selection. However, our approach improves on their results, supporting the idea that considering only the label space information (ignoring label-feature dependency information) in the label selection step can lead to label sets that are not easy to predict whose training error will be propagated through to final model predictions.

The SLEEC and FastXML methods are ensemble-based methods using multiple nonlinear models and can be expected to outperform single model methods such as ours. SLEEC aims to partition the sample space into smaller tractable clusters to obtain a nonlinear embedding and trained model for each partition. FastXML finds a partitioning tree by using nonlinear binary classifiers to partition the samples at each node, which is a very complicated and unstable problem for high-dimensional spaces. Therefore, for both SLEEC and FastXML methods, they need an ensemble of the learners in order to overcome this instability and achieve a good prediction accuracy. Table 3 shows that SLEEC performs best on the Medim-amil and FastXML performs best on the Delicious dataset. This shows that finding a representative subset using a linear method is not a consistent assumption for these datasets than the low-rank and tree-based assumptions. However, for Bibtex dataset, our proposed method is competitive with the best results, and for Eurlex and Wiki10-31k, our method is substantially better than both SLEEC and FastXML, a notable achievement for

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline  & **Proposed** & PD-sparse & LEML & CPIST & CS & ML-CSSP \\ \hline \multicolumn{5}{l|}{Bibtex} \\ \hline P@1 & **64.56\(\pm\)**0.79 & 61.29\(\pm\)0.65 & 62.54\(\pm\)0.52 & 62.38\(\pm\)0.63 & 58.87\(\pm\)0.61 & 44.98\(\pm\)1.15 \\ \cline{2-5} P@3 & **39.51\(\pm\)**0.34 & 35.82\(\pm\)0.46 & 38.41\(\pm\)0.42 & 37.84\(\pm\)0.48 & 33.53\(\pm\)0.49 & 30.43\(\pm\)0.59 \\ \cline{2-5} P@5 & **28.80\(\pm\)**0.26 & 25.74\(\pm\)0.30 & 28.21\(\pm\)0.24 & 27.62\(\pm\)0.27 & 23.72\(\pm\)0.29 & 23.53\(\pm\)0.37 \\ \cline{2-5}  & \multicolumn{5}{l|}{Dickions} \\ \hline p@1 & _65.13\(\pm\)_0.39 & 51.82\(\pm\)1.40 & **65.67\(\pm\)**0.73 & 63.57\(\pm\)0.88 & 61.36\(\pm\)0.38 & 63.04\(\pm\)1.28 \\ \cline{2-5} P@3 & 59.07\(\pm\)0.41 & 44.18\(\pm\)1.04 & **60.55\(\pm\)**0.48 & 59.95\(\pm\)0.43 & 56.46\(\pm\)0.33 & 56.26\(\pm\)1.13 \\ \cline{2-5} P@5 & 54.52\(\pm\)0.34 & 38.95\(\pm\)0.94 & **56.68\(\pm\)**0.43 & 55.31\(\pm\)0.50 & 52.07\(\pm\)0.30 & 50.16\(\pm\)0.83 \\ \cline{2-5}  & \multicolumn{5}{l|}{Medium} \\ \hline P@1 & **84.25\(\pm\)**0.27 & 81.86\(\pm\)4.08 & _84.07\(\pm\)_0.31 & 83.35\(\pm\)0.33 & _81.82\(\pm\)_5.92 & 78.89\(\pm\)0.23 \\ \cline{2-5} P@3 & 67.25\(\pm\)0.24 & 62.52\(\pm\)2.31 & 67.20\(\pm\)0.23 & 66.18\(\pm\)0.22 & 67.83\(\pm\)0.42 & 60.93\(\pm\)0.24 \\ \cline{2-5} P@5 & **52.90\(\pm\)**0.15 & 41.11\(\pm\)1.51 & 52.80\(\pm\)0.18 & 51.46\(\pm\)0.20 & 52.80\(\pm\)2.61 & 44.27\(\pm\)0.20 \\ \cline{2-5}  & \multicolumn{5}{l|}{Eurles} \\ \hline P@1 & **81.04\(\pm\)**0.81 & 76.43\(\pm\)1.04 & 63.40\(\pm\)1.58 & 72.28\(\pm\)0.99 & 58.52\(\pm\)1.06 & 62.09\(\pm\)2.12 \\ \cline{2-5} P@3 & **67.91\(\pm\)**0.97 & 60.37\(\pm\)0.74 & 50.35\(\pm\)1.44 & 58.16\(\pm\)1.11 & 45.51\(\pm\)0.71 & 48.39\(\pm\)1.31 \\ \cline{2-5} P@5 & **56.81\(\pm\)**0.97 & 49.72\(\pm\)0.74 & 41.28\(\pm\)1.07 & 47.73\(\pm\)0.97 & 32.47\(\pm\)0.58 & 40.11\(\pm\)1.10 \\ \cline{2-5}  & \multicolumn{5}{l|}{Wiki0.3lik} \\ \hline p@1 & **86.05** & 82.14 & 73.47 & - & - & - \\ \cline{2-5} P@3 & **76.85** & 69.68 & 62.43 & - & - & - \\ \cline{2-5} P@5 & **67.77** & 58.76 & 54.35 & - & - & - \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics

\begin{table}
\begin{tabular}{c|c|c|c} \hline  & **Proposed** & SLEEC & FastXML \\ \hline P@1 & _64.56\(\pm\)_0.79 & **65.08\(\pm\)**0.65 & 63.42\(\pm\)0.67 \\ \hline P@3 & _39.57\(\pm\)_0.34 & **39.64\(\pm\)**0.39 & 39.23\(\pm\)0.57 \\ \hline P@5 & _28.80\(\pm\)_0.26 & **28.87\(\pm\)**0.32 & 28.86\(\pm\)0.38 \\ \hline  & Dolicons & & \\ \hline P@1 & 65.13\(\pm\)_0.39 & 67.59\(\pm\)_0.53 & **69.64\(\pm\)**0.58 \\ \hline P@3 & 59.07\(\pm\)_0.41 & 63.18\(\pm\)0.59 & **64.12\(\pm\)**0.75 \\ \cline{2-5} P@5 & 54.52\(\pm\)0.34 & 56.56\(\pm\)0.54 & **59.27\(\pm\)**0.65 \\ \hline  & Medium+illa & & \\ \hline P@1 & 84.25\(\pm\)_0.27 & **87.82\(\pm\)**0.33 & 84.22\(\pm\)0.27 \\ \cline{2-5}  & \multicolumn{2}{l|}{Fuzzy} \\ \hline P@3 & 67.29\(\pm\)_0.24 & **73.45\(\pm\)**0.30 & 67.33\(\pm\)0.20 \\ \hline P@5 & 52.90\(\pm\)_0.15 & **59.17\(\pm\)**0.34 & 53.04\(\pm\)0.18 \\ \hline  & Eurles & & \\ \hline P@1 & **81.04\(\pm\)**0.81 & 79.26\(\pm\)_0.26 & 76.13\(\pm\)_0.26 \\ \cline{2-5}  & \multicolumn{2}{l|}{Wiki0.3lik} \\ \hline P@3 & **67.91\(\pm\)**0.97 & 64.30\(\pm\)_0.88 & 59.90\(\pm\)_1.58 \\ \hline P@5 & **56.81\(\pm\)**0.97 & 52.33\(\pm\)_0.80 & 50.39\(\pm\)_1.40 \\ \hline  & Wiki0.3lik} \\ \hline P@1 & **86.05** & 85.88 & 83.05 \\ \cline{2-5}  & **76.85** & 72.98 & 67.47 \\ \hline P@5 & **67.77** & 62.70 & 57.76 \\ \hline \end{tabular}
\end{table}
Table 2: Non-ensemble models with k=100 or 30 (Mediamill). Best in **bold** and not significantly different to best at p=0.05 in _italics_.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline  & **Number of Features** & Number of Labels & Training Points & Testing Points \\ \hline Bittex & Text & 1836 & 159 & 4 

a single model approach.

### Ablation study

The ablation study results in Table 4 shows how different data sets benefit from different parts of our proposed framework, with all parts contributing substantially to at least one data set. We have reported the results by considering only label-label dependency information (\(f_{pen}\)), label-feature dependency information (\(f_{score}\)) and combining all 3 parts (\(f_{pen}\), \(f_{score}\) and outlier information). The results support the assertions that considering only the label space information (ignoring label-feature dependency information) in the label selection step causes prediction error of labels with high training error to be propagated to the whole label space and that it is important to also select outlier labels that are hard to predict from other selected labels.

We also investigated the effect of changing the subset size S on the final prediction quality (we have ignored the outlier effect in these experiments). Figure 1 shows an initial marked increase in performance with subset size, however the results gets more stable when the subset size gets larger. This observation, which is consistent with the submodular property, provides a clue that using a more complicated training model, like a non-linear model, for a smaller selected set of labels may lead to higher performance than increasing the subset size while using a linear model.

Figure 1: Precision score changes by subset size without outliers.

 

Conclusion and Future Work

We propose a novel approach for extreme multi-label classification that simplifies the problem by selecting an informative and easily modelled subset of labels and subsequently mapping back to the full set of labels. While the method is very well applicable to text datasets, it is applicable as a general ML method for different domains. Our novel label selection mechanism follows three principles: A new submodular maximisation framework that combines label-label dependencies and label training error together with a mechanism to identify outlier labels that are hard to reconstruct. Modelling only the most informative labels helps to avoid transforming the label space to a new embedding space leading to accumulation of training and embedding errors. We use a greedy approach for our monotone submodular framework with linear cost and good theoretical convergence. Extensive experiments using a linear prediction model on selected labels conducted on five standard real-world datasets demonstrate that our method achieves better performance than single model approaches, and better or comparable performance to ensemble based methods. In future, we can improve our model by using nonlinear training model instead of a simple linear regression model for the selected subset of the labels. Moreover, ablation study results suggest that a nonlinear propagation model to reconstruct the full label set may be of benefit.

## Acknowledgments

This work was partially funded by grants #16214415 and #16248016 of the Hong Kong Research Grants Council, ITS/319/16FP of Innovation Technology Commission, and RDC 1718050-0 of EMOS.AI.

## References

* Agrawal et al. (2013) Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma. 2013. Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In _Proceedings of WWW-13_, pages 13-24, Seoul, Korea.
* Balasubramanian and Lebanon (2012) Krishnakumar Balasubramanian and Guy Lebanon. 2012. The landmark selection method for multiple output prediction. In _Proceedings of ICML-12_, pages 983-990, Edinburgh, Scotland.
* Barezi et al. (2017) Elham J Barezi, James T Kwok, and Hamid R Rabiee. 2017. Multi-label learning in the independent label sub-spaces. _Pattern Recognition Letters_, 97:8-12.
* Beck and Teboulle (2009) Amir Beck and Marc Teboulle. 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. _SIAM journal on Imaging Sciences_, 2(1):183-202.
* Bhatia et al. (2015a) Kush Bhatia, Himanshu Jain, Purushottam Kar, Prateek Jain, and Manik Varma. 2015a. Locally non-linear embeddings for extreme multi-label learning. In _NIPS_, Montreal, Canada.
* Bhatia et al. (2015b) Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. 2015b. Sparse local embeddings for extreme multi-label classification. In _NIPS_, pages 730-738.
* Bi and Kwok (2013) Wei Bi and James Kwok. 2013. Efficient multi-label classification with many labels. In _Proceedings of ICML-13_, pages 405-413, Atlanta.
* Boyd (2011) Stephen Boyd. 2011. Alternating direction method of multipliers. In _Talk at NIPS workshop on optimization and machine learning_.
* Boyd et al. (2011) Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. _Foundations and Trends(r) in Machine Learning_, 3(1):1-122.
* Chen and Lin (2012) Yao-Nan Chen and Hsuan-Tien Lin. 2012. Feature-aware label space dimension reduction for muliabel classification. In _NIPS_, pages 1529-1537, Harrahs and Hareys.
* Gygli et al. (2015) Michael Gygli, Helmut Grabner, and Luc Van Gool. 2015. Video summarization by learning submodular mixtures of objectives. In _Proceedings CVPR 2015_, pages 3090-3098.
* He and Yuan (2012) Bingsheng He and Xiaoming Yuan. 2012. On the o(1/n) convergence rate of the douglas-rachford alternating direction method. _SIAM Journal on Numerical Analysis_, 50(2):700-709.
* Hsu et al. (2009) Daniel Hsu, Sham Kakade, John Langford, and Tong Zhang. 2009. Multi-label prediction via compressed sensing. In _NIPS_, volume 22, pages 772-780, Vancouver CANADA.
* Katakis et al. (2008) Ioannis Katakis, Grigorios Toumakas, and Ioannis Vlahavas. 2008. Multilabel text classification for automated tag suggestion. In _Proceedings of the ECMLPKDD_, page 75.
* Kempe et al. (2003) David Kempe, Jon Kleinberg, and Eva Tardos. 2003. Maximizing the spread of influence through a social network. In _Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 137-146. ACM.
* Lin et al. (2009) Hui Lin, Jeff Bilmes, and Shasha Xie. 2009. Graph-based submodular selection for extractive summarization. In _IEEE Workshop on ASRU_, pages 381-386 

Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. 2014. Multi-label classification via feature-aware implicit label space encoding. In _Proceedings of ICML-14_, pages 325-333, Beijing, China.
* Liu et al. (2005) Tie-Yan Liu, Yiming Yang, Hao Wan, Hua-Jun Zeng, Zheng Chen, and Wei-Ying Ma. 2005. Support vector machines classification with a very large-scale taxonomy. _Acm Sigkdd Explorations Newsletter_, 7(1):36-43.
* Mencia and Furnkranz (2008) Eneldo Loza Mencia and Johannes Furnkranz. 2008. Efficient pairwise multilabel classification for large-scale problems in the legal domain. In _Machine Learning and Knowledge Discovery in Databases_, pages 50-65. Springer.
* Mirrokni and Zadimoghaddam (2015) Vahab Mirrokni and Morteza Zadimoghaddam. 2015. Randomized composable core-sets for distributed submodular maximization. In _Proceedings of the forty-seventh annual ACM symposium on Theory of computing_, pages 153-162. ACM.
* Mirzasoleiman et al. (2015) Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrak, and Andreas Krause. 2015. Lazier than lazy greedy. In _AAAI_, pages 1812-1818.
* Nemhauser et al. (1978) George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. 1978. An analysis of approximations for maximizing submodular set functionsi. _Mathematical Programming_, 14(1):265-294.
* Nesterov (2004) Y Nesterov. 2004. Introductory lectures on convex programming: a basic course, volume i.
* Ohsaka et al. (2014) Naoto Ohsaka, Takuya Akiba, Yuichi Yoshida, and Ken-ichi Kawarabayashi. 2014. Fast and accurate influence maximization on large networks with pruned monte-carlo simulations. In _AAAI_, pages 138-144.
* Prabhu et al. (2018) Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In _Proceedings of the 2018 World Wide Web Conference on World Wide Web_, pages 993-1002. International World Wide Web Conferences Steering Committee.
* Prabhu and Varma (2014) Yashoteja Prabhu and Manik Varma. 2014. Fastxml: A fast, accurate and stable tree-classifier for extreme multi-label learning. In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data mining_, pages 263-272, New York City. ACM.
* Snoek et al. (2006) Cees GM Snoek, Marcel Worring, Jan C Van Gemert, Jan-Mark Geusbroek, and Arnold WM Smeulders. 2006. The challenge problem for automated detection of 101 semantic concepts in multimedia. In _14th Annual ACM International Conference on Multimedia_, pages 421-430. ACM.
* Tai and Lin (2012) Farbound Tai and Hsuan-Tien Lin. 2012. Multilabel classification with principal label space transformation. _Neural Computation_, 24(9):2508-2542.
* Tsamardinos and Borboudakis (2010) Ioannis Tsamardinos and Giorgos Borboudakis. 2010. Permutation testing improves bayesian network learning. _Machine Learning and Knowledge Discovery in Databases_, pages 322-337.
* Tschiatschek et al. (2014) Sebastian Tschiatschek, Rishbh K Iyer, Haochen Wei, and Jeff A Bilmes. 2014. Learning mixtures of submodular functions for image collection summarization. In _Advances in neural information processing systems_, pages 1413-1421.
* Tsoumakas et al. (2008) G Tsoumakas, I Katakis, and I Vlahavas. 2008. Effective and efficient multilabel classification in domains with large number of labels. In _Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD'08)_, pages 30-44.
* Tsoumakas et al. (2010) Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2010. Mining multi-label data. In _Data Mining and Knowledge Discovery Handbook_, pages 667-685. Springer.
* Yen et al. (2017) Ian EH Yen, Xiangru Huang, Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. 2017. Ppdsparse: A parallel primal-dual sparse method for extreme classification. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 545-553. ACM.
* Yen et al. (2016) Ian EH Yen, Xiangru Huang, Kai Zhong, Pradeep Ravikumar, and Inderjit S Dhillon. 2016. Pd-sparse: A primal and dual sparse approach to extreme multiclass and multilabel classification. In _Proceedings of ICML-16_.
* Yu et al. (2014)Hsiang-Fu Yu, Prateek Jain, Purushottam Kar, and Inderjit S Dhillon. 2014. Large-scale multi-label learning with missing labels. In _Proceedings of ICML-14_, volume 32, Beijing, China.
* Zhang and Zhang (2010) Min-Line Zhang and Kun Zhang. 2010. Multi-label learning by exploiting label dependency. In _16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 999-1008. ACM.
* Zubiaga (2012) Arkaitz Zubiaga. 2012. Enhancing navigation on wikipedia with social tags. _arXiv preprint arXiv:1202.5469_