Large-scale Multi-label Text Classification - Revisiting Neural Networks
Authors: Jinseok Nam, Jungi Kim, Iryna Gurevych, Johannes Fürnkranz
ICLR 2014

Abstract: Large-scale datasets with multi-labels are becoming readily available, and the demand for large-scale multi-label classification algorithm is also increasing. In this work, we propose to utilize a single-layer Neural Networks approach in large-scale multi-label text classification tasks with recently proposed learning techniques. We carried out experiments on six textual datasets with varying characteristics and size, and show that a simple Neural Networks model equipped with recent advanced techniques for Neural Networks components such as an activation layer, optimization, and generalization techniques performs as well as or even outperforms the previous state-of-the-art approaches on large-scale datasets with diverse characteristics.


Peer review:
[Review]:
Thanks for the helpful reviews. We are currently working on improving our paper along your suggestions. We noticed that we have not been clear enough about a few important points, which we would like to clarify in this first reply (more details will follow later):

1. learning-to-rank vs. multi-label classification
Although we frequently talk about ranking in our paper, our objective is not learning- to-rank. Learning-to-rank aims at ranking a set of objects (such as documents), whereas our goal is to assign a set of labels to a given document. (as opposed to conventional multi-class classification, where only a single label is assigned to the document). Mutli-label classification is often framed as a ranking problem, but in this case the labels need to be ranked for a given document (as opposed to ranking the documents themselves). Many commonly used loss functions for multi-label classifi- cation focus on a good ranking of the labels (a good ranking is one where all relevant labels tend to be ranked before all irrelevant labels).

2. The pairwise hinge loss and the pairwise exponential loss to minimize ranking loss
It is said that the pairwise hinge loss in RankSVM [2] and the pairwise expo- nential loss in BP-MLL are natural choices as the surrogate loss for the ranking loss [4]. However, these surrogate losses are not consistent with the ranking loss that we want to minimize.

3. Are BP-MLL and BR (linear SVMs) reasonable choices for the baselines?
Neural Network-based algorithms are particularly interesting for multilabel classifi- cation because they allow to model dependencies between the occurrence of labels, whereas the standard binary relevance approach assumes that the occurrence of a la- bel for an example is independent of the occurrence of other labels for this examples, an assumption that is typically wrong in practice. BP-MLL is a well-known NN archi- tecture which exploits a pairwise error function instead of the traditional cross entropy loss.
The main claim that we want to make in this paper is that Neural-Network based approaches to multilabel classification may benefit from several recent advancements that have been developed in Deep Learning, as well as that the minimization of the pairwise error function may be replaced with something simpler such the cross entropy loss.
Thus BP-MLL is a natural benchmark, because this is the prototypical Neural- Network-based multilabel classification algorithm, which is often used as a baseline [5].
Binary relevance, on the other hand, is in many domains not a strong baseline for the reasons discussed above, but in particular in text domains it is still commonly used and has shown very good results. Several recent works have shown that the BR approach may outperform their counterparts which consider label dependencies between labels. Also note that recent analyses has shown that ranking losses may be minimized by loss functions on the individual labels [1, 4], which may be part of an explanation why binary relevance with SVMs as base classifiers tends to often performs well in practice even though it does not take label dependencies into account.

Reviewer 1

- Section 3: this is a nice idea, for choosing data dependent thresholds - is it novel (for multi-label classification)?:
The basic idea of this sort of threshold has been discussed in several papers [2, 8, 6]. Instead of minimizing bipartite misclassification errors, which is sum of the number of false positives and false negatives, we use F1 score as a reference measure.

- Given the use of ReLU transfer functions, what different did L1 regularization on the hidden activations make?:
We follow the recent findings from [8] where deep neural networks are constructed by using ReLUs at the hidden layers, together with L1 regularization on the activations. Even though, as we expected, stronger L1 regularization makes averaged value of positive hidden activations decrease, we did not find meaningful differences.

- Figs 3a is great. Fig 3b is a little misleading, though - you chose the one dataset where adding dropout helped. What does that curve look like on a dataset where adding dropout hurt?:
The performance of NNs with dropout or without dropout is highly correlated with properties of datasets. When we train NNs on EUR-Lex and Delicious dataset, the networks tend to overfit severely such as the red dashed lines in Figure 3 (b). Both datasets have a relatively large number of labels compared to the number of training documents and unique terms. Precisely, on the Delicious dataset, the number of distinct labels is larger than the number of unique words, and each document is associated with nearly 20 labels on average. Additionally, one thirds of labels on EUR-Lex dataset does not appear in the training data split, that is the label distribution of training data and that of test data are different. To prevent overfitting due to such characteristics of datasets, we tried to regularize the models with L1 and L2 penalty as well as Dropout, but only works Dropout. On the rest of the datasets, even though we increase the number of units in the hidden layer up to 2000 and 4000, no such severe overfitting observed. We conjecture that this is because why dropout does not help training. In that case, otherwise, it introduces undesirable noise to the models.
We will include figures for the cases where dropout does not help.

Reviewer 2

- the comparison made in the paper between the proposed network and BP-MLL is not valid: influences of architecture, choice of transfer functions and loss are all mixed. In the end of the 'Plateaus' paragraph, it is suggested that ReLUs allow to outperform BP-MLL but no experience with BP-MLL with ReLU has been conducted. Perhaps the loss of BP-MLL (PWE) could be as efficient as cross entropy with ReLU. Is BP-MLL used as a benchmark as a network or simply as a loss (exponential)? What are the results with hinge loss?:
We used BP-MLL as it has been proposed where the hidden units and the output units are tanh, and the error function is the pairwise error function. In comparison of ReLU and tanh in the hidden layer of BP-MLL, tanh often performs better than ReLU. We will add additional experimental results with respect to the type of hidden units in BP-MLL.

- The curves form Figure 2 are more original but I don’t agree with its creation. What justifies that what is observed with such a weird networks (a single hidden unit) can transfer to more generic feed-forward NNs?:
It is hard to draw error as a function of parameters in general neural networks as in Figure 2 because the number of all possible configurations of parameters in NN equals to '1 + the number of hidden layers' assuming that each hidden layer has a single unit. 

- the conclusions regarding the ”plateaus” are not really obvious: it seems that there are plateaus for all settings.:
We can only say given all the same settings such as input data, output targets and weight initialization methods, a curve is more steep than the other. What we want to show in Figure 2 is that the use of different cost functions yields the different curves where PWE consists of larger plateaus compared to CE. Obviously, using the ReLU unit in the hidden layer gives rise to absolutely flat region as in Figure 2(b), which implies it is impossible to escape from such a region once the hidden unit’s activation is determined to zero. That is a downside of the use of ReLUs as Reviewer 1 pointed out.

- Couldn’t it be interesting to try to learn thresholds using the class probabilities outputted by the network?:
To learn instance-wise threshold predictor, the class probabilities are used to estimate the best performing threshold on training instances from which we learn the threshold predictor. If one wants to train label-wise threshold predictor, a variant of ScutFBR might be considered [7, 3].


References
[1] Krzysztof Dembczynski, Wojciech Kotlowski, and Eyke Hüllermeier. Consistent multilabel ranking through univariate losses. In ICML, 2012.
[2] André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In NIPS, 2001.
[3] Rong-En Fan and Chih-Jen Lin. A study on threshold selection for multi-label classification. Technical Report, National Taiwan University, 2007.
[4] Wei Gao and Zhi-Hua Zhou. On the consistency of multi-label learning. In COLT, 2011.
[5] Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. Random k-Labelsets for Multilabel Classi- fication. IEEE Trans. Knowl. Data Eng., 23(7):1079–1089, 2011.
[6] Yiming Yang and Siddharth Gopal. Multilabel classification with meta-level features in a learning-to-rank framework. Machine Learning, pages 1–22, 2011.
[7] Yiming Yang. A study of thresholding strategies for text categorization. In SIGIR, 2001.
[8] Min-Ling Zhang Min-Ling Zhang and Zhi-Hua Zhou Zhi-Hua Zhou. Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization. IEEE Trans. Knowl. Data Eng., 18(10): 1338–1351, 2006.