Large-scale Multi-label Text Classification - Revisiting Neural Networks
Authors: Jinseok Nam, Jungi Kim, Iryna Gurevych, Johannes Fürnkranz
ICLR 2014

Abstract: Large-scale datasets with multi-labels are becoming readily available, and the demand for large-scale multi-label classification algorithm is also increasing. In this work, we propose to utilize a single-layer Neural Networks approach in large-scale multi-label text classification tasks with recently proposed learning techniques. We carried out experiments on six textual datasets with varying characteristics and size, and show that a simple Neural Networks model equipped with recent advanced techniques for Neural Networks components such as an activation layer, optimization, and generalization techniques performs as well as or even outperforms the previous state-of-the-art approaches on large-scale datasets with diverse characteristics.


Peer review:
[Review]:
This paper tackles the problem of multi-label classification using a single-layer neural network. Several options are considered such as thresholding, using various transfer functions or dropouts.

The paper is full of imprecisions, writing errors that make it hard to read (the paragraph 'Computational Expenses' of Section 2.2 is perhaps the worse from this point of view), even if it is possible to get most of the content.

The whole paper is based on a comparison with BP-MLL, which is quite problematic for two reasons. First, it is not clear that BP-MLL is the best suitable baseline. What motivated this choice? Their pairwise exponential loss function is not standard, most ranking methods instead choose the hinge loss). What is their architecture? We need more arguments to assess this as a strong benchmark. Besides, the comparison made in the paper between the proposed network and BP-MLL is not valid: influences of architecture, choice of transfer functions and loss are all mixed. In the end of the 'Plateaus' paragraph, it is suggested that ReLUs allow to outperform BP-MLL but no experience with BP-MLL with ReLU has been conducted. Perhaps the loss of BP-MLL (PWE) could be as efficient as cross entropy with ReLU. Is BP-MLL used as a benchmark as a network or simply as a loss (exponential)? What are the results with hinge loss?

Most results about neural networks are not particularly new. It is already quite well known that (1) Dropout prevent overfitting and (2) networks with ReLUs are easier to optimize than those with Tanh. In that sense, Figure 3, which is quite nice and sound, or Section 5 do not bring much new results to practitioners already used to Deep Learning (such as the ICLR audience I guess). 

The curves form Figure 2 are more original but I don't agree with its creation. What justifies that what is observed with such a weird networks (a single hidden unit) can transfer to more generic feed-forward NNs? Besides, the conclusions regarding the 'plateaus' are not really obvious: it seems that there are plateaus for all settings. Is is expected that the curves with tanh appear to be symetric w.r.t. the origin?

Other comments:
- I don't see what means 'Revisiting Neural Networks' in the title of the paper.
- In Equation (2), what is the definition of $y_l$?
- Legend of Fig 1(b): CE w/ ReLU -> CE W/ tanh
- ReLU is used from the beginning of Section 2 but defined in Section 2.3.
- How is defined Delta for ADAGRAD?
- Couldn't it be interesting to try to learn thresholds using the class probabilities outputted by the network?
- The metrics used in experiments could be described.
- Discussion on the training efficiency of linear SVMs in Section 6.2 seems irrelevant here.