Large-scale Multi-label Text Classification - Revisiting Neural Networks
Authors: Jinseok Nam, Jungi Kim, Iryna Gurevych, Johannes Fürnkranz
ICLR 2014

Abstract: Large-scale datasets with multi-labels are becoming readily available, and the demand for large-scale multi-label classification algorithm is also increasing. In this work, we propose to utilize a single-layer Neural Networks approach in large-scale multi-label text classification tasks with recently proposed learning techniques. We carried out experiments on six textual datasets with varying characteristics and size, and show that a simple Neural Networks model equipped with recent advanced techniques for Neural Networks components such as an activation layer, optimization, and generalization techniques performs as well as or even outperforms the previous state-of-the-art approaches on large-scale datasets with diverse characteristics.


Peer review:
[Review]:
The authors claim that a simple two-layer fully connected neural net can outperform or meet state of the art accuracies on large, multi-label classification tasks, using rectified linear units and dropout.  They make novel use of L2 regression to compute a data-dependent threshold to map model outputs to class labels.

While the paper is interesting, the baselines seem weak, in particular, the comparison against a neural net ranker with exponential versus cross-entropy loss.  The authors make a point several times in the paper with which I disagree - that ranking approaches do not scale to large datasets.  The existence of effective web search engines is strong evidence to the contrary.


Specific comments
*****************

There are many typos and grammatical errors but few that change the meaning, so below I only mention the latter.  

Eq. (2): For clarity's sake it might be worth mentioning that f_0 has range [0,1] and that the labels y are in {0,1}.

Fig. (2): Yes, but the high slope of ReLU can also cause problems if there is noise in the data (which appears as outliers).

I think that a much better comparison than the one with BP-MLL would be to compare with RankNet, a neural net ranker that uses a cross entropy error function and thus is much closer to the cost function used in the paper.  It does seem to make sense to rank all outputs corresponding to positive labels above all outputs corresponding to negative; the question left open by your comparison is whether the problem lies with the use of the exponential cost in BP-MLL.  Unfortunately there are at least two confounding factors that you have not separated in comparing BP-MLL versus your method: the use of ranking at all, and the choice of ranking cost function.  So I think it's important to compare against a cross entropy ranking function. 

'The ReLU disables negative activation so that the number of parameters to be learned decreases during the training.' - what do you mean?

Section 2.3: define the Delta used in Adagrad

Section 3: this is a nice idea, for choosing data dependent thresholds - is it novel (for multi label classification)?

Sec. 4.1: it would be useful to give brief definitions of the ranking measures used ('Rank loss, one-error, etc.'), since, except for MAP they (or at least, these names for them) are not well known.

Sec 4.2: my impression is that your choice of SVM baseline is weak (also, few details are given). Binary relevance seems like a poor choice for the multilabel task.  Why not compare to SVM rankers?  And since you're comparing to a nonlinear system, reporting results using nonlinear kernels would be good to be more complete (although linear SVMs usually do work well on text classification, this is a different task).

Table 2: 'B and R followed by BR are to represent' should read 'B and R following BR represent' and would be even better written as simply 'B and R subscripts represent'

The results in Table 2 seem to be mixed. Sometimes dropout helps, sometimes it hurts. Sometimes the SVMs win, often not.  The only take-away that seems safe to me, is that if you want the best performing system, then try all these methods (and other, stronger baselines - see above) on your data, and pick the best.  I think the most interesting result is that NNs (lumping with and without dropout together) beat linear SVMs, which is usually the strongest performing method for text classification.  But this task is different, and so it would have made more sense to compare against SVM rankers.

Given the use of ReLU transfer functions, what different did L1 regularization on the hidden activations make?

Figs 3a is great.  Fig 3b is a little misleading, though - you chose the one dataset where adding dropout helped.  What does that curve look like on a dataset where adding dropout hurt?

Section 6.1:  'the presence of a specific label may suppress or exhibit the probability of presence of other labels' - I think you just mean that some sets of labels tend to occur together, and this is ignored in binary relevance methods, but please explain this more clearly.