A New Multi-input Model with the Attention Mechanism for Text Classification

Abstract: Recently, deep learning has made extraordinary achievements in text classification. However, most of present models, especially convolutional neural network (CNN), do not extract long-range associations, global representations, and hierarchical features well due to their relatively shallow and simple structures. This causes a negative effect on text classification. Moreover, we find that there are many express methods of texts. It is appropriate to design the multi-input model to improve the classification effect. But most of models of text classification only use words or characters and do not use the multi-input model. Inspired by the above points and Densenet (Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.), we propose a new text classification model, which uses words, characters, and labels as input. The model, which is a deep CNN with a novel attention mechanism, can effectively leverage the input information and solve the above issues of the shallow model. We conduct experiments on six large text classification datasets. Our model achieves the state of the art results on all datasets compared to multiple baseline models.

Reviews and author responses:

Summary

This paper introduces a new model architecture for doing text classification. The main contribution is proposing a deeper CNN approach, using both word and character embeddings (as well as label embeddings with attention). The paper claims improved performance over baselines. 

Decision

I reject the paper for 3 main reasons:
1) Very misleading claims regarding establishing a new state of the art. The baselines used for comparison don't include any of the best existing published results. 
2) Lack of positioning within the literature. In particular, no mention nor discussion of Transformers (self-attention) networks, including BERT and XLNet approaches, which are the state of the art in text classification. 
3) Lack of justification/explanation for the proposed architecture. One key argument made is that current models are shallow, but it appears that only CNN models are considered for that comparison. More discussion is needed to understand why the new aspects of the proposed network are importantly different from other existing approaches.

Additional details for decision

The results from this paper are significantly inferior to the best results published. With a few quick searches, I found that there are several approaches performing better than the proposed model on every dataset considered in the analysis, as you can see below.

http://nlpprogress.com/english/text_classification.html
https://github.com/sebastianruder/NLP-progress/blob/master/english/sentiment_analysis.md
https://paperswithcode.com/sota/text-classification-on-yahoo-answers

Extra notes (not factoring in decision)

- Consider spacing out the 3 rightmost blocks in Figure 1, I found the layout confusing and there's space available.
- In section 3, I would have liked more explanation for the motivation of the various design choices.

This paper presents a multi-input model for text classification.  The presentation of this paper is very poor.  The novelty is very limited.  The justification of the proposed architecture is not persuasive.  The experiment design has many flaws.  All the compared baselines are extremely weak.  The authors need to follow the experiment setup in more recent text classification papers.  Authors need at least to compare their methods to BERT.  The current version of this paper is definitely not an ICLR publication.  

This paper described a multi-input model for text classification. This paper is a bit hard to read compared to other submissions. I highly recommend authors to have native speaker to proofread this before submission. The paper mentioned "labels" as input to embedding layer as well as attention mechanism. However, it will be more helpful if authors can provide examples of labels. Moreover, the experiment results (e.g., Table 3 and 5) did not include label. I wonder if the methods used as baselines are still the state-of-the-art as language model pre-training might have better results. It will be more convincing if authors include additional baselines such as fine-tuning BERT.

Minor issue: the references in Table 3 (in brackets) are hard to find. 

Based on above reason, I would reject this paper.

Meta-review: This paper proposes a CNN-based text classification model that uses words, characters, and labels as its input. It also presents an attention block to replace the pooling operation that is typically used in a CNN. The proposed method is evaluated on six benchmark classification datasets, achieving reasonably good results.

While the proposed method performs reasonably well compared to baselines in the papers, all reviewers pointed out that there is no discussion or comparison with existing SotA based on pretrained models (e.g., BERT, XLNet), which would strengthen the main claim of the paper. All three reviewers also suggested that the writing of the paper could be improved. The authors did not respond to these reviews, so there was little discussion needed to arrive at a consensus.

I agree with all reviewers and recommend to reject the paper.

Decision: reject