A New Multi-input Model with the Attention Mechanism for Text Classification
Authors: Junhao Qiu
Ronghua Shi
Fangfang Li (the corresponding author)
Jinjing Shi
Wangmin Liao
Venue:  (2020)
Keywords: Natural Language Processing
Text Classification
Densent
Multi-input Model
Attention Mechanism

Abstract: Recently, deep learning has made extraordinary achievements in text classification. However, most of present models, especially convolutional neural network (CNN), do not extract long-range associations, global representations, and hierarchical features well due to their relatively shallow and simple structures. This causes a negative effect on text classification. Moreover, we find that there are many express methods of texts. It is appropriate to design the multi-input model to improve the classification effect. But most of models of text classification only use words or characters and do not use the multi-input model. Inspired by the above points and Densenet (Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.), we propose a new text classification model, which uses words, characters, and labels as input. The model, which is a deep CNN with a novel attention mechanism, can effectively leverage the input information and solve the above issues of the shallow model. We conduct experiments on six large text classification datasets. Our model achieves the state of the art results on all datasets compared to multiple baseline models.

Under review as a conference paper at ICLR 2020
A NEW MULTI-INPUT MODEL WITH THE ATTENTION MECHANISM FOR TEXT CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, deep learning has made extraordinary achievements in text classifica- tion. However, most of present models, especially convolutional neural network (CNN), do not extract long-range associations, global representations, and hier- archical features well due to their relatively shallow and simple structures. This causes negative effect on text classification. Moreover, many express methods of texts. It is appropriate to design the multi-input model improve the classification effect. But most of models of text classification only use words or characters and do not use the multi-input model. Inspired by the above points and Densenet (Huang et al. (2017), we propose new text classi- fication model, which uses words, characters, and labels as input. The model, which is deep CNN with novel attention mechanism, can effectively leverage the input information and solve the above issues of the shallow model. We con- duct experiments on six large text classification datasets. Our model achieves the state of the art results on all datasets compared to multiple baseline models.
find that there are
INTRODUCTION
Text classification, including sentiment analysis (Pang et al. 20002; Yang & Cardie (014)), topic classification (Tong & Koller (2002)), and spam detection (Jindl & Liu (2007)) an important subdomain of natural language processing (NLP). It is widely used in public opinion monitoring, advertising filtering, user and product analysis and other fields. In recent years, deep learning has been frequently used in text classification. Compared with traditional methods (Wang & Manning 20122) that rely on hand-crafted features, deep learning methods are employed to learn features from texts by a variety of neural network structures, especially recurrent neural network (RNN) and convolutional neural network (CNN), and attention mechanisms. There are some examples of text classification in Table 1. Both the first and the second cases should be classified the topic of science and technology, and the third case should be classified as the topic of animals. Most of key words phrases of casel and case2, which determine the category, moons, and Saturn), but the location of them is different.It can be well captured by CNN, because of the position invariance of CNN. However, the key words or phrases of case3 are scattered. The dependencies between them are long distance. Compared with casel and case2, these key words or phrases are relatively hard to capture by straightforward or shallow models. And case3 contains lot of geographical names, which make it possible to be misclassified as geographic topics.
similar (e.g.. NASA,
By the above examples, we find problems in present models: These models are shallow, especially CNN. It limits the final classification effect. Specifically, texts have the hierarchical structure which includes characters. words. phrase, sentences, and documents. The shallow models are hard to capture the hierarchical, high-level, long distance or global features form texts, which is why the models are limited.. Nowadays, the amount of text information on the Internet is increasing rapidly. Thus, the model which can cope with texts of various lengths and levels is needed, especially deep models of complicated texts. Compared with the CNN models (He et al. (2016)) Huang et al. (017)) of computer vision (CV), the CNN models of text classification deeper models can be tried to improve the result of text classification, as they have done in CV.
very shallow. It indicates that
Moreover, most of models utilize words character the single input, but the multi-input model is well suited to text classification. The natural language has a variety of carriers or expressions, such words and characters, which can be related to others. This is very helpful for designing multiple

---

Under review as a conference paper at ICLR 2020
Table 1: Examples of text classification. The Bold is the key word phrase.
casel:
moons were spotted around Saturn by Cassini space probe, raising the total to 33 moons for the ringed planet, NASA said on Mon- day
case2:
NASA's Cassini spaceeraft has spied satellite-rich Saturn, the space agency said Monday.
little around
cuse.:
Rendall's serotine (Neommiciia remdalli) species of vesper bat.ii found in Benin Botswana Burkinu Faso Cameroon Central Africun Re- public Chad Republic of the Congo Democratic Republic the Congo Gambia Ghana Kenya Malawi Mali Mozambique Niger Nigeria Rwanda Senegal Sierra Leone Somalia South Africa Sodan Tanzania Uganda and Zambia Its natural habitats are dry savanna moist savanna subtropical tropicll dry shrubland subrropical or tropical moist shrubland threatened by habitat loss.
inputs. We think the multi-input model can obtain better results because it provides more useful information. Some related works (Amplayo et al. 2018); Xue & Li )018)) have been done in order to solve this problem, but most of works used multi-input models in some special subdomains such as user sentiment analysis. Those are not the general methods for text classification.
In order to solve these problems. we propose a novel deep multi-input model for text classification. Our model uses words, characters, and labels as inputs. These are proved to be beneficial for text classification previous works. Both the multi-input model and deep model have relatively more parameters and more expensive computing cost, and more effective feature extraction is needed on the multi-input model. Thus, we use structure that is similar to Dense block of Densenet (Huang et al. (2017)) to alleviate the above problems, because it has narrow feature maps and can reduce the numbers of parameters. Moreover, we employ a new attention mechanism, which allows capturing of dependencies without regard to their distance in inputs, to replace the global pooling. It can strengthen feature extraction from several inputs. We conduct extensive experiments on several public text classification datasets. The results show that model outperforms all baseline models on all datasets, and has fewer parameters in comparison to similar works.
Our primary contributions are as follows:
1. We propose a novel multi-input model for text classification. Our model inputs include words, characters, and labels. Most of previous works only use a single input such as the word or the multi-input model in certain subdomains like user sentiment analysis (Amplayo et al. (2018).
2. Inspired by Densenet, Compared with shallow and straightforward models, our model can extract more global or long-term features, and have fewer parameters.
design 42-layers network. is deeper than most of present models.
3. We use a new attention mechanism in our model. It is used to replace the global pooling that commonly used in CNN. Compared with global pooling, our method can effectively utilize inputs information to capture key part more precisely. Compared with using the attention mechanism at the beginning the model, this method can significantly reduce parameters and computation, because the feature maps of high layers are much smaller than that after embedding.
2 RELATED WORK
Deep learning has been shown their values in many domains of artificial intelligence, including text classification. This paper will introduce some significant works about it in text classification.
For the past few years, deep learning or deep neural networks have been considered the most preva- lent machine learning method in the field of text classification. Most of them use pre-trained word embedding methods such as word2vec (Mikolov et al. (2013) or glove (Pennington et al. (2014)) and models based CNN RNN, but some of them employ the word embedding without pre- trained (Joulin et al. (2017) or other embedding methods such as utilizing characters (Zhang et al. 2015)).
2

---

Under review as a conference paper at ICLR 2020
casofiration Neck
attrrtcnn Nock
compres Nock
compress Nick
black
compresu Hick
dosiariis liak
Dance
Tass pad, urill
Eeary
mas pool, atride
euppen blck
3coor 12M
umbalding
Tree
venc
durractirs
lnbels
wend
clarostes
embhdddnng beck
Figure 1: Our model architecture
CNN is regarded as the predominant model the computer vision due to its friendliness in obtaining local features and processing in parallel. Recently it also has achieved high performance in NLP. In text classification, Kim (2144) combines several ID CNNs to classify sentence. The convolution filter widths and feature maps are various between these ID CNNs. Similarly, Kim (2014 also use convolutional layers with several kinds convolution filter widths. And they propose the dynamic k-max pooling, the k of which is related the sentence length and the network depth. Zhang al. (2015) propose character-level CNN model. The model consists of convolutional layers and uses English letters, digits and punctuations instead of the word represent text. Their work shows that characters can be used as the input of text classification and the characters method can obtain good results. Zhang al.) (2017) propose a representation method which use the convolutional encoder and deconvolutional decoder to classify text. The convolutional encoder has the stride structure and no more than layers. Xue Li (2018) propose CNN model with a gating mechanism figure out the tasks of aspect based sentiment analysis and achieve excellent performance. It is enlightened from recent advances in computer vision (He et al. (2016)), Conneau et al. (2017) design a 29-layer deep CNN model to improve the performance on large datasets.
The NLP input and sequence input are highly compatible. Therefore, RNN variants, such as long short term memory (LSTM) (Hochreiter & Schmidhuber (1997)), often used in NLP tasks due to the suitable for sequence features. Tang et al. (2015a) propose the gated recurrent neural network, which employs LSTT and gated neural network to solve the document sentiment classification task. The attention mechanisms, especially Transformer (Vaswani et al.) (201)). have achieved great results in many fields of NLP. Most attention mechanisms are used with RNN to extract features more accurately. Zhou et al. (2016) propose an attention mechanism based on bidirectional LSTM in cross-lingual sentiment classification. Yang al. (2016) design a hierarchical attention network (HAN) that can better capture document structure features. It uses bidirectional GRU and attention mechanism on words and sentences, respectively. Moreover, Lai al. (2015) propose the recurrent convolutional neural networks (RCNN), which add a recurrent structure before convolutional layers. can utilize the left and right contents the word to improve the performance of classification.
Moreover, labels, as the potential information, can effectively extend the input to help improve the precision of the model (Yogatama et al. (2015)). However, it is not employed much in text classi- fication. Tang et al. (2015b) propose a heterogeneous text network, which utilizes embedding including the labeled information and different levels of word co-occurrence information. This embedding has a good performance for the particular task. Zhang al. (2018) propose multi- task label-emen -ebbeddib to transform labels into semantic vectors, which utilizes semantic correlations among tasks enhance performances.
3

---

Under review as a conference paper at ICLR 2020
3 METHODS
Our model is briefly illustrated in Figure 1. It mainly consists of embedding block, Dense block, compression block, attention block and classification block. The embedding block, including words, characters, and labels, is the first part of this model. There are a convolutional layer and a pooling layer after the embedding block. And then the combination of Dense block and compression block is repeated several times, but it is an attention block after the final Dense block instead of a compression block. The classification block the final part. It consists of three full-comeectd layers, followed by dropout. The max pooling is used for all pooling layers, and batch normalization (BN) (Szegedy) al.] (2015) is applied after all convolutional layers. The activation function is the rectified linear unit (RELU) for all convolutional layers and full-connected layers.
Embedding block The word embedding and the label embedding are similar to common pre- trained word embedding, but we use three consecutive operations after the word embedding layer: dropout, adding Gaussian noise and BN. Inspired by character-level CNN (Zhang et al. (2155). the characters embedding utilizes multi-layer CNN to encode the character information. The CNN encoder has a max pooling layer in the last. And to reduce the parameters, its structure refers to the Inception (Szegedy et al. (2015). We also three identical consecutive operations as the word-level embedding after the character-level embedding layer. Finally, we concatenate the words embedding part and the characters embedding part as the first output, which is used as the input of the convolutional layer after the embedding block. The labels embedding part is the second output, which is used as one of the inputs of attention block. Comparing with using the word embedding only or the character embedding only, our methods can express text more fully and extract more detailed information, which can help to improve the model performance. Figure shows the embedding block architecture.
Dense block The structural design of this block is based on Densenet (Huang et al. (017)). ) so we call it Dense block. Dense block consists of several convolutional blocks, which have two 1D convolutional layers. The filter widths of the first convolution layer and the second are and 3, and the size of feature maps of the first layer is 4 times that of the second layer. It can reduce the number of feature maps, and thus to make computational cheaper. Each convolution block is directly connected to all the other convolutional blocks. In a Dense block which has X convolutional blocks, there are X(X 1)/2 connections between convolutional blocks. However, in X layers typical CNN, there are only X connections. In other words, the final output includes all previous convolutional blocks output and the initial input, not just the output of the last convolutional layer as the typical CNN model. In this block, features are utilized efficiently to reduce relearning, and the change of gradient from loss function can affect each layer more directly to address the problem of vanishing-gradient. vanishing- gradient. It is substantially beneficial for designing very deep models. Thus, our model can be deeper and perform better than the typical CNN model. Figure 1 shows the Dense block architecture.
Noticeably, the input size of the convolutional block will grow gradually, because each convolution block input includes the output of all the previous convolution blocks and the initial input. If m is the size of feature maps of the second layer of each convolution block, and that of the first should be 4m, the input feature maps of the next convolutional block will increase by m compared to the previous. Define M as the size of input feature maps of the h-th convolutional block:
M = Mo + x (h 1)
(I)
g g=ixn
(2)
Where Mo is the size of input feature maps of the first convolutional block, g are the number of growth feature maps, i is the number of channels in the growth feature maps, n is the dimensions of feature maps. We call the hyperparameter i the growth rate of the model. It can control the amount of information that each convolutional block gives the final output. The growth rate is usually small but enough to obtain an excellent result. Therefore, the feature maps of convolutional blocks are very narrow. It makes model become very deep and can better extract long- long-range associations, global representations, and hierarchical features. It is why this model is called the multiple shortcut connections CNN.
model have fewer parameters. Since Dense block is used repeatedly, our
4

---

Under review as a conference paper at ICLR 2020
wrighted
Densc bock
indtesinilarity
Satures
Iabeh cunbedding
otteation
Figure 2: The attention block architecture.
Compression block The feature maps of our model, especially the number of channels, are in- creased due to the Dense block. If the Dense block is repeated several times, the feature maps may become huge. After the Dense block, we design structure which is used to compress the feature maps avoid this problem. The ID max pooling though perform downsampling to reduce the number of feature maps, but the number of channels is usually not changed. Thus, a 1D convo- lutional layer with 1 convolution filter width is used before the max pooling layer. The number of channels of this convolutional layer is compressed to that of the previous Dense block input. We call this combination of convolutional layer and max pooling layer the compression block. After the compression block, the feature maps have been compressed on both dimensions and channels. In other words, this block makes feature maps not only not become huge and harmful, but also reduce periodically to improve the model efficiency. Figure shows the compress block and the classification block architecture.
Attention block The traditional CNN methods for text classification usually use global pooling or k-max pooling after the last convolutional layer. The attention block is different from conventional pooling operations, which is roughly filtered out one several features. It utilizes the relation between labels and extracted features to give suitable weights to extracted features. Most labels text usually short and recapitulative, which are much simpler than common texts. Thus, extracting label features do not need complex structures such as that of common texts to increase computational complexity. We choose the label embedding and the features extracting from texts and characters as the block input. It has smaller and narrower feature maps in comparison to designing an attention mechanism after the embedding block, it has fewer parameters and avoids using more complex attention structures to deal with huge feature maps. Figure 2 shows the attention block architecture.
The input of this block are the labels embedding and the output of the last compression block. All labels are concatenated as a sentence. In the embedding block, we embed the label sentence as well the word-level embedding. Then we use average pooling to map each label to a vector. Besides, we employ one Dense block to extract global text features further and convert the features to the dimension of label-embedding. Define L = [L,,, L.2...., L.,] as the label-embedding. V [V.,, V. V.,al as the extracted features of sample, which are gotten from the features extracting step. Where the number of classes, the dimension the extracted feature maps. We use the cosine similarity C' dd to describe the degree of correlation between L and V.
Ct.1
C
Cij
(3)
Ca.
V L.. is the cosine similarity between the i-th feature vector V., and j-th label- MV L embedding vector L.j- The cosine similarity is a simple way that is difficult to obtain relative information between labels and extracted features fully. To generate more precise attention weights,
Where Ct
5

---

Under review as a conference paper at ICLR 2020
Table 2: Summary statistics of datasets.
Datasets
Classes
Train Samples
Test Samples
Average Words
Word Lengths
Character Lengths
Batch
Sizes
Sogou
5
450,000
60,000
578
1,300
5,000
32
Yahoo
10
1400,000
60,000
112
223
1358
128
Dbpedia
Yelp
560,000
38,000
153
337
1350
128
Yelp
650,000
50,000
155
305
1194
128
Table 3:
Test error rates (%) six datasets. The Bold is the best result.
Dbpedia
Yelp.p
Yelp.f
Sogou
Linear model [30]
7.64
1.31
4,36
40.14
28.96
2.81
Fastllxtt
1.4
4,3
Regien.emnb [16]
7.2
1.1
4.7
35.1
26.3
LSTM [30))
13.94
1.45
5,26
41,83
29.16
4.82
char-CNN [30)
1.55
4.88
37.95
word
Wond-CNN CNN (30)
8.55
1.37
4,60
39.58
28.84
4.39
D-LSTM [28]
7.9
1.3
7.4
40.4
26.3
5.1
VDCNN
1.29
char CRNN [23]
8.64
1.43
5.51
38.18
28.26
4.80
Our model (word)
7.17
1,10
3,95
34.85
26.22
2.40
(wordsshuracter)
we utilize the CNN structure to capture further location correlation. We use multi-layer CNN structure with small convolution filter width. It results fewer parameters and larger coverage of the convolution filter. Therefore, this block has advantage when facing with large feature maps of complex models. After multi-layer CNN structures, the max pooling is employed to select the most relevant vector h [b1. h2..... ha] between labels and extracted features. We can compute the i-th attention score/weight a, by h:
exp(h:) 0, Σj=1 exp(hy)
(4)
Where h; is the i-th element of h. The final text representation t is the sum of weighted features:
t V.
The final output is used as the input of the classification block. The attention block can assign weights between labels information and features extracting by words and characters more accurately. In other words, can take full advantage of a large amount of information the multi-input model. Thus, multi-input model with this label-attention mechanism capable of improve classification accuracy effectively.
4 EXPERIMENTS
4.1 DATASETS
We experiment six public datasets] which are proposed by Zhang et (2015). Table 2 shows the summary statistics. AG and Sogou are news topic classification datasets. Dbpedia is an ontology classification dataset. Yahoo is topic classification dataset about Q&A. Yelp.p and Yelp.f are review sentiment classification datasets, the former is coarse granularity (two classes: positive and negative), and the latter fine granularity (five classes). Sogou is the Chinese pinyin dataset, the others are English text datasets.
4.2 THE EXPERIMENT SETUP
We randomly selected 10000 documents from the training data as the validation set. word embed- ding part, tokenized all the English texts with NLTK (Bird & Loper (2004) and used the 300d

---

Under review as a conference paper at ICLR 2020
Table 4: Comparison of model parameters
Methods
CNN
1STM
VDCNN
Our model
9-layers
17-kyyers
29-ayers
49-layers
Parameters
541
1.8 M
2.2
4.3M
4.6M
7.8M
1.1-1.2 M
Table 5: Test error rates (%) for different variations of our model. We run all experiments AG and Yelp.p.
Methols
Errbekling
Dasse Hock
Convoktiin Block
Growt Rite
MG
Yelp
moohlighobll prolleg)
wordt charates
32
7,26
4.51
Darmodel oiepler
6/22
1.97
7.36
3.71
6,96
3.70
Our model
6.92
3.82
32
6.84
4.94
6.55
1.75
16
48
7.00
3.57
64
6.72
3.85
glove 840B vectors by Pennington et al. (2014) initialization. In addition, used the word2vec (Mikolov et al. (2013) to train the pinyin representation, which is used as initialization of the Sogou dataset. In character embedding part, the processing
like Zhang et al. (2015). The characters consist of letters (all uppercase letters were converted to lowercase), digits and punctuations. We initialized the character embedding weights using a Gaussian distribution. The initial mean and stan- dard deviation were (0,0.05). Word and character lengths were set according to the datasets. They are also shown in Table 2. The drop rate of both parts was set to 0.5. Empirically, A convolutional layer of 1 128 convolutions of size 3 was applied to the first after embedding block. We used 4 Dense blocks in our model, each including 4 convolution blocks. And the growth rate was set to 32. The max pooling with size 3 and stride 2 was used for all pooling layers except the global max pooling. The feature maps were halved in both dimension and channel after each compression blocks. We performed three full-connected layers with size 256 and dropout rate 0.2 0.5 in the classification block. Moreover, in the attention block, the processing was similar to the word embedding part. We employed 3 convolutional layers with filter width 3 or 4, the number of channels of which the same as the number of the classes. Finally, we selected Nadam as the optimizer. The initial learning rate was set 0.002. We used cross entropy the loss function. All experiments were performed on the NVidia 1080Ti GPU, using different mini-bater sizes on different datasets as shown in Table
4.3 BASELINES
We compared our method using with various baseline methods, including CNN, RNN, the hybrid method, and other methods. The CNN based methods consist of the word-CNN, char-CNN [30] and the 29-layers very deep CNN (VDCNN) (Conneau et al. (2017)). The RNN based methods consist of LSTM (Zhou et al. )2016) and the discriminative LSTM (D-LSTM) (Yggatama al. (017)) The char-CRNN (Xiao & Cho (206)) is the hybrid method based on the combination of CNN and RNN. The other methods include the linear model (Zhang et al. (2055), the FastText (Joulin et al. 2017)), and a region embedding method (Qiao et al.). We also tested our model without characters inputs.
4.4 RESULTS
Error rates Table shows the test error rates results. The simple methods (rows 1-3) perform better small datasets such as AG than the large such as Yelp.f. The sophisticated methods (rows 7-9) are the opposite. The result is consistent with common sense that the demand degree of the number of training data is usually positively correlated with model complexity. Simple models may even have an advantage on small datasets. The best simple baseline is region embedding method (Qiao et al.), and the best complex baseline is VDCNN (Conneau al. (2017). Compared with

---

Under review as a conference paper at ICLR 2020
them, our methods including 2 embedding structures variants work well both small and large data sets. It ranks the 1st (word+character) or 2nd (word) on all datasets. The results of long lengths datasets validates that our model can extract more global long-term features. And the result of our model with the combination embedding of words and characters outperforms that of only word embedding, which validates the effectiveness of the characters input. We observe error reduction are less significant on Yahoo, Sogou, and Dbpedia. There may be some reasons: (i) The training samples of Sogou and Dbpedia are relatively small, and Yahoo have special text structure (Q&A). Both may require more data to training for complex methods because all the complex baselines do not perform well on those datasets. (iz) The number of classes of Dbpedia and that of Yahoo are fourteen and ten respectively, and the others are no more than five. It is relatively more difficult to train label-attention mechanism to get the most similar labels exactly when there are relatively more classes. (iii) Sogou is a Chinese pinyin dataset. Unlike other datasets, the pre-train pinyin word vector is got by Sogou dataset. Compared with 300d glove 840B vectors (Pennington et al. (2014). the number of pre-train samples of pinyin is much smaller. Thus, the representation of words and labels may become less effective on Sogou dataset.
Parameters We compared the parameters with CNN, LSTM, and VDCNN. Table 4 shows the parameters results. Except for shallow CNN with very few parameters, our model has far fewer parameters than LSTM and the shallowest VDCNN which have the least number of parameters. And our model (42-layers) is deeper than all the baseline except for 49-layers VDCNN. It means that method avoids excessive parameters while achieving low error rate.
Impact of structures We compared the impact of using different structures on our model as shown in Table 5, including embedding structures, attention block structures, and other hyperparameters. We tested 2 embedding structures variants (using global pooling and attention block): only word embedding and the combination embedding of words and characters. We also tested two attention block structures: a single convolutional layer with large convolution filter width (55) and three convolutional layers with small convolution filter width (3). Besides, we compared the impacts of different growth rates, the number of multiple direct connections blocks, and the number of convolution blocks on the results. The results show that the combination embedding is better than only word embedding (rows 1 and 2, 4 and 7), our model with attention block outperformed that without attention block (rows and 4, and 3, and 7), three convolutional layers attention block is better than the single convolutional layer (rows 3 and 7), and the best combination is 4 Dense block with 4 convolution blocks and growth rate i = 32 (rows 5 -16).
CONCLUSION AND FUTURE WORK
This paper proposes a novel deep CNN model with new label-embedding attention mechanism for text classification. Our model adds shortcut connections between any two convolutional layers. Compared to the traditional deep model and attention mechanism, our method has fewer parameters, deeper structure, and takes full ad advantage of the inputs information. Therefore, our model achieves the state of the art results on several public text classification datasets. Moreover, we also analyze the effects of different structures on the method and the reasons for the relatively poor performance of the method. In the future, we will apply and improve our method the mabli problem and the classification of obscure label information.
REFERENCES
Reinald Kim Amplayo, Jihyeok Kim, Sua Sung, and Seungwon Hwang. Cold-start aware user and product attention for sentiment classification. In 56th Annual Meeting of the Association for Computational Linguistics, ACL2018 2018, pp. 2535-2544. Association for Computational Linguistics (ACL), 2018
Steven Bird and Edward Loper. Nitk: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214 2004.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional net- works for text classification. In European Chapter of the Association for Computational Linguis- fies EACL' 17, 2017.
8

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 4700-4708, 2017.
Nitin Jindal and Bing Liu. Review spam detection. In Proceedings of the 16th international confer- ence on World Wide Web, pp. 1189-1190. ACM, 2007.
Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov, Bag of tricks for efficient text classification. EACL 2017, pp. 427, 2017.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1751,, 2014.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp- 2267-2273. AAAI Press, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 31 3111-3119, 2013.
Bo Pang. Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up?:
sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volie 10, pp- 79 86. Association for Computational Linguistics, 2002.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Chao Qiao, Bo Huang, Guocheng Niu, Daren Li, Daxiang Dong, Wei He, Dianhai Yu, and Hua Wu. A new method of region embedding for text classification.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Duyu Tang, Bing Qin, and Ting Liu. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1432, 2015a.
Jian Tang, Meng Qu, and Qiaozhu Mei. Pte: Predictive text embedding through large-scale hetero- geneous text networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1165-1174. ACM, 2015b.
Simon Tong and Daphne Koller. Support vector machine active learning with application sto text classification. Machine Learning Research, 2(1):999-1006, 2002.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 0998-6008, 2017.
Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th annual meeting of the association for computational linguistics: Short popers-volume 2, pp. 90-94. Association for Computational Linguistics, 2012.

---

Under review as a conference paper at ICLR 2020
Yijun Xiao and Kyunghyun Cho. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv: 162.00666200007,, 2016.
Wei Xue and Tao Li. Aspect based sentiment analysis with gated convolutional networks. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Lingusstics (Volume 1: Long Pupers), pp. 2514-2523, 2018.
Bishan Yang and Claire Cardie. Context-aware learning for sentence-level sentiment analysis with posterior regularization. In Proceedings of the 52nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Pupers), pp. 325-335, 2014.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguisties: human language technologies,
pp. 1480-1489, 2016.
D Yogatama, C Dyer, W Ling, and P Blunsom. Generative and discriminative text classification with recurrent neural networks. In Thirty-fourthh International Conference on Machine Learning (ICML 2017), International Machine Learning Society, 2017.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. Embedding methods for fine grained entity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 291-2966 2015.
Honglun Zhang, Liqiang Xiao, Wenqing Chen, Yongkun Wang, and Yaohui Jin. Multi-task label embedding for text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4545 4553, 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Yizhe Zhang. Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, and Lawrence Carin. De- convolutional paragraph representation learning. In Advances in Neural Information Processing Systems, pp. 4179, 2017.
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. Atteetion-based Istm network for cross- coos-linggal sen- timent classification. In Proceedings of the 2016 conference on empirical methods natural language processing, pp. 247-256, 2016.
10


Reviews:
[{"review_id": "SygNcwUaqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper described a multi-input model for text classification. This paper is a bit hard to read compared to other submissions. I highly recommend authors to have native speaker to proofread this before submission. The paper mentioned \"labels\" as input to embedding layer as well as attention mechanism. However, it will be more helpful if authors can provide examples of labels. Moreover, the experiment results (e.g., Table 3 and 5) did not include label. I wonder if the methods used as baselines are still the state-of-the-art as language model pre-training might have better results. It will be more convincing if authors include additional baselines such as fine-tuning BERT.\n\nMinor issue: the references in Table 3 (in brackets) are hard to find. \n\nBased on above reason, I would reject this paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HJl-jdkaKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper introduces a new model architecture for doing text classification. The main contribution is proposing a deeper CNN approach, using both word and character embeddings (as well as label embeddings with attention). The paper claims improved performance over baselines. \n\nDecision\n\nI reject the paper for 3 main reasons:\n1) Very misleading claims regarding establishing a new state of the art. The baselines used for comparison don't include any of the best existing published results. \n2) Lack of positioning within the literature. In particular, no mention nor discussion of Transformers (self-attention) networks, including BERT and XLNet approaches, which are the state of the art in text classification. \n3) Lack of justification/explanation for the proposed architecture. One key argument made is that current models are shallow, but it appears that only CNN models are considered for that comparison. More discussion is needed to understand why the new aspects of the proposed network are importantly different from other existing approaches.\n\nAdditional details for decision\n\nThe results from this paper are significantly inferior to the best results published. With a few quick searches, I found that there are several approaches performing better than the proposed model on every dataset considered in the analysis, as you can see below.\n\nhttp://nlpprogress.com/english/text_classification.html\nhttps://github.com/sebastianruder/NLP-progress/blob/master/english/sentiment_analysis.md\nhttps://paperswithcode.com/sota/text-classification-on-yahoo-answers\n\nExtra notes (not factoring in decision)\n\n- Consider spacing out the 3 rightmost blocks in Figure 1, I found the layout confusing and there's space available.\n- In section 3, I would have liked more explanation for the motivation of the various design choices.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJg7Gw_0KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-input model for text classification.  The presentation of this paper is very poor.  The novelty is very limited.  The justification of the proposed architecture is not persuasive.  The experiment design has many flaws.  All the compared baselines are extremely weak.  The authors need to follow the experiment setup in more recent text classification papers.  Authors need at least to compare their methods to BERT.  The current version of this paper is definitely not an ICLR publication.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a CNN-based text classification model that uses words, characters, and labels as its input. It also presents an attention block to replace the pooling operation that is typically used in a CNN. The proposed method is evaluated on six benchmark classification datasets, achieving reasonably good results.

While the proposed method performs reasonably well compared to baselines in the papers, all reviewers pointed out that there is no discussion or comparison with existing SotA based on pretrained models (e.g., BERT, XLNet), which would strengthen the main claim of the paper. All three reviewers also suggested that the writing of the paper could be improved. The authors did not respond to these reviews, so there was little discussion needed to arrive at a consensus.

I agree with all reviewers and recommend to reject the paper.
Decision: Reject