A New Multi-input Model with the Attention Mechanism for Text Classification

1 INTRODUCTION . Text classification , including sentiment analysis ( Pang et al . ( 2002 ) ; Yang & Cardie ( 2014 ) ) , topic classification ( Tong & Koller ( 2002 ) ) , and spam detection ( Jindal & Liu ( 2007 ) ) , is an important subdomain of natural language processing ( NLP ) . It is widely used in public opinion monitoring , advertising filtering , user and product analysis and other fields . In recent years , deep learning has been frequently used in text classification . Compared with traditional methods ( Wang & Manning ( 2012 ) ) that rely on hand-crafted features , deep learning methods are employed to learn features from texts by a variety of neural network structures , especially recurrent neural network ( RNN ) and convolutional neural network ( CNN ) , and attention mechanisms . There are some examples of text classification in Table 1 . Both the first and the second cases should be classified as the topic of science and technology , and the third case should be classified as the topic of animals . Most of key words or phrases of case1 and case2 , which determine the category , are similar ( e.g. , NASA , moons , and Saturn ) , but the location of them is different.It can be well captured by CNN , because of the position invariance of CNN . However , the key words or phrases of case3 are scattered . The dependencies between them are long distance . Compared with case1 and case2 , these key words or phrases are relatively hard to capture by straightforward or shallow models . And case3 contains a lot of geographical names , which make it possible to be misclassified as geographic topics . By the above examples , we find a problems in present models : These models are shallow , especially CNN . It limits the final classification effect . Specifically , texts have the hierarchical structure which includes characters , words , phrase , sentences , and documents . The shallow models are hard to capture the hierarchical , high-level , long distance or global features form texts , which is why the models are limited .. Nowadays , the amount of text information on the Internet is increasing rapidly . Thus , the model which can cope with texts of various lengths and levels is needed , especially deep models of complicated texts . Compared with the CNN models ( He et al . ( 2016 ) ; Huang et al . ( 2017 ) ) of computer vision ( CV ) , the CNN models of text classification are very shallow . It indicates that deeper models can be tried to improve the result of text classification , as they have done in CV . Moreover , most of models utilize words or character as the single input , but the multi-input model is well suited to text classification . The natural language has a variety of carriers or expressions , such as words and characters , which can be related to others . This is very helpful for designing multiple inputs . We think the multi-input model can obtain better results because it provides more useful information . Some related works ( Amplayo et al . ( 2018 ) ; Xue & Li ( 2018 ) ) have been done in order to solve this problem , but most of works used multi-input models in some special subdomains such as user sentiment analysis . Those are not the general methods for text classification . In order to solve these problems , we propose a novel deep multi-input model for text classification . Our model uses words , characters , and labels as inputs . These are proved to be beneficial for text classification in previous works . Both the multi-input model and deep model have relatively more parameters and more expensive computing cost , and more effective feature extraction is needed on the multi-input model . Thus , we use a structure that is similar to Dense block of Densenet ( Huang et al . ( 2017 ) ) to alleviate the above problems , because it has narrow feature maps and can reduce the numbers of parameters . Moreover , we employ a new attention mechanism , which allows capturing of dependencies without regard to their distance in inputs , to replace the global pooling . It can strengthen feature extraction from several inputs . We conduct extensive experiments on several public text classification datasets . The results show that our model outperforms all baseline models on all datasets , and has fewer parameters in comparison to similar works . Our primary contributions are as follows : 1 . We propose a novel multi-input model for text classification . Our model inputs include words , characters , and labels . Most of previous works only use a single input such as the word or use the multi-input model in certain subdomains like user sentiment analysis ( Amplayo et al . ( 2018 ) ) . 2 . Inspired by Densenet , we design a 42-layers network . It is deeper than most of present models . Compared with shallow and straightforward models , our model can extract more global or long-term features , and have fewer parameters . 3 . We use a new attention mechanism in our model . It is used to replace the global pooling that commonly used in CNN . Compared with global pooling , our method can effectively utilize inputs information to capture key part more precisely . Compared with using the attention mechanism at the beginning of the model , this method can significantly reduce parameters and computation , because the feature maps of high layers are much smaller than that after embedding . 2 RELATED WORK . Deep learning has been shown their values in many domains of artificial intelligence , including text classification . This paper will introduce some significant works about it in text classification . For the past few years , deep learning or deep neural networks have been considered the most prevalent machine learning method in the field of text classification . Most of them use pre-trained word embedding methods such as word2vec ( Mikolov et al . ( 2013 ) ) or glove ( Pennington et al . ( 2014 ) ) and models based on CNN or RNN , but some of them employ the word embedding without pretrained ( Joulin et al . ( 2017 ) ) or other embedding methods such as utilizing characters ( Zhang et al . ( 2015 ) ) . CNN is regarded as the predominant model in the computer vision due to its friendliness in obtaining local features and processing in parallel . Recently it also has achieved high performance in NLP . In text classification , Kim ( 2014 ) combines several 1D CNNs to classify sentence . The convolution filter widths and feature maps are various between these 1D CNNs . Similarly , Kim ( 2014 ) also use convolutional layers with several kinds convolution filter widths . And they propose the dynamic k-max pooling , the k of which is related to the sentence length and the network depth . Zhang et al . ( 2015 ) propose a character-level CNN model . The model consists of 6 convolutional layers and uses English letters , digits and punctuations instead of the word to represent text . Their work shows that characters can be used as the input of text classification and the characters method can obtain good results . Zhang et al . ( 2017 ) propose a representation method which use the convolutional encoder and deconvolutional decoder to classify text . The convolutional encoder has the stride structure and no more than 4 layers . Xue & Li ( 2018 ) propose a CNN model with a gating mechanism to figure out the tasks of aspect based sentiment analysis and achieve excellent performance . It is enlightened from recent advances in computer vision ( He et al . ( 2016 ) ) , Conneau et al . ( 2017 ) design a 29-layer deep CNN model to improve the performance on large datasets . The NLP input and the sequence input are highly compatible . Therefore , RNN variants , such as long short term memory ( LSTM ) ( Hochreiter & Schmidhuber ( 1997 ) ) , are often used in NLP tasks due to the suitable for sequence features . Tang et al . ( 2015a ) propose the gated recurrent neural network , which employs LSTM and gated neural network to solve the document sentiment classification task . The attention mechanisms , especially Transformer ( Vaswani et al . ( 2017 ) ) , have achieved great results in many fields of NLP . Most attention mechanisms are used with RNN to extract features more accurately . Zhou et al . ( 2016 ) propose an attention mechanism based on bidirectional LSTM in cross-lingual sentiment classification . Yang et al . ( 2016 ) design a hierarchical attention network ( HAN ) that can better capture document structure features . It uses bidirectional GRU and attention mechanism on words and sentences , respectively . Moreover , Lai et al . ( 2015 ) propose the recurrent convolutional neural networks ( RCNN ) , which add a recurrent structure before convolutional layers . It can utilize the left and right contents of the word to improve the performance of classification . Moreover , labels , as the potential information , can effectively extend the input to help improve the precision of the model ( Yogatama et al . ( 2015 ) ) . However , it is not employed much in text classification . Tang et al . ( 2015b ) propose a heterogeneous text network , which utilizes an embedding including the labeled information and different levels of word co-occurrence information . This embedding has a good performance for the particular task . Zhang et al . ( 2018 ) propose a multitask label-embedding to transform labels into semantic vectors , which utilizes semantic correlations among tasks to enhance performances . 3 METHODS . Our model is briefly illustrated in Figure 1 . It mainly consists of embedding block , Dense block , compression block , attention block and classification block . The embedding block , including words , characters , and labels , is the first part of this model . There are a convolutional layer and a pooling layer after the embedding block . And then the combination of Dense block and compression block is repeated several times , but it is an attention block after the final Dense block instead of a compression block . The classification block is the final part . It consists of three full-connected layers , followed by dropout . The max pooling is used for all pooling layers , and batch normalization ( BN ) ( Szegedy et al . ( 2015 ) ) is applied after all convolutional layers . The activation function is the rectified linear unit ( RELU ) for all convolutional layers and full-connected layers . Embedding block The word embedding and the label embedding are similar to common pretrained word embedding , but we use three consecutive operations after the word embedding layer : dropout , adding Gaussian noise and BN . Inspired by character-level CNN ( Zhang et al . ( 2015 ) ) , the characters embedding utilizes multi-layer CNN to encode the character information . The CNN encoder has a max pooling layer in the last . And to reduce the parameters , its structure refers to the Inception ( Szegedy et al . ( 2015 ) ) . We also use three identical consecutive operations as the word-level embedding after the character-level embedding layer . Finally , we concatenate the words embedding part and the characters embedding part as the first output , which is used as the input of the convolutional layer after the embedding block . The labels embedding part is the second output , which is used as one of the inputs of attention block . Comparing with using the word embedding only or the character embedding only , our methods can express text more fully and extract more detailed information , which can help to improve the model performance . Figure 1 shows the embedding block architecture . Dense block The structural design of this block is based on Densenet ( Huang et al . ( 2017 ) ) , so we call it Dense block . Dense block consists of several convolutional blocks , which have two 1D convolutional layers . The filter widths of the first convolution layer and the second are 1 and 3 , and the size of feature maps of the first layer is 4 times that of the second layer . It can reduce the number of feature maps , and thus to make computational cheaper . Each convolution block is directly connected to all the other convolutional blocks . In a Dense block which has X convolutional blocks , there are X ( X − 1 ) /2 connections between convolutional blocks . However , in X layers typical CNN , there are only X connections . In other words , the final output includes all previous convolutional blocks output and the initial input , not just the output of the last convolutional layer as the typical CNN model . In this block , features are utilized efficiently to reduce relearning , and the change of gradient from loss function can affect each layer more directly to address the problem of vanishing-gradient . It is substantially beneficial for designing very deep models . Thus , our model can be deeper and perform better than the typical CNN model . Figure 1 shows the Dense block architecture . Noticeably , the input size of the convolutional block will grow gradually , because each convolution block input includes the output of all the previous convolution blocks and the initial input . If m is the size of feature maps of the second layer of each convolution block , and that of the first should be 4m , the input feature maps of the next convolutional block will increase by m compared to the previous . Define M as the size of input feature maps of the h-th convolutional block : M = M0 + g × ( h− 1 ) ( 1 ) g = i× n ( 2 ) Where M0 is the size of input feature maps of the first convolutional block , g are the number of growth feature maps , i is the number of channels in the growth feature maps , n is the dimensions of feature maps . We call the hyperparameter i the growth rate of the model . It can control the amount of information that each convolutional block gives the final output . The growth rate is usually small but enough to obtain an excellent result . Therefore , the feature maps of convolutional blocks are very narrow . It makes our model have fewer parameters . Since Dense block is used repeatedly , our model become very deep and can better extract long-range associations , global representations , and hierarchical features . It is why this model is called the multiple shortcut connections CNN . Compression block The feature maps of our model , especially the number of channels , are increased due to the Dense block . If the Dense block is repeated several times , the feature maps may become huge . After the Dense block , we design a structure which is used to compress the feature maps to avoid this problem . The 1D max pooling though can perform downsampling to reduce the number of feature maps , but the number of channels is usually not changed . Thus , a 1D convolutional layer with 1 convolution filter width is used before the max pooling layer . The number of channels of this convolutional layer is compressed to that of the previous Dense block input . We call this combination of convolutional layer and max pooling layer as the compression block . After the compression block , the feature maps have been compressed on both dimensions and channels . In other words , this block makes feature maps not only not become huge and harmful , but also reduce periodically to improve the model efficiency . Figure 1 shows the compress block and the classification block architecture . Attention block The traditional CNN methods for text classification usually use global pooling or k-max pooling after the last convolutional layer . The attention block is different from conventional pooling operations , which is roughly filtered out one or several features . It utilizes the relation between labels and extracted features to give suitable weights to extracted features . Most labels text usually short and recapitulative , which are much simpler than common texts . Thus , extracting label features do not need complex structures such as that of common texts to increase computational complexity . We choose the label embedding and the features extracting from texts and characters as the block input . It has smaller and narrower feature maps in comparison to designing an attention mechanism after the embedding block , so it has fewer parameters and avoids using more complex attention structures to deal with huge feature maps . Figure 2 shows the attention block architecture . The input of this block are the labels embedding and the output of the last compression block . All labels are concatenated as a sentence . In the embedding block , we embed the label sentence as well as the word-level embedding . Then we use average pooling to map each label to a vector . Besides , we employ one Dense block to extract global text features further and convert the features to the dimension of label-embedding . Define L = [ L : ,1 , L : ,2 , . . . , L : ,k ] as the label-embedding , V = [ V : ,1 , V : ,2 , . . . , V : ,d ] as the extracted features of a sample , which are gotten from the features extracting step . Where k is the number of classes , d is the dimension of the extracted feature maps . We use the cosine similarity C ∈ Rd×k to describe the degree of correlation between L and V . C = C1,1 . . . Ci , j . . . Cd , k ( 3 ) Where Ci , j = V T : ,i·L : ,j |V T : ,i|·|L : ,j | is the cosine similarity between the i-th feature vector V : ,i and j-th labelembedding vector L : ,j . The cosine similarity is a simple way that is difficult to obtain relative information between labels and extracted features fully . To generate more precise attention weights , we utilize the CNN structure to capture further location correlation . We use a multi-layer CNN structure with small convolution filter width . It results in fewer parameters and larger coverage of the convolution filter . Therefore , this block has an advantage when facing with large feature maps of complex models . After multi-layer CNN structures , the max pooling is employed to select the most relevant vector h = [ h1 , h2 , . . . , hd ] between labels and extracted features . We can compute the i-th attention score/weight ai by h : ai = exp ( hi ) ∑d j=1 exp ( hj ) ( 4 ) Where hi is the i-th element of h. The final text representation t is the sum of weighted features : t = d∑ i=1 aiV : ,i ( 5 ) The final output t is used as the input of the classification block . The attention block can assign weights between labels information and features extracting by words and characters more accurately . In other words , it can take full advantage of a large amount of information of the multi-input model . Thus , our multi-input model with this label-attention mechanism is capable of improve classification accuracy effectively .

Summary: This paper presents a multi-input model for text classification.  The presentation of this paper is very poor.  The novelty is very limited.  The justification of the proposed architecture is not persuasive.  The experiment design has many flaws.  All the compared baselines are extremely weak.  The authors need to follow the experiment setup in more recent text classification papers.  Authors need at least to compare their methods to BERT.  The current version of this paper is definitely not an ICLR publication.