Venue: NAACL 2016

# Deep LSTM based Feature Mapping for Query Classification

 Yangyang Shi

Kaisheng Yao

Le Tian

Daxin Jiang

Microsoft

{yangshi,kaisheng.yao,letian,djiang}@microsoft.com

###### Abstract

Traditional convolutional neural network (cnn) based query classification uses linear feature mapping in its convolution operation. The recurrent neural network (rnn), differs from a cnn in representing word sequence with their ordering information kept explicitly. We propose using a deep long-short-term-memory (dlstm) based feature mapping to learn feature representation for cnn. The dlstm, which is a stack of lstm units, has different order of feature representations at different depth of lstm unit. The bottom lstm unit equipped with input and output gates, extracts the first order feature representation from current word. To extract higher order nonlinear feature representation, the lstm unit at higher position gets input from two parts. First part is the lower lstm unit's memory cell from previous word. Second part is the lower lstm unit's hidden output from current word. In this way, the dlstm captures the nonlinear nonconsecutive interaction within _n_-grams. Using an architecture that combines a stack of the dlstm layers with a tradition cnn layer, we have observed new state-of-the-art query classification accuracy on benchmark data sets for query classification.

## 1 Introduction

Convolutional neural networks (cnns) have achieved significant improvements for query classification. cnns capture the correlations of spatial or temporal structures with different resolutions using their temporal convolution operators. A pooling strategy on these local correlations extracts invariant regularities.

However, cnns use simple linear operations on _n_-gram vectors that are formed by concatenating word vectors. The linear operation together with the concatenation may not be sufficient to model the nonconsecutive dependency and interaction within the _n_-grams. For example, in the query "not a total loss", nonconsecutive dependency "not loss" is the key information that is not well addressed by the linear operation with simple concatenation.

In this paper, we propose to use deep long-short-term-memory (dlstm) based feature mapping to capture high order nonlinear feature representations. lstm Hochreiter and Schmidhuber (1997) is one type of recurrent neural networks (rnns) that have achieved remarkable performance in natural language processing and speech recognition Sutskever et al. (2014); Graves et al. (2013).

The dlstm is a stack of lstm units where different order of nonlinear feature representation is captured by lstm units at different depth. The bottom lstm unit extracts the first order feature representation from current word. The lstm unit at the higher position captures the higher order feature representation relying on the outputs from lstm units at lower position, specifically, the memory cell from lower lstm unit at previous word position and the hidden output from lower lstm unit at current word position. Using dlstm, linear feature mapping in traditional cnn can be obviously extended to non-linear feature mapping. Moreover, the memory cell together with different gates in lstm unit are able to model the nonconsecutive feature interaction and

information decaying based on context. For example, in the query "not so good", the proposed dlstm is expected to keep the information of "not" and "good" in the memory, and to decay the information about "so" via the forget gates.

Similar to cnns where multiple convolution operations are used, we propose to stack different dlstm feature mappings together to model multiple level nonlinear feature representations. The bottom blstm layer takes the original word sequence as input. The dlstm layer at lower position fed its output to the adjacent higher dlstm layer. In the proposed models, the concatenation of the multiple level feature representations are further reduced by the pooling operation. The prediction output is finally made based on the reduced feature representations.

We evaluated the proposed method on three benchmark data sets: Standford Sentiment Treebank dataset Socher et al. (2013), TREC Text Retrieval Conference) question type classification data set Li and Roth (2002) and ATIS Airline Travel Information Systems) dataset Hemphill et al. (1990). On Standford Sentiment Treebank dataset, our model obtains 51.9% accuracy on fine-grained classification and 88.7% accuracy on binary classification. The svm based method uses a large amount of engineered features, and it outperforms lstm and rnn based methods on TREC question type classification dataset. The dltsm outperforms other neural network based methods without using engineered features. On ATIS data, dlstm achieves 97.9% F1 score, which is better than the previous best F1 score of 95.6% using the same data settings.

## 2 Related Work

Deep neural networks Bengio (2009); Deng and Yu (2014); Hinton et al. (2006) dominates natural language processing Socher (2012); Collobert et al. (2011); Gao et al. (2014). They have achieved cutting-edge performance in various tasks such as language modeling Mikolov et al. (2010); Sundermeyer et al. (2012), machine translation Bahdanau et al. (2014); Cho et al. (2014); Jean et al. (2015), slot filling Yao et al. (2014a); Shi et al. (2015a) and syntactic parsing Wang et al. (2015); Collobert et al. (2011). For query classifications, recurrent neural networks (rnns) and convolutional neural networks (cnns) have emerged as top performing architectures Zhang and Wallace (2016); Kim (2014); Kalchbrenner et al. (2014); Ravuri and Stolcke (2015a).

Due to its superior ability to memorize long distance dependencies, lstms have been applied to extract the sentence-level continuous representation Ravuri and Stolcke (2015a); Tang et al. (2015); Tai et al. (2015). When the lstm is applied to model a sentence, memory cell from the ending word in the sentence carries the information of the whole sentence. The lstm hidden vector from the ending word is directly used as sentence feature representation in Ravuri and Stolcke (2015a). Alternatively, a sentence is represented by the average of lstm hidden vectors from its words Tang et al. (2015). Inspired from recursive neural networks Socher et al. (2011a), lstm is further combined with a tree structure to model sentence representation Tai et al. (2015).

cnns have been originally developed for image processing Lecun et al. (1998). They are firstly applied by Collobert et al. (2008, 2011) for natural language processing tasks using max-over-time pooling method to aggregate convolution layer vectors. cnns have also been applied to spoken language understanding Shi et al. (2015b), information retrieval Shen et al. (2014) and semantic parsing Yih et al. (2015). Kalchbrenner et al. (2014) proposed to extend cnns max-over-time pooling to _k_-max pooling for sentence modeling. Remarkable query classification performance on different benchmark datasets have been achieved by integrating cnns with different feature mapping channels and pre-trained word vectors Zhang and Wallace (2015); Kim (2014). Recently, Mou et al. (2015) proposed to model sentences by tree structured cnns.

cnns and lstms are complementary in their modeling capabilities; cnns are good at capturing local invariant regularities and lstms are good at modeling temporal features. The combination of cnns and lstms achieves improved performances in speech recognition Sainath et al. (2015) and query classification Tang et al. (2015); Zhou et al. (2015). In these models, the basic architecture is the lstm that models sequence representation from local features captured by cnns.

Different from the above methods, our method use lstm units to model the nonlinear and non 

consecutive local features. cnns are placed on top of these local features for query classification. Our motivation is to use lstm replace the linear feature mapping in convolution operation where the feature mapping is a multiplication of the word vectors with a filter matrix. So our proposed model is still cnn based model but using dlstm as feature mapping for convolution operation.

Our work is closely related to tensor product based cnns(Lei et al., 2015) that expand cnn feature representation capacity with non-consecutive _n_-grams. They improve the query modeling from two aspects. Firstly, tensor products enable the nonlinear feature vector interactions between adjacent words. Secondly, an exponentially decaying weight is applied to represent non-consecutive _n_-gram features. Instead of using tensor products as feature mapping, we propose to apply dlstm to address these two aspects. Nonlinear feature mapping can be achieved by the dlstm that equipped with nonlinear activation function. The nonsconsecutive feature interaction is well addressed by the memory cell and different gates in lstm unit. In particular, the forget gate is able to decay the information according to the context rather than a fixed decaying weight in tensor product based cnns.

## 3 CNN Based Query Classification Using DLSTM Feature Mapping

### Linear Feature Mapping in CNN

Let \(k\)-dimensional vector \(x_{t}\in\mathbb{R}^{k}\) be the continuous feature representation of the \(t\)th word in a sentence. A sentence with \(l\) words is represented by \(x_{0:l-1}=[x_{0};x_{2};...;x_{l-1}]\) that is a concatenation of all word vectors. The traditional cnn(Collobert et al., 2011; Kim, 2014) takes such sentence feature vector as input.

Different filters \(M_{j}\in\mathbb{R}^{nd*h}\) are applied in convolution operation to map each \(n\)-gram feature vector \(x_{t:t+n-1},t\in(0,l-n)\) to an \(h\)-dimensional feature vector \(c_{r,j}\).

\[c_{t,j}=M_{j}^{T}\cdot x_{t:t+n-1}+b_{j}, \tag{1}\]

where \(b_{j}\) is the bias in filter \(j\).

The resulting feature vector \(c_{t,j}\) are often passed through non-linear element-wise transformations (e.g. the hyperbolic tangent and rectifier linear unit) as well as pooling operations. After aggregation or reduction by different pooling operations such as the max-over-time pooling (Collobert et al., 2011; Kim, 2014) and the average pooling (Lei et al., 2015), a constant dimensional feature vector is generated for sentences with various lengths.

In traditional cnns, the concatenated word vectors are mapped linearly to feature coordinates as shown in Equation (1). Such linear feature mapping can be improved from the following two aspects, one is to extend linear mapping to nonlinear mapping. The other one is to improve the consecutive feature mapping to nonconsecutive feature mapping. For example, in the query "not a total loss", "not loss" is the key sentiment. By using nonconsecutive feature representation, the information about "not loss" could be addressed. Lei et al. (2015) extends the linear feature mapping to tensor based feature mapping. To model the nonconsecutive _n_-grams, a decaying weight is applied to control the information carryover. In this paper, we propose to replace the linear feature mapping using dlstm that captures the nonlinear and nonconsecutive feature interaction within _n_-grams. Rather than setting a fixed decaying weight, the proposed architecture is able to control the information decaying according to the context information.

### Feature Mapping Based on Deep Long Short Term Memory

Figure 1 gives the basic architecture of a three-order nonlinear feature mapping in dlstm. The bottom lstm extract the first order information from word input vector \(x_{t}\). It is equipped with input gate and output gate. The input gate automatically controls the information saving in memory cell that will be passed to higher order lstm unit. The output gate modifies the information from the memory cell to represent current word.

\[i_{0,t}=sigmoid(W_{i}x_{t}+b_{i}) \tag{2}\] \[\tilde{c}_{0,t}=\tanh(W_{c}x_{t}+b_{c})\] (3) \[c_{0,t}=i_{0,t}*\tilde{c}_{0,t}\] (4) \[o_{0,t}=sigmoid(W_{o}x_{t}+V_{o}c_{0,t}+b_{o})\] (5) \[h_{0,t}=o_{0,t}*\tanh(c_{0,t}) \tag{6}\]

On top of the bottom lstm unit, we analogously 

stack two lstm unitslst1 and lstm2 to extract nonlinear feature representations from bigram and trigram, respectively. The lstm\({}_{j}\) is formulated as follows:

\[i_{j,t}=sigmoid(W_{v}x_{t}+U_{i}h_{j-1,t}+b_{i}) \tag{7}\] \[\tilde{c}_{j,t}=\tanh(W_{c}x_{t}+U_{c}h_{j-1,t}+b_{c})\] (8) \[f_{j,t}=sigmoid(W_{f}x_{t}+U_{f}h_{j-1,t}+b_{f})\] (9) \[c_{j,t}=i_{j,t}*\tilde{c}_{j,t}*c_{j-1,t-1}+f_{j,t}*c_{j-1,t}\] (10) \[o_{j,t}=sigmoid(W_{o}x_{t}+U_{o}h_{j-1,t}+V_{o}c_{j,t}+b_{o})\] (11) \[h_{j,t}=o_{j,t}*\tanh(c_{j,t}) \tag{12}\]

Due to the effect from different gates that controls the information saving, expressing and decaying, lstm\({}_{1}\) and lstm\({}_{2}\) are able to model the non-consecutive interaction in _n_-grams. Take "not so good" as a example. lstm\({}_{0}\) extract the nonlinear feature mapping from word "good" as \(h_{0,2}\). The lstm\({}_{1}\) takes \(c_{0,1}\) (carries the information from word "so") and \(h_{0,2}\) as input. Due to the effect of forget gate, we expect the output \(h_{1,2}\) from lstm\({}_{1}\) to address more on word "good" rather than "so". By further stacking lstm\({}_{2}\), information about the word "not" and "good" should be emphasized by the proposed dlstm.

Note the sum of the resulting outputs from these lstm units is used as the high order feature representation of a _n_-gram ending with word \(x_{t}\). So the original sequence input \(x_{0:l-1}\) is mapped to a sequence of feature vector \(z_{0:l-1}=[z_{0},z_{1},...,z_{l-1}]\), where \(z_{j}=h_{0,j}+h_{1,j}+h_{2,j}\).

The proposed dlstm architecture is characterized by the following two features:

1. Weight Sharing: lstm\({}_{1}\) and lstm\({}_{2}\) are identical lstm units that share the same weights. The bottom lstm unit lstm\({}_{0}\) also shares the corresponding weights with other lstm units such as \(W_{i}\), \(W_{c}\) and \(W_{o}\). By sharing weights among different lstm units, we can effectively reduce the risk of model over-fitting issue. At the same time, lstm is good at capturing temporal regularities. By sharing weights, the lstm units can learn the temporal dependencies from being exposed to different order of _n_-grams during the training.
2. Memory Cell Interaction: To model the nonlinear feature interaction in _n_-gram vectors, traditional lstm unit is modified by Equation (10) in which the memory cell stores the interaction of different order memory cells. In this way, the feature interaction in _n_-grams is characterized by the memory cell interactions.

To stack the lstm unit deeper, the depth-gated lstm Yao et al. (2015) and the highway network Srivastava et al. (2015); Zhang et al. (2015) also allow the memory cell flow across lstm units at different depth. There are three basic differences between these architectures with the proposed dlstm. Firstly, in their architectures, lstm units at different depth are different lstms that have different weight matrices. In our model, the lstm units in dlstm share weight matrices with each other. Secondly, in their proposed architecture, the memory cell is carried over to higher lstm unit for facilitating model training. Because the networking training becomes more difficult with increasing model depth. In our 

dlstm, the lstm unit at higher position takes the memory cell from lower lstm unit mainly for feature interaction in \(n\)-grams. Finally, an additional "depth" gate is applied in their architecture to control the information flow across different layers. In our model, the input gate in higher lstm unit controls the interaction between the memory cells extracted from previous word and current word.

### The Architecture

Figure 3 gives the whole architecture of the proposed query classification system. A dltm layer first maps the input sequence to a sequence of high order nonlinear feature representations \(z^{0}\). Instead of being directly used for query classification, the feature representation \(z^{0}\) is further processed by a stack of dlstm layers illustrated in previous section. In such stacked dlstm layers, the output \(z^{i}\) of the _i_th dlstm layer, is used as the input for the \(i+1\)th dlstm layer parameterized by a different set of weight matrices. As shown in Figure 3, the resulting feature representations \(z^{0},z^{1},...,z^{d}\) of all these layers are concatenated. Finally, an average pooling is applied to reduce the sentence feature representation to a fixed dimensional vector that is further fed to a softmax function to obtain the prediction output.

### Learning and Regularization

In the classification layer, the prediction output is obtained by the following softmax function.

\[softmax(y)_{j}=\frac{\exp(y_{j})}{\sum_{i=1}^{i=m}\exp(y_{i})}, \tag{13}\]

where \(y\) is a \(m\)-dimensional vector. The model is trained by minimizing cross-entropy on the given training data set. To avoid overfitting during training, \(L2\) regularization and dropout Hinton et al. (2012) are used. The \(L2\) regularization is applied to constrain all weight matrices using the same regularization weight. The dropout is only applied to the output of each dlstm layer.

In the training, the model weights are updated using mini-batch stochastic gradient descent sgd. We adapt a per-feature learning rate control method (AdaGrad) Duchi et al. (2011) to dynamically tune the learning rate as follows:

\[\alpha_{t,i}=\frac{\alpha}{\sqrt{\sum_{j=1}^{t}g_{j,i}^{2}+\epsilon}}, \tag{14}\]

where \(\alpha_{t,i}\) is the learning rate for weight \(i\) at epoch \(t\). \(\sum_{j=1}^{l}g_{j,i}\) sums all the historical gradients of weight \(i\). A small positive \(\epsilon\) is applied to make the AdaGrad robust. \(\epsilon\) is usually set to \(1e-5\).

## 4 Experiments

### Datasets

We evaluate the proposed query classification models on sentence sentiment classification, question type categorization and query intent detection tasks.

For sentence sentiment classification, the Stanford Sentiment Treebank Socher et al. (2013) is used. In this dataset, 11855 English sentences are annotated at both sentence level and phrases level with fine-grained labels (very positive, positive, neutral, negative and very negative). We use the provided data split, which has 8544 sentences for training, 1101 sentences for developing and 2210 sentences for testing. This dataset also provides a binary classification variant that ignores the neutral

Figure 2: cnn based query classification using dlstm feature mapping. The input sequence is represented by a \(k\times l\) matrix where column \(t\) is the word vector for the \(t\)th word in the sequence. The word vectors are mapped by a stack of dlstm layers to multi-level feature representations \(z^{0},...,z^{d}\). As illustrated in Figure 1, each level feature representation is the sum of outputs from different lstm units. The multi-level features are concatenated and reduced to a \(dh\)-dimensional vector where \(d\) is the number of dlstm layers, \(h\) is the output size of each lstm unit. A classification layer gives the prediction output.

 

sentences. The binary classification task in this dataset has 6920 sentences for training, 872 sentences for developing and 1821 sentences for testing. There are in total 17835 unique running words for fine-grained dataset and 16185 for binary version dataset.

For query intent detection, ATIS (airline travel information system) dataset Hemphill et al. (1990); Yao et al. (2014b) is used. This dataset is mainly about the air travel domain with 26 different intents such as "flight", "\(ground_{s}ervice\)" and "city". There are 893 utterances for testing (ATIS-III, Nov93 and Dec94), and 4978 utterances for training (rest of ATIS-III and ATIS-II). There are 899 unique running words and 22 intents in the training data.

The question type classification task is to classify a question into a specific type, which is a very important step in question answering system. In TREC (Text Retrieval Conference) data Li and Roth (2002), all the questions are divided into 6 categories, including "human", "entity", "location", "description", "abbreviation" and "numeric". The dataset in total has 5952 questions, 5452 of them for training, the rest for testing. The vocabulary size of TREC dataset is 9592.

Following previous work Iyyer et al. (2015); Tai et al. (2015); Lei et al. (2015), we used word vectors pre-trained on large unannotated corpora to achieve better generalization capability. In this paper, we used a publicly available 300 dimensional GloVe word vectors that are trained using Common Crawl with 840B tokens and 2.2M vocabulary size.

### Settings

We implemented our model based on Theano library Bastien et al. (2012). All our models are trained on Nvidia Tesla K40m.

We performed extensive hyperparameter selection based on Stanford Sentiment Treebank Binary version of validation data. The selected hyperparameters were directly used for all datasets. To investigate the robustness of the proposed method, we ran each configuration 10 times using different random initialization (random seed ranges from 1 to 10).

For final models, we set the initial learning rate to 0.1, \(L2\) regularization weight to \(1e-5\), the dropout probability to 0.5 and mini-batch size to 64. We use hidden layer size 256 for all the models described in the experiments. The number of the dlstm layers and the number of the lstm units in each dlstm are both set to 3. So basically there are 9 latm units are used for each word position.

For all models, we set maximum iteration number 100 to terminate the training process. For sentiment classification task, during the training, the model with the best classification accuracy on validation data was used as final model for testing. For question type classification and query intent detection, there wasn't validation data. So we simply use the model trained at the 100th iteration as the final model for testing.

### Results on Stanford Sentiment Treebank

Table 1 lists results for s

\begin{table}
\begin{tabular}{p{113.8pt}|c|c} \hline model & Fine & Binary \\ \hline svmLei et al. (2015) & 38.3 & 81.3 \\ NbowLei et al. (2015) & 44.5 & 82.0 \\ Para-vecLe and Mikolov (2014) & 48.7 & 87.8 \\ DANIyyer et al. (2015) & 48.2 & 86.8 \\ \hline RAESocher et al. (2011b) & 43.2 & 82.4 \\ MVRNNSocher et al. (2012) & 44.4 & 82.9 \\ RNTN Socher et al. (2013) & 45.7 & 85.4 \\ DRNNIrsoy and Cardiie (2014) & 49.8 & 86.8 \\ RLSTMTai et al. (2015) & 51.0 & 88.0 \\ \hline CLSTMZhou et al. (2015) & 49.2 & 87.8 \\ DCNNKalchbrenner et al. (2014) & 48.5 & 86.9 \\ CNN-MCKim (2014) & 47.4 & 88.1 \\ CNN-nostaticKim (2014) & 48.0 & 87.2 \\ TCNN Lei et al. (2015) & 50.6 & 87.0 \\ TCNN+phrasesLei et al. (2015) & 51.2 & 88.6 \\ \hline ours & 49.2 & 87.2 \\ ours+phrases & **51.9** & **88.7** \\ \hline \end{tabular}
\end{table}
Table 1: Standford Sentiment Treebank Classification accuracy results. “Fine” denotes the accuracy on the fine-grained dataset with 5 labels. “Binary” denotes binary classification results.

\begin{table}
\begin{tabular}{p{113.8pt}|c} \hline model & Acc \\ \hline discriminativeTur et al. (2010) & 95.5 \\ svmShi et al. (2015b) & 95.6 \\ joint-RNNShi et al. (2015b) & 95.2 \\ ours & **97.9** \\ \hline \end{tabular}
\end{table}
Table 2: ATIS intent classification accuracy comparison of different models.

 

There are four blocks in the table. The bottom block gives the results from our model. The third blocks are methods related to cnns. The second block shows the results from recursive neural network based approaches. The other baseline methods are listed in the top block.

The top block shows that the traditional methods such as svm using ngram features and neural network using bag-of-words features (**Nbow**) perform much worse than **Para-vec** and **DAN** using word vectors that are pre-trained on large amount of unlabeled data. **Para-vec** builds a logistic regression on top of paragraph vectors. **DAN** is a deep neural network takes the average of word vectors as input.

In addition to pre-trained word vectors, syntactic compositional information can be used to improve the sentiment classification accuracy. **RAE** is a tree structured Autoencoder model based on pretrained word vectors from Wikipedia. **MVRNN** further improves the recursive neural network by assigning each node with a matrix to learn the meaning change of neighboring words and phrases. To address large amount of different vectors and matrices involved in **MVRNN**, **RNTN** proposed to use one single tensor based function to model all nodes. By making the tree-structured recursive neural networks deeper, significant improvement has been achieved by **DRNN**. According to our knowledge, the best compositional information based model is achieved by **RLSTM** that combines lstm unit with tree-structure.

By comparing the classification accuracy between second blocks and third blocks, we see that cnn based models in general perform better than recursive neural network based methods. Another advantage of cnn based methods is that they can be generalized to any language without dependency over compositional information. **DCNN** uses a dynamic \(k\)-max pooling operator function in cnn. To explore the task specific word vectors and the general word vectors pre-trained on large News dataset, **CNN-MC** equips cnn with two feature mapping channels. **CNN-nostatic** gives the results by only making use of general word vectors. The best published classification results are achieved by **TCNN** that is tensor based cnn.

In this paper, the proposed method is closely related to **TCNN**. Instead of using tensor products to replace linear convolution operation, our method exploits the nonlinear feature mapping through dlstm. Rather than setting specific decaying weight to model non-consecutive _n_-gram features in tensor based cnn, the different gates automatically adjust the information storing, removing and outputting according to context.

Following the work of **TCNN**, to leverage the phrases level annotation in Standford Sentiment Treebank, all phrases and their corresponding labels are added to training data as additional sequences. The bottom line of Table 1 shows that our models achieved the state-of-the-art performance on sentiment classification task.

For the best settings described above, we ran each model 10 times with different random initialization. The average and standard deviation for fine-grained classification are 50.7% and 1.04%, for binary classification 88% and 0.41%. Comparing with **TCNN**, our model is more sensitive to the parameter random initialization. In the future, some efforts should be used to analyze and address this issue.

### Results on ATIS

ATIS dataset is widely used to test spoken language understanding system. As shown in Table 2, svm using _n_-grams performs better than simple rnn and cnn based approach. **joint-RNN** is a query classification and slot filling joint training model where cnn is applied on top of slot tagging rnn for query classification. In this way, **joint-RNN** actually implicitly makes use of slot tag information for query classification. However, **joint-RNN** doesn't take ad

\begin{table}
\begin{tabular}{l|c} \hline model & Acc \\ \hline svm[36] & **95.0** \\ Para-vec[10] & 91.8 \\ AdaSent[23] & 92.4 \\ CNN-MC[30] & 92.2 \\ CNN-nostatic[31] & 93.6 \\ DCNN[18] & 93.0 \\ LSTM[24] & 93.2 \\ BiLSTM[24] & 93.0 \\ CLSTM[24] & 94.6 \\ \hline ours & 94.8 \\ \hline \end{tabular}
\end{table}
Table 3: TREC Question type Classification accuracy comparison of different models.

 

vantage of word vectors trained on large amount of unlabeled data. Based on pre-trained word vectors, our models obtain more than 2% absolute classification accuracy improvement over the published best model.

In ATIS data, about 70% of queries is categorized to "flight" intent. Recent work using rnn for utterance classification Ravuri and Stolcke (2015); Ravuri and Stolcke (2015) simplifies it to a "flight" VS "others" binary classification task. In their paper, using word based lstm, they achieve 97.55% classification accuracy. By using extra name entity features, word based gated rnn obtains 98.42% classification accuracy.

### Results on TREC Question Type Classification

Table 3 gives the TREC question type classification accuracy of our models with other baseline models. Different from the sentiment classification task, the shallow models using diverse engineered feature performs better than cnn and lstm based models. Previous best classification results on TREC data is achieved by svm using unigrams, bigrams, whword, head word, POS tags, hypernyms, WordNet synsets and a bunch of hand-coded rules.

**AdaSent** is a self adaptive hierarchical sentence model based on gating networks with level pooling. As shown in Table 3, cnn and lstm achieve similar performances on question type classification. Recently **CLSTM** achieves substantial improvement over previous neural network based methods. In **CLSTM**, cnn is used to extract high level phrase representation. Such local segment representation is

Figure 4: Example predictions given by our model trained on Stanford Sentiment Treebank fine-grained data. The expected sentiment score of each word is plotted in the figure. The score range from \(-2\) to \(2\), where a score \(-2\) means very “negative”, \(0\) stands for “neutral” and \(2\) means “very positive”.

Figure 3: Stanford Sentiment Treebank Binary Classification accuracy comparison among models using the same parameter configuration except the number of dlstm layers. For each number of dlstm layers, \(10\) models are run independently using different random initialization. Horizontal axis gives the validation accuracy. Vertical axis shows the test accuracy.

 

fed into lstm to model whole sequence representation. Different with \(\mathbf{CLSTM}\) that is an lstm based sequence model with cnn for local feature extraction, our model is cnn based model using dlstm for non-linear feature mapping. Our model outperforms previous neural network based models without relying on task specific feature engineering.

### Deep Architecture

One critical hyperparameter in the proposed method is the number of dlstm layers. On sentiment binary classification task, we run our model 10 times by keeping all the hyperparameters the same except the number of dlstm layers using different random initialization. As observed from Figure 3, the better performance is achieved by deeper architecture. Our model achieves the best classification result by stacking 3 dlstm layers that actually leverages 9 different lstm units to extract the nonlinear feature from _n_-grams.

### Examples

Figure 4 demonstrates some examples and their sentiments predicted by our model trained on fine-grained classification data. In order to see how the nonlinear feature mapping captures the sentiment at each word position in the query, we follow the strategy used in Lei et al. (2015) where the softmax function is directly applied on the concatenated feature mapping without passing through the average pooling layer. So the sentiment distribution \(p_{t}\) at _r_th word is computed as \(p_{t}=W^{T}[z_{t}^{0},z_{t}^{1},...,z_{t}^{d}]\). The expected value over the probability distribution \(\sum_{s=-2}^{2}s.p_{t}\) is used as the sentiment score that is plotted in Figure 4. In the figure, the sentiment score ranges from \(-2\) to \(2\), where \(-2\) means very negative, \(2\) mean very positive and \(0\) means neutral.

Five examples are illustrated in the figure where the first row gives the synthetic examples to show that our model is able to model the nonsconsecutive interaction within _n_-grams. For example, in query "hardly to be bad", even though word "hardly" is not directly modifying word "bad", our model still be able to capture such sentiment changes.

The second row of the figure shows the examples from fine-grained classification testing data. Both the example show that our model to some degree can capture sentiment of the satire. Especially the last example, our model actually gives negative prediction, even no word in the query really means negative.

## 5 Conclusions

We have proposed a deep long-short-term-memory (dlstm) nonlinear nonconsecutive feature mapping architecture to replace traditional linear mapping in the convolutional neural network based query classification. Each lstm unit in the dlstm is responsible for capturing different order feature representation from word segments. The bottom lstm unit equipped with input gate and output gate, extracts the nonlinear feature from unigram. The higher lstm unit in the dlstm takes the outputs from lower lstm units as input. In such way, the higher lstm unit is able to capture nonlinear feature representation from higher order _n_-grams. The sum of different lstm units is used as the output of the dlstm layer. The dlstm output rather than being directly used as input to convolutional neural network for query classification, is passed through a stacked dlstm layers. The query is finally represented by the concatenation of the outputs from the stacked dlstm layers.

We evaluated the proposed models on three benchmark datasets-Stanford Sentiment Treebank dataset, TREC dataset and ATIS dataset. On both sentiment classification dataset and ATIS dataset, our model achieved the state-of-the-art performance. On TREC question type classification, svm based model using extra engineered features still performed better than our model. But we noticed that the proposed method outperformed all the other neural network based approaches.

## References

* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _CoRR_.
* Bastien et al. (2012) Frederic Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. In _Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop_.
* Bengio (2009) Yoshua Bengio. 2009. Learning deep architectures for ai. _Found. Trends Mach. Learn._, 2:1-127 

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _EMNLP_.
* Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In _The Proceedings of the International Conference on Machine Learning_, pages 160-167.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Deng and Yu (2014) Li Deng and Dong Yu. 2014. Deep learning: Methods and applications. _Found. Trends Signal Process._, 7:197-387.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_, 12:2121-2159.
* Gao et al. (2014) Jianfeng Gao, Xiaodong He, Wen-tau Yih, and Li Deng. 2014. Learning continuous phrase representations for translation modeling. In _Proceedings of ACL_, pages 699-709.
* Graves et al. (2013) Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In _The proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_, pages 6645-6649.
* Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The atis spoken language systems pilot corpus. In _The Proceedings of the Workshop on Speech and Natural Language_, pages 96-101.
* Hinton et al. (2006) Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. _Neural Comput._, 18(7):1527-1554.
* Hinton et al. (2012) Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. _CoRR_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Irsoy and Cardie (2014) Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In _Proceedings of NIPS_, pages 2096-2104.
* Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In _Proceedings of ACL_, pages 1681-1691.
* Jean et al. (2015) Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In _Proceedings of ACL_, pages 1-10.
* Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In _Proceedings of ACL_, June.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751, October.
* Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proceedings of ICML_, pages 1188-1196.
* Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. _Proceedings of the IEEE_, 86(11):2278-2324, Nov.
* Lei et al. (2015) Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2015. Molding cnns for text: non-linear, non-consecutive convolutions. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1565-1575.
* Li and Roth (2002) Xin Li and Dan Roth. 2002. Learning question classifiers. In _Proceedings of the 19th International Conference on Computational Linguistics_, COLING '02, pages 1-7.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _The Proceedings of Interspeech_, pages 1045-1048.
* Mou et al. (2015) Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2015. Tree-based convolution: A new neural architecture for sentence modeling. _CoRR_.
* Ravuri and Stolcke (2015a) Suman Ravuri and Andreas Stolcke. 2015a. A comparative study of neural network models for lexical intent classification. In _The Proceedings of IEEE Automatic Speech Recogntion and Understanding Workshop_.
* Ravuri and Stolcke (2015b) Suman Ravuri and Andreas Stolcke. 2015b. Recurrent neural network and lstm models for lexical utterance classification. In _Proceedings of Interspeech_.
* Sainath et al. (2015) Tara N. Sainath, Oriol Vinyals, Andrew W. Senior, and Hasim Sak. 2015. Convolutional, long short-term memory, fully connected deep neural networks. In _Proceedings of ICASSP_, pages 4580-4584.
* Shen et al. (2014) Yelong Shen, Xiaodong he, Jianfeng Gao, Li Deng, and Gregoire Mesnil. 2014. Learning semantic representations using convolutional neural networks for web search. In _Proceedings of WWW_. WWW 2014, April.
* Shi et al. (2015a) Yangyang Shi, Kaisheng Yao, Hu Chen, Yi-Cheng Pan, and Mei-Yuh Hwang. 2015a. Semi-supervised spoken language understanding using recurrent transductive support vector machines. In _Proceeding of ASRU_ 

Yangyang Shi, Kaisheng Yao, Hu Chen, Yi-Cheng Pan, Mei-Yuh Hwang, and Baolin Peng. 2015b. Contextual spoken language understanding using recurrent neural networks. In _The Proceedings of International Conference on Acoustics, Speech and Signal Processing_.
* Silva et al. (2010) J. Silva, L. Coheur, A. C. Mendes, and Andreas Wichert. 2010. From symbolic to sub-symbolic information in question classification. _Artificial Intelligence Review_.
* Socher et al. (2011a) Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011a. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In _Proceedings of ICML_.
* Socher et al. (2011b) Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In _Proceedings of EMNLP_, pages 151-161.
* Socher et al. (2012) Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642.
* Socher (2012) Richard Socher. 2012. New directions in deep learning: Structured models, tasks, and datasets. _Neural Information Processing Systems (NIPS) Workshop on Deep Learning and Unsupervised Feature Learning_.
* Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jurgen Schmidhuber. 2015. Highway networks. _CoRR_, abs/1505.00387.
* Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In _INTERSPEECH_, pages 194-197.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_, pages 3104-3112.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of ACL_, pages 1556-1566. Association for Computational Linguistics, July.
* Tang et al. (2015) Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1422-1432.
* Tur et al. (2010) G. Tur, D. Hakkani-Tur, and L. Heck. 2010. What is left to be understood in atis? In _Spoken Language Technology Workshop (SLT), 2010 IEEE_, pages 19-24.
* Wang et al. (2015) Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2015. A unified tagging solution: Bidirectional LSTM recurrent neural network with word embedding. _CoRR_.
* Yao et al. (2014a) Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014a. Spoken language understanding using long short-term memory neural networks. In _The Proceedings of IEEE workshop on Spoken Language Technology_, pages 189-194.
* Yao et al. (2014b) Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014b. Spoken language understanding using long short-term memory neural networks. In _The Proceedings of IEEE workshop on Spoken Language Technology_, pages 189-194.
* Yao et al. (2015) Kaisheng Yao, Trevor Cohn, Katerina Vylomova, Kevin Duh, and Chris Dyer. 2015. Depth-gated LSTM. _CoRR_.
* Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _Proceedings of ACL_, pages 1321-1331.
* Zhang and Wallace (2015) Ye Zhang and Byron Wallace. 2015. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. _CoRR_.
* Zhang and Wallace (2016) Ye Zhang and Byron Wallace. 2016. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. In _CoRR_.
* Zhang et al. (2015) Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, and James Glass. 2015. Highway long short-term memory rnns for distant speech recognition. _CoRR_.
* Zhao et al. (2015) Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. _CoRR_.
* Zhou et al. (2015) Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. 2015. A C-LSTM neural network for text classification. _CoRR_, abs/1511.08630.