Venue: NAACL 2016

# Sequential Short-Text Classification with

Recurrent and Convolutional Neural Networks

 Ji Young Lee1

MIT

jjylee@mit.edu &Franck Dernoncourt2

MIT

francky@mit.edu

Footnote 1: These authors contributed equally to this work.

###### Abstract

Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.

## 1 Introduction

Short-text classification is an important task in many areas of natural language processing, including sentiment analysis, question answering, or dialog management. Many different approaches have been developed for short-text classification, such as using Support Vector Machines (SVMs) with rule-based features Silva et al. (2011), combining SVMs with naive Bayes Wang and Manning (2012), and building dependency trees with Conditional Random Fields Nakagawa et al. (2010). Several recent studies using ANNs have shown promising results, including convolutional neural networks (CNNs) Kim (2014); Blunsom et al. (2014); Kalchbrenner et al. (2014) and recursive neural networks Socher et al. (2012).

Most ANN systems classify short texts in isolation, i.e., without considering preceding shorttexts. However, short texts usually appear in sequence (e.g., sentences in a document or utterances in a dialog), and therefore using information from preceding short texts may improve the classification accuracy. Previous works on sequential short-text classification are mostly based on non-ANN approaches, such as Hidden Markov Models (HMMs) Reithinger and Klesen (1997); Stolcke et al. (2000); Surendran and Levow (2006), maximum entropy Ang et al. (2005), naive Bayes Lendvai and Geertzen (2007), and conditional random fields (CRFs) Kim et al. (2010); Quarteroni et al. (2011).

Inspired by the performance of ANN-based systems for non-sequential short-text classification, we introduce a model based on recurrent neural networks (RNNs) and CNNs for sequential short-text classification, and evaluate it on the dialog act classification task. A dialog act characterizes an utterance in a dialog based on a combination of pragmatic, semantic, and syntactic criteria. Its accurate detection is useful for a range of applications, from speech recognition to automatic summarization Stolcke et al. (2000). Our model achieves state-of-the-art results on three different datasets.

## 2 Model

Our model comprises two parts. The first part generates a vector representation for each short text using either the RNN or CNN architecture, as discussed in Section 2.1 and Figure 1. The second part classifies the current short text based on the vector representations of the current as well as a few preceding short texts, as presented in Section 2.2 and Figure 2.

We denote scalars with italic lowercases (e.g.,

\(k,\,b_{f}\)), vectors with bold lowercases (e.g., \(\mathbf{s},\,\mathbf{x}_{i}\)), and matrices with italic uppercases (e.g., \(W_{f}\)). We use the colon notation \(\mathbf{v}_{i:j}\) to denote the sequence of vectors \((\mathbf{v}_{i},\mathbf{v}_{i+1},\ldots,\mathbf{v}_{j})\).

### Short-text representation

A given short text of length \(\ell\) is represented as the sequence of \(m\)-dimensional word vectors \(\mathbf{x}_{1:\ell},\) which is used by the RNN or CNN model to produce the \(n\)-dimensional _short-text representation_\(\mathbf{s}\).

#### 2.1.1 RNN-based short-text representation

We use a variant of RNN called Long Short Term Memory (LSTM) Hochreiter and Schmidhuber (1997). For the \(t^{th}\) word in the short-text, an LSTM takes as input \(\mathbf{x}_{t},\mathbf{h}_{t-1},\mathbf{c}_{t-1}\) and produces \(\mathbf{h}_{t},\mathbf{c}_{t}\) based on the following formulas:

\[\begin{split}&\mathbf{i}_{t}=\sigma(W_{i}\mathbf{x}_{t}+U_{i} \mathbf{h}_{t-1}+\mathbf{b}_{i})\\ &\mathbf{f}_{t}=\sigma(W_{f}\mathbf{x}_{t}+U_{f}\mathbf{h}_{t-1}+ \mathbf{b}_{f})\\ &\tilde{\mathbf{c}}_{t}=\mbox{tanh}(W_{c}\mathbf{x}_{t}+U_{c} \mathbf{h}_{t-1}+\mathbf{b}_{c})\\ &\mathbf{c}_{t}=\mathbf{f}_{t}\odot\mathbf{c}_{t-1}+\mathbf{i}_{ t}\odot\tilde{\mathbf{c}}_{t}\\ &\mathbf{o}_{t}=\sigma(W_{o}\mathbf{x}_{t}+U_{o}\mathbf{h}_{t-1}+ \mathbf{b}_{o})\\ &\mathbf{h}_{t}=\mathbf{o}_{t}\odot\mbox{tanh}(\mathbf{c}_{t})\\ \end{split}\]

where \(W_{j}\in\mathbb{R}^{n\times m},~{}U_{j}\in\mathbb{R}^{n\times n}\) are weight matrices and \(\mathbf{b}_{j}\in\mathbb{R}^{n}\) are bias vectors, for \(j\in\{i,f,c,o\}\). The symbols \(\sigma(\cdot)\) and \(\mbox{tanh}(\cdot)\) refer to the elementwise sigmoid and hyperbolic tangent functions, and \(\odot\) is the element-wise multiplication. \(\mathbf{h}_{0}=\mathbf{c}_{0}=\mathbf{0}\).

In the pooling layer, the sequence of vectors \(\mathbf{h}_{1:\ell}\) output from the RNN layer are combined into a single vector \(\mathbf{s}\in\mathbb{R}^{n}\) that represents the short-text, using one of the following mechanisms: last, mean, and max pooling. Last pooling takes the last vector, i.e., \(\mathbf{s}=\mathbf{h}_{\ell}\), mean pooling averages all vectors, i.e., \(\mathbf{s}=\frac{1}{\ell}\sum_{t=1}^{\ell}\mathbf{h}_{t}\), and max pooling takes the elementwise maximum of \(\mathbf{h}_{1:\ell}\).

#### 2.1.2 CNN-based short-text representation

Using a _filter_\(W_{f}\in\mathbb{R}^{h\times m}\) of height \(h,\) a convolution operation on \(h\) consecutive word vectors starting from \(t^{th}\) word outputs the scalar _feature_

\[c_{t}=\textrm{ReLU}(W_{f}\bullet X_{t:t+h-1}+b_{f})\]

where \(X_{t:t+h-1}\in\mathbb{R}^{h\times m}\) is the matrix whose \(i^{th}\) row is \(\mathbf{x}_{i}\in\mathbb{R}^{m},\) and \(b_{f}\in\mathbb{R}\) is a bias. The symbol \(\bullet\) refers to the dot product and ReLU\((\cdot)\) is the elementwise rectified linear unit function.

We perform convolution operations with \(n\) different filters, and denote the resulting features as

Figure 1: RNN (left) and CNN (right) architectures for generating the vector representation \(\mathbf{s}\) of a short text \(\mathbf{x}_{1:\ell}\). For CNN, Conv refers to convolution operations, and the filter height \(h=3\) is used in this figure.

 

\({\bf c}_{t}\in\mathbb{R}^{n},\) each of whose dimensions comes from a distinct filter. Repeating the convolution operations for each window of \(h\) consecutive words in the short-text, we obtain \({\bf c}_{1:\ell-h+1}.\) The short-text representation \({\bf s}\in\mathbb{R}^{n}\) is computed in the max pooling layer, as the element-wise maximum of \({\bf c}_{1:\ell-h+1}.\)

### Sequential short-text classification

Let \({\bf s}_{i}\) be the \(n\)-dimensional short-text representation given by the RNN or CNN architecture for the \(i^{th}\) short text in the sequence. The sequence \({\bf s}_{i-d_{1}-d_{2}\,:\,i}\) is fed into a two-layer feedforward ANN that predicts the class for the \(i^{th}\) short text. The hyperparameters \(d_{1},d_{2}\) are the history sizes used in the first and second layers, respectively.

The first layer takes as input \({\bf s}_{i-d_{1}-d_{2}\,:\,i}\) and outputs the sequence \({\bf y}_{i-d_{2}\,:\,i}\) defined as

\[{\bf y}_{j}=\mbox{tanh}\left(\sum_{d=0}^{d_{1}}W_{-d}\,{\bf s}_{j- d}+{\bf b}_{1}\right),~{}\forall j\in[i-d_{2},i]\]

where \(W_{0},W_{-1},W_{-2}\in\mathbb{R}^{k\times n}\) are the weight matrices, \({\bf b}_{1}\in\mathbb{R}^{k}\) is the bias vector, \({\bf y}_{j}\in\mathbb{R}^{k}\) is the _class representation_, and \(k\) is the number of classes for the classification task.

Similarly, the second layer takes as input the sequence of class representations \({\bf y}_{i-d_{2}:i}\) and outputs \({\bf z}_{i}\in\mathbb{R}^{k}\):

\[{\bf z}_{i}=\mbox{softmax}\left(\sum_{j=0}^{d_{2}}W_{-j}\,{\bf y}_{ i-j}+{\bf b}_{2}\right)\]

where \(U_{0},U_{-1},U_{-2}\in\mathbb{R}^{k\times k}\) and \({\bf b}_{2}\in\mathbb{R}^{k}\) are the weight matrices and bias vector.

The final output \({\bf z}_{i}\) represents the probability distribution over the set of \(k\) classes for the \(i^{th}\) short-text: the \(j^{th}\) element of \({\bf z}_{i}\) corresponds to the probability that the \(i^{th}\) short-text belongs to the \(j^{th}\) class.

## 3 Datasets and Experimental Setup

### Datasets

We evaluate our model on the dialog act classification task using the following datasets:

* DSTC 4: Dialog State Tracking Challenge 4 Kim et al. (2015); Kim et al. (2016).
* MRDA: ICSI Meeting Recorder Dialog Act Corpus Janin et al. (2003); Shriberg et al. (2004). The 5 classes are introduced in Ang et al. (2005).
* SwDA: Switchboard Dialog Act Corpus Jurafsky et al. (1997).

For MRDA, we use the train/validation/test splits provided with the datasets. For DSTC 4 and SwDA, only the train/test splits are provided.1 Table 1 presents statistics on the datasets.

Footnote 1: All train/validation/test splits can be found at [https://github.com/Franck-Dernoncourt/naacl2016](https://github.com/Franck-Dernoncourt/naacl2016)

### Training

The model is trained to minimize the negative log-likelihood of predicting the correct dialog acts of the utterances in the train set, using stochastic gradient descent with the Adadelta update rule Zeiler (2012). At each gradient descent step, weight matrices, bias vectors, and word vectors are updated. For regularization, dropout is applied after the pooling layer, and early stopping is used on the validation set with a patience of 10 epochs.

## 4 Results and Discussion

To find effective hyperparameters, we varied one hyperparameter at a time while keeping the other ones fixed. Table 2 presents our hyperparameter choices.

We initialized the word vectors with the 300-dimensional word vectors pretrained with word2vec

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline
**Dataset** & \(|C|\) & \(|V|\) & Train & Validation & Test \\ \hline DSTC 4 & 89 & 6k & 24 (21k) & 5 (5k) & 6 (6k) \\ MRDA & 5 & 12k & 51 (78k) & 11 (16k) & 11 (15k) \\ SwDA & 43 & 20k & 1003 (193k) & 112 (23k) & 19 (5k) \\ \hline \end{tabular}
\end{table}
Table 1: Dataset overview. \(|C|\) is the number of classes, \(|V|\) the vocabulary size. For the train, validation and test sets, we indicate the number of dialogs (i.e., sequences) followed by the number of utterances (i.e., short texts) in parenthesis.

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Hyperparameter** & **Choice** & **Experiment Range** \\ \hline LSTM output dim. \((n)\) & 100 & 50 – 1000 \\ LSTM pooling & max & max, mean, last \\ LSTM direction & unidir. & unidir., bidir. \\ CNN num. of filters \((n)\) & 500 & 50 – 1000 \\ CNN filter height \((h)\) & 3 & 1 – 10 \\ Dropout rate & 0.5 & 0 – 1 \\ Word vector dim. \((m)\) & 200, 300 & 25 – 300 \\ \hline \end{tabular}
\end{table}
Table 2: Experiments ranges and choices of hyperparameters. Unidir refers to the regular RNNs presented in Section 2.1.1, and bidir refers to bidirectional RNNs introduced in Schuster and Paliwal (1997).

 

on Google News Mikolov et al. (2013); Mikolov et al. (2013) for DSTC 4, and the 200-dimensional word vectors pretrained with GloVe on Twitter Pennington et al. (2014) for MRDA and SwDA, as these choices yielded the best results among all publicly available word2vec, GloVe, SENNA Collobert (2011); Collobert et al. (2011) and RNNLM Mikolov et al. (2011) word vectors.

The effects of the history sizes \(d_{1}\) and \(d_{2}\) for the short-text and the class representations, respectively, are presented in Table 3 for both the LSTM and CNN models. In both models, increasing \(d_{1}\) while keeping \(d_{2}=0\) improved the performances by 1.3-4.2 percentage points. Conversely, increasing \(d_{2}\) while keeping \(d_{1}=0\) yielded better results, but the performance increase was less pronounced: incorporating sequential information at the short-text representation level was more effective than at the class representation level.

Using sequential information at both the short-text representation level and the class representation level does not help in most cases and may even lower the performances. We hypothesize that short-text representations contain richer and more general information than class representations due to their larger dimension. Class representations may not convey any additional information over short-text representations, and are more likely to propagate errors from previous misclassifications.

Table 4 compares our results with the state-of-the-art. Overall, our model shows competitive results, while requiring no human-engineered features. Rigorous comparisons are challenging to draw, as many important details such as text preprocessing and train/valid/test split may vary, and many studies fail to perform several runs despite the randomness in some parts of the training process, such as weight initialization.

## 5 Conclusion

In this article we have presented an ANN-based approach to sequential short-text classification. We demonstrate that adding sequential information im

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**Model** & DSTC 4 & MRDA & SwDA \\ \hline CNN & 65.5 & **84.6** & **73.1** \\ LSTM & **66.2** & 84.3 & 69.6 \\ Majority class & 25.8 & 59.1 & 33.7 \\ SVM & 57.0 & – & – \\ Graphical model & – & 81.3 & – \\ Naive Bayes & – & 82.0 & – \\ HMM & – & – & 71.0 \\ Memory-based Learning & – & – & 72.3 \\ Interlabeler agreement & – & – & 84.0 \\ \hline \end{tabular}
\end{table}
Table 4: Accuracy (%) of our models and other methods from the literature. The majority class model predicts the most frequent class. SVM: Dernoncourt et al. (2016). Graphical model: Ji and Bilmes (2006). Naive Bayes: Lendvai and Geertzen (2007). HMM: Stolcke et al. (2000). Memory-based Learning: Rotaru (2002). All five models use features derived from transcribed words, as well as previous predicted dialog acts except for Naive Bayes. The interlabeler agreement could not be obtained for MRDA, and DSTC 4 was labeled by a single annotator. For the CNN and LSTM models, the presented results are the test set accuracy of the run with the highest accuracy on the validation set.

\begin{table}
\begin{tabular}{|p{28.5pt} p{28.5pt}|p{28.5pt} p{28.

 

proves the quality of the predictions, and the performance depends on what sequential information is used in the model. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.

## Acknowledgments

We warmly thank Regina Barzilay, Tommi Jaakkola and the anonymous reviewers for their helpful feedback and suggestions.

## References

* [Ang et al.2005] Jeremy Ang, Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In _ICASSP (1)_, pages 1061-1064.
* [Blunsom et al.2014] Phil Blunsom, Edward Grefenstette, Nal Kalchbrenner, et al. 2014. A convolutional neural network for modelling sentences. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
* [Cho et al.2014] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. _arXiv preprint arXiv:1409.1259_.
* [Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _The Journal of Machine Learning Research_, 12:2493-2537.
* [Collobert2011] Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In _International Conference on Artificial Intelligence and Statistics_, number EPFL-CONF-192374.
* [Dernoncourt et al.2016] Franck Dernoncourt, Ji Young Lee, Trung H. Bui, and Hung H. Bui. 2016. Adobe-MIT submission to the DSTC 4 Spoken Language Understanding pilot task. In _7th International Workshop on Spoken Dialogue Systems (IWSDS)_.
* [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* [Janin et al.2003] Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The ICSI meeting corpus. In _Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03). 2003 IEEE International Conference on_, volume 1, pages I-364. IEEE.
* [Ji and Bilmes2006] Gang Ji and Jeff Bilmes. 2006. Backoff model training using partially observed data: application to dialog act tagging. In _Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics_, pages 280-287. Association for Computational Linguistics.
* [Jurafsky et al.1997] Dan Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBDDAMSL shallow-discourse-function annotation coders manual. _Institute of Cognitive Science Technical Report_, pages 97-102.
* [Kalchbrenner et al.2014] Nak Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. _arXiv preprint arXiv:1404.2188_.
* [Kim et al.2010] Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 862-871. Association for Computational Linguistics.
* [Kim et al.2015] Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Bancls, Jason Williams, and Matthew Henderson. 2015. Dialog State Tracking Challenge 4: Handbook.
* [Kim et al.2016] Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Bancls, Jason Williams, and Matthew Henderson. 2016. The Fourth Dialog State Tracking Challenge. In _Proceedings of the 7th International Workshop on Spoken Dialogue Systems (IWSDS)_.
* [Kim2014] Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1746-1751. Association for Computational Linguistics.
* [Landvai and Geertzen2007] Piroska Lendvai and Jeroen Geertzen. 2007. Token-based chunking of turn-internal dialogue act sequences. In _Proceedings of the 8th SIGDIAL Workshop on Discourse and Dialogue_, pages 174-181.
* [Mikolov et al.2011] Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In _Proc. of the 2011 ASRU Workshop_, pages 196-201.
* [Mikolov et al.2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* [Mikolov et al.2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and 

their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* [Nakagawa et al.2010] Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 786-794. Association for Computational Linguistics.
* [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: global vectors for word representation. _Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014)_, 12:1532-1543.
* [Quarteroni et al.2011] Silvia Quarteroni, Alexei V Ivanov, and Giuseppe Riccardi. 2011. Simultaneous dialog act segmentation and classification from human-human spoken conversations. In _Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on_, pages 5596-5599. IEEE.
* [Reithinger and Klesen1997] Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In _EuroSpeech_. Citeseer.
* [Rotaru2002] Mihai Rotaru. 2002. Dialog act tagging using memory-based learning. _Term project, University of Pittsburgh_, pages 255-276.
* [Schuster and Paliwal1997] Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _Signal Processing, IEEE Transactions on_, 45(11):2673-2681.
* [Shriberg et al.2004] Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, DTIC Document.
* [Silva et al.2011] Joao Silva, Luisa Coheur, Ana Cristina Mendes, and Andreas Wichert. 2011. From symbolic to sub-symbolic information in question classification. _Artificial Intelligence Review_, 35(2):137-154.
* [Socher et al.2012] Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 1201-1211. Association for Computational Linguistics.
* [Stolcke et al.2000] Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Esslykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. _Computational linguistics_, 26(3):339-373.
* [Surendran and Levow2006] Dinoj Surendran and Giana-Anne Levow. 2006. Dialog act tagging with support vector machines and hidden markov models. In _INTERSPEECH_.
* [Wang and Manning2012] Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2_, pages 90-94. Association for Computational Linguistics.
* [Zeiler2012] Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. _arXiv preprint arXiv:1212.5701_.