Venue: NAACL 2016

# Recurrent Support Vector Machines For Slot Tagging In Spoken Language Understanding

 Yangyang Shi

Kaisheng Yao

Hu Chen

Dong Yu

Yi-Cheng Pan

Mei-Yuh Hwang

Microsoft

{yanshi,kaisheng.yao,huch,dong.yu,ycpan,mehwang}@microsoft.com

###### Abstract

We propose recurrent support vector machine (rsvm) for slot tagging. This model is a combination of the recurrent neural network (rnn) and the structured support vector machine. rnn extracts features from the input sequence. The structured support vector machine uses a sequence-level discriminative objective function. The proposed model therefore combines the sequence representation capability of an rnn with the sequence-level discriminative objective. We have observed new state-of-the-art results on two benchmark datasets and one private dataset. rsvm obtained statistical significant 4% and 2% relative average F1 score improvement on ATIS dataset and Chunking dataset, respectively. Out of eight domains in Cortana live log dataset, rsvm achieved F1 score improvement on seven domains. Experiments also show that rsvm significantly speeds up the model training by skipping the weight updating for non-support vector training samples, compared against training using rnn with crf of minimum cross-entropy objectives.

## 1 Introduction

One of the key tasks in natural language understanding [5, 3, 6]: lipa_tml_2003, has just been used to tag the labels user queries with semantic tags. It is a sequence labeling problem that transcribes a sequence of observations \(X=[x(1),x(2),...,x(M)]\) to a sequence of discrete labels \(Y=[y(1),y(2),...,y(M)]\). For example, in the query "show me flights from Seattle to Boston", the words "Seattle" and "Boston" should be labeled, respectively, as the _from-city-name_ slot and the _to-city-name_slot.

Recently recurrent neural networks (rnns) and their variants achieved state-of-the-art performances on slot tagging tasks [8, 9, 7, 10, 11, 12, 13, 14, 15]. One direction to improve the sequence labeling is to strengthen the model memorization capability by designing dedicated special structures, for example, using long-short-term-memory (lstm) networks [12, 13, 14], gated rnn and rnn with external memory (rnn-em) [2]. The other direction is to optimize the sequence-level discrimination criterion. For example, recurrent conditional random fields (rcrfs) [13] is trained to optimize the sequence conditional likelihood rather than minimizing frame level cross-entropy applied in conventional rnn based sequence labeling [2, 13, 14, 15, 16].

In this paper, we propose recurrent support vector machines (rsyms) to improve the discrimination ability of rnns. Different from rcrfs and conventional rnns that in essence apply the multinomial logistic regression on the output layer, rsvms optimize the sequence-level max-margin training criterion used by structured support vector machines [15] on the output layer of rnns. There are several advantages of using sequence-level max-margin training over maximum likelihood or minimum cross-entropy. Firstly, the sequence-level max-margin criterion is a _global un-normalized_ criterion in which there is no computation cost for normalization. Secondly, using max-margin training, only training samples from support vectors generate non zero errors. In other words, model training can be sped up by skipping the weight updating for non-support vector training samples. Finally, as proven in [11], margin maximization is equivalent to minimization of an upper bound on the generalization errors. Max-margin training has no assumption about the model distribution. To use maximum likelihood or minimum cross-entropy, it assumes that the model distribution is peaked. However, especially in natural language processing where the ambiguity is ubiquitous, this assumption does not hold.



For example, "seven eleven" can be labeled as time tag or place name (super market name) tag. The conditional probability of tag given "seven eleven" should not be sharp for time or place name.

Recently, svm is also applied on top of a deep neural network for speech recognition [25]. In their work, a cutting-plane algorithm [19] is used, which is computationally expensive for speech recognition tasks. In this paper, we use the stochastic gradient descent algorithm sgd [30] for model training. The loss function is critical to the sequence level max-margin training criterion, which defines the margin. In this paper, we apply the sequence level hard loss function rather than traditional Hamming loss function [47]. In sequence level hard loss function, the wrong sequence is assigned loss one without considering the number of wrong slot labels in the sequence. In the experiments on two bench mark datasets, namely the ATIS [18, 6] and the CoNLL 2000 Chunking dataset 1, and private Cortana live log dataset, rsvms outperformed previous results.

Footnote 1: See [http://www.cnts.ua.ac.be/conllZ000/chunking](http://www.cnts.ua.ac.be/conllZ000/chunking).

## 2 Recurrent Support Vector Machines

In this section, we propose rsvm that uses the structured svm algorithm [19] to estimate the weights for rnn and label transition probabilities based on the entire training sequence. The training objective in rsvm is the following constrained optimization.

\[\min_{W,A}\frac{1}{2}||W||^{2}_{2}+\frac{1}{2}||A||^{2}_{2}+C\sum_ {k=1}^{K}\zeta_{k}\] \[s.t.f(Y^{(k)*})+\zeta_{k}\geq f(Y^{(k)})+L(Y^{(k)})\quad\forall Y ^{(k)} \tag{1}\] \[L(Y^{(k)})\geq 0\quad\forall Y^{(k)}\] (2) \[\zeta_{k}\geq 0\] (3) where \[f(Y^{(k)})=\sum_{i=1}^{T}a_{y_{k}(t-1)y_{k}(t)}+W^{T}_{y_{k}(t)}H(t) \tag{4}\]

where \(C\) is regularization weight for empirical loss. \(Y^{(k)}\) represents the slot label sequence \(y^{(k)}(1:T)\) for training sample \(X^{(k)}\). \(Y^{(k)*}\) is the ground truth slot sequence \(y^{(k)*}(1:T)\) for \(X^{(k)}\). \(a_{y^{(k)}(t-1)y^{(k)}(t)}\) is one element in matrix \(A\), representing the weight for the slot label transition features from \(y^{(k)}(t-1)\) to \(y^{(k)}(t)\). \(L(Y^{(k)})\) defines the loss function of a possible slot label sequence for a training sample \(X^{(k)}\) , which is actually used as a margin to separate the score \(f(Y^{(k)*})\) for ground truth slot label sequence with all other possible slot sequences in Eq. (1). \(\zeta^{(k)}\) is the slack variable that penalize the slot label sequence that violates the margin constraint.

The constrained optimization problem can be transformed to an unconstrained optimization problem as

\[\min_{W,A} F(W,A)=\frac{1}{2C}||W||^{2}_{2}+\frac{1}{2C}||A||^{2}_{2}\] \[+\sum_{k=1}^{K}[\max_{Y^{(k)}\neq Y^{(k)*}}\bigl{(}f(Y^{(k)})+L( Y^{(k)})\bigr{)}-f(Y^{(k)*})]_{+}. \tag{5}\]

where \([x]_{+}\) is the Hinge function that maps \(x\) to zero when \(x\) is smaller than zero, otherwise \([x]_{+}=x\).

The loss function \(L(Y^{(k)})\) is critical to the structured svm training. The following two types of loss functions have been investigated:

\[L(Y^{(k)}) =\sum_{t=1}^{T}1\big{(}y^{(k)}(t)\neq y^{(k)*}(t)\big{)} \tag{6}\] \[L(Y^{(k)}) =1\big{(}y^{(k)}(1:T)\neq y^{(k)*}(1:T)\big{)} \tag{7}\]

Eq. (6) is Hamming loss that is applied by [47] for structured svm sequence labeling. Eq. (7) is sequence level hard loss function that always give loss one to wrong slot label sequences no matter how many words are labeled with wrong slot labels. In our experiment, we find that the margin defined by Eq. (7) gives the best performance.

### Training Procedure For Recurrent Support Vector Machines

Fig. 1 depicts the architecture of rsvm that can be viewed as the conventional knn unrolled over the sequence \(x(1:T)\). For each single training sample \(x(1:T)\), a forward inference and a backward learning are carried out to sweep over the network shown in Fig. 1.

In the forward inference, an unnormalized slot score vector \(y(t)\) is computed based on each word input \(x(t)\) and its corresponding auxiliary feature \(Cx(t)\). The word input and auxiliary feature are encoded in one-hot representation. As shown in Fig. 1, a slot label lattice is generated for the training sample \(x(1:T)\). Using Viterbi algorithm, two best slot label sequences \(Y^{(k)\text{top}}\) and \(Y^{(k)\text{second}}\) are derived from the lattice. In the decoding phase, only the best slot label sequence is computed.

In backward learning, the sub-gradient [13] are calculated to update the weights for rsvms. When \(Y^{(k)\text{top}}\neq Y^{(k)*}\) the sub-gradient is

\[\frac{\partial\zeta_{k}}{\partial\theta}=\frac{\partial f(Y^{(k)\text{top}})}{ \partial\theta}-\frac{\partial f(Y^{(k)*})}{\partial\theta}. \tag{8}\]

When \(Y^{(k)\text{top}}=y^{(k)*}\) and \(y^{(k)*}-Y^{(k)\text{second}}<L(Y^{(k)})\), the 

sub-gradient is

\[\frac{\partial\zeta_{k}}{\partial\theta}=\frac{\partial f(Y^{(k)} ^{\text{second}})}{\partial\theta}-\frac{\partial f(Y^{(k)}^{*})}{\partial\theta}. \tag{9}\]

When \(Y^{(k)}\textsuperscript{top}=Y^{(k)}^{*}\) and \(y^{(k)}^{*}-Y^{(k)}\textsuperscript{second}\geq L(Y^{(k)})\), the subgradient is zero. Our experiment show that the rsvm training can be substantially sped up by skipping the backward weight updating for non-support vector training samples that obtain zero subgradient.

In Eq. (8) and (9), \(\theta\) represents the weights in rsvms. Specifically, the weights \(W\), \(A\), \(U\) and \(V\) are updated using sequence level mini-batch method. The weights \(O\) connecting hidden layers are updated using Backpropagation Through Time (bptt) (werbos, 1990).

## 3 Experiments

### Data

To evaluate performances of the proposed model, three sets of experiments were conducted. The first set of experiments are based on ATIS dataset Hemphill et al. (1990b); Yao et al. (2014b). There are 893 queries from ATIS-III, Nov93 and Dec94 for testing, and 4978 utterances from the rest of ATIS-III and ATIS-II used for training. The training data contains 127 unique slot tags.

The second dataset used in the experiment is CoNLL 2000 Chunking dataset. Chunking is also called shallow parsing that assigns syntactic labels to segments of a sequence of words. Chunking and slot tagging are typical sequence labeling problems. In this paper, we use the chunking task to further verify the performance of the proposed rsvm model. In the CoNLL 2000 Chunking task, the training data are from sections 15-18 of WSJ data and the test data are from section 20. In the training data, there are 220663 tokens with 19123 unique words and additional 45 different types of PartOf-Speech (pos).

The last dataset is Cortana live log dataset which is constituted by 8 domains, namely alarm, calender, communication, note, ondevice, places, reminder and weather. In total, there are 71 slots. There are 42506 queries used for training and 5290 queries for testing. The data distribution is described in Table. 1. The last column of Table. 1 shows the average query length (the number of words in one query) on different domains.

### Settings

In this paper, we use a predefined maximum iteration number to terminate the training. The learning rate is dynamically adjusted using AdaGrad Duchi et al. (2011).

In all the experiments, we set the hidden layer size to 300 and initial learning rate to 0.1. In rsvms, the surrounding two words of the current word are used as auxiliary feature which is represented as bag of words. We set the maximum iteration to 20 for ATIS and 30 for Chunking and Cortana live log. For each dataset, we trained 10 models with the same parameter settings except using different random initialization.

### Results on ATIS

ATIS is a well studied benchmark dataset. Table. 2 gives the slot tagging F1 scores achieved by different models in the literature, using the same data settings. There are three blocks in Table. 2. The top block gives the F1 score obtained by crf and simple rnn. The middle block gives the results obtained by applying advanced rnn architectures such as lstm, Gated rnn and rnn with external memories (rnn-em) Peng and Yao (2015). These advanced rnns improves rnn by enhancing its memory (sequence representation capability). The bottom block gives results using the proposed rsvms.

The bottom part gives F1 scores of the proposed rsvm method. We show results generated by 10 models with different random seeds. The average F1 score of rsvm is similar to the best score of rnn-em. F1 score distribution of 10 rsvm models gets significant improvement over the average score of rnn-em (z-test \(p-value=0.0002<<0.05\)). Fundamentally, the proposed rsvm

Figure 1: Recurrent support vector machines for slot tagging. \(U\) is the weight matrix connecting the word input to the hidden layer, \(V\) is the weight matrix connecting auxiliary feature input to the hidden layer, \(O\) is the weight matrix connecting previous hidden state to the current hidden state and \(W\) is the weight matrix connecting the hidden layer to the output layer. \(A\) represents the weight for slot label transition features.

\begin{table}
\begin{tabular}{l|c|r|r} \hline domain & train & test & length \\ \hline alarm & 3816 & 452 & 4.3 \\ calendar & 4138 & 475 & 4.5 \\ communication & 9551 & 1262 & 3.8 \\ note & 829 & 139 & 4.6 \\ endevice & 4384 & 572 & 2.4 \\ places & 6167 & 753 & 4.7 \\ reminder & 5359 & 720 & 4.1 \\ weather & 8270 & 917 & 3.4 \\ \hline \end{tabular}
\end{table}
Table 1: Cortana live log data distribution.

 

is based on simple rnn. Comparing lstm and rnn-em, the proposed model has simpler topology. Note that in [13] and [20], their advanced models are trained using local normalization method without using sequence level optimization. So the superiority of the proposed rsvm may come from the sequence training and the powerful discriminant capability of svm. Applying the proposed rsvm method to lstm or other advanced rnn can be a promising direction for future work.

### Results on CoNLL 2000 Chunking

Table. 3 gives the F1 scores of different models on CoNLL 2000 chunking experiment. To our best knowledge, the first neural network nn based chunking model is proposed in [8]. Using four basic natural language processing tasks, namely pos tagging, chunking, name entity recognition and semantic role labeling, they demonstrate the ability of nn to discover hidden representations. In their work, only simple input feature is used. There is not any task-specific feature engineering work in their proposed system. Their model purely relies on the nn feature representation that are learned from large amount of unlabeled data. As shown in Table. 3, their system performs better than all the previous systems on CoNLL 2000 chunking dataset.

The performance of Bidirectional lstm (blstm), rcrf and the proposed rsvm on chunking task further confirms the conclusion in [7] that nn is able to discover the internal representations that are useful for different natural language processing tasks. Additionally, the results of blstm, rcrf and rsvm, indicate that rnns have better capabilities to discover the sequence representation than nn. The average F1 score of rcrf and rsvm are 94.9% and 95.0%, respectively. Comparing the F1 score distribution, rsvm achieves the significant improvement over rcrf (paired t-test \(p-value=0.012<0.05\)). As shown in Table. 3, replacing the crf objective function with structured svm max-margin criterion could generate further improvement. The average performance of rsvm is better than the best result of rcrf shown in the table.

### Results on Internal Live dataset

In this section, we compare different slot models on different domains based on Cortana live log data.

Table. 4 compares the F1 score on crf, rnn, rcrf, joint-rnn and the proposed rsvm on alarm, calendar, communication, and note. Table. 5 presents the F1 score of different models on the rest domains. "rnn" denotes the Elman type of rnn for slot tagging which uses current word information, previous slot output information and context window information (surrounding four words) [16]. "rcrf" represents the rcrf slot tagging models that use the same feature as "rnn" [16]. "joint-rnn" [17] also uses the same features as "rnn" and "rcrf". However, "joint-rnn" implicitly makes use of query domain, intent and slot information by training the domain classifier, intent classifier and slot labeling jointly via multi-task learning.

Overall, the proposed rsvm obtains significant improvement over crf, rnn, rcrf and joint-rnn on alarm, communication, note and reminder (z-test \(p-value<5E-5\)). On the calendar, places and weather, rsvm achieves similar performance as joint-rnn. Even joint

\begin{table}
\begin{tabular}{l|c} \hline model & F1(\%) \\ \hline crf [17] & 92.9 \\ dbn [13] & 93.2 \\ rnn [13] & 94.1 \\ rnn-Jordan[17] & 94.3 \\ rnn-embed[13] & 94.4 \\ rnn-joint [17] & 94.6 \\ rnn-hybrid[17] & 95.1 \\ \hline deep-lstm [16] & 95.0 \\ grmn-max[20] & 94.7 \\ rnn-em-min[20] & 94.7 \\ rnn-em-average[20] & 95.0 \\ rnn-em-max[20] & 95.2 \\ \hline rsvm-min & 94.9 \\ rsvm-average & 95.2 \\ rsvm-max & 95.5 \\ \hline \end{tabular}
\end{table}
Table 2: F1 score (in %) for slot tagging on ATIS achieved by different models using only lexical feature. ”-min”, “-max”, and ”-average” each denotes minimum, maximum and average F1 scores for a corresponding method.

\begin{table}
\begin{tabular}{l|c} \hline model & F1(\%) \\ \hline sym [18] & 93.5 \\ gen-Winnow[13] & 93.9 \\ syn [18] & 93.9 \\ crf [17] & 94.3 \\ crf [16] & 94.4 \\ crf [16] & 94.5 \\ rsrvm-max & 94.5 \\ rssvm-average & 95.0 \\ rssvm-max & 95.1 \\ \hline \end{tabular}
\end{table}
Table 3: F1 score (in %) for chunking on CoNLL 2000 shared task using different models. All these models use word features as well as pos features. ”-min”, ”-max”, and ”-average” each denotes minimum, maximum and average F1 scores for a corresponding method.

 

rnn is built on the basis of conventional rnn using local normalization, it actually takes the sequence representation information implicitly from domain and intent classification. However, in endevice domain, rcrf performs the best and the proposed rsvm model performs even worse than crf. We notice that, in endevice model, user queries tend to be short, with on average 2.4 words in a query, shown in Table. 1. Also the loss function in the proposed model only uses the top and the second most hypothesis, which may be less informative, especially with short sentences, as compared to using all hypothesis in rcrf.

Table. 6 gives the overall performance comparison of different models in internal live log dataset using the weighted average F1 score over all domains. In this table, we can find that the proposed rsvm on average can achieve 0.6% and 0.7% F1 score improvement over joint-rnn and rcrf, respectively.

### Training Speed Up In rsvm

Using max-margin criteria, backward weight updating only happens to support vector samples. While using cross-entropy or maximum likelihood based training criteria, backward weight updating has to sweep over the whole training data. Fig. 2 shows that rsvm can substantially speed up the model training by skipping the backward weight updating for non-support vector samples. As depicted in Fig. 2, rsvm only executes backward weight updating for 337 training samples (7% of whole training data) at epoch 20.

## 4 Conclusions

We have proposed a recurrent support vector machine (rsvm) which applies the structured svm on top of the conventional rnn for slot tagging. Different from previous rnn sequence training approaches that use maximum conditional likelihood as objective function, the proposed method uses sequence level max-margin criterion with hard loss function. The model is trained to discriminate the score of ground-truth slot sequences with respect to other competing slot sequences by a margin. Viterbi algorithm is used in decoding to select a slot sequence that gives the largest score. To verify the performance of the proposed method, three datasets, namely ATIS dataset, CoNLL 2000 Chunking dataset and Cortana live log dataset, were used. The proposed rsvm achieved a new state-of-the-art performances on these datasets. In addition, rsvm showed substantial training speed up by skipping the weight updating for non-support vector training samples. On ATIS data, after 20 epoches, backward weight updating only happened for almost 7% of whole training samples.

The proposed rsvm is built on top of conventional rnn structure. Though rsvm doesn't have advanced topology used in lstm and rnn-em, it achieves comparable or better performances. Therefore, the improvement comes from its sequence level max-margin criterion. For future works, we plan to apply the structured svM on top of other advanced models.

\begin{table}
\begin{tabular}{l|c} \hline model & F1(\%) \\ \hline crf & 92.6 \\ rnn & 92.8 \\ rcrf & 93.9 \\ joint-rnn & 94.0 \\ \hline rsvm-min & 94.0 \\ rsvm-average & 94.6 \\ rsvm-max & 95.2 \\ \hline \end{tabular}
\end{table}
Table 6: The weighted average F1 score of different slot tagging models over all the domains.

Figure 2: Training sample usage comparison for RSVM and RCRF on ATIS data in backward weight updating in each training epoch.

\begin{table}
\begin{tabular}{l|r|r|r|r} \hline model & adarm & calen & commu & note \\ \hline crf & 93.2 & 93.7 & 91.6 & 76.5 \\ rnn & 93.8 & 95.2 & 92.5 & 77.0 \\ rcrf & 93.9 & 95.9 & 93.2 & 76.9 \\ joint-RNN & 94.9 & 96.4 & 92.9 & 74.9 \\ \hline rsvm-min & 95.8 & 95.8 & 93.3 & 86.9 \\ rsvm-aver & 95.9 & 96.3 & 93.9 & 88.3 \\ rsvM-max & 96.2 & 96.6 & 94.9 & 89.7 \\ \hline \end{tabular}
\end{table}
Table 4: F1 score comparison on different slot tagging models on alarm, calendar, communication and note.

 



## References

* [Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* De Mori (2007) Renato De Mori. 2007. Spoken language understanding: a survey. In _The Proceedings of IEEE Workshop on Automatic Speech Recognition Understanding_, pages 365-376.
* Deoras and Sarikaya (2013) Anoop Deoras and Ruhi Sarikaya. 2013. Deep belief network based semantic taggers for spoken language understanding. In _ISCA Interspeech_. ISCA, September.
* Dinarelli et al. (2008) Marco Dinarelli, Alessandro Moschitti, and Giuseppe Riccardi. 2008. Joint generative and discriminative models for spoken language understanding. In _The Proceeding of Spoken Language Technology Workshop_, pages 61-64.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _Journal of Machine Learning Research_, 12:2121-2159.
* Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. 2013. Speech recognition with deep recurrent neural networks. In _The Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_, pages 6645-6649.
* Graves (2012) Alex Graves. 2012. _Supervised Sequence Labelling with Recurrent Neural Networks_, volume 385. Springer.
* He and Young (2003) Yulan He and S. Young. 2003. A data-driven spoken language understanding system. In _The Proceedings of Automatic Speech Recognition and Understanding_, pages 583-588.
* Hemphill et al. (1990a) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990a. The atis spoken language systems pilot corpus. In _The Proceedings of the DARPA Speech and Natural Language Workshop_, pages 96-101.
* Hemphill et al. (1990b) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990b. The attis spoken language systems pilot corpus. In _The Proceedings of the Workshop on Speech and Natural Language_, pages 96-101.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Joachims et al. (2009) Thorsten Joachims, Thomas Finley, and Chun-Nam John Yu. 2009. Cutting-plane training of structural svms. _Machine Learning_, 77(1):27-59.
* Kudo and Matsumoto (2000) Taku Kudo and Yuji Matsumoto. 2000. Use of support vector learning for chunking identification. In _The proceedings of Conference on Natural Language Learning (CoNLL) and Second Learning Language In Logic WorkShop_.
* Kudo and Matsumoto (2001) Taku Kudo and Yuji Matsumoto. 2001. Chunking with support vector machines. In _The Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics in Language Technologies_, pages 1-8.
* McDonald et al. (2005) Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Flexible text segmentation with structured multilingual classification. In _The Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing_, pages 987-994.
* Mesnil et al. (2015) Gregoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, and Geoffrey Zweig. 2015. Using recurrent neural networks for slot filling in spoken language understanding. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, pages 530-539.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafhat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _The Proceedings of Interspeech_, pages 1045-1048.
* Nguyen and Guo (2007) Nam Nguyen and Yunsong Guo. 2007. Comparisons of sequence labeling algorithms and extensions. _International Conference on Machine Learning_, pages 681-688.
* Panagiotakopoulos and Tsampouka (2013) Constantinos Panagiotakopoulos and Petroula Tsampouka. 2013. The stochastic gradient descent for the primal l1-svm optimization revisited. _CoRR_.
* Peng and Yao (2015) Baolin Peng and Kaisheng Yao. 2015. Recurrent neural networks with external memory for language understanding. _CoRR_, abs/1506.00195.
* Prez-Ortiz et al. (2001) Juan Antonio Perez-ortiz, Mikel L. Forcada, and Department De Llenguages I Sistemes. 2001. Part-of-speech tagging with recurrent neural networks. In _The Proceedings of the International Joint Conference on Neural Networks_.
* Ratliff et al. (2007) Nathan Ratliff, J. Andrew (Drew) Bagnell, and Martin Zinkevich. 2007. (online) subgradient methods for structured prediction. In _Eleventh International Conference on Artificial Intelligence and Statistics (ASIstats)_.
* Sha and Pereira (2005) Fei Sha and Fernando Pereira. 2005. Shallow parsing with conditional random fields. In _The Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics and Human Language Technologies_.
* Shi et al. (2015) Yangyang Shi, Kaisheg Yao, Hu Chen, Yi-Cheng Pan, Mei-Yuh Hwang, and Baolin Peng. 2015. Contextual spoken language understanding using recurrent neural networks. In _The Proceedings of International Conference on Acoustics, Speech and Signal Processing_.
* Volume 1_, COLING '08, pages 841-848.
* Tsochantaridis et al. (2005) Ioannis Tsochantaridis, Thorben Joachims, Thomas Hofmann, and Yasemin Altun. 2005. Large margin methods for structured and interdependent output variables. _Journal Of Machine Learning Research_, 6:1453-1484.
* Vapnik (1995) Vladimir N. Vapnik. 1995. _The Nature of Statistical Learning Theory_.
* Wang et al. (2005) Ye-Yi Wang, Li Deng, and Alex Acero. 2005. Spoken language understanding -- an introduction to the statistical framework. _IEEE Signal Processing Magazine_, 22(5):16-31.
* Wang et al. (2015) Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2015. A unified tagging solution: Bidirectional LSTM recurrent neural network with word embedding. _CoRR_.
* Werbos (1990) P.J. Werbos. 1990. Backpropagation through time: what it does and how to do it. _The Proceedings of the IEEE_, 78(10):1550-1560.
* Xu and Sarkaya (2013) Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular CRF for joint intent detection and slot filling. In _The Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_, pages 78-83.
* Yao et al. (2013) Kaisheg Yao, Geoffrey Zweig, Mei-Yuhu Hwang, Yangyang Shi, and Dong Yu. 2013. Recurrent neural networks for

language understanding. In _The Proceedings of Interspeech_, pages 2524-2528.
* [Yao et al.2014a] Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014a. Spoken language understanding using long short-term memory neural networks. In _The Proceedings of IEEE workshop on Spoken Language Technology_, pages 189-194.
* [Yao et al.2014b] Kaisheng Yao, Baolin Peng, Geoffrey Zweig, Dong Yu, Xiaolong Li, and Feng Gao. 2014b. Recurrent conditional random field for language understanding. In _The Proceedings of International Conference of Acoustic, Speech and Signal Processing_, pages 4105-4109.
* [Zhang et al.2001] Tong Zhang, Fred Damerau, and David Johnson. 2001. Text chunking based on a generalization of winnow. _Journal of Machine Learning Research_, 2:615-637.
* 4279, April.