Venue: EMNLP 2017

# No Need to _Pay Attention_:

Simple Recurrent Neural Networks Work!

(for Answering "Simple" Questions)

Ferhan Ture and Oliver Jojic

Comcast Labs, Washington, DC 20005

{ferhan_ture, oliver_jojic}@cable.comcast.com

###### Abstract

First-order factoid question answering assumes that the question can be answered by a single fact in a knowledge base (KB). While this does not seem like a challenging task, many recent attempts that apply either complex linguistic reasoning or deep neural networks achieve 65%-76% accuracy on benchmark sets. Our approach formulates the task as two machine learning problems: detecting the entities in the question, and classifying the question as one of the relation types in the KB. We train a recurrent neural network to solve each problem. On the SimpleQuestions dataset, our approach yields substantial improvements over previously published results -- even neural networks based on much more complex architectures. The simplicity of our approach also has practical advantages, such as efficiency and modularity, that are valuable especially in an industry setting. In fact, we present a preliminary analysis of the performance of our model on real queries from Comcast's X1 entertainment platform with millions of users every day.

## 1 Introduction

First-order factoid question answering (QA) assumes that the question can be answered by a single fact in a knowledge base (KB). For example, "How old is Tom Hanks" is about the [age] of [Tom Hanks]. Also referred to as _simple_ questions by Bordes et al. (2015), recent attempts that apply either complex linguistic reasoning or attention-based complex neural network architectures achieve up to 76% accuracy on benchmark sets Golub and He (2016); Yin et al. (2016). While it is tempting to study QA systems that can handle more complicated questions, it is hard to reach reasonably high precision for unrestricted questions. For more than a decade, successful industry applications of QA have focused on first-order questions. This bears the question: are users even interested in asking questions beyond first-order (or are these use cases more suitable for interactive dialogue)? Based on voice logs from a major entertainment platform with millions of users every day, Comcast X1, we find that most existing use cases of QA fall into the first-order category.

Our strategy is to tailor our approach to first-order QA by making strong assumptions about the problem structure. In particular, we assume that the answer to a first-order question is a _single_ property of a _single_ entity in the KB, and decompose the task into two subproblems: (a) detecting entities in the question and (b) classifying the question as one of the relation types in the KB. We simply train a vanilla recurrent neural network (RNN) to solve each subproblem Elman (1990). Despite its simplicity, our approach (RNN-QA) achieves the highest reported accuracy on the SimpleQuestions dataset. While recent literature has focused on building more complex neural network architectures with attention mechanisms, attempting to generalize to broader QA, we enforce stricter assumptions on the problem structure, thereby reducing complexity. This also means that our solution is efficient, another critical requirement for real-time QA applications. In fact, we present a performance analysis of RNN-QA on Comcast's X1 entertainment system, used by millions of customers every day.

## 2 Related work

If knowledge is presented in a structured form (e.g., knowledge base (KB)), the standard ap

proach to QA is to transform the question and knowledge into a compatible form, and perform reasoning to determine which fact in the KB answers a given question. Examples of this approach include pattern-based question analyzers (Buscaldi et al., 2010), combination of syntactic parsing and semantic role labeling (Bilotti et al., 2007, 2010), as well as lambda calculus (Berant et al., 2013) and combinatory categorical grammars (CCG) (Reddy et al., 2014). A downside of these approaches is the reliance on linguistic resources/heuristics, making them language- and/or domain-specific. Even though Reddy et al. (2014) claim that their approach requires less supervision than prior work, it still relies on many English-specific heuristics and hand-crafted features. Also, their most accurate model uses a corpus of paraphrases to generalize to linguistic diversity. Linguistic parsers can also be too slow for real-time applications.

In contrast, an RNN can detect entities in the question with high accuracy and low latency. The only required resources are word embeddings and a set of questions with entity words tagged. The former can be easily trained for any language/domain in an unsupervised fashion, given a large text corpus without annotations (Mikolov et al., 2013; Pennington et al., 2014). The latter is a relatively simple annotation task that exists for many languages and domains, and it can also be synthetically generated. Many researchers have explored similar techniques for general NLP tasks (Collobert et al., 2011), such as named entity recognition (Lu et al., 2015; Hammerton, 2003), sequence labeling (Graves, 2008; Chung et al., 2014), part-of-speech tagging (Huang et al., 2015; Wang et al., 2015), chunking (Huang et al., 2015).

Deep learning techniques have been studied extensively for constructing parallel neural networks for modeling a joint probability distribution for question-answer pairs Hsu et al. (2016); Yang et al. (2014); He et al. (2015); Mueller and Thyagarajan (2016) and re-ranking answers output by a retrieval engine (Rao et al., 2016; Yang et al., 2016). These more complex approaches might be needed for general-purpose QA and sentence similarity, where one cannot make assumptions about the structure of the input or knowledge. However, as noted in Section 1, first-order factoid questions can be represented by an entity and a relation type, and the answer is usually stored in a structured knowledge base. Dong et al. (2015) similarly assume that the answer to a question is at most two hops away from the target entity. However, they do not propose how to obtain the target entity, since it is provided as part of their dataset. Bordes et al. (2014) take advantage of the KB structure by projecting entities, relations, and subgraphs into the same latent space. In addition to finding the target entity, the other key information to first-order QA is the relation type corresponding to the question. Many researchers have shown that classifying the question into one of the predefined types (e.g., based on patterns (Zhang and Lee, 2003) or support vector machines (Buscaldi et al., 2010)) improves QA accuracy.

## 3 Approach

**(a) From Question to Structured Query.** Our approach relies on a _knowledge base_, containing a large set of _facts_, each one representing a binary [subject, relation, object] relationship. Since we assume _first-order questions_, the answer can be retrieved from a single fact. For instance, "How old is Sarah Michelle Gellar?" can be answered by the fact:

[Sarah Michelle Gellar, bornOn, 4/14/1977]

The main idea is to dissect a first-order factoid natural-language question by converting it into a structured query: {entity "Sarah Michelle Gellar", relation: bornOn}. The process can be modularized into two machine learning problems, namely _entity detection_ and _relation prediction_. In the former, the objective is to _tag_ each question word as either entity or not. In the latter, the objective is to _classify_ the question into one of the \(K\) relation types. We modeled both using an RNN.

We use a standard RNN architecture: Each word in the question passes through an embedding lookup layer \(E\), projecting the one-hot vector into a \(d\)-dimensional vector \(x_{t}\). A recurrent layer combines this _input representation_ with the hidden layer representation from the _previous word_ and applies a non-linear transformation to compute the hidden layer representation for the _current word_. The hidden representation of the final recurrent layer is projected to the output space of \(k\) dimensions and normalized into a probability distribution via soft-max.

In _relation prediction_, the question is classified 

into one of the 1837 classes (i.e., relation types in Freebase). In the _entity detection_ task, each word is classified as either _entity_ or _context_ (i.e., \(k=2\)).

Given a new question, we run the two RNN models to construct the structured query. Once every question word is classified as entity (denoted by E) or context (denoted by C), we can extract entity phrase(s) by grouping consecutive entity words. For example, for question "How old is Michelle Gellar", the output of entity detection is [C C C E E], from which we can extract a single entity "Michelle Gellar". The output of relation prediction is bornOn. The inferred structured query \(q\) becomes the following:

{_entityText:_ "michelle gellar", _relation_: bornOn}
**(b) Entity Linking.** The textual reference to the entity (entityText in \(q\)) needs to be linked to an actual entity node in our KB. In order to achieve that, we build an _inverted index_\(I_{\text{entity}}\) that maps all \(n\)-grams of an entity (\(n\in\{1,2,3\}\)) to the entity's alias text (e.g., name or title), each with an associated \(TF\)-\(IDF\) score. We also map the exact text (\(n=\infty\)) to be able to prioritize exact matches.

Following our running example, let us demonstrate how we construct \(I_{\text{entity}}\). Let us assume there is a node \(e_{i}\) in our KB that refers to the actress "Sarah Michelle Gellar". The alias of this entity node is the name, which has three unigrams ("sarah", "michelle", "gellar"), two bigrams ("sarah michelle", "michelle gellar") and a single trigram (i.e., the entire name). Each one of these \(n\)-grams gets indexed in \(I_{\text{entity}}\) with \(TF\)-\(IDF\) weights. Here is how the weights would be computed for unigram "sarah" and bigram "michelle gellar" (\(\Rightarrow\) denotes mapping):

\[I_{\text{entity}}(\text{``sarah''})\Rightarrow\{\text{node}:e_{ i},\] \[\text{score $:TF$-$IDF$(\text{``sarah''};\text{sarah michelle gellar"})}\}\] \[\text{$I_{\text{entity}}(\text{``michelle gellar''})}\Rightarrow\{ \text{node}:e_{i},\] \[\text{score $:TF$-$IDF$(\text{``michelle gellar''},}\] \[\qquad\qquad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \ 

SimpleQuestions was built by Bordes et al. (2014) to serve as a larger and more diverse factoid QA dataset.1 Freebase facts are sampled in a way to ensure a diverse set of questions, then given to human annotators to create questions from, and get labeled with corresponding entity and relation type. There are a total of 1837 unique relation types that appear in SimpleQuestions.

Footnote 1: [https://github.com/google-research/language-learning-and-plays-international-language-for-the-question-answering-of-the-questions-using-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-of-the-question-of-the-question-of-the-the-question-of](https://github.com/google-research/language-learning-and-plays-international-language-for-the-question-answering-of-the-questions-using-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-answering-of-the-question-of-the-question-of-the-question-of-the-the-question-of) 

deduce that our RNN-based entity detection is the reason our approach performs so well.

**Error Analysis.** In order to better understand the weaknesses of our approach, we performed a _blame analysis_: Among 2537 errors in the test set, 15% can be blamed on entity detection -- the relation type was correctly predicted, but the detected entity did not match the ground truth. The reverse is true for 48% cases.4 We manually labeled a sample of 50 instances from each blame scenario. When entity detection is to blame, 20% was due to spelling inconsistencies between question and KB, which can be resolved with better text normalization during indexing (e.g., "la kings" refers to "Los Angeles Kings"). We found 16% of the detected entities to be correct, even though it was not the same as the ground truth (e.g., either "New York" or "New York City" is correct in "what can do in new york?"); 18% are inherently ambiguous and need clarification (e.g., "where bin laden got killed?" might mean "Osama" or "Salem"). When blame is on relation prediction, we found that the predicted relation is reasonable (albeit different than ground truth) 29% of the time (e.g., "what was nikola tesla known for" can be classified as profession or notable_for).

Footnote 4: In remaining 37% incorrect answers, both models fail, so the blame is shared.

**RNN-QA in Practice.** In addition to matching the state of the art in effectiveness, we also claimed that our simple architecture provides an efficient and modular solution. We demonstrate this by applying our model (without any modifications) to the entertainment domain and deploying it to the Comcast X1 platform serving millions of customers every day. Training data was generated synthetically based on an internal entertainment KB. For evaluation, 295 unique question-answer pairs were randomly sampled from real usage logs of the platform.

We can draw two important conclusions from Table 2: First of all, we find that almost all of the user-generated natural-language questions (278/295\(\sim\)95%) are first-order questions, supporting the significance of first-order QA as a task. Second, we show that even if we simply use an open-sourced deep learning toolkit (keras.io) for implementation and limit the computational resources to 2 CPU cores per thread, RNN-QA answers 75% of questions correctly with very reasonable latency.

## 6 Conclusions and Future work

We described a simple yet effective approach for QA, focusing primarily on first-order factual questions. Although we understand the benefit of exploring task-agnostic approaches that aim to capture semantics in a more general way (e.g., Kumar et al. (2015)), it is also important to acknowledge that there is no "one-size-fits-all" solution as of yet.

One of the main novelties of our work is to decompose the task into two subproblems, entity detection and relation prediction, and provide solutions for each in the form of a RNN. In both cases, we have found that bidirectional networks are beneficial, and that two layers are sufficiently deep to balance the model's ability to fit versus its ability to generalize.

While an ablation study revealed the importance of both entity detection and relation prediction, we are hoping to further study the degree of which improvements in either component affect QA accuracy. Drop-out was tuned to 10% based on validation accuracy. While we have not implemented attention directly on our model, we can compare our results side by side on the same benchmark task against prior work with complex attention mechanisms (e.g., Yin et al. (2016)). Given the proven strength of attention mechanisms, we were surprised to find our simple approach to be clearly superior on SimpleQuestions.

Even though deep learning has opened the potential for more generic solutions, we believe that taking advantage of problem structure yields a more accurate and efficient solution. While first-order QA might seem like a solved problem, there is clearly still room for improvement. By revealing that 95% of real use cases fit into this paradigm, we hope to convince the reader that this is a valuable problem that requires more _attention_.

\begin{table}
\begin{tabular}{c|c}
**Error** & **Count** \\ \hline Correct & 220 \\ \hline Incorrect entity & 16 \\ \hline Incorrect relation & 42 \\ \hline Not first-order question & 17 \\ \hline \hline Total Latency & 76\(\pm\)16 ms \\ \end{tabular}
\end{table}
Table 2: Evaluation of RNN-QA on real questions from X platform.

 



## References

* [Berant et al.2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 1533-1544. ACL.
* Bilotti et al.2010] Matthew W Bilotti, Jonathan Elsas, Jaime Carbonell, and Eric Nyberg. 2010. Rank Learning for Factoid Question Answering with Linguistic and Semantic Constraints. In _Proceedings of the 19th ACM International Conference on Information and Knowledge Management_, CIKM '10, pages 459-468, New York, NY, USA. ACM.
* Bilotti et al.2007] Matthew W Bilotti, Paul Ogilvie, Jamie Callan, and Eric Nyberg. 2007. Structured Retrieval for Question Answering. In _Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '07, pages 351-358, New York, NY, USA. ACM.
* Bordes et al.2014] Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. _arXiv preprint arXiv:1406.3676_.
* Bordes et al.2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. _CoRR_, abs/1506.02075.
* Buscaldi et al.2010] Davide Buscaldi, Paolo Rosso, Jose Manuel Gomez-Soriano, and Emilio Sanchis. 2010. Answering Questions with an N-gram Based Passage Retrieval Engine. _J. Intell. Inf. Syst._, 34(2):113-134.
* Chung et al.2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. _CoRR_, abs/1412.3555.
* Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) format scratch. _J. Mach. Learn. Res._, 12:2493-2537.
* Dong et al.2015] Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multicolumn convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume I: Long Papers_, pages 260-269. The Association for Computer Linguistics.
* Elman1990] Jeffrey L Elman. 1990. Finding structure in time. _Cognitive science_, 14(2):179-211.
* Golub and He2016] David Golub and Xiaodong He. 2016. Character-level question answering with attention. _arXiv preprint arXiv:1604.00727_.
* Graves2008] Alex Graves. 2008. _Supervised sequence labelling with recurrent neural networks_. Ph.D. thesis, Technical University Munich.
* Hammerton2003] James Hammerton. 2003. Named entity recognition with long short-term memory. In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4_, pages 172-175. Association for Computational Linguistics.
* He et al.2015] Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multiperspective sentence similarity modeling with convolutional neural networks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1576-1586. The Association for Computational Linguistics.
* Hsu et al.2016] Wei-Ning Hsu, Yu Zhang, and James R. Glass. 2016. Recurrent neural network encoder with attention for community question answering. _CoRR_, abs/1603.07044.
* Huang et al.2015] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv preprint arXiv:1508.01991_.
* Kumar et al.2015] Ankit Kumar, Ozan Irsov, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. _CoRR_, abs/1506.07285.
* Lu et al.2015] Zefu Lu, Lei Li, and Wei Xu. 2015. Twisted recurrent network for named entity recognition. In _Bay Area Machine Learning Symposium_.
* Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Mueller and Thyagarajan2016] Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA._, pages 2786-2792. AAAI Press.
* Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, volume 14, pages 1532-1543.
* Rao et al.2016] Jinfeng Rao, Hua He, and Jimmy Lin. 2016. Noise-contrastive estimation for answer selection with deep neural networks. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, pages 1913-1916. ACM.
* Reddy et al.2014] Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _TACL_, 2:377-392

Peilu Wang, Yao Qian, Frank K Soong, Lei He, and Hai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. _arXiv preprint arXiv:1510.06168_.
* Yang et al. (2016) Liu Yang, Qingyao Ai, Jiafeng Guo, and W Bruce Croft. 2016. anmm: Ranking short answer texts with attention-based neural matching model. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, pages 287-296. ACM.
* Yang et al. (2014) Min-Chul Yang, Nan Duan, Ming Zhou, and Hae-Chang Rim. 2014. Joint relational embeddings for knowledge-based question answering. In _EMNLP_, pages 645-650.
* Yin et al. (2016) Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and Hinrich Schutze. 2016. Simple question answering by attentive convolutional neural network. _arXiv preprint arXiv:1606.03391_.
* Zhang and Lee (2003) Dell Zhang and Wee Sun Lee. 2003. Question Classification Using Support Vector Machines. In _Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval_, SIGIR '03, pages 26-32, New York, NY, USA. ACM.