Venue: ACL 2017

# Question Answering on Knowledge Bases and Text

using Universal Schema and Memory Networks

Rajarshi Das\({}^{\spadesuit}\)Manzil Zaheer\({}^{\diamondsuit}\)Siva Reddy\({}^{\clubsuit}\)Andrew McCallum\({}^{\spadesuit}\)College of Information and Computer Sciences, University of Massachusetts Amherst \({}^{\heartsuit}\)School of Computer Science, Carnegie Mellon University \({}^{\spadesuit}\)School of Informatics, University of Edinburgh {rajarshi, mccallum}@cs.umass.edu, manzil2@cs.cmu.edu siva.reddy@ed.ac.uk

###### Abstract

Existing question answering methods infer answers either from a knowledge base or from raw text. While knowledge base (KB) methods are good at answering compositional questions, their performance is often affected by the incompleteness of the KB. Au contraire, web text contains millions of facts that are absent in the KB, however in an unstructured form. _Universal schema_ can support reasoning on the union of both structured KBs and unstructured text by aligning them in a common embedded space. In this paper we extend universal schema to natural language question answering, employing _memory networks_ to attend to the large body of facts in the combination of text and KB. Our models can be trained in an end-to-end fashion on question-answer pairs. Evaluation results on Spades fill-in-the-blank question answering dataset show that exploiting universal schema for question answering is better than using either a KB or text alone. This model also outperforms the current state-of-the-art by 8.5 \(F_{1}\) points.

+
Footnote †: Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 358–365 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics [https://doi.org/10.18653/v1/P17-2057](https://doi.org/10.18653/v1/P17-2057)

## 1 Introduction

Question Answering (QA) has been a longstanding goal of natural language processing. Two main paradigms evolved in solving this problem: 1) answering questions on a knowledge base; and 2) answering questions using text.

Knowledge bases (KB) contains facts expressed in a fixed schema, facilitating compositional reasoning. These attracted research ever since the early days of computer science, e.g., BASEBALL [Green Jr et al., 1961]. This problem has matured into learning semantic parsers from parallel question and logical form pairs [Zelle and Mooney, 1996, Zettlemoyer and Collins, 2005], to recent scaling of methods to work on very large KBs like Freebase using question and answer pairs [Berant et al., 2013]. However, a major drawback of this paradigm is that KBs are highly incomplete [Dong et al., 2014]. It is also an open question whether KB relational structure is expressive enough to represent world knowledge [Stanovsky et al., 2014, Gardner and Krishnamurthy, 2017]

The paradigm of exploiting text for questions started in the early 1990s [Kupiec, 1993]. With the advent of web, access to text resources became abundant and cheap. Initiatives like TREC QA competitions helped popularizing this paradigm [Voorhees et al., 1999]. With the recent advances in deep learning and availability of large public datasets, there has been an explosion of research in a very short time [Rajpurkar et al., 2016, Trischler et al., 2016, Nguyen et al., 2016, Wang and Jiang, 2016, Lee et al., 2016, Xiong et al., 2016, Seo et al., 2016]. Still, text representation is unstructured and does not allow the compositional reasoning which structured KB supports.

An important but under-explored QA paradigm is where KB and text are exploited together [Ferucci et al., 2010]. Such combination is attractive because text contains millions of facts not present in KB, and a KB's generative capacity represents infinite number of facts that are never seen in text. However QA inference on this combination is challenging due to the structural non-uniformity of KB and text. _Distant supervision_ methods [Bunescu and Mooney, 2007, Mintz et al., 2009, Riedel et al., 2010, Yao et al., 2010, Zeng et al., 2015] address this problem partially by means of aligning text patterns with KB. But the rich and ambiguous nature of language allows a fact to be expressed in many different forms which these models fail to capture. _Universal schema_[Riedel et al., 2013] avoids the alignment problem by jointly embedding KB facts

and text into a uniform structured representation, allowing interleaved propagation of information. Figure 1 shows a universal schema matrix which has pairs of entities as rows, and Freebase and textual relations in columns. Although universal schema has been extensively used for relation extraction, this paper shows its applicability to QA. Consider the question USA has elected_blank_, our first african-american president with its answer Barack Obama. While Freebase has a predicate for representing presidents of USA, it does not have one for 'african-american' presidents. Whereas in text, we find many sentences describing the presidency of Barack Obama and his ethnicity at the same time. Exploiting both KB and text makes it relatively easy to answer this question than relying on only one of these sources.

Memory networks (MemNN; Weston et al.2015) are a class of neural models which have an external memory component for encoding short and long term context. In this work, we define the memory components as observed cells of the universal schema matrix, and train an end-to-end QA model on question-answer pairs.

The contributions of the paper are as follows (a) We show that universal schema representation is a better knowledge source for QA than either KB or text alone, (b) On the SPADES dataset (Bisk et al., 2016), containing real world fill-in-the-blank questions, we outperform state-of-the-art semantic parsing baseline, with 8.5 \(F_{1}\) points. (c) Our analysis shows how individual data sources help fill the weakness of the other, thereby improving overall performance.

## 2 Background

Problem DefinitionGiven a question \(q\) with words \(w_{1},w_{2},\ldots,w_{n}\), where these words contain one_blank_ and at least one entity, our goal is to fill in this_blank_ with an answer entity \(q_{a}\) using a knowledge base \(\mathcal{K}\) and text \(\mathcal{T}\). Few example question answer pairs are shown in Table 2.

Universal SchemaTraditionally universal schema is used for relation extraction in the context of knowledge base population. Rows in the schema are formed by entity pairs (e.g. USA, NYC), and columns represent the relation between them. A relation can either be a KB relation, or it could be a pattern of text that exist between these two entities in a large corpus. The embeddings of entities and relation types are learned by low-rank matrix factorization techniques. Riedel et al.Riedel et al.2013 treat textual patterns as static symbols, whereas recent work by Verga et al.Verga et al.2016 replaces them with distributed representation of sentences obtained by a RNN. Using distributed representation allows reasoning on sentences that are similar in meaning but different on the surface form. We too use this variant to encode our textual relations.

Memory NetworksMemNNs are neural attention models with external and differentiable memory. MemNNs decouple the memory component from the network thereby allowing it store external information. Previously, these have been successfully applied to question answering on KB where the memory is filled with distributed representation of KB triples (Bordes et al., 2015), or for read

Figure 1: Memory network attending the facts in the universal schema (matrix on the left). The color gradients denote the attention weight on each fact.

 

ing comprehension (Sukhbaatar et al., 2015; Hill et al., 2016), where the memory consists of distributed representation of sentences in the comprehension. Recently, key-value MemNN are introduced (Miller et al., 2016) where each memory slot consists of a key and value. The attention weight is computed only by comparing the question with the key memory, whereas the value is used to compute the contextual representation to predict the answer. We use this variant of MemNN for our model. Miller et al. (2016), in their experiments, store either KB triples or sentences as memories but they do not explicitly model multiple memories containing distinct data sources like we do.

## 3 Model

Our model is a MemNN with universal schema as its memory. Figure 1 shows the model architecture.

Memory:Our memory \({\mathcal{M}}\) comprise of both KB and textual triples from universal schema. Each memory cell is in the form of key-value pair. Let \((s,r,o)\in{\mathcal{K}}\) represent a KB triple. We represent this fact with distributed key \(\mathbf{k}\in\mathbb{R}^{2d}\) formed by concatenating the embeddings \(\mathbf{s}\in\mathbb{R}^{d}\) and \(\mathbf{r}\in\mathbb{R}^{d}\) of subject entity \(s\) and relation \(r\) respectively. The embedding \(\mathbf{o}\in\mathbb{R}^{d}\) of object entity \(o\) is treated as its value \(\mathbf{v}\).

Let \((s,~{}[w_{1},\ldots,arg_{1},\ldots,arg_{2},w_{n}],~{}o)\in{\mathcal{T}}\) represent a textual fact, where \(arg_{1}\) and \(arg_{2}\) correspond to the positions of the entities 's' and 'o'. We represent the key as the sequence formed by replacing \(arg_{1}\) with '_s'_ and \(arg_{2}\) with a special '_blank_' token, i.e., \(k=[w_{1},~{}\ldots,s,~{}\ldots~{}\text{,}\text{blank\_},~{}w_{n}]\) and value as just the entity 'o'. We convert \(k\) to a distributed representation using a bidirectional LSTM (Hochreiter and Schmidhuber, 1997; Graves and Schmidhuber, 2005), where \(\mathbf{k}\in\mathbb{R}^{2d}\) is formed by concatenating the last states of forward and backward LSTM, i.e., \(\mathbf{k}=\begin{bmatrix}\mathsf{LSTM}(k);&\mathsf{LSTM}(k)\\ \end{bmatrix}\). The value \(\mathbf{v}\) is the embedding of the object entity \(o\). Projecting both KB and textual facts to \(\mathbb{R}^{2d}\) offers a unified view of the knowledge to reason upon. In Figure 1, each cell in the matrix represents a memory containing the distributed representation of its key and value.

Question Encoder:A bidirectional LSTM is also used to encode the input question \(q\) to a distributed representation \(\mathbf{q}\in\mathbb{R}^{2d}\) similar to the key encoding step above.

Attention over cells:We compute attention weight of a memory cell by taking the dot product of its key \(\mathbf{k}\) with a contextual vector \(\mathbf{c}\) which encodes most important context in the current iteration. In the first iteration, the contextual vector is the question itself. We only consider the memory cells that contain at least one entity in the question. For example, for the input question in Figure 1, we only consider memory cells containing USA. Using the attention weights and values of memory cells, we compute the context vector \(\mathbf{c_{t}}\) for the next iteration \(t\) as follows:

where \(\mathbf{c_{0}}\) is initialized with question embedding \(\mathbf{q}\), \(\mathbf{W_{p}}\) is a projection matrix, and \(\mathbf{W_{t}}\) represents the weight matrix which considers the context in previous hop and the values in the current iteration based on their importance (attention weight). This multi-iterative context selection allows multi-hop reasoning without explicitly requiring a symbolic query representation.

Answer Entity Selection:The final contextual vector \(c_{t}\) is used to select the answer entity _q\({}_{a}\)_ (among all 1.8M entities in the dataset) which has the highest inner product with it.

## 4 Experiments

### Evaluation Dataset

We use Freebase (Bollacker et al., 2008) as our KB, and ClueWeb (Gabrilovich et al., 2013) as our text source to build universal schema. For evaluation, literature offers two options: 1) datasets for text-based question answering tasks such as answer sentence selection and reading comprehension; and 2) datasets for KB question answering.

Although the text-based question answering datasets are large in size, e.g., SQuAD (Rajpurkar et al., 2016) has over 100k questions, answers to these are often not entities but rather sentences which are not the focus of our work. Moreover these texts may not contain Freebase entities at all, making these skewed heavily towards text. Coming to the alternative option, WebQuestions (Berant et al., 2013) is widely used for QA on Freebase. This dataset is curated such that all questions can be answered on Freebase alone. But since our goal is to explore the impact of universal schema, testing on a dataset completely answerable on a KB is not ideal. WikiMovies dataset (Miller et al., 2016) also has similar properties. Gardner and Krishnamurthy (2017) created a dataset with motivations similar to 

ours, however this is not publicly released during the submission time.

Instead, we use Spades(Bisk et al., 2016) as our evaluation data which contains fill-in-the-blank cloze-styled questions created from ClueWeb. This dataset is ideal to test our hypothesis for following reasons: 1) it is large with 93K sentences and 1.8M entities; and 2) since these are collected from Web, most sentences are natural. A limitation of this dataset is that it contains only the sentences that have entities connected by at least one relation in Freebase, making it skewed towards Freebase as we will see (SS 4.4). We use the standard train, dev and test splits for our experiments. For text part of universal schema, we use the sentences present in the training set.

### Models

We evaluate the following models to measure the impact of different knowledge sources for QA.

OnlyKB:In this model, MemNN memory contains only the facts from KB. For each KB triple \((e_{1},r,e_{2})\), we have two memory slots, one for \((e_{1},r,e_{2})\) and the other for its inverse \((e_{2},r^{i},e_{1})\).

OnlyTEXT:Spades contains sentences with blanks. We replace the blank tokens with the answer entities to create textual facts from the training set. Using every pair of entities, we create a memory cell similar to as in universal schema.

EnsembleThis is an ensemble of the above two models. We use a linear model that combines the scores from, and use an ensemble to combine the evidences from individual models.

UniSchemaThis is our main model with universal schema as its memory, i.e., it contains memory slots corresponding to both KB and textual facts.

### Implementation Details

The dimensions of word, entity and relation embeddings, and LSTM states were set to \(d=\)50. The word and entity embeddings were initialized with word2vec Mikolov et al. (2013) trained on 7.5 million ClueWeb sentences containing entities in Freebase subset of Spades. The network weights were initialized using Xavier initialization Glorot and Bengio (2010). We considered up to a maximum of 5k KB facts and 2.5k textual facts for a question. We used Adam Kingma and Ba (2015) with the default hyperparameters (learning rate=1e-3, \(\beta_{1}\)=0.9, \(\beta_{2}\)=0.999, e=1e-8) for optimization. To overcome exploding gradients, we restricted the magnitude of the \(\ell_{2}\) norm of the gradient to 5. The batch size during training was set to 32.

To train the UniSchema model, we initialized the parameters from a trained OnlyKB model. We found that this is crucial in making the UniSchema to work. Another caveat is the need to employ a trick similar to _batch normalization_Ioffe and Szegedy (2015). For each minibatch, we normalize the mean and variance of the textual facts and then scale and shift to match the mean and variance of the KB memory facts. Empirically, this stabilized the training and gave a boost in the final performance.

### Results and Discussions

Table 1 shows the main results on Spades. UniSchema outperforms all our models validating our hypothesis that exploiting universal schema for QA is better than using either KB or text alone. Despite Spades creation process being friendly to Freebase, exploiting text still provides a significant improvement. Table 2 shows some of the questions which UniSchema answered but OnlyKB failed. These can be broadly classified into (a) relations that are not expressed in Freebase (e.g., african-american presidents in sentence 1); (b) intentional facts since curated databases only represent concrete facts rather than intentions (e.g., threating to leave in sentence 2); (c) comparative predicates like first, second, largest, smallest (e.g., sentences 3 and 4); and (d) providing addi

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Dev. F\({}_{1}\)** & **Test F\({}_{1}\)** \\ \hline Bisk et al. (2016) & 32.7 & 31.4 \\ OnlyKB & 39.1 & 38.5 \\ OnlyText & 25.3 & 26.6 \\ Ensemble. & 39.4 & 38.6 \\ UniSchema & **41.1** & **39.9** \\ \hline \hline \end{tabular}
\end{table}
Table 1: QA results on Spades.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}} \hline \hline  & **Question** & **Answer** \\ \hline
1. USA have elected \_blank\_, our first african-american president. & Obama \\
2. Angelina has reportedly been threatening to leave Jblank... & Brad\_Pitt \\
3. Spanish is more often a second and weaker language among many \_blank\_.. & Latinos \\
4. \_blank\_ is the third largest city in the United\_States. & Chicago \\
5. \_blank\_ was Belshazar ’s father. & Nabonidus \\ \hline \hline \end{tabular}
\end{table}
Table 2: A few questions on which OnlyKB fails to answer but UniSchema succeeds.

 

tional type constraints (e.g., in sentence 5, Freebase does not have a special relation for father. It can be expressed using the relation parent along with the type constraint that the answer is of gender male).

We have also anlalyzed the nature of UniSchema attention. In 58.7% of the cases the attention tends to prefer KB facts over text. This is as expected since KBs facts are concrete and accurate than text. In 34.8% of cases, the memory prefers to attend text even if the fact is already present in the KB. For the rest (6.5%), the memory distributes attention weight evenly, indicating for some questions, part of the evidence comes from text and part of it from KB. Table 3 gives a more detailed quantitative analysis of the three models in comparison with each other.

To see how reliable is UniSchema, we gradually increased the coverage of KB by allowing only a fixed number of randomly chosen KB facts for each entity. As Figure 2 shows, when the KB coverage is less than 16 facts per entity, UniSchema outperforms OnlyKB by a wide-margin indicating UniSchema is robust even in resource-scarce scenario, whereas OnlyKB is very sensitive to the coverage. UniSchema also outperforms Ensemble showing joint modeling is superior to ensemble on the individual models. We also achieve the state-of-the-art with 8.5 \(F_{1}\) points difference. Bisk et al. use graph matching techniques to convert natural language to Freebase queries whereas even without an explicit query representation, we outperform them.

## 5 Related Work

A majority of the QA literature that focused on exploiting KB and text either improves the inference on the KB using text based features Krishnamurthy and Mitchell (2012); Reddy et al. (2014); Joshi et al. (2014); Yao and Van Durme (2014); Yih et al. (2015); Neelakantan et al. (2015b); Guu et al. (2015); Xu et al. (2016b); Choi et al. (2015); Savenkov and Agichtein (2016) or improves the inference on text using KB Sun et al. (2015).

Limited work exists on exploiting text and KB jointly for question answering Gardner and Krishnamurthy (2017) is the closest to ours who generate a open-vocabulary logical form and rank candidate answers by how likely they occur with this logical form both in Freebase and text. Our models are trained on a weaker supervision signal without requiring the annotation of the logical forms.

A few QA methods infer on curated databases combined with OpenIE triples Fader et al. (2014); Yahya et al. (2016); Xu et al. (2016a). Our work differs from them in two ways: 1) we do not need an explicit database query to retrieve the answers Neelakantan et al. (2015a); Andreas et al. (2016); and 2) our text-based facts retain complete sentential context unlike the OpenIE triples Banko et al. (2007); Carlson et al. (2010).

## 6 Conclusions

In this work, we showed universal schema is a promising knowledge source for QA than using KB or text alone. Our results conclude though KB is preferred over text when the KB contains the fact of interest, a large portion of queries still attend to text indicating the amalgam of both text and KB is superior than KB alone.

Figure 2: Performance on varying the number of available KB facts during test time. UniSchema model consistently outperforms OnlyKB





## Acknowledgments

We sincerely thank Luke Vilnis for helpful insights. This work was supported in part by the Center for Intelligent Information Retrieval and in part by DARPA under agreement number FA8750-13-2-0020. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

## References

* Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to Compose Neural Networks for Question Answering. In _NAACL_.
* Banko et al. (2007) Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open Information Extraction from the Web. In _IJCAI_.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In _EMNLP_.
* Bisk et al. (2016) Yonatan Bisk, Siva Reddy, John Blitzer, Julia Hockenmaier, and Mark Steedman. 2016. Evaluating Induced CCG Parsers on Grounded Semantic Parsing. In _EMNLP_.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In _ICDM_.
* Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. _CoRR_ .
* Bunescu and Mooney (2007) Razvan C. Bunescu and Raymond J. Mooney. 2007. Learning to extract relations from the web using minimal supervision. In _ACL_.
* Carlson et al. (2010) Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Jr. Estevam R. Hruschka, and Tom M. Mitchell. 2010. Toward an Architecture for Neverending Language Learning. In _AAAI_.
* Choi et al. (2016) Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, and Jonathan Berant. 2016. Hierarchical question answering for long documents. _arXiv preprint arXiv:1611.01839_ .
* Choi et al. (2015) Eunsol Choi, Tom Kwiatkowski, and Luke Zettlemoyer. 2015. Scalable Semantic Parsing with Partial Ontologies. In _ACL_.
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge Vault: A Web-scale Approach to Probabilistic Knowledge Fusion. New York, NY, USA, KDD '14.
* Fader et al. (2014) Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In _KDD_. ACM, pages 1156-1165.
* Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, and others. 2010. Building Watson: An overview of the DeepQA project. _AI magazine_ .
* Gabrilovich et al. (2013) Evgeniy Gabrilovich, Michael Ringgaard, and Amarang Subramanya. 2013. Face1: Freebase annotation of clueweb corpora. ([http://lemurproject.org/clueweb09/](http://lemurproject.org/clueweb09/).
* Gardner and Krishnamurthy (2017) Matt Gardner and Jayant Krishnamurthy. 2017. OpenVocabulary Semantic Parsing with both Distributional Statistics and Formal Knowledge. In _AAAI_.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _AISTATS_.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. _Neural Networks_ .
* Green Jr et al. (1961) Bert F Green Jr, Alice K Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: an automatic question-answerer. In _Papers presented at the May 9-11, 1961, western joint IRE-AIEE-ACM computer conference_. ACM, pages 219-224.
* Guu et al. (2015) K. Guo, J. Miller, and P. Liang. 2015. Traversing knowledge graphs in vector space. In _EMNLP_.
* Hill et al. (2016) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children's books with explicit memory representations. _ICLR_ .
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_ .
* Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _ICML_. JMLR Workshop and Conference Proceedings.
* Joshi et al. (2014) Mandar Joshi, Uma Sawant, and Soumen Chakrabarti. 2014. Knowledge Graph and Corpus Driven Segmentation and Answer Inference for Telegraphic Entity-seeking Queries. In _EMNLP_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. _ICLR_ .



* Krishnamurthy and Mitchell (2012) Jayant Krishnamurthy and Tom Mitchell. 2012. Weakly Supervised Training of Semantic Parsers. In _EMNLP_.
* Kupiec (1993) Julian Kupiec. 1993. MURAX: A robust linguistic approach for question answering using an on-line encyclopedia. In _SIGIR_. ACM.
* Lee et al. (2016) Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. _arXiv preprint arXiv:1611.01436_ .
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _NIPS_.
* Miller et al. (2016) Alexander H. Miller, Adam Fisch, Jesse Dodge, AmirHossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. _In EMNLP_ .
* Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In _ACL_.
* Neelakantan et al. (2015a) Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2015a. Neural programmer: Inducing latent programs with gradient descent. _arXiv preprint arXiv:1511.04834_ .
* Neelakantan et al. (2015b) Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. 2015b. Compositional vector space models for knowledge base completion. In _ACL_.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading COmprehensions Dataset. _CoRR_ abs/1611.09268.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _EMNLP_. Austin, Texas.
* Reddy et al. (2014) Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _TACL_ 2.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _ECML PKDD_.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In _NAACL_.
* Savenkov and Agichtein (2016) Denis Savenkov and Eugene Agichtein. 2016. When a knowledge base is not enough: Question answering over knowledge bases with external text data. In _SIGIR_. ACM.
* Seo et al. (2016) Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Query-reduction networks for question answering. _arXiv preprint arXiv:1606.04582_ .
* Stanovsky et al. (2014) Gabriel Stanovsky, Omer Levy, and Ido Dagan. 2014. Proposition Knowledge Graphs. _COLING 2014_ .
* Sakhbbaatar et al. (2015) Sainbayar Sakhbbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In _NIPS_.
* Sun et al. (2015) Huan Sun, Hao Ma, Wen-tau Yih, Chen-Tse Tsai, Jingjing Liu, and Ming-Wei Chang. 2015. Open domain question answering via semantic enrichment. In _WWW_. ACM.
* Trischler et al. (2016) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. NewsQA: A Machine Comprehension Dataset. _CoRR_ abs/1611.09830.
* Verga et al. (2016) Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016. Multilingual relation extraction using compositional universal schema .
* Voorhees et al. (1999) Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In _Trec_. volume 99, pages 77-82.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. _arXiv preprint arXiv:1608.07905_ .
* Weston et al. (2015) Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. _In ICLR_ .
* Xiong et al. (2016) Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic Coattention Networks For Question Answering. _arXiv preprint arXiv:1611.01604_ .
* Xu et al. (2016a) Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016a. Hybrid Question Answering over Knowledge Base and Free Text. In _COLING_.
* Xu et al. (2016b) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016b. Question Answering on Freebase via Relation Extraction and Textual Evidence. In _ACL_.
* Yahya et al. (2016) Mohamed Yahya, Denilson Barbosa, Klaus Berberich, Qiuyue Wang, and Gerhard Weikum. 2016. Relationship queries on extended knowledge graphs. In _Proceedings of the Ninth ACM International Conference on Web Search and Data Mining_. ACM, pages 605-614.
* Yao et al. (2010) Limin Yao, Sebastian Riedel, and Andrew McCallum. 2010. Collective cross-document relation extraction without labelled data. In _EMNLP_.
* Yao and Durme (2014) Xuchen Yao and Benjamin Van Durme. 2014. Information Extraction over Structured Data: Question Answering with Freebase. In _ACL_ 

Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In _ACL_.
* Zelle and Mooney (1996) John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In _AAAI_. Portland, Oregon.
* Zeng et al. (2015) Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In _EMNLP_.
* Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In _UAI_. Edinburgh, Scotland.