Venue: ACL 2017

# Generating Natural Answers by Incorporating

Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning

Shizhu He\({}^{1}\), Cao Liu\({}^{1,2}\), Kang Liu\({}^{1}\) &Jun Zhao\({}^{1,2}\)

\({}^{1}\) National Laboratory of Pattern Recognition, Institute of Automation,

Chinese Academy of Sciences, Beijing, 100190, China

\({}^{2}\) University of Chinese Academy of Sciences, Beijing, 100049, China

{shizhu.he, cao.liu, kliu, jzhao}@nlpr.ia.ac.cn

###### Abstract

Generating answer with natural language sentence is very important in real-world question answering systems, which needs to obtain a right answer as well as a coherent natural response. In this paper, we propose an end-to-end question answering system called COREQ in sequence-to-sequence learning, which incorporates copying and retrieving mechanisms to generate natural answers within an encoder-decoder framework. Specifically, in COREQA, the semantic units (words, phrases and entities) in a natural answer are dynamically predicted from the vocabulary, copied from the given question and/or retrieved from the corresponding knowledge base jointly. Our empirical study on both synthetic and real-world datasets demonstrates the efficiency of COREQA, which is able to generate correct, coherent and natural answers for knowledge inquired questions.

+
Footnote †: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 199–208 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics [https://doi.org/10.18653/v1/P17-1019](https://doi.org/10.18653/v1/P17-1019)

## 1 Introduction

Question answering (QA) systems devote to providing exact answers, often in the form of phrases and entities for natural language questions Woods [17]; Ferrucci et al. [20]; Lopez et al. [2011]; Yih et al. [2015], which mainly focus on analyzing questions, retrieving related facts from text snippets or knowledge bases (KBs), and finally predicting the answering semantic units-**SU** (word-s, phrases and entities) through ranking Yao and Van Durme [2014] and reasoning Kwok et al. [2001].

However, in real-world environments, most people prefer the correct answer replied with a more natural way. For example, most existing commercial products such as Siri1 will reply a natural answer "_Jet Li is 1.64m in height._" for the question "_How tall is Jet Li?_", rather than only answering one entity "_1.64m_". Basic on this observation, we define the "natural answer" as the natural response in our daily communication for replying factual questions, which is usually expressed in a complete/partial natural language sentence rather than a single entity/phrase. In this case, the system needs to not only parse question, retrieve relevant facts from KB but also generate a proper reply. To this end, most previous approaches employed message-response patterns. Figure 1 schematically illustrates the major steps and features in this process. The system first needs to recognize the topic entity "_Jet Li_" in the question and then extract multiple related facts <_Jet Li, gender, Male>_, <_Jet Li, birthplace, Beijing> and <Jet Li, nationality, Singapore_> from KB. Based on the chosen facts and the commonly used message-response patterns "_where was %entity from?_" -_"%entity was born in %birthplace, %pronoun is %rationality citizen."2_, the system could finally generate the natural answer McTear et al. [2016].

Footnote 1: [http://www.apple.com/ios/siri/](http://www.apple.com/ios/siri/)

Footnote 2: In this pattern, %_entity_ indicates the placeholder of the topic entity, %_property_ indicates the property value of the topic entity.

In order to ge

Figure 1: Incorporating copying and retrieving mechanisms in generating a natural answer.



products need lots of Natural Language Processing (NLP) tools and pattern engineering McTear et al. (2016), which not only suffers from high costs of manual annotations for training data and patterns, but also have low coverage that cannot flexibly deal with variable linguistic phenomena in different domains. Therefore, this paper devotes to develop an end-to-end paradigm that generates natural answers without any NLP tools (e.g. POS tagging, parsing, etc.) and pattern engineering. This paradigm tries to consider question answering in an end-to-end framework. In this way, the complicated QA process, including analyzing question, retrieving relevant facts from KB, and generating correct, coherent, natural answer-s, could be resolved jointly.

Nevertheless, generating natural answers in an end-to-end manner is not an easy task. The key challenge is that the words in a natural answer may be generated by different ways, including: _1)_ the common words usually are predicted using a (conditional) language model (e.g. "_born_" in Figure 1); _2)_ the major entities/phrases are selected from the source question (e.g. "_Jet Li_"); _3)_ the answering entities/phrases are retrieved from the corresponding KB (e.g. "_Beijing_"). In addition, some words or phrases even need to be inferred from related knowledge (e.g. "_He_" should be inferred from the value of "_gender_"). And we even need to deal with some morphological variants (e.g. "_Singapore_" in KB but "_Singaporean_" in answer). Although existing end-to-end models for KB-based question answering, such as GenQA Yin et al. (2016), were able to retrieve facts from KBs with neural models. Unfortunately, they cannot copy SUs from the question in generating answers. Moreover, they could not deal with complex questions which need to utilize multiple facts. In addition, existing approaches for conversational Dialogue systems are able to generate natural utterances Serban et al. (2016); Li et al. (2016) in sequence-to-sequence learning (Seq2Seq). But they cannot interact with KB and answer information-inquired questions. For example, CopyNet Gu et al. (2016) is able to copy words from the original source in generating the target through incorporating copying mechanism in conventional Seq2Seq learning, but they cannot retrieve SUs from external memory (e.g. KBs, Texts, etc.).

Therefore, facing the above challenges, this paper proposes a neural generative model called COREQA with Seq2Seq learning, which is able to reply an answer in a natural way for a given question. Specifically, we incorporate **CO**pying and **RE**trieving mechanisms within Seq2Seq learning. COREQA is able to analyze the question, retrieve relevant facts and generate a sequence of SUs using a hybrid method with a completely end-to-end learning framework. We conduct experiments on both synthetic data sets and real-world datasets, and the experimental results demonstrate the efficiency of COREQA compared with existing end-to-end QA/Dialogue methods.

In brief, our main contributions are as follows:

* We propose a new and practical question answering task which devotes to generating natural answers for information inquired questions. It can be regarded as a fusion task of QA and Dialogue.
* We propose a neural network based model, named as COREQ, by incorporating copying and retrieving mechanism in Seq2Seq learning. In our knowledge, it is the first end-to-end model that could answer complex questions in a natural way.
* We implement experiments on both synthetic and real-world datasets. The experimental results demonstrate that the proposed model could be more effective for generating correct, coherent and natural answers for knowledge inquired questions compared with existing approaches.

## 2 Background: Neural Models for Sequence-to-Sequence Learning

### RNN Encoder-Decoder

Recurrent Neural Network (RNN) based Encoder-Decoder is the backbone of Seq2Seq learning Cho et al. (2014). In the Encoder-Decoder framework, an encoding RNN first transform a source sequential object \(X=[x_{1},...,x_{L_{X}}]\) into an encoded representation **c**. For example, we can utilize the basic model: \(\textbf{h}_{t}=f(x_{t},\textbf{h}_{t-1});\textbf{c}=\phi(\textbf{h}_{1},..., \textbf{h}_{L_{X}})\), where \(\{\textbf{h}_{t}\}\) are the RNN hidden states, **c** is the context vector which could be assumed as an abstract representation of \(X\). In practice, gated RNN variants such as LSTM Hochreiter and Schmidhuber (1997) and GRU Chung et al. (2014) are commonly used for learning long-term dependencies. And the another encoding 

tricks is Bi-directional RNN, which connect two hidden states of positive time direction and negative time direction. Once the source sequence is encoded, another decoding RNN model is to generate a target sequence \(Y=[y_{1},...,y_{L_{Y}}]\), through the following prediction model: \(\textbf{s}_{t}=f(y_{t-1},\textbf{s}_{t-1},\textbf{c});~{}p(y_{t}|y_{<t},X)=g(y_{ t-1},\textbf{s}_{t},\textbf{c})\), where \(\textbf{s}_{t}\) is the RNN hidden state at time \(t\), the predicted target word \(y_{t}\) at time \(t\) is typically performed by a \(softmax\) classifier over a settled vocabulary (e.g. 30,000 words) through function \(g\).

### The Attention Mechanism

The prediction model of classical decoders for each target word \(y_{i}\) share the same context vector **c**. However, a fixed vector is not enough to obtain a better result on generating a long targets.The attention mechanism in the decoding can dynamically choose context \(\textbf{c}_{t}\) at each time step Bahdanau et al. (2014), for example, representing \(\textbf{c}_{t}\) as the weighted sum of the source states \(\left\{\mathbf{h}_{t}\right\}\),

\[\textbf{c}_{t}=\sum\nolimits^{L_{X}}_{i=1}\alpha_{ti}\textbf{h}_{i};~{}~{} \alpha_{ti}=\frac{e^{\rho(\textbf{s}_{t-1},\textbf{h}_{i})}}{\sum\nolimits_{ i^{\prime}}e^{\rho(\textbf{s}_{t-1},\textbf{h}_{i}^{\prime})}} \tag{1}\]

where the function \(\rho\) use to compute the attentive strength with each source state, which usually adopts a neural network such as multi-layer perceptron (MLP).

### The Copying Mechanism

Seq2Seq learning heavily rely on the "meaning" for each word in source and target sequences, however, some words in sequences are "no-meaning" symbols and it is improper to encode them in encoding and decoding processes. For example, generating the response _"Of course, read"_ for replying the message _"Can you read the word 'read'?"_ should not consider the meaning of the second _"read"_. By incorporating the copying mechanism, the decoder could directly copy the sub-sequences of source into the target Vinyals et al. (2015). The basic approach is to jointly predict the indexes of the target word in the fixed vocabulary and/or matched positions in the source sequences Gu et al. (2016); Gulcehre et al. (2016).

## 3 Coreqa

To generate natural answers for information required questions, we should first recognize key topics in the question, then extract related facts from KB, and finally fusion those instance-level knowledge with some global-level "smooth" and "glue" words to generate a coherent reply. In this section, we present COREQA, a differentiable Seq2Seq model to generate natural answers, which is able to analyze the question, retrieve relevant facts and predict SUs in an end-to-end fashion, and the predicted SUs may be predicted from the vo

Figure 2: The overall diagram of COREQ.

 

cabulary, copied from the given question, and/or retrieved from the corresponding KB.

### Model Overview

As illustrated in Figure 2, COREQA is an encoder-decoder framework plugged with a KB engineer. A knowledge retrieval module is firstly employed to retrieve related facts from KB by question analysis (see Section 3.2). And then the input question and the retrieved facts are transformed into the corresponding representations by **Encoders** (see Section 3.3). Finally, the encoded representations are feed to **Decoder** for generating the target natural answer (see Section 3.4).

### Knowledge (facts) Retrieval

We mainly focus on answering the information inquired questions (factual questions, and each question usually contains one or more topic entities). This paper utilizes the gold topic entities for simplifying our design. Given the topic entities, we retrieve the related facts from the corresponding KB. KB consists of many relational data, which usually are sets of inter-linked subject-property-object (_SPO_) triple statements. Usually, question contains the information used to match the _subject_ and _property_ parts in a fact triple, and answer incorporates the _object_ part information.

### Encoder

The encoder transforms all discrete input symbol-s (including words, entities, properties and properties' values) and their structures into numerical representations which are able to feed into neural models [15].

#### 3.3.1 Question Encoding

Following [8], a bi-directional RN-N [7] is used to transform the question sequence into a sequence of concatenated hidden states with two independent RNNs. The forward and backward RNN respectively obtain \(\{\overrightarrow{\textbf{h}}_{1},...,\overrightarrow{\textbf{h}}_{L_{X}}\}\) and \(\{\overleftarrow{\textbf{h}}_{L_{X}},...,\overleftarrow{\textbf{h}}_{1}\}\). The concatenated representation is considered to be the short-term memory of question (\(\textbf{M}_{Q}=\{\textbf{h}_{t}\},\textbf{h}_{t}=|\overrightarrow{\textbf{ h}}_{t},\overleftarrow{\textbf{h}}_{L_{X}-t+1}|\)). \(\textbf{q}=[\overrightarrow{\textbf{h}}_{L_{X}},\overleftarrow{\textbf{h}}_ {1}]\) is used to represent the entire question, which could be used to compute the similarity between the question and the retrieved facts.

#### 3.3.2 Knowledge Base Encoding

We use \(s\), \(p\) and \(o\) denote the subject, property and object (value) of one fact f, and \(\textbf{e}_{s}\), \(\textbf{e}_{p}\) and \(\textbf{e}_{o}\) to denote its corresponding embeddings. The fact representation **f** is then defined as the concatenation of \(\textbf{e}_{s}\), \(\textbf{e}_{p}\) and \(\textbf{e}_{o}\). The list of all related facts' representations, \(\{\textbf{f}\}=\{\textbf{f}_{1}\), ..., \(\textbf{f}_{L_{F}}\}\) (refer to \(\textbf{M}_{KB}\), \(L_{F}\) denotes the maximum of candidate facts), is considered to be a short-term memory of KB while answering questions about the topic entities.

In addition, given the distributed representation of question and candidate facts, we define the matching scores function between question and facts as \(S(q,f_{j})=DNN_{1}(\textbf{q},\textbf{f}_{j})=tanh(\textbf{W}_{2}\cdot tanh( \textbf{W}_{1}\cdot[\textbf{q},\textbf{f}_{j}]+\textbf{b}_{1})+\textbf{b}_{2})\),, where \(DNN_{1}\) is the matching function defined by a two-layer perceptron, \([\cdot,\cdot]\) denotes vector concatenation, and \(\textbf{W}_{1}\), \(\textbf{W}_{2}\), \(\textbf{b}_{1}\) and \(\textbf{b}_{2}\) are the learning parameters. In fact, we will make a slight change of the matching function because it will also depend on the state of decoding process at different times. The modified function is \(S(q,s_{t},f_{j})=DNN_{1}(\textbf{q},\textbf{s}_{t},\textbf{f}_{j})\) where \(s_{t}\) is the hidden state of decoder at time t.

### Decoder

The decoder uses an RNN to generate a natural answer based on the short-term memory of question and retrieved facts which represented as \(\textbf{M}_{Q}\) and \(\textbf{M}_{KB}\), respectively. The decoding process of COREQA have the following differences compared with the conventional decoder:

**Answer words prediction**: COREQ predicts SUs based on a mixed probabilistic model of three modes, namely the _predict-mode_, the _copy-mode_ and the _retrieve-mode_, where the first mode predicts words with the vocabulary, and the two latter modes pick SUs from the questions and matched facts, respectively;

**State update**: the predicted word at step \(t-1\) is used to update \(\textbf{s}_{t}\), but COREQA uses not only its word embedding but also its corresponding positional attention informations in \(\textbf{M}_{Q}\) and \(\textbf{M}_{KB}\) ;

**Reading short-Memory M\({}_{Q}\) and M\({}_{KB}\)**: M\({}_{Q}\) and M\({}_{KB}\) are fed into COREQA with two ways, the first one is the "meaning" with embeddings and the second one is the positions of different words (properties' values).

#### 3.4.1 Answer Words Prediction

The generated words (entities) may come from vocabulary, source question and matched KB. Accordingly, our model use three correlative output 

layer: _shortlist_ prediction layer, _question location_ copying layer and _candidate-facts location_ retrieving layer, respectively. And we use the \(softmax\) classifier of the above three cascaded output layers to pick SUs. We assume a vocabulary \({\cal V}=\{v_{1},...,v_{N}\}\cup\{\text{UNK}\}\), where UNK indicates any out-of-vocabulary (OOV) words. Therefore, we have adopted another two set of SUs \({\cal X}_{Q}\) and \({\cal X}_{KB}\) which cover words/entities in the source question and the partial KB. That is, we have adopted the instance-specific vocabulary \({\cal V}\cup{\cal X}_{Q}\cup{\cal X}_{KB}\) for each question. It's important to note that these three vocabularies \({\cal V}\), \({\cal X}_{Q}\) and \({\cal X}_{KB}\) may overlap.

At each time step \(t\) in the decoding process, given the RNN state \(\textbf{s}_{t}\) together with **M\({}_{Q}\)** and **M\({}_{KB}\)**, the probabilistic function for generating any target SU \(y_{t}\) is a "mixture" model as follow

\[p(y_{t}|\textbf{s}_{t},y_{t-1},\textbf{M}_{Q},\textbf{M}_{KB})= \tag{2}\] \[p_{pr}(y_{t}|\textbf{s}_{t},y_{t-1},\textbf{c}_{t})\cdot p_{m}( pr|\textbf{s}_{t},y_{t-1})+\] \[p_{co}(y_{t}|\textbf{s}_{t},y_{t-1},\textbf{M}_{Q})\cdot p_{m}( co|\textbf{s}_{t},y_{t-1})+\] \[p_{re}(y_{t}|\textbf{s}_{t},y_{t-1},\textbf{M}_{KB})\cdot p_{m}( re|\textbf{s}_{t},y_{t-1})\]

where \(pr\), \(co\) and \(re\) stand for the predict-mode, the copy-mode and the retrieve-mode, respectively, \(p_{m}(\cdot|\cdot)\) indicates the probability model for choosing different modes (we use a \(softmax\) classifier with two-layer MLP). The probability of the three modes are given by

\[p_{pr}(y_{t}|\cdot)=\frac{1}{Z}e^{\psi_{pr}(y_{t})} \tag{3}\] \[p_{co}(y_{t}|\cdot)=\frac{1}{Z}\sum\nolimits_{j:Q_{j}=y_{t}}e^{ \psi_{co}(y_{t})}\] \[p_{re}(y_{t}|\cdot)=\frac{1}{Z}\sum\nolimits_{j:KB_{j}=y_{t}}e^{ \psi_{re}(y_{t})}\]

where \(\psi_{pr}(\cdot)\), \(\psi_{co}(\cdot)\) and \(\psi_{re}(\cdot)\) are score functions for choosing SUs in predict-mode (from \({\cal V}\)), copy-mode (from \({\cal X}_{Q}\)) and retrieve-mode (from \({\cal X}_{KB}\)), respectively. And \(Z\) is the normalization term shared by the three modes, \(Z=e^{\psi_{pr}(v)}+\sum\nolimits_{j:Q_{j}=w}e^{\psi_{co}(v)}+\sum\nolimits_{ j:KB_{j}=v}e^{\psi_{re}(v)}\). And the three modes could compete with each other through a \(softmax\) function in generating target SUs with the shared normalization term (as shown in Figure 2. Specifically, the scoring functions of each mode are defined as follows:

_P_**redict-mode**: Some generated words need reasoning (e.g. "He" in Figure 1) and morphological transformation (e.g. "Singaporean" in Figure 1). Therefore, we modify the function as \(\psi_{pr}(y_{t}=v_{i})=\textbf{v}_{i}^{T}\textbf{W}_{pr}[\textbf{s}_{t}, \textbf{c}_{q_{t}},\textbf{c}_{bb_{t}}]\) , where \(\textbf{v}_{i}\in\mathcal{R}^{d_{o}}\) is the word vector at the output layer (not the input word embedding), \(\textbf{W}_{pr}\in\mathcal{R}^{(d_{h}+d_{i}+d_{f})\times d_{o}}\) (\(d_{i}\), \(d_{h}\) and \(d_{f}\) indicate the size of input word vector, RNN decoder hidden state and fact representation respectively), and \(\textbf{c}_{q_{t}}\) and \(\textbf{c}_{kb_{t}}\) are the temporary memory of reading \(\textbf{M}_{Q}\) and \(\textbf{M}_{KB}\) at time t (see Section 3.4.3).

_Copy_**-mode**: The score for "copying" the word \(x_{j}\) from question Q is calculated as \(\psi_{co}(y_{t}=x_{j})=DNN_{2}(\textbf{h}_{j},\textbf{s}_{t},\textbf{hist}_{Q})\) , where \(DNN_{2}\) is a neural network function with a two-layer MLP and \(\textbf{hist}_{Q}\in\mathcal{R}^{L_{X}}\) is an accumulated vector which record the attentive history for each word in question (similar with the coverage vector in (Tu et al., 2016)).

_Retrieve-mode_: The score for "retrieving" the entity word \(v_{j}\) from retrieval facts ("Object" part) is calculated as \(\psi_{re}(y_{t}=v_{j})=DNN_{3}(\textbf{f}_{j},\textbf{s}_{t},\textbf{hist}_{KB})\) , where \(DNN_{3}\) is also a neural network function and \(\textbf{hist}_{KB}\in\mathcal{R}^{L_{F}}\) is an accumulated vector which record the attentive history for each fact in candidate facts.

#### 3.4.2 State Update

In the generic decoding process, each RNN hidden state \(\textbf{s}_{t}\) is updated with the previous state \(\textbf{s}_{t-1}\), the word embedding of previous predicted symbol \(y_{t-1}\), and an optional context vector \(\textbf{c}_{t}\) (with attention mechanism). However, \(y_{t-1}\) may not come from vocabulary \({\cal V}\) and not owns a word vector. Therefore, we modify the state update process in COREQA. More specifically, \(y_{t-1}\) will be represented as concatenated vector of \([\textbf{e}(y_{t-1}),\textbf{r}_{q_{t-1}},\textbf{r}_{kb_{t-1}}]\), where \(\textbf{e}(y_{t-1})\) is the word embedding associated with \(y_{t-1}\), \(\textbf{r}_{q_{t-1}}\) and \(\textbf{r}_{kb_{t-1}}\) are the weighted sum of hidden states in \(\textbf{M}_{Q}\) and \(\textbf{M}_{KB}\) corresponding to \(y_{t-1}\) respectively.

\[\begin{split}&\textbf{r}_{q_{t}}=\sum\nolimits_{j=1}^{L_{X}}\rho_{tj} \textbf{h}_{j},\textbf{r}_{kb_{t}}=\sum\nolimits_{j=1}^{L_{F}}\delta_{tj} \textbf{f}_{j}\\ &\rho_{tj}=\left\{\begin{array}{cc}\dfrac{1}{K_{1}}p_{co}(x_{j}| \cdot),& x_{j}=y_{t}\\ \mathbf{0}& otherwise\\ \end{array}\right.\\ &\delta_{tj}=\left\{\begin{array}{cc}\dfrac{1}{K_{2}}p_{re}(f_{j}| \cdot),& object(f_{j})=y_{t}\\ \mathbf{0}&otherwise\\ \end{array}\right.\end{split} \tag{4}\]

where \(object(f)\) indicate the "object" part of fact \(f\) (see Figure 2), and \(K_{1}\) and \(K_{2}\) are the normalization terms which equal \(\sum\nolimits_{j^{\prime}:x_{j}^{\prime}=y_{t}}p_{co}(x_{j}^{\prime}|\cdot)\) and \(\sum\nolimits_{j^{\prime}:object(f_{j}^{\prime})=y_{t}}p_{re}(f_{j}^{\prime}|\cdot)\), respectively, and it 

could consider the multiple positions matching \(y_{t}\) in source question and KB.

#### 3.4.3 Reading short-Memory M\({}_{Q}\) and M\({}_{KB}\)

COREQA employ the attention mechanism at decoding process. At each decoder time t, we selective read the context vector **c\({}_{q_{t}}\)** and **c\({}_{kb_{t}}\)** from the short-term memory of question **M\({}_{Q}\)** and retrieval facts **M\({}_{KB}\)** (alike to Formula 1). In addition, the accumulated attentive vectors **hist\({}_{Q}\)** and **hist\({}_{KB}\)** are able to record the positional information of SUs in the source question and retrieved facts.

### Training

Although some target SUs in answer are copied and retrieved from the source question and the external KB respectively, COREQA is fully differential and can be optimized in an end-to-end manner using back-propagation. Given the batches of the source questions \(\{X\}_{M}\) and target answers \(\{Y\}_{M}\) both expressed with natural language (symbolic sequences), the objective function is to minimize the negative log-likelihood:

\[\mathcal{L}=-\frac{1}{N}\sum_{k=1}^{M}\sum_{t=1}^{L_{Y}}log[p(y_{t}^{(k)}|y_{< t}^{(k)},X^{(k)}] \tag{5}\]

where the superscript \((k)\) indicates the index of one question-answer (Q-A) pair. The network is no need for any additional labels for training models, because the three modes sharing the same \(softmax\) classifier for predicting target words, they can learn to coordinate with each other by maximizing the likelihood of observed Q-A pairs.

## 4 Experiments

In this section, we present our main experimental results in two datasets. The first one is a small synthetic dataset in a restricted domain (only involving four properties of persons) (Section 4.1). The second one is a big dataset in open domain, where the Q-A pairs are extracted from community QA website and grounded against a KB with an Integer Linear Programming (ILP) method (Section 4.2). COREQA and all baseline models are trained on a NVIDIA TITAN X GPU using TensorFlow3 tools, where we used the Adam Kingma and Ba (2014) learning rule to update gradients in all experimental configures. The sources codes and data will be released at the personal homepage of the first author4.

Footnote 3: [https://www.tensorflow.org/](https://www.tensorflow.org/)

### Natural QA in Restricted Domain

Task:The QA systems need to answer question-s involving 4 concrete properties of birthdate (including year, month and day) and gender). Through merely involving 4 properties, there are plenty of QA patterns which focus on different aspects of birthdate, for example, "_What year were you born?_" touches on "year", but "_When is your birthday?_" touches on "month and day".

Dataset:Firstly, 108 different Q-A pattern-s have been constructed by two annotators, one in charge of raising question patterns and another one is responsible for generating corresponding suitable answer patterns, e.g. When is %\(e\) birthday? \(\rightarrow\) She was born in %\(m~{}\%d\texttt{th}\). where the variables %\(e\), %\(y\), %\(m\), %\(d\) and %\(g\) (deciding _she_ or _he_) indicates the person's name, birth year, birth month, birth day and gender, respectively. Then we randomly generate a KB which contains 80,000 person entities, and each entity including four facts. Given KB fact-s, we can finally obtain specific Q-A pairs. And the sampling KB, patterns, and the generated Q-A pairs are shown in Table 1. In order to maintain the diversity, we randomly select 6 patterns for each person. Finally, we totally obtain 239,934 sequences pairs (half patterns may be unmatched because of "gender" property).

Experimental Setting:The total 239,934 Q-A pairs are split into training (90%) and testing set (10%). The baseline includes 1) generic RNN Encoder-Decoder (marked as _RNN_), 2) Seq2Seq with attention (marked as _RNN+atten_), 3) _CopyNet_, and 4) _GenQA_. For a fair comparison, we use bi-directional LSTM for encoder and another LST-M for decoder for all Seq2Seq models, with hidden layer size = 600 and word embedding dimen

\begin{table}
\begin{tabular}{|c|c|} \hline  & **Examples** (e.g. KB facts \\
**Q-A Patterns** & (e2,year,1987);(e2,month,6); \\  & (e2,day,20);(e2,gender,male)) \\ \hline When is \(\%e\) birthday? & When is e2 birthday? \\ He was born in \(\%m~{}\%d\texttt{th}\). & He was born in June 20th. \\ \hline What year were \(\%e\) born? & What year were e2 born? \\ \(\%e\) is born in \(\%y\) year. & e2 is born in 1987 year. \\ \hline \end{tabular}
\end{table}
Table 1: Sample KB facts, patterns and their generated Q-A pairs.

 

sion = 200. We set \(L_{F}\) as 5.

**Metrics:** We adopt (**automatic evaluation (AE)** to test the effects of different models. **AE** considers the precisions of the entire predicted answer-s and four specific properties, and the answer is complete correct only when all predicted properties' values is right. To measure the performance of the proposed method, we select following metrics, including \(P_{g}\)5, \(P_{y}\), \(P_{m}\) and \(P_{d}\) which denote the precisions for 'gender', 'year', 'month' and 'day' properties, respectively. And \(P_{A}\), \(R_{A}\) and \(F1_{A}\) indicate the precision, recall and F1 in the complete way.

Footnote 5: The “gender” is right when the entity name (e.g. ‘e2’) or the personal pronoun (e.g. ’She’) in answer is correct.

**Experimental Results:** The **AE** experimental results are shown in Table 2. It is very clear from Table 2 that COREQA significantly outperforms all other compared methods. The reason of the GenQA's poor performance is that all synthetic questions need multiple facts, and GenQA will "safely" choose the most frequent property ("gender") for all questions. We also found the performances on "year" and "day" have a little worse than other properties such as "gender", it may because there have more ways to answer questions about "year" and "day".

**Discussion:** Because of the feature of directly "hard" copy and retrieve SUs from question and KB, COREQA could answer questions about unseen entities.To evaluate the effects of answering questions about unseen entities, we re-construct 2,000 new person entities and their corresponding facts about four known properties, and obtain 6,081 Q-A pairs through matching the sampling patterns mentioned above. The experimental results are shown in Table 3, it can be seen that the performance did not fall too much.

### Natural QA in Open Domain

Task:To test the performance of the proposed approach in open domains, we modify the task of GenQA Yin et al. (2016) for supporting multi-facts (a typical example is shown in Figure 1). That is, a natural QA system should generate a sequence of SUs as the natural answer for a given natural language question through interacting with a KB.

Dataset:GenQA have released a corpus6, which contains a crawling KB and a set of ground Q-A pairs. However, the original Q-A pairs only matched with just one single fact. In fact, we found that a lot of questions need more than one fact (about 20% based on sampling inspection). Therefore, we crawl more Q-A pairs from Chinese community QA website Baidu Zhidao7). Combined with the originally published corpus, we create a lager and better-quality data for natural question answering. Specifically, an Integral Linear Programming (ILP) based method is employed to automatically construct "grounding" Q-A pairs with the facts in KB (inspired by the work of adopting ILP to parse questions Yahya et al. (2012)). In ILP, the main constraints and considered factors are listed below: 1) the "subject" entity and "object" entity of a triple have to match with question words/phrases (marked as _subject mention_) and answer words/phrases (marked as _object mention_) respectively; 2) any two _subject mentions_ or _object mentions_ should not overlap; 3) a _mention_ can match at most one entity; 4) the edit distance between the Q-A pair and the matched candidate fact (use a space to joint three parts) is smaller, they are more relevant. Finally, we totally obtain 619,199 instances (an instance contains a question, an answer, and multiple facts), and the number of instances that can match one and multiple facts in KB are 499,809 and 119,390, respectively. Through the evaluation of 200 sampling instances, we estimate that approximate 81% matched facts are helpful for the generating answers. However, strictly speaking, only 44% instances are truly correct grounding. In fact, grounding the Q-A pairs from community QA website is a very challenge problem, we will leave it in the future work.

Footnote 6: [https://github.com/jxfeb/Generative_QA](https://github.com/jxfeb/Generative_QA)

**Experimental Setting:** The dataset is split into training (90%) and testing set (10%). The sen

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline
**Models** & \(P_{g}\) & \(P_{y}\) & \(P_{m}\) & \(P_{d}\) & \(P_{A}\) & \(R_{A}\) & \(F1_{A}\) \\ \hline RNN & 72.2 & 0 & 1.1 & 0.2 & 0 & 27.5 & 0 \\ RNN+atten & 55.8 & 1.1 & 11.3 & 9.5 & 1.7 & 34 & 3.2 \\ CopyNet & 75.2 & 8.7 & 28.3 & 5.8 & 3.7 & 32.5 & 6.7 \\ GenQA & 73.4 & 0 & 0 & 0 & 0 & 27.1 & 0 \\ \hline COREQA & **100** & **84.8** & **93.4** & **81** & **87.4** & **94** & **90.6** \\ \hline \end{tabular}
\end{table}
Table 2: The AE results (%) on synthetic test data.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline
**Entities** & \(P_{g}\) & \(P_{y}\) & \(P_{m}\) & \(P_{d}\) & \(P_{A}\) & \(R_{A}\) & \(F1_{A}\) \\ \hline Seen & 100 & 84.8 & 93.4 & 81 & 87.4 & 94 & 90.6 \\ \hline Unseen & 75.1 & 84.5 & 93.5 & 81.2 & 63.8 & 85.1 & 73.1 \\ \hline \end{tabular}
\end{table}
Table 3: The AE (%) for seen and unseen entities.

 

tences in Chinese are segmented into word sequences with Jieba8 tool. And we use the words with the frequency larger than 3, which covering 98.4% of the word in the corpus. For a fair comparison, we use bi-directional LSTM for the encoder and another LSTM for decoder for all Seq2Seq models, with hidden layer size = 1024 and word embedding dimension = 300. We select CopyNet (more advanced Seq2Seq model) and GenQA for comparison. We set \(L_{F}\) as 10.

Footnote 7: [https://github.com/kysj/jieba](https://github.com/kysj/jieba)

**Metrics:** Besides adopting the **AE** as a metric (same as GenQA Yin et al. (2016)), we additionally use **manual evaluation (ME)** as another metric. **ME** considers three aspects about the quality of the generated answer (refer to Asghar et al. (2016)): 1) **correctness**; 2) syntactical **fluency**; 3) **coherence** with the question. We employ two annotators to rate such three aspects of CopyNet, GenQA and COREQA. Specifically, we sample 100 questions, and conduct \(C^{2}_{3}=3\) pair-wise comparisons for each question and count the winning times of each model (comparisons may both win or both lose).

**Experimental Results:** The **AE** and **ME** results are shown in Table 4 and Table 5, respectively. Meanwhile, we separately present the results according to the number of the facts which a question needs in KB, including just one single fact (marked as **Single**), multiple facts (marked as **Multi**) and all (marked as **Mixed**). In fact, we train two separate models for **Single** and **Multi** questions for the unbalanced data . From Table 4 and Table 5, we can clearly observe that COREQA significantly outperforms all other baseline models. And COREQ could generate a better natural answer in three aspects: correctness, fluency and coherence. CopyNet cannot interact it with KB which is important to generate correct answers. For example, for "Who is the director of The_Little_Chinese_Seamstress?", if without the fact (The_Little_Chinese_Seamstress, director, Dai_Siji), QA systems cannot generate a correct answer.

**Case Study and Error Analysis:** Table 6 gives some examples of generated by COREQA and the gold answers to the questions in test set. It is very clearly seen that the parts of generating SUs are predicted from the vocabulary, and other SUs are copied from the given question (marked as **bold**) and retrieved from the KB (marked as underline). And we analyze sampled examples and believe that there are several major causes of errors: **1)** did not match the right facts (ID 6); **2)** the generated answers contain some repetition of meaningless words (ID 7); **3)** the generated answers are not coherence natural language sentences (ID 8).

## 5 Related Work

Seq2Seq learning is to maximize the likelihood of predicting the target sequence \(Y\) conditioned on the observed source sequence \(X\) Sutskever et al. (2014), which has been applied successfully to a large number of NLP tasks such as Machine Translation Wu et al. (2016) and Dialogue Vinyals and Le (2015). Our work is partially inspired by the recent work of QA and Dialogue which have adopted Seq2Seq learning. CopyNet Gu et al. (2016) and Pointer Networks (Vinyals et al., 2015; Gulcehre et al., 2016) which could incorporate copying mechanism in conventional Seq2Seq learning. Different from our application which deals with knowledge inquired questions and generates natural answers, CopyNet Gu et al. (2016) and Pointer Networks (Gulcehre et al., 2016) can only copy words from the original input sequence. In contrast, COREQA is able to retrieve SUs from external memory. And GenQA Yin et al. (2016) can only deal with the simple questions which could be answered by one fact, and it also did not incorporate the copying mechanism in Seq2Seq learning.

Moreover, our work is also inspired by Neural Abstract Machine Graves et al. (2016); Yin et al. (2015); Liang et al. (2016) which could retrieve facts from KBs with neural models. Unlike natural answer, Neural Abstract Machine Mou et al. (2016) is concentrating on obtaining concrete answer en

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Models** & **Single** & **Multi** & **Mixed** \\ \hline CopyNet & 9.7 & 0.8 & 8.7 \\ GenQA & 47.2 & 28.9 & 45.1 \\ \hline COREQA & **58.4** & **42.7** & **56.6** \\ \hline \end{tabular}
\end{table}
Table 4: The AE accuracies (%) on real world test data.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**Models** & **Correctness** & **Fluency** & **Coherence** \\ \hline CopyNet & 0 & 13.3 & 3.3 \\ GenQA & 26.7 & 33.3 & 20 \\ \hline COREQA & **46.7** & **50** & **60** \\ \hline \end{tabular}
\end{table}
Table 5: The ME results (%) on sampled mixed test data.

 

tities with neural network based reasoning.

## 6 Conclusion and Future Work

In this paper, we propose an end-to-end system to generate natural answers through incorporating copying and retrieving mechanisms in sequence-to-sequence learning. Specifically, the sequences of SUs in the generated answer may be predicted from the vocabulary, copied from the given question and retrieved from the corresponding K-B. And the future work includes: a) lots of questions cannot be answered directly by facts in a KB (e.g. "_Who is Jet Li's father-in-law?_"), we plan to learn QA system with latent knowledge (e.g. K-B embedding Bordes et al. (2013)); b) we plan to adopt memory networks Sukhbaatar et al. (2015) to encode the temporary KB for each question.

## Acknowledgments

The authors are grateful to anonymous reviewers for their constructive comments. The work was supported by the Natural Science Foundation of China (No.61533018) and the National High Technology Development 863 Program of China (No.2015AA015405).

## References

* Asghar et al. (2016) Nabiha Asghar, Pascal Poupart, Jiang Xin, and Hang Li. 2016. Online sequence-to-sequence reinforcement learning for open-domain conversational agents. _arXiv preprint arXiv:1612.03929_ .
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Advances in neural information processing systems_. pages 2787-2795.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_ .
* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. _arXiv preprint arXiv:1412.3555_ .
* Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building watson: An overview of the deepqa project. _AI magazine_ 31(3):59-79.
* Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. _Nature_ 538(7626):471-476.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and O.K. Victor Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 1631-1640.
* Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 140-149.
* Galves et al. (2016)

\begin{table}
\begin{tabular}{|c|p{142.3pt}|p{142.3pt}|p{142.3pt}|} \hline
**ID** & **Question** & **Gold Answer** & **Generated Natural Answer** \\ \hline
1 & \multirow{2}{*}{Which country old _Harveyers_ of Bayer comes from?} & British born in Canada & \multicolumn{1}{p{142.3pt}}{} \\  & & & \\ \cline{2-4}  & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0 pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) \\ \hline
2 & \multicolumn{2}{c|}{Which ethnic groups is _Blaze home_ from?} & Han, now he is the DSG of the UN & DSO of the UN & DSO of U.S. Hua \\ \hline
3 & \multicolumn{2}{c|}{\(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) & \(\text{\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\hskip-0.6pt\raisebox{-0.6pt}[0.0pt]{$\bigcirc$}}\) \\ \hline
4 & \multicolumn{2}{c|}{When his Moscow?} & He is a football player & An excellent football player \\ \hline
5 & \multicolumn{2}{c|}{Web is the director of _The Little Chinese Scantness_?} & Dai Stipe & **Director** Dai Sijie \\ \hline
6 & \multicolumn{2}{c|}{Web do not have the movie _In Triangle_?} & Tsui Hark, Johnny To, Hang Lam & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}
7 & \multicolumn{2}{c|}{Web knows some details of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}  & \multicolumn{2}{c|}{Web do not know the details of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}
8 & \multicolumn{2}{c|}{Web do not know the details of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}  & \multicolumn{2}{c|}{Web do not know the details of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & yes, yes \\ \hline
8 & \multicolumn{2}{c|}{Web do not know the relations of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}  & \multicolumn{2}{c|}{Web do not know the relations of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & yes, yes \\ \hline
8 & \multicolumn{2}{c|}{Web do not know the relations of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \cline{2-4}  & \multicolumn{2}{c|}{Web do not know the relations of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & yes \\ \hline
8 & \multicolumn{2}{c|}{Web do not know the relations of _X Morning_?} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & yes \\ \hline \end{tabular}
\end{table}
Table 6: Examples of the generated natural answers by COREQA.

 

* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ .
* Kwok et al. (2001) Cody Kwok, Oren Etzioni, and Daniel S Weld. 2001. Scaling question answering to the web. _ACM Transactions on Information Systems (TOIS)_ 19(3):242-262.
* Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1192-1202.
* Liang et al. (2016) Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. _arXiv preprint arXiv:1611.00020_ .
* Lopez et al. (2011) Vanessa Lopez, Victoria Uren, Marta Sabou, and Enrico Motta. 2011. Is question answering fit for the semantic web?: a survey. _Semantic Web_ 2(2):125-155.
* McTear et al. (2016) Michael McTear, Zoraida Collejas, and David Griol. 2016. _The Conversational Interface: Talking to Smart Devices_. Springer Publishing Company, Incorporated, 1st edition.
* Mou et al. (2016) Lili Mou, Zhengdong Lu, Hang Li, and Zhi Jin. 2016. Coupling distributed and symbolic execution for natural language queries. _arXiv preprint arXiv:1612.02741_ .
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE Transactions on Signal Processing_ 45(11):2673-2681.
* Serban et al. (2016) Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16)_.
* Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In _Advances in neural information processing systems_. pages 2440-2448.
* Sutskever et al. (2014) Illya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_. pages 3104-3112.
* Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Coverage-based neural machine translation. _arXiv preprint arXiv:1601.04811_ .
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Advances in Neural Information Processing Systems_. pages 2692-2700.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. _arXiv preprint arXiv:1506.05869_ .
* Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. _CoRR_ abs/1410.3916.
* Woods (1977) William A Woods. 1977. Lunar rocks in natural english: Explorations in natural language question answering. In _Linguistic structures processing_. pages 521-569.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Kriktun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_ .
* Yahya et al. (2012) Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012. Natural language questions for the web of data. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. Association for Computational Linguistics, pages 379-390.
* Yao and Van Durme (2014) Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 956-966.
* Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 1321-1331.
* Yin et al. (2016) Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative question answering. In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_.
* Yin et al. (2015) Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2015. Neural enquirer: Learning to query tables. _arXiv preprint arXiv:1512.00965_ .