We download web pages returned in the first 3 result pages and take all the text pieces which have no more than 5 sentences and include at least one question word as candidate evidences. As evidence retrieval is beyond the scope of this work, we simply use TF-IDF values to re-rank these candidates. For each question in the training set, we provide the top 10 ranked evidences to annotate (“Annotated Evidence” in Table TABREF20 ). An evidence is annotated as positive if the question can be answered by just reading the evidence without any other prior knowledge, otherwise negative. Only evidences whose annotations are agreed by at least two annotators are retained. We also provide trivial negative evidences (“Retrieved Evidence” in Table TABREF20 ), i.e. evidences that do not contain golden standard answers. For each question in the validation and test sets, we provide one major positive evidence, and maybe an additional positive one to compute features. Both of them are annotated. Raw retrieved evidences are also provided for evaluation purpose (“Retrieved Evidence” in Table TABREF20 ). The dataset will be released on the project page http://idl.baidu.com/WebQA.html.

Evaluation Method	The performance is measured with precision (P), recall (R) and F1-measure (F1) DISPLAYFORM0  where INLINEFORM0 is the list of correctly answered questions, INLINEFORM1 is the list of produced answers, and INLINEFORM2 is the list of all questions . As WebQA is collected from web, the same answer may be expressed in different surface forms in the golden standard answer and the evidence, e.g. “北京 (Beijing)” v.s. “北京市 (Beijing province)”. Therefore, we use two ways to count correctly answered questions, which are referred to as “strict” and “fuzzy” in the tables: Strict matching: A question is counted if and only if the produced answer is identical to the golden standard answer; Fuzzy matching: A question is counted if and only if the produced answer is a synonym of the golden standard answer; And we also consider two evaluation settings: Annotated evidence: Each question has one major annotated evidence and maybe another annotated evidence for computing q-e.comm and e-e.comm features (Section SECREF14 ); Retrieved evidence: Each question is provided with at most 20 automatically retrieved evidences (see Section SECREF5 for details).

The reason is that we can enumerate all possible label sequences implicitly by dynamic programming for CRF during predicting but this is not possible for “Softmax( INLINEFORM4 -1)” , which indicates CRF is a better choice. “Noise” in Table TABREF24 means whether we inject noise data or not (Section SECREF34 ). As all evidences are positive under the annotated evidence setting, the ability for recognizing unreliable evidence will be useless. Therefore, the performance of our model with and without noise is comparable under the annotated evidence setting. However, the ability is important to improve the performance under the retrieved evidence setting because a large amount of the retrieved evidences are negative ones. As a result, we observe significant improvement by injecting noise data for this setting.

Word-based v.s. Character-based Input	Our model achieves fuzzy matching F1 scores of 69.78% and 70.97% on character-based input in annotated and retrieved evidence settings respectively (Table TABREF46 ), which are only 3.72 and 3.72 points lower than the corresponding scores on word-based input respectively. The performance is promising, demonstrating that our model is robust and effective.

Effect of Evidence LSTMs Structures	We investigate the effect of evidence LSTMs layer number, layer width and cross layer links in this section. The results are shown in Figure TABREF46 . For fair comparison, we do not use cross layer links in Figure TABREF46 (a) (dotted lines in Figure FIGREF4 ), and highlight the results with cross layer links (layer width 64) with circle and square for retrieved and annotated evidence settings respectively. We can conclude that: (1) generally the deeper and wider the model is, the better the performance is; (2) cross layer links are effective as they make the third evidence LSTM layer see information in both directions.

Conclusion and Future Work	In this work, we build a new human annotated real-world QA dataset WebQA for developing and evaluating QA system on real-world QA data. We also propose a new end-to-end recurrent sequence labeling model for QA. Experimental results show that our model outperforms baselines significantly. There are several future directions we plan to pursue. First, multi-entity factoid and non-factoid QA are also interesting topics. Second, we plan to extend our model to multi-evidence cases. Finally, inspired by Residual Network  , we will investigate deeper and wider models in the future.

Model Settings	If not specified, the following hyper-parameters will be used in the reset of this section: LSTM layer width INLINEFORM0 (Section SECREF7 ), word embedding dimension INLINEFORM1 (Section SECREF9 ), feature embedding dimension INLINEFORM2 (Section SECREF9 ). The word embeddings are initialized with pre-trained embeddings using a 5-gram neural language model  and is fixed during training. We will show that injecting noise data is important for improving performance on retrieved evidence setting in Section SECREF37 . In the following experiments, 20% of the training evidences will be negative ones randomly selected on the fly, of which 25% are annotated negative evidences and 75% are retrieved trivial negative evidences (Section SECREF5 ). The percentages are determined empirically. Intuitively, we provide the noise data to teach the model learning to recognize unreliable evidence. For each evidence, we will randomly sample another evidence from the rest evidences of the question and compare them to compute the e-e.comm feature (Section SECREF14 ). We will develop more powerful models to process multiple evidences in a more principle way in the future.

Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering	While question answering (QA) with neural network, i.e. neural QA, has achieved promising results in recent years, lacking of large scale real-word QA dataset is still a challenge for developing and evaluating neural QA system. To alleviate this problem, we propose a large scale human annotated real-world QA dataset WebQA with more than 42k questions and 556k evidences. As existing neural QA methods resolve QA either as sequence generation or classification/ranking problem, they face challenges of expensive softmax computation, unseen answers handling or separate candidate answer generation component. In this work, we cast neural QA as a sequence labeling problem and propose an end-to-end sequence labeling model, which overcomes all the above challenges. Experimental results on WebQA show that our model outperforms the baselines significantly with an F1 score of 74.69% with word-based input, and the performance drops only 3.72 F1 points with more challenging character-based input.

WebQA Dataset	In order to train and evaluate open-domain factoid QA system for real-world questions, we build a new Chinese QA dataset named as WebQA. The dataset consists of tuples of (question, evidences, answer), which is similar to example in Figure FIGREF3 . All the questions, evidences and answers are collected from web. Table TABREF20 shows some statistics of the dataset. The questions and answers are mainly collected from a large community QA website Baidu Zhidao and a small portion are from hand collected web documents. Therefore, all these questions are indeed asked by real-world users in daily life instead of under controlled conditions. All the questions are of single-entity factoid type, which means (1) each question is a factoid question and (2) its answer involves only one entity (but may have multiple words). The question in Figure FIGREF3 is a positive example, while the question “Who are the children of Albert Enistein?” is a counter example because the answer involves three persons. The type and correctness of all the question answer pairs are verified by at least two annotators. All the evidences are retrieved from Internet by using a search engine with questions as queries.

Effect of Word Embedding	As stated in Section SECREF34 , the word embedding INLINEFORM0 is initialized with LM embedding and kept fixed in training. We evaluate different initialization and optimization methods in this section. The evaluation results are shown in Table TABREF40 . The second row shows the results when the embedding is optimized jointly during training. The performance drops significantly. Detailed analysis reveals that the trainable embedding enlarge trainable parameter number and the model gets over fitting easily. The model acts like a context independent entity tagger to some extend, which is not desired. For example, the model will try to find any location name in the evidence when the word “在哪 (where)” occurs in the question. In contrary, pre-trained fixed embedding forces the model to pay more attention to the latent syntactic regularities. And it also carries basic priors such as “梨 (pear)” is fruit and “李世石 (Lee Sedol)” is a person, thus the model will generalize better to test data with fixed embedding. The third row shows the result when the embedding is randomly initialized and jointly optimized. The performance drops significantly further, suggesting that pre-trained embedding indeed carries meaningful priors.

Evidence LSTMs	The three-layer evidence LSTMs processes evidence INLINEFORM0 INLINEFORM1 to produce “features” for the CRF layer. The first LSTM layer takes evidence INLINEFORM0 , question representation INLINEFORM1 and optional features as input. We find the following two simple common word indicator features are effective: Question-Evidence common word feature (q-e.comm): for each word in the evidence, the feature has value 1 when the word also occurs in the question, otherwise 0. The intuition is that words occurring in questions tend not to be part of the answers for factoid questions.  Evidence-Evidence common word feature (e-e.comm): for each word in the evidence, the feature has value 1 when the word occurs in another evidence, otherwise 0. The intuition is that words shared by two or more evidences are more likely to be part of the answers. Although counterintuitive, we found non-binary e-e.comm feature values does not work well. Because the more evidences we considered, the more words tend to get non-zero feature values, and the less discriminative the feature is.

For example, bAbI  and the 30M Factoid Question-Answer Corpus  are artificially synthesized; the TREC datasets  , Free917  and WebQuestions  are human generated but only have few thousands of questions; SimpleQuestions  and the CNN and Daily Mail news datasets  are large but generated under controlled conditions. Thus, a new large-scale real-world QA dataset is needed. A new design choice for answer production besides sequence generation and classification/ranking: Without loss of generality, the methods used for producing answers in existing neural QA works can be roughly categorized into the sequence generation type and the classification/ranking type. The former generates answers word by word, e.g.  ,  ,  . As it generally involves INLINEFORM0 computation over a large vocabulary, the computational cost is remarkably high and it is hard to produce answers with out-of-vocabulary word. The latter produces answers by classification over a predefined set of answers, e.g.  , or ranking given candidates by model score, e.g.  . Although it generally has lower computational cost than the former, it either also has difficulties in handling unseen answers or requires an extra candidate generating component which is hard for end-to-end training.

Above all, we need a new design choice for answer production that is both computationally effective and capable of handling unseen words/answers. In this work, we address the above two challenges by a new dataset and a new neural QA model. Our contributions are two-fold: Experimental results show that our model outperforms baselines with a large margin on the WebQA dataset, indicating that it is effective. Furthermore, our model even achieves an F1 score of 70.97% on character-based input, which is comparable with the 74.69% F1 score on word-based input, demonstrating that our model is robust.

Factoid QA as Sequence Labeling	In this work, we focus on open-domain factoid QA. Taking Figure FIGREF3 as an example, we formalize the problem as follows: given each question Q, we have one or more evidences E, and the task is to produce the answer A, where an evidence is a piece of text of any length that contains relevant information to answer the question. The advantage of this formalization is that evidences can be retrieved from web or unstructured knowledge base, which can improve system coverage significantly. Inspired by  , we introduce end-to-end sequence labeling as a new design choice for answer production in neural QA. Given a question and an evidence, we use CRF  to assign a label to each word in the evidence to indicate whether the word is at the beginning (B), inside (I) or outside (O) of the answer (see Figure FIGREF3 for example). The key difference between our work and  is that  needs a lot work on feature engineering which further relies on POS/NER tagging, dependency parsing, question type analysis, etc. While we avoid feature engineering, and only use one single model to solve the problem.

Furthermore, compared with sequence generation and classification/ranking methods for answer production, our method avoids expensive INLINEFORM0 computation and can handle unseen answers/words naturally in a principled way. Formally, we formalize QA as a sequence labeling problem as follows: suppose we have a vocabulary INLINEFORM0 of size INLINEFORM1 , given question INLINEFORM2 and evidence INLINEFORM3 , where INLINEFORM4 and INLINEFORM5 are one-hot vectors of dimension INLINEFORM6 , and INLINEFORM7 and INLINEFORM8 are the number of words in the question and evidence respectively. The problem is to find the label sequence INLINEFORM9 which maximizes the conditional probability under parameter INLINEFORM10 DISPLAYFORM0  In this work, we model INLINEFORM0 by a neural network composed of LSTMs and CRF.

Overview	Figure FIGREF4 shows the structure of our model. The model consists of three components: (1) question LSTM for computing question representation; (2) evidence LSTMs for evidence analysis; and (3) a CRF layer for sequence labeling. The question LSTM in a form of a single layer LSTM equipped with a single time attention takes the question as input and generates the question representation INLINEFORM0 . The three-layer evidence LSTMs takes the evidence, question representation INLINEFORM1 and optional features as input and produces “features” for the CRF layer. The CRF layer takes the “features” as input and produces the label sequence. The details will be given in the following sections.

Long Short-Term Memory (LSTM)	Following  , we define INLINEFORM0 as a function mapping its input INLINEFORM1 , previous state INLINEFORM2 and output INLINEFORM3 to current state INLINEFORM4 and output INLINEFORM5 : DISPLAYFORM0  where INLINEFORM0 are parameter matrices, INLINEFORM1 are biases, INLINEFORM2 is LSTM layer width, INLINEFORM3 is the INLINEFORM4 function, INLINEFORM5 , INLINEFORM6 and INLINEFORM7 are the input gate, forget gate and output gate respectively.

Question LSTM	The question LSTM consists of a single-layer LSTM and a single-time attention model. The question INLINEFORM0 is fed into the LSTM to produce a sequence of vector representations INLINEFORM1 DISPLAYFORM0  where INLINEFORM0 is the embedding matrix and INLINEFORM1 is word embedding dimension. Then a weight INLINEFORM2 is computed by the single-time attention model for each INLINEFORM3 DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 . And finally the weighted average INLINEFORM2 of INLINEFORM3 is used as the representation of the question DISPLAYFORM0

Sequence Labeling	Following  ,  , we use CRF on top of evidence LSTMs for sequence labeling. The probability of a label sequence INLINEFORM0 given question INLINEFORM1 and evidence INLINEFORM2 is computed as DISPLAYFORM0  where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 is the number of label types, INLINEFORM3 is the transition weight from label INLINEFORM4 to INLINEFORM5 , and INLINEFORM6 is the INLINEFORM7 -th value of vector INLINEFORM8 .

The second LSTM layer stacks on top of the first LSTM layer, but processes its output in a reverse order. The third LSTM layer stacks upon the first and second LSTM layers with cross layer links, and its output serves as features for CRF layer. Formally, the computations are defined as follows DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are one-hot feature vectors, INLINEFORM2 and INLINEFORM3 are embeddings for the features, and INLINEFORM4 and INLINEFORM5 are the feature embedding dimensions. Note that we use the same word embedding matrix INLINEFORM6 as in question LSTM.

Evaluation on the Entire WebQA Dataset	In this section, we evaluate our model on the entire WebQA dataset. The evaluation results are shown in Table TABREF24 . Although producing multi-word answers is harder, our model achieves comparable results with the one-word answer subset (Table TABREF23 ), demonstrating that our model is effective for both single-word and multi-word word settings. “Softmax” in Table TABREF24 means we replace CRF with INLINEFORM0 , i.e. replace Eq. ( EQREF19 ) with DISPLAYFORM0  CRF outperforms INLINEFORM0 significantly in all cases. The reason is that INLINEFORM1 predicts each label independently, suggesting that modeling label transition explicitly is essential for improving performance. A natural choice for modeling label transition in INLINEFORM2 is to take the last prediction into account as in  . The result is shown in Table TABREF24 as “Softmax( INLINEFORM3 -1)”. However, its performance is only comparable with “Softmax” and significantly lower than CRF.

Training	The objective function of our model is INLINEFORM0  where INLINEFORM0 is the golden label sequence, and INLINEFORM1 is training set. We use a minibatch stochastic gradient descent (SGD)  algorithm with rmsprop  to minimize the objective function. The initial learning rate is 0.001, batch size is 120, and INLINEFORM0 . We also apply dropout  to the output of all the LSTM layers. The dropout rate is 0.05. All these hyper-parameters are determined empirically via grid search on validation set.

Effect of Question Representations	In this section, we compare the single-time attention method for computing INLINEFORM0 ( INLINEFORM1 , Eq. ( EQREF12 , EQREF13 )) with two widely used options: element-wise max operation INLINEFORM2 : INLINEFORM3 and element-wise average operation INLINEFORM4 : INLINEFORM5 . Intuitively, INLINEFORM6 can distill information in a more flexible way from { INLINEFORM7 }, while INLINEFORM8 tends to hide the differences between them, and INLINEFORM9 lies between INLINEFORM10 and INLINEFORM11 . The results in Table TABREF41 suggest that the more flexible and selective the operation is, the better the performance is.

Baselines	We compare our model with two sets of baselines: MemN2N  is an end-to-end trainable version of memory networks  . It encodes question and evidence with a bag-of-word method and stores the representations of evidences in an external memory. A recurrent attention model is used to retrieve relevant information from the memory to answer the question. Attentive and Impatient Readers  use bidirectional LSTMs to encode question and evidence, and do classification over a large vocabulary based on these two encodings. The simpler Attentive Reader uses a similar way as our work to compute attention for the evidence. And the more complex Impatient Reader computes attention after processing each question word. The key difference between our model and the two readers is that they produce answer by doing classification over a large vocabulary, which is computationally expensive and has difficulties in handling unseen words. However, as our model uses an end-to-end trainable sequence labeling technique, it avoids both of the two problems by its nature.

Effect of q-e.comm and e-e.comm Features	As shown in Table TABREF41 , both the q-e.comm and e-e.comm features are effective, and the q-e.comm feature contributes more to the overall performance. The reason is that the interaction between question and evidence is limited and q-e.comm feature with value 1, i.e. the corresponding word also occurs in the question, is a strong indication that the word may not be part of the answer.

All the evidences will be processed by our model independently and answers are voted by frequency to decide the final result. Note that a large amount of the evidences are negative and our model should not produce any answer for them.

As the answer for each question in our WebQA dataset only involves one entity (Section SECREF5 ), we distinguish label Os before and after the first B in the label sequence explicitly to discourage our model to produce multiple answers for a question. For example, the golden labels for the example evidence in Figure FIGREF3 will became “Einstein/O1 married/O1 his/O1 first/O1 wife/O1 Mileva/B Marić/I in/O2 1903/O2”, where we use “O1” and “O2” to denote label Os before and after the first B . “Fuzzy matching” is also used for computing golden standard labels for training set. For each setting, we will run three trials with different random seeds and report the average performance in the following sections.

Comparison with Baselines	As the baselines can only predict one-word answers, we only do experiments on the one-word answer subset of WebQA, i.e. only questions with one-word answers are retained for training, validation and test. As shown in Table TABREF23 , our model achieves significant higher F1 scores than all the baselines. The main reason for the relative low performance of MemN2N is that it uses a bag-of-word method to encode question and evidence such that higher order information like word order is absent to the model. We think its performance can be improved by designing more complex encoding methods  and leave it as a future work. The Attentive and Impatient Readers only have access to the fixed length representations when doing classification. However, our model has access to the outputs of all the time steps of the evidence LSTMs, and scores the label sequence as a whole. Therefore, our model achieves better performance.

Introduction	Question answering (QA) with neural network, i.e. neural QA, is an active research direction along the road towards the long-term AI goal of building general dialogue agents  . Unlike conventional methods, neural QA does not rely on feature engineering and is (at least nearly) end-to-end trainable. It reduces the requirement for domain specific knowledge significantly and makes domain adaption easier. Therefore, it has attracted intensive attention in recent years. Resolving QA problem requires several fundamental abilities including reasoning, memorization, etc. Various neural methods have been proposed to improve such abilities, including neural tensor networks  , recursive networks  , convolution neural networks  ,  ,  , attention models  ,  ,  , and memories  ,  ,  ,  ,  , etc. These methods achieve promising results on various datasets, which demonstrates the high potential of neural QA. However, we believe there are still two major challenges for neural QA: System development and/or evaluation on real-world data: Although several high quality and well-designed QA datasets have been proposed in recent years, there are still problems about using them to develop and/or evaluate QA system under real-world settings due to data size and the way they are created.

Question: What was the inter-annotator agreement?
Answer: correctness of all the question answer pairs are verified by at least two annotators