Introduction
Recent work on sequence-to-sequence neural networks have shown remarkable progress in many domains such as speech recognition, computer vision and language processing. These neural models can be used to map a sequence to another sequence, which has direct applications in natural language understanding BIBREF0 . One of the major advantages of this framework is that it requires a limited amount of feature engineering while still improving state-of-the-art results. This allows researchers to work on tasks for which domain knowledge may not be available. Question answering (QA) systems can directly benefit from this novel approaches, because it only requires mapping between questions and their answers. Due to the complexity of this mapping, most of the previous research in this domain used a complex pipeline of conventional linguistically-based NLP techniques, such as parsing, part-of-speech tagging and co-reference resolution BIBREF1 .
In this work, we build an open-domain QA system with sequence-to-sequence neural network models. The system was trained with question-answer pairs on the world knowledge represented in the DBpedia repository. Although building an open-domain system allows us to use an extensive amount of data stored in DBpedia, one of the challenges is to store this large amount of knowledge in the neural network architecture. Although these systems generally involve a smaller learning pipeline, they require a significant amount of training data.
Figure FIGREF1 illustrates how a sequence-to-sequence neural network can be trained on question-answering pairs. First, an sequence-to-sequence framework reads the source sentence, i.e. question, using an encoder to build a dense vector, a sequence of non-zero values that represents the meaning of a question. A decoder, processes this vector to predict an answer. In this manner, these encoder-decoder models can capture long-range dependencies in languages, e.g., gender agreements or syntax structures.
Related Work
Recent approaches on building QA systems are dominated by the usage of neural networks. BIBREF2 present an approach for conversational modelling, which uses a sequence-to-sequence neural model. Their model predicts the next sentence given the previous sentences for an IT helpdesk domain, as well as for an open-domain trained on a subtitles dataset. For an open-domain dialogue generation, BIBREF3 propose using adversarial training. Therefore, reinforcement learning is used to train the system that produces sequences that are indistinguishable from human-generated dialogue utterances. For this, they jointly train two systems, a generative model, which produces response sequences and a discriminator to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, guiding the system to generate dialogues that mostly resemble human dialogues. BIBREF4 demonstrate an end-to-end neural network model for generative QA. Their model is built on the encoder-decoder framework for sequence-to-sequence learning, while equipped with the ability to query a knowledge-base, which they demonstrate on the Chinese encyclopedia web site. The authors show that the proposed model is capable of generating natural and right answers by referring to the facts in the knowledge base. Additionally to the previous approaches, BIBREF5 extend the hierarchical recurrent encoder-decoder neural network to the open domain dialogue system and demonstrate that this model is competitive with state-of-the-art neural language models and back-off n-gram models. They illustrate limitations of similar approaches and show how the performance can be improved by bootstrapping the learning from a larger question-answer pair corpus and from pre-trained word embeddings. A heuristic that guides the development of neural baseline systems for the extractive QA task is described in BIBREF6 , which serves as guideline for the development of two neural baseline systems. Their RNN-based system, called FastQA, demonstrates good performance for extractive question answering due to the awareness of question words while processing the context. Additionally they introduce a composition function that goes beyond simple bag-of-words modelling. BIBREF7 demonstrate an approach to non-factoid answer generation with a separate component, which bases on BiLSTM to determine the importance of segments in the input. In contrast to other attention-based models, they determine the importance while assuming the independence of questions and candidate answers. BIBREF8 present their Gated-Attention reader for answering cloze-style questions over documents. The reader features a novel multiplicative gating mechanism in combination with a multi-hop architecture, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection.
The work mentioned above employ neural networks, but incorporate only a small amount of question-answering pairs to build their QA systems. Work on QA, which uses the DBpedia knowledge on the other hand, use much more complex linguistically-based NLP techniques to generate an answer for a given question. BIBREF9 propose a general purpose a QA system using Wikipedia data as its knowledge source, which can answer wh-interrogated questions. Their QA system includes tasks of named entity tagging, question classification, information retrieval and answer extraction. BIBREF10 present a QA approach on the DBpedia dataset. With predefined templates and dependency parsing on questions they obtain the entity and property mention from the question. They split the QA process into two steps; first they build an entity-centric index to search the target entity, whereby in the second step they expand the property mention with WordNet and ConceptNet to match the target entity with the searched entity property. Similarly, BIBREF11 present a QA system over Linked Data (DBpedia), which focuses on construct a bridge between the users and the Linked Data. Based on the consisting of subject-property-object (SPO) triples, each natural language question firstly is transformed into a triple-based representation (query triple). Then, the corresponding resources in DBpedia, including class, entity, property, are mapped for the phrases in the query triples. Finally, the optimal SPARQL query is generated as the output result. With this approach, their system can not only deal with the single-relation questions but also complex questions containing multi-relations. Differentially to the approaches mentioned above, we combine the neural network approaches with the large number of facts (> 20M n-triples) extracted from the DBpedia repository.
Experimental Setting
In this section, we give an overview of the sequence-to-sequence framework used in this experiment as well as information on the datasets used in our experiment. Furthermore, we give insights into the techniques to evaluate the correctness of the provided answers.
Sequence-to-sequence Neural Network Toolkit
OpenNMT BIBREF12 is a generic deep learning framework mainly specialised in sequence-to-sequence (seq2seq) models covering a variety of tasks such as machine translation, summarisation, speech processing and question answering. We used the default neural network training parameters, i.e. 2 hidden layers, 500 hidden LSTM units, input feeding enabled, batch size of 64, 0.3 dropout probability and a dynamic learning rate decay. We train the network for 13 epochs and report the results in Section SECREF5 .
A common problem in training a neural network is the computational complexity, which cause that the vocabulary has to be limited to a specific threshold. Because of this the neural network can not learn expressions of rare and unknown words, e.g. domain-specific expressions. Therefore, if the training method does not see a specific word or phrase multiple times during training, it will not learn the interpretation of the word. This challenge is even more evident in sequence-to-sequence models used for summarisation, question answering or machine translation. Therefore the vocabulary is often limited only to 50,000 or 100,000 words (in comparison to 300,000 or more unique words in our training data sets, see Table TABREF5 ). To overcome this limitation, different methods were suggested, i.e. character based neural model BIBREF13 , BIBREF14 or using subword units, e.g. Byte Pair Encoding (BPE). The latter one was successfully adapted for word segmentation specifically for the NMT scenario BIBREF15 . BPE BIBREF16 is a form of data compression that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. Instead of merging frequent pairs of bytes as shown in the original algorithm, characters or character sequences are merged for the purposes of natural language generation. To achieve this, the symbol vocabulary is initialised with the character vocabulary, and each word is represented as a sequence of characters—plus a special end-of-word symbol, which allows to restore the original tokenisation after the generating the answer based on the given question. This process is repeated as many times as new symbols are created.
Training Datasets for the QA system
For our QA system we used the DBpedia repository and the Subtitles corpus. Due to the nature of training a sequence-to-sequence neural model, questions and answers need to be aligned. The statistics on the used data are shown in Table TABREF5 .
In our work, we use the DBpedia BIBREF17 repository (version 2016-04). The DBpedia project aims to extract structured knowledge from the knowledge added to the Wikipedia repository. DBpedia allows users to semantically query relationships and properties of Wikipedia resources, including links to other related datasets.
For our experiment on training a QA system, we extracted (see Section SECREF4 ) the knowledge stored in Dbpedia Infobox Properties, which with its n-triple structure represents answers to particular questions. From this dataset we extracted more than 21M question-answer entries.
Since the extracted information form the DBpedia n-triples does not represent a natural language, we added to the extracted n-triples answers with dialogues from the OpenSubtitles dataset BIBREF18 . Since this dataset is stored in an XML structure with time codes, only sentences were extracted, where the first sentence ends with a question mark and the second sentence does not end with a question mark. Additionally, to ensure a better consistency between an question and the answer, the second sentence has to follow the first sentence by less than 20 seconds. From the 14M sentence corpus of question-answer pairs provided by the OpenNMT project, we used 5M dialogue entries to modify the language generation part. Table TABREF10 shows some examples from the OpenSubtitles dataset.
Evaluation Dataset
For the automatic evaluation of the QA system, we used 10,000 entries extracted from the DBpedia Infobox Property repository (Table TABREF5 ).
Evaluation Metrics
The automatic evaluation of the proposed QA system is based on the correspondence between the generated answer and gold standard. For the automatic evaluation we used the BLEU BIBREF19 , METEOR BIBREF20 and chrF BIBREF21 metrics. BLEU (Bilingual Evaluation Understudy) is calculated for individual generated segments (n-grams) by comparing them with a dataset of of the gold standard. Considering the shortness of the questions, we report besides the four-gram overlap (BLEU) also scores based on the unigram overlap (BLEU-1). Those scores, between 0 and 100 (perfect overlap), are then averaged over the whole evaluation dataset to reach an estimate of the overall quality of the generated answers.
INLINEFORM0
METEOR (Metric for Evaluation of Translation with Explicit ORdering) is based on the harmonic mean of precision and recall, whereby recall is weighted higher than precision. Along with exact word (or phrase) matching it has additional features, i.e. stemming, paraphrasing and synonymy matching. In contrast to BLEU, the metric produces good correlation with human judgement at the sentence or segment level. Differently to the BLEU metric, which penalises the hypothesis, if words were generated with a different inflection, chrF3 is a character n-gram metric, which showed very good correlations with human judgements for morphologically rich languages.
INLINEFORM0
chrP represents the percentage of n-grams in the generated answer which has a counterpart in the gold standard answer, whereby chrR represents the percentage of character n-grams in the gold standard, which are also present in the predicted answer. INLINEFORM0 is a parameter which assigns INLINEFORM1 times more importance to recall than to precision.
Methodology
The training approach to the sequence-to-sequence neural model requires an aligned dataset of questions and answers, which are aligned on a sentence level. To train our QA system on generic knowledge, we extracted this required information from the DBpedia Infobox Property repository.
Evaluation
In this section we report the results of the automatic and a manual evaluation of the answers generated by the neural models.
Automatic Evaluation
The results of the automatic evaluation using the BLEU, METEOR and chrF metric are shown in Table TABREF17 . We observer BLEU and Meteor scores, which evaluate the generated answers on based on a n-gram overlap, of 16.48 and 13.97, respectively. BLEU-1 and the chrF score, which perform on a single word or character level, are slightly higher, i.e. 56.0 and 26.84.
Manual Evaluation
Besides the automatic evaluation of answer generation over the DBpedia Infobox properties, we performed a detailed manual evaluation of the answers produced by the neural networks.
We first evaluated, which answers, based on the DBpedia properties (of the property-subject pair) were correctly generated based on the METEOR score. Table TABREF18 illustrates that answers, based on the property next single of or basin countries of were mostly generated correctly. On the other hand, answers, based on question containing the properties operating system of or intercommunality of were mostly generated wrongly.
Finally, Table TABREF19 presents different examples of answers, based on the Dbpedia Infobox Properties. Although the properties domain of or capital of appear only once in the evaluation set, they were answered correctly. Furthermore, the neural networks often provide answers, which are synonyms or closely related words to the gold standard answers. As an example, the generated answer lawyer is an synonym word of attorney, and usa can also be referred to as united states only. Similarly, a closely related word to french is france, although this cannot be counted as a correctly generated answer to the question nationality of jean michel diot. At last, the proposed approach of using neural networks also demonstrated various wrongly generated answered. As an example, clark duke is not an football player and edith roger was born in vestby, norway and not in vienna , austria as answered by the proposed neural network approach.
Conclusions
We presented the work on a QA system trained with a sequence-to-sequence neural model with the DBpedia knowledge and movie dialogues. Although the automatic evaluation shows a low overlap of generated answers compared to the gold standard, a manual inspection of the showed promising outcomes from the experiment. Due to the nature of the training dataset, short answers are preferred, since they are more likely to have a lower log-likelihood score than the longer ones. Nevertheless, we observed several correct answers, which shows the availability of storing the entire DBpedia knowledge into neural networks. Our future work will focus on providing longer answers, as well as focusing on answering more complex questions.
Acknowledgement
This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 (Insight).

Question: Which movie subtitles dataset did they use?
Answer: the OpenSubtitles dataset BIBREF18

OpenSubtitles dataset BIBREF18