Automatic question answering
By: Yana Arbuzova
Tags: NLP, Information Retrieval, Machine Learning, Neural Networks, Deep Learning

Querying Information from structured and unstructured data has become very important. There is a lot of textual data, FAQ, newspapers, articles, documentation, user cases, customer service requests, etc. Quite hard to keep in mind all that information. Who won the last football Euro Cup? What is Bitcoin? When is the birthday of a famous singer? To find some information, we need to search a document, spend a couple of minutes reading before you find the answer. Nazar Grycshuk, Data Scientist at Sigma Software and the author of this article, believes that automatic QnA system could save the most valuable resource in our lives — time.

First, let’s discuss what types of question are there, and what answers to such questions users can expect.

All questions can be divided into several types:

General questions, with Yes/No answers

Wh-Questions, start with: who, what, where, when, why, how, how many

Choice Questions, where you have some options inside the question

Factoid questions, where the complete answer can be found inside a text. The answer to such questions consist of one or several words that go one after another

In this article, we will focus on the factoid question answering approach that can process a wide range of questions and covers one of the most difficult type of them.

The approach that will be discussed in this article is based on an algorithm designed specifically for factoid question answering. As to other question types (general, Wh-questions, choice questions), the algorithm could be applied after some adjustments. Also, some ideas could be reused to create your own approach.

Main Factoid Question Answering Approach

Typically, a question answering system consists of several logical blocks:

Data source,

Information retrieval (IR) system ,

, Machine reading comprehension model (MR model),

One or more additional blocks, such as modules for text pre-processing, answers post processing, checking, and stabilization.

Let’s see how these blocks interact with each other:

As you can see in the picture, when the question is transferred to the system, the first step to be done is the search of the texts that can contain the answer to the question. The sources of such texts can include various web pages, documents stored on a local disc, Wikipedia, etc.

The next step is to select the most relevant texts from all the information that was found. For this purpose, the following approaches could be used: TF-IDF similarity, Jaccard index, Word embeddings, Deep Learning text similarity algorithms. All of these approaches have pros and cons, different time/memory needs for pre-processing and getting similarity index. There is no silver bullet or best approach for all cases. Which approach is the best for a concrete task depends on several factors like domain area, dataset, amount of data. The problem is complex and has many facets, every month a couple of articles about IR/Text similarity task are published.

Information Retrieval System

According to Wikipedia, “an open domain question answering system aims at returning an answer in response to the user’s question. The returned answer is in the form of short texts rather than a list of relevant documents.”

If you choose to answer open-domain questions, the source of information can be: your own documents, Common Crawl, Wikipedia, some ontologies like DBpedia or Google knowledge graph, also you can try Google custom search engine.

If you decide to use Wikipedia, there are 2 approaches. The first one is searching in downloaded Wikipedia pages. The second one is extracting entities (e.g. person’s name, location, dates) from a question and then extracting all wiki pages with those entities. However, there is a possibility that a Wiki page with the entity does not cover your question, and you will not find the answer in the text.

In our project Google custom search engine was chosen, because it returns the most relevant information not only from Wiki, but also from other indexed pages. Search could be adjusted by adding special parameters, location, topics, website types, etc. Also, Google custom search engine returns a list of URLs with a short summary, that could be used as an answer passage for the next steps.

Here is an example of an automatically found answer to the same question by using different approaches.

At the previous step, we end up with a list of relevant documents. The next step is to split documents into passages and find the most relevant ones.

The following algorithms were used:

Ranking of passages with tf-idf/Jaccard/cosines distance based on word embeddings

Deep Learning (DL) approach like Bi-LSTM model with avg pooling and attention (https://arxiv.org/pdf/1511.04108.pdf)

Depending on data, 0.63–0.85 F1 score could be archived. As a result of this step, we will get a list of passages sorted by a similarity index. Let’s take 5–10 most relevant to the next step.

Machine Reading

The last step is the most interesting one. We have a question and a passage. The idea is to extract the answer from the passage. There is a bunch of approaches that can be applied here. For example, you can build an abstract representation for the question and passage, get a question type (so you will know the answer type), and get an answer. But in practice, this approach is quite complicated even for close domain question with restricted questions types and entities.

Most of the modern approaches are based on Deep Learning. To use these approaches, we need to have datasets. There is a list of datasets, such as SQuAD, CNN\Dayli Mail, MSMarco, TriviaQA, WikiReading, to train neural network (NN).

We’ve trained our model on SQuAD dataset v1 and v2. V1 contains 100,000+ question-answer pairs with content from 500+ documents related to different topics. In v2, there were added 50K questions without answers. The models trained and tested on this dataset can be compared via a leaderboard. The structure of the dataset looks as follows:

There is another type of datasets, where the answer to the question is not in the context. An example of such dataset type is MS Marco. There is 1M+ Bing user query, 160К+ Answers.

Here is a list of good NN architectures for Question Answering that can be helpful in creating a question answering system.

AoA https://arxiv.org/abs/1607.04423 r-NET https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf ReasoNet https://arxiv.org/abs/1609.05284 FusionNet https://arxiv.org/abs/1711.07341 HyperQA https://github.com/tjurczyk/HyperQA/blob/master/LICENSE BiDAF https://allenai.github.io/bi-att-flow/ RDF https://www.semanticscholar.org/paper/Natural-language-question-answering-over-RDF%3A-a-Zou-Huang/cb4d277a51da6894fe5143013978567ef5f805c8 Reinforcement Mnemonic Reading https://arxiv.org/pdf/1705.02798.pdf

For our task, R-Net model was used. The choice was made in favor of this model because of its good score on state-of-the-art leader boards and a well-described architecture. The R-Net model is based on Bidirectional Recurrent Network where question and passage are processed separately. After that, a gated attention-based recurrent network matches a question and a passage to get a question-aware representation for the passage. Then, the self-matched attention aggregates evidences for the whole passage. The output layer predicts the start and end positions of the answer inside the passage.

Image from www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf.

Answers to Random Questions

Our Test on Factoid Question Dataset: 62%(EM) Exact Match

Summary:

We end up with quite good machine reading model in combination with Wiki-based and Google custom search Information retrieval system. The system quality can be improved by combining different information sources, post processing with combining and scoring results, train new architectures and build ensemble model, increase dataset for training MR model with some specific cases.

This approach can be applied for real business cases as a part of Question answering/user support system. If you are building a question-answering system and use NLP engine, like Rasa NLU, Dialogflow, Luis, this NLP engine can answer predefined questions. However, if there is no predefined intent, you can call this automatic QnA system to search in documents and return the answer. Continuous learning can be organized by getting feedback from users. For now, the quality of this model can’t outperform people, but it is really close to it (70–80%) depending on a dataset. Every month the quality of such models increases. It means that in the very near future such models may answer questions with the same accuracy as humans, but much faster.

I hope this article will help you in creating your question answering system without spending time on trying non-optimal approaches or tools.

Sigma Software AI expertise

Sigma Software gathered AI experts under its roof and supports them in investigating specifics of machine learning, computer vision, natural language processing, robotics, and other aspects of AI. The experience our experts get in such a way they apply in real-world projects and products ranging from chatbots to automated procurement planning solutions. Learn more about Sigma Software AI expertise.