Systematic review of question answering over knowledge bases

IET SoftwareVolume 16, Issue 1 p. 1-13 REVIEWOpen Access Systematic review of question answering over knowledge bases Arnaldo Pereira, Corresponding Author Arnaldo Pereira [email protected] orcid.org/0000-0003-3361-6269 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, Portugal Correspondence Arnaldo Pereira, Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, 3810-193 Aveiro, Portugal. Email: [email protected]Search for more papers by this authorAlina Trifan, Alina Trifan orcid.org/0000-0001-7613-1435 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, PortugalSearch for more papers by this authorRui Pedro Lopes, Rui Pedro Lopes orcid.org/0000-0002-9170-5078 Research Centre in Digitalization and Intelligent Robotics (CeDRI), Polytechnic Institute of Bragança, Bragança, PortugalSearch for more papers by this authorJosé Luís Oliveira, José Luís Oliveira orcid.org/0000-0002-6672-6176 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, PortugalSearch for more papers by this author Arnaldo Pereira, Corresponding Author Arnaldo Pereira [email protected] orcid.org/0000-0003-3361-6269 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, Portugal Correspondence Arnaldo Pereira, Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, 3810-193 Aveiro, Portugal. Email: [email protected]Search for more papers by this authorAlina Trifan, Alina Trifan orcid.org/0000-0001-7613-1435 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, PortugalSearch for more papers by this authorRui Pedro Lopes, Rui Pedro Lopes orcid.org/0000-0002-9170-5078 Research Centre in Digitalization and Intelligent Robotics (CeDRI), Polytechnic Institute of Bragança, Bragança, PortugalSearch for more papers by this authorJosé Luís Oliveira, José Luís Oliveira orcid.org/0000-0002-6672-6176 Institute of Electronics and Informatics Engineering of Aveiro (IEETA), University of Aveiro, Aveiro, PortugalSearch for more papers by this author First published: 24 June 2021 https://doi.org/10.1049/sfw2.12028Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Over the years, a growing number of semantic data repositories have been made available on the web. However, this has created new challenges in exploiting these resources efficiently. Querying services require knowledge beyond the typical user's expertise, which is a critical issue in adopting semantic information solutions. Several proposals to overcome this difficulty have suggested using question answering (QA) systems to provide user-friendly interfaces and allow natural language use. Because question answering over knowledge bases (KBQAs) is a very active research topic, a comprehensive view of the field is essential. The purpose of this study was to conduct a systematic review of methods and systems for KBQAs to identify their main advantages and limitations. The inclusion criteria rationale was English full-text articles published since 2015 on methods and systems for KBQAs. Sixty-six articles were reviewed to describe their underlying reference architectures. 1 INTRODUCTION Question answering (QA) refers to systems that allow users to use natural language (NL) interfaces to ask questions and receive concise answers. The first solution was created in the 1960s to answer questions asked in English about baseball games from information saved in a list-structured database [1]. Some years later, with the emergence of the relational data model, considerable effort was put into developing natural language interfaces for databases (NLIDB). However, just five years after the creation of the World Wide Web, Androutsopoulos et al. reported the lack of interest in investigating NLIDB [2]. In those days, the focus was on information retrieval techniques to create web search engines using the keyword-based search paradigm. Meanwhile, QA over text advanced [3], and the Semantic Web (SW) vision formulated in 2001 by Berners-Lee et al. [4] brought attention to semantic data. Search engines have come to offer direct answers to some user questions in recent years [5]. Instead of just presenting a list of links to documents where the answer is likely to be found, the idea is to satisfy the need for information without further searching and navigation. Questions whose answer is an entity are the ideal candidate for this type of approach, and using large semantic databases that capture general knowledge has become of great value. In this context, triples extraction to answer questions is priceless and continues to motivate research work. Since the formulation of SW principles and standards, semantic representations to capture knowledge in the life sciences have become common practice [6]. As a result, there was an explosion in the creation of ontologies and Resource Description Framework (RDF) databases made available online. The usual access to this data has been through visual navigation interfaces and commonly through SPARQL Protocol and RDF Query Language (SPARQL) endpoints, but these access methods have problems. The first approach is not rich enough to answer more complex questions, and the second is not suitable for users who have not mastered the use of formal querying languages. Therefore, this reality is pressing for new question answering over knowledge base (KBQA) solutions for biomedical data. A couple of examples from the life sciences illustrate the use of these systems. Asiaee et al. applied a KBQA solution to parasite immunology [7], and Hamon et al. created a querying platform for linked biomedical data [8]. Other KBQA systems retrieve information from open knowledge databases, such as DBpedia or Wikidata, or use proprietary enterprise knowledge graphs, such as Google Knowledge Graph or Bing Satori [9]. We have identified the publication of surveys and overviews exclusively or partially about KBQA since 2015. Höffner et al. [10] surveyed 72 papers published until the middle of 2015 and mapped against a set of challenges. By the depth of its analysis and selection of articles, this study is a reference work to understand the contributions that had emerged until that moment. Other contributions, such as those of Mishra and Jain [11], Dimitrakis et al. [12], Affolter et al. [13], and Ojokoh and Adebisi [14], are narrative reviews, meaning that they do not use systematic search methods. Unlike the latter studies, our review follows a systematic search strategy that provides a greater guarantee that essential contributions have not gone unnoticed. We also focus on the presentation of KBQA reference architectures, which none of the previous works addresses. Our attention to architectural decomposition allows us to highlight state-of-the-art practices for particular subtasks. The remainder of this paper comprises four sections. Section 2 overviews the current aspects of KBQA. In Section 3, we introduce the methods for retrieving and selecting the articles surveyed and present the quantitative results of that selection. In Section 4, we critically summarize our findings. We wind up the paper with the conclusions in Section 5. 2 BACKGROUND The purpose of QA systems is to allow users to create questions in NL without a formal query language. Considering the nature of the data sources, we can divide these solutions into three major groups: QA over unstructured data (e.g., text), QA over semi-structured data (e.g., tables), and QA over structured data (e.g. graph data sets). Hybrid systems operate in two or more kinds of data sources. Regarding the scope of data, on the one hand, we can consider domain-specific solutions when the data schema is narrowed down to a particular body of knowledge (e.g., biomedical data) that limits the question types that are accepted. On the other hand, open-domain systems consider data on generic subjects specified by general ontologies. RDF is the data model for the SW, established in a suite of normative specifications by the World Wide Web Consortium [15]. An RDF triple (or statement) has three components: the subject, which is an Internationalized Resource Identifier (IRI) or a blank node (bnode); the predicate, which is an IRI; and the object, which is an IRI, a literal, or a bnode [16]. A set of RDF triples is an RDF graph, and an RDF data set is a collection of RDF graphs. Several RDF serialization formats are available, such as Turtle, TriG, and JSON-LD (JavaScript Object Notation for Linked Data). We realize the real power of the triples when considering large data sets. An RDF store (or triplestore) is a proper database for the storage and retrieval of triples. For quick reference, we can list some well-known solutions: OpenLink Virtuoso (https://virtuoso.openlinksw.com/), Eclipse RDF4J (formerly Sesame) (http://rdf4j.org/), Apache Jena (https://jena.apache.org/), and GraphDB (http://graphdb.ontotext.com/). By default, we use SPARQL [17] for querying RDF graphs considering one of four query forms. The SELECT construction returns variables and their bindings directly, and the CONSTRUCT clause returns a single RDF graph. The ASK statement returns a boolean indicating whether a query pattern matches, and a DESCRIBE query returns an RDF graph describing the resources found [18]. Several benchmarks and evaluation campaigns have promoted the advancement of KBQA systems. The Question Answering on Linked Data (QALD) challenge launched in 2011 is the oldest running campaign, and its ninth edition provided a training data set with 408 questions in 11 different languages for the open-domain semantic QA over DBpedia task [19]. To shorten the size limitations of the QALD data set, the Large-Scale Complex Question Answering Dataset (LC-QuAD) provides 30,000 questions with corresponding SPARQL queries for DBpedia and Wikidata [20]. Free917 is another benchmark and consists of 917 utterances using Freebase with a meaning representation in a variant of lambda calculus [21]. To avoid using logical forms, Berant et al. [22] created the WebQuestions data set containing 5810 question-answer pairs, to which Yih et al. [23] added SPARQL queries to create WebQuestionsSP. Then, Bordes et al. [24] achieved, with SimpleQuestions, a significant scale-up of the numbers with 108,442 questions for possible rephrasing in the form (subject, relationship, ?). Finally, the BioASQ series of challenges has a task on domain-specific semantic QA on biomedical data to evaluate systems outputting relevant triples and text snippets [25]. To meet the challenges posed in implementing KBQA solutions, it is important to identify the most common architectures. From the analysis of the papers we selected for our work, we found that they are classified by four different architectures. Semantic parsing pipelines are solutions based on semantic parsing, which uses a pipe and filter style where data flows to generate a formal query from the original input in NL. It is the most straightforward architectural style of KBQA systems and relies on connecting components to form a pipeline, as shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint General architecture for semantic parsing pipelines. The direction of the arrows denotes the direction of the data flows The idea is to apply several data transformations from the question in NL to the logical form or formal query. To achieve that, we use natural language processing (NLP) techniques such as tokenization, named-entity recognition (NER), part-of-speech (POS) tagging, dependency parsing, and entity linking (EL). In addition, steps for query generation and answer generation are required. An alternative way of using semantic parsing is based on the observation that executing a formal query is equivalent to finding a subgraph, as depicted in Figure 2. FIGURE 2Open in figure viewerPowerPoint Subgraph matching approach Systems capable of answering complex questions (e.g., questions that cannot be reduced to a simple triple pattern) require a higher degree of sophistication than the systems presented so far. A template is a query skeleton with an arbitrary degree of complexity that fits the knowledge base (KB) to be questioned and has slots that must be filled with information from entities and relations. Naturally, the quality of the system depends on the effort put into creating the templates. In the early stages, the use of templates was mainly accomplished through manual annotations; in more recent times, automatic approaches have emerged. These systems rely on the manual or automatic creation of a template database assuming an architectural configuration such as that shown in Figure 3. FIGURE 3Open in figure viewerPowerPoint General architecture for template-based question answering over knowledge bases In the offline phase, it is necessary to create templates. This involves considering pairs of questions and answers used to obtain successively more abstract representations that are used to generate pairs of question-query templates after alignment. The online phase is straightforward: a question is matched with a template to produce a query template, the slots are filled with entities and relations, and the answer is provided by issuing the query candidate. End-to-end solutions perform sequence-to-sequence translation or apply methods to extract triples directly from the KB. The selection of the final answer is based on the representations of the questions in NL obtained by applying machine learning techniques, as can be seen in Figure 4. After extracting the candidate answers from the KB, they are evaluated against a predefined score using a specialized function. FIGURE 4Open in figure viewerPowerPoint General architecture for question answering over knowledge bases based on information extraction Höffner et al. [10] highlighted significant challenges faced by semantic QA. The lexical gap occurs when the surface forms used in a question are different from those used in the KB. The ambiguity stemming from the fact that the same word can represent various entities is also problematic. Another significant problem is finding answers to questions manoeuvring several units combined in complex queries requiring ordered, aggregated, or filtered outputs. Equally challenging is multilingualism, which concerns two distinct realities that may or may not co-occur. The first involves the problem of using the same interface to ask questions in several NLs, and the second has to do with the possibility of the KB data being multilingual. In addition, systems relying on languages other than English end up receiving far less attention from the scientific community, limiting the number of available solutions. For instance, very few developers have participated in challenges like QALD with multilingual systems. Some systems try to prevent difficulties by using controlled natural languages (CNLs), which are constructions that restrict in some way the lexicon, syntax, or semantics of the NL from which they start. This review will not focus on multilingualism or the use of CNL interfaces. 3 PAPER SELECTION Computer science and software engineering can benefit from using systematic literature reviews to synthesize the best evidence about the state of the art, especially for mature topics for which a large number of studies may not be appropriately acknowledged [26]. In this work, we followed a strict methodology. To enable repeatability, we have created a replication package that is available at https://osf.io/hxyvw. We have used PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [27] to report the protocol's execution and present the findings. Search methodology The following research questions guided our study: RQ1.Which methods of KBQA have been proposed? RQ2.What are the solved and unsolved challenges? Going through past surveys and overviews [10-14], we collected the keywords shown in Table 1 mapped against the Population, Intervention, Comparison, Outcomes (PICO) structure [28]. As pointed out previously, Höffner et al. [10] presented a systematic study of KBQA systems until mid-2015. We have built upon that. Our search query followed the template TABLE 1. Slot values for the population, intervention, comparison, and outomes template PICO Slot Values Population 'Knowledge Base*' OR 'Knowledge Graph*' OR 'Semantic Web' OR 'Linked Data*' OR 'RDF Data*' OR 'data web' Intervention Question-Answer* OR 'natural language que*' OR 'Natural Language Interface' Comparison SPARQL OR 'Query Graph*' Outcomes QALD* OR SimpleQuestions OR WebQuestions OR WebQSP OR LC-QuAD Abbreviations: SPARQL, SPARQL Protocol and RDF Query Language; QALD, Question Answering on Linked Data. ((Population OR Comparison) AND (Intervention OR Outcomes)) Then we used Scopus, Web of Science, IEEE Xplore, and the ACM Digital Library to find KBQA papers. We emphasize that our research questions and search string allow us to retrieve all papers reviewed by Höffner et al. [10], in addition to the most recent ones. Next, we considered the following inclusion criteria: I1.Studies on methods and systems for question answering over knowledge bases. I2.Papers focussing on a specific challenge of KBQA solutions (answering complex questions, module reusability etc.). For the exclusion criteria, we have E1.Books, surveys and overviews, tutorials, talks, panel sessions, conference reviews, editorials, abstracts, a summary of a workshop or challenge, dissertations, or grey literature. Not available in English. Being unable to retrieve the full text. E2.If faced with multiple papers by the same author about the same subject, we will keep only those needed to report the main contribution. The main threats to validity are losing relevant studies in the search step and risking the rejection of relevant studies in the review phase. The PICO strategy minimizes the first problem, ensuring high recall. In addition, we have queried four bibliographic databases. To mitigate the second problem, we allocated two people (the first two authors) to select and review papers. We have performed content analysis to collect information about the challenges addressed, proposed solutions, future work, and architectural styles. The fourth author participated as a referee when disagreements arose. Selection results Figure 5 shows the flow diagram of the paper selection procedure. After applying the inclusion and exclusion criteria, we selected the 66 papers listed in Table 2. Figure 6 presents the relations of the keywords of the papers selected for qualitative analysis. Naturally, the term 'question answering' appears in the most articles. It is interesting to note that the 'Semantic Web' entry is very prominent only until 2017. The most recent significant cluster is linked to the term 'knowledge graph', showing a greater interest in this type of structure, which is very suitable for applying deep learning techniques. FIGURE 5Open in figure viewerPowerPoint Study selection TABLE 2. Publications selected ID Paper name Architectural style Year 1 Answering questions with complex semantic constraints on open knowledge bases Sem. parsing pipe. 2015 2 Applying semantic parsing to question answering over linked data: Addressing the lexical gap [29] Sem. parsing pipe. 2015 3 HAWK—hybrid question answering using linked data Sem. parsing pipe. 2015 4 How to build templates for RDF question/answering - An uncertain graph similarity join approach [30] Template-based 2015 5 ISOFT at QALD-5: Hybrid question answering system over linked data and text data Sem. parsing pipe. 2015 6 More accurate question answering on Freebase Template-based 2015 7 QAnswer—Enhanced entity matching for question answering over linked data [31] Sem. parsing pipe. 2015 8 Question answering over Freebase with multi-column convolutional neural networks [32] Info. extraction 2015 9 Question answering via phrasal semantic parsing Sem. parsing pipe. 2015 10 Semantic parsing via staged query graph generation: Question answering with knowledge base [33] Subgraph matching 2015 11 [email protected]: LIMSI participation at [email protected] Subgraph matching 2015 12 SINA: Semantic interpretation of user queries for question answering on interlinked data Sem. parsing pipe. 2015 13 TR Discover: A natural language interface for querying and analysing interlinked datasets [34] Sem. parsing pipe. 2015 14 AskNow: A framework for natural language query formalization in SPARQL [35] Sem. parsing pipe. 2016 15 CFO: Conditional focussed neural question answering with large-scale knowledge bases Info. extraction 2016 16 Character-level question answering with attention Info. extraction 2016 17 Constraint-based question answering with knowledge graph Subgraph matching 2016 18 GRU-RNN based question answering over knowledge base Info. extraction 2016 19 Hybrid question answering over knowledge base and free text Sem. parsing pipe. 2016 20 Knowledge base question answering based on deep learning models Info. extraction 2016 21 Neural generative question answering Info. extraction 2016 22 Qanary—A methodology for vocabulary-driven open question answering systems [36] Sem. parsing pipe. 2016 23 QuerioDALI: Question answering over dynamic and linked knowledge graphs [37] Sem. parsing pipe. 2016 24 Question answering on Freebase via relation extraction and textual evidence [38] Info. extraction 2016 25 The value of semantic parse labelling for knowledge base question answering [23] Sem. parsing pipe. 2016 26 When a knowledge base is not enough: Question answering over knowledge bases with external text data [39] Info. extraction 2016 27 An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge [40] Info. extraction 2017 28 Automated template generation for question answering over knowledge graphs [41] Template-based 2017 29 End-to-end representation learning for question answering with weak supervision Subgraph matching 2017 30 Improved neural relation detection for knowledge base question answering [42] Info. extraction 2017 31 Introducing feedback in Qanary: How users can Interact with QA systems [43] Sem. parsing pipe. 2017 32 KBQA: Learning question answering over QA corpora and knowledge bases Template-based 2017 33 Matching natural language relations to knowledge graph properties for question answering Sem. parsing pipe. 2017 34 Natural language supported relation matching for question answering with knowledge graphs Info. extraction 2017 35 Neural network-based question answering over knowledge graphs on word and character levels [9] Info. extraction 2017 36 QAESTRO—semantic-based composition of question answering pipelines [44] Sem. parsing pipe. 2017 37 Querying biomedical linked data with natural language questions [8] Sem. parsing pipe. 2017 38 Question answering on knowledge bases and text using universal schema and memory networks Info. extraction 2017 39 Trill: A reusable front-end for QA systems [45] Sem. parsing pipe. 2017 40 An attention-based word-level Interaction model for knowledge base relation detection Info. extraction 2018 41 Answering natural language questions by subgraph matching over knowledge graphs [46] Subgraph matching 2018 42 Formal query generation for question answering over knowledge bases [47] Sem. parsing pipe. 2018 43 Frankenstein: A platform enabling reuse of question answering components [48] Sem. parsing pipe. 2018 44 Never-ending learning for open-domain question answering over knowledge bases [49] Template-based 2018 45 Novel knowledge-based system with relation detection and textual evidence for question answering research Sem. parsing pipe. 2018 46 Question answering over knowledge graphs: Question understanding via template decomposition Template-based 2018 47 Svega: Answering natural language questions over knowledge base with semantic matching Subgraph matching 2018 48 Why reinvent the wheel: Let's build question answering systems together [50] Sem. parsing pipe. 2018 49 Answer-enhanced path-aware relation detection over knowledge base Info. extraction 2019 50 Complex query augmentation for question answering over knowledge graphs [51] Sem. parsing pipe. 2019 51 ComQA: Question answering over knowledge base via semantic matching Subgraph matching 2019 52 Deep query ranking for question answering over knowledge bases Sem. Parsing pipe. 2019 53 Handling modifiers in question answering over knowledge graphs Sem. parsing pipe. 2019 54 Knowledge base question answering with a matching-aggregation model and question-specific contextual relations Info. extraction 2019 55 Knowledge base question answering with attentive pooling for question representation Info. extraction 2019 56 Learning to answer complex questions over knowledge bases with query composition Subgraph matching 2019 57 Learning to rank query graphs for complex question answering over knowledge graphs [52] Subgraph matching 2019 58 Message passing for complex question answering over knowledge graphs Subgraph matching 2019 59 Pretrained transformers for simple question answering over knowledge graphs [53] Info. extraction 2019 60 A BERT-based approach with relation-aware attention for knowledge base question answering [54] Info. extraction 2020 61 A state-transition framework to answer complex questions over knowledge base Subgraph matching 2020 62 Data-driven construction of SPARQL queries by approximate question graph alignment in question answering over knowledge graphs Subgraph matching 2020 63 Exploring sequence-to-sequence models for SPARQL pattern composition [55] Template-based 2020 64 Formal query building with query structure prediction for complex question answering over knowledge base Info. extraction 2020 65 Improving question answering over incomplete KBs with knowledge-aware reader Info. extraction 2020 66 Knowledge base question answering via encoding of complex query graphs Info. extraction 2020 FIGURE 6Open in figure viewerPowerPoint Relationships between keywords Figure 7 shows the distribution of the selected articles divided by types of architecture and distributed over years. FIGURE 7Open in figure viewerPowerPoint Distribution of papers by year and architecture As we can see, there is a consistent decline in the use of pipeline-based approaches. On the other hand, after an increase in subgraph matching solutions, we saw a slight drop in 2020. After a boom in 2016, the proposals for information extraction fell to a plateau still higher than the other proposals. Finally, template-based systems fluctuated to an annual maximum of two proposals in 2017 and 2018. 4 QA OVER KNOWLEDGE BASES SOLUTIONS Methods (Research Question 1) 4.1.1 Semantic parsing pipelines Hamon et al. [8] obtain answers from linked biomedical data using a four-step method that begins with the linguistic and semantic annotation of the input question. Similarly, a simple pipeline is proposed by Lopez et al. [37] for the QuerioDALI solution. In the first step, the system performs an NER to classify named entities related to the KB domain. At the next level, an EL filter binds a unique identity to each of the entities identified in the previous step. Then, the system uses fusion and ranking of possible answers. The lexical gap is a central issue in solving QA challenges. This refers to the mismatch between the vocabularies: on one hand, the user's vocabulary, and on the other hand, the vocabulary used in the data set of interest. The importance of the problem led to the emergence of several strategies. Hakimov et al. [29] use a combinatorial categorical grammar with handcrafted lexical items and lambda-type calculus expressions to obtain semantic representations. In this way, the input utterances must comply with the grammar. As is naturally emphasized by the authors, performance improves according to the lexicon size. The same conclusion is reached by Yih et al. [23], which shows that learning from labelled semantic parsers significantly improves overall performance. TR Discover, a solution by Song et al. [34], uses feature-based grammar. First, it parses an NL question to its first-order logic representation, which is, in turn, translated into SPARQL. Dubey et al. [35] also use a logical representation. Users can put queries in English to a target RDF KB. They are first normalized into an intermediary canonical syntactic form, called normalized query structure, and then translated into SPARQL queries. The direct use of an ontology makes it possible to reduce ambiguity. Ruseti et al. [31] use the structured KB DBpedia and a Wikipedia-based approach to match phrases from the question to entities in the ontology. Different solutions for matching each type of entity were developed, and the most probable interpretation is converted in a SPARQL query. Missing prope