A Fundamental Evaluation of Candidate Answers
Generation for Question Answering Using Wikidata
Ryoga Nakagawa1,∗,† , Kouji Kozaki1,∗,†
1Osaka Electro-Communication University, 18-8 Hatsuchou, Neyagawa-shi, Osaka, Japan, 572-0833

Abstract
Question answering (QA) is an import application of knowledge graphs (KGs). In order to evaluate
knowledge structures within a knowledge graph, we consider an investigation into the details of correspondences between questions and knowledge graph. In this paper, we focus on Wikidata as a knowledge
graph in the general domain and evaluate a basic question answering method using Wikidata, aiming toconduct a fundamental evaluation of its content as a knowledge graph.

Keywords
Knowledge graph, Wikidata, Question answering, Natural language question answering, SPARQL

1. Introduction
Knowledge graphs are widely used in various fields and have been applied to tasks such as
natural language processing, question answering, information retrieval, and problem-solving.
In question answering (QA), a knowledge graph enables users to ask questions in naturallanguage, and the system generates answers based on the knowledge graph. Specifically, thesystem interprets the meaning of a question, extracts relevant entities and relationships fromthe knowledge graph, and combines them to generate answers.
Several existing works focus on developing query answering systems using knowledge graphs.
For instance, T. Ploumis et al. developed a query answering system using Wikidata based on
the dependency analysis of question text[1]. X. Hu, et.al proposed a method for question
answering over knowledge graph based on SPARQL queries and keyword search [2]. Such an
approach plays an important role in improving the performance of QA systems. These studies
utilize datasets from various domains for QA and conduct experiments to evaluate the proposedsystems.
However, to evaluate knowledge structures within a knowledge graph, we are considering amore in-depth investigation into correspondences between questions and knowledge graph.
In this paper, we uses Wikidata [3] as an open knowledge graph in the general domain andevaluate a basic question answering method using Wikidata, aiming to conduct fundamental
evaluation of its contents as a knowledge graph.
IJCKG 2023 Poster and Demo track
Corresponding author.
†These authors contributed equally.
Envelope-Open mi23a002@oecu.jp (R. Nakagawa); kozaki@osakac.ac.jp (K. Kozaki)
Orcid 0000-0003-4578-4980 (K. Kozaki)∗© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR Workshop Proceedings (CEUR-WS.org)

2. Test Dataset for Question Answering using Wikidata
In this study, we have a test dataset for question answering using Wikidata. While thereare several datasets for question answering using Wikidata, they do not support Japanese
language. Therefore, we utilized the Japanese quiz dataset[4] which was developed using
Japanese Wikipedia for a competition of QA system primarily focused on NLP. This dataset
provides provides question text and its answer, which is described as one of Wikipedia articles.
Consequently, we can also obtain Wikidata ID corresponding to the article because Wikidata
contains resources related to all Wikipedia articles in principle. In order to determine how manyquestions can be correctly answered using Wikidata, we conducted manual SPARQL queries.
It’s important to note that some questions cannot be answered using any SPARQL query due tothe limitations in the graph structures of Wikidata, which may lack the necessary knowledge
to answer certain questions. From the quiz dataset, We choose 50 questions and formulated
SPARQL queries. As a result, we were able to create SPARQL queries that provided correct
answers for 33 out of the 50 questions.

3. Evaluation of Candidate Answers Generation
To generate candidate answers, it is necessary to map the words in the question text to Wikidata
IDs and then use the mapping results to generate SPARQL queries to find answers using Wikidata.
The former corresponds to Entity Linking, and the latter to SPARQL query generation. In thefollowing sections, we evaluate these two methods using the test dataset provided in Section 2.
As an example, we will use the question shown in (Figure 1 (a)) to illustrate these methods.

3.1. Evaluation of Entity Linking
Out of 50 question in the test dataset, we evaluated entity linking using the 33 questions forwhich we could obtain correct answers using SPARQL queries on Wikidata. For the entity
linking method, we extracted words from question text and searched Wikidata for entities thatmatched the extracted words using an exact string match. In the example question mentioned
above, four entities with Wikidata IDs were obtained from the search results (see Figure 1 (b)).
The evaluation was conducted manually using Wikidata search API to avoid any potential
issues caused by morphological analysis.
Next, we compare the obtained IDs from a question text to the SPARQL query for the questionin the test dataset. If all the IDs in the query are included in the obtained IDs, it means that theFigure 1: An example of (a)question text, (b)entity linking and (c)SPARQL queryquery could be generated using them. In the case of the example of Figure 1(a), all the IDs shownin the query in Figure 1(c) are included in the obtained IDs shown in in Figure 1(b). However,it’s worth noting that Wikidata IDs of properties are not obtained by the entity linking method,
so they are replaced with variables such as “?p1”,“?p2” in the query. In the evaluation results ofentity linking, it was found that for 27 of the 33 questions, all the IDs required to generate theSPARQL queries were obtained. However, for 6 questions, the questions text included some IDs
that were not obtained by the entity linking method due to differences in spelling between thequestion text and Wikidata.

3.2. Evaluation of SPARQL Query Generation
For the 27 questions which all the required IDs could be obtained through entity linking, weevaluate SPARQL query generation to obtain the candidate answers using the graph structure of
Wikidata. In this study, we explore an exhaustive combination of the entities obtained throughentity linking. When two IDs are obtained through entity linking, four SPARQL patterns aregenerated. For instance, when two IDs are obtained, such as ID1 and ID2, four patterns of
SPARQL queries are generated as illustrated in Figure 2. In this exampel, variables like “?p1”,“?p2”
represent properties, and “?ans” represents the answer to the question. If the SPARQL query
accurately represents the question’s condition, the correct answer can be returned as the valueof “?ans” . However, if there are multiple entities that satisfy the search conditions, a single
query pattern can yield multiple candidate answers.
To evaluate the query generation, we executed the generated queries on SPARQL endpoint of
Wikidata and verified whether their query results contained the correct answer to the questions.
As a results, out of the 27 questions, the method successfully generated SPARQL queries that
produced the correct answer for 23 questions. However, in the case of the remaining fourquestions, timeouts occurred during SPARQL query processing, preventing us from obtaining
the correct answers..

Figure 2: Patterns for SPARQL query generation using two IDs

4. Discussion
Figure 3 displays an overview of the evaluation results. For our testing, we used 50 questions
from the Japanese quiz dataset. Through manual efforts, We obtained SPARQL queries capableof deriving correct answers from Wikidata for 33 of these questions (Figure 3(1)). These 33
questions with SPARQL queries constitute the correct data for evaluating question answering
using Wikidata. Among the remaining 17 questions, 7 required aliases (alternative labels) toFigure 3: The overview of evaluation results

obtain the correct answers using SPARQL queries to Wikidata. For the remaining 10 questions,
we could obtain the correct answers using SPARQL due to the lack of knowledge defined in
Wikidata. We evaluated fundamental methods of entity linking and SPARQL query generation
using the 33 questions and their associated queries in the correct dataset.
Through the evaluation of entity linking, we obtained all the IDs required for the SPARQL
queries for 27 of the 33 questions. We then applied these IDs to query generation and obtained
the correct answers using one of the generated SPARQL queries for 23 of the 27 questions.
In other words, we were able to obtain the correct answers from Wikidata for 82.2% of thequestions when we could obtain enough IDs through entity linking. We believe that we could
obtain correct answers for the remaining questions by improving the querying performance
because the primary reason why we could not obtain the answers was the timeout problem
during SPARQL query processing. When we compare the final results in (3) and the 33 questionsin the correct dataset, we found that we could obtain correct answers from Wikidata for 69.7%
of questions. To improve these results, we need to enhance the accuracy of entity linking.
In this way, this study conducts a fundamental evaluation of candidate answers generation
for question answering using Wikidata. The evaluation results could serve as baseline data todevelopment of question answering methods using Wikidata. As part of our future works, weplan to extend the number of questions for the test datasets and evaluation results. We willconsider using not only manual efforts but also automatic methods to prepare them using entity
linking and SPARQL query generation, as discussed in Section 3. Additionally, we are in theprocess of developing a QA system by extending these methods.

References[1] T. Ploumis, I. Perikos, F. Grivokostopoulou, I. Hatzilygeroudis, A factoid based question
answering system based on dependency analysis and wikidata (2021) 1–7.
[2] X. Hu, J. Duan, D. Dang, Natural language question answering over knowledge graph: themarriage of sparql query and keyword search, Knowledge and Information Systems 63(2021) 819–844.
[3] Wikidata, https://www.wikidata.org/, 2022. Accessed on 2022/10/25.
[4] Jaqket japanese on the subject of quizzes qa data-set, https://jaqket.s3.ap-northeast-1.
amazonaws.com/data/aio_01/dev1_questions.json, 2020. Accessed on 2022/10/25.