Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction (2024)

Abstract: Advancements in AI and natural language processing have revolutionized machine-human language interactions, with question answering (QA) systems playing a pivotal role. The knowledge base question answering (KBQA) task, utilizing structured knowledge graphs (KG), allows for handling extensive knowledgeintensive questions. However, a significant gap exists in KBQA datasets, especially for lowresource languages. Many existing construction pipelines for these datasets are outdated and inefficient in human labor, and modern assisting tools like Large Language Models (LLM) are not utilized to reduce the workload. To address this, we have designed and implemented a modern, semi-automated approach for creating datasets, encompassing tasks such as KBQA, Machine Reading Comprehension (MRC), and Information Retrieval (IR), tailored explicitly for low-resource environments. We executed this pipeline and introduced the PUGG dataset, the first Polish KBQA dataset, and novel datasets for MRC and IR. Additionally, we provide a comprehensive implementation, insightful findings, detailed statistics, and evaluation of baseline models.

heading: 1 Introduction
text: Question answering (QA) systems serve as a sophisticated interface between humans and computers. To further enhance their utility, we need QA systems capable of answering questions based on extensive knowledge (Petroni et al., 2021). The knowledge base question answering (KBQA) task addresses this need by using structured knowledge graphs (KG) to provide accurate and relevant answers (Lan et al., 2021). KBQA leverages these graphs, which are rich with interconnected entities and relationships, to decode complex queries and deliver precise answers. Importantly, systems that reason over KGs are more resistant to the phenomenon of hallucinations, common in large lan-
guage models (LLM) (Baek et al., 2023). Additionally, the inherent flexibility of KGs facilitates easy modification and updating, ensuring the use of only the most current and accurate facts.
However, a significant gap exists in KBQA datasets. Many are schematic and not natural in their language, or they rely on discontinued knowledge graphs (Lan et al., 2021; Steinmetz and Sattler, 2021; Jiang and Usbeck, 2022). By natural we refer to naturally occurring questions (Kwiatkowski et al., 2019). While a broader range of KBQA datasets is available for English, most low-resource languages, including Polish, lack such resources (Korablinov and Braslavski, 2020). This scarcity is part of a broader issue prevalent in the field of NLP concerning low-resource languages (Augustyniak et al., 2022). Recognizing this gap, we set out to create a KBQA dataset for Polish. We faced several challenges during extensive studies of existing works to find the most efficient methods for dataset creation. Many datasets were built on simpler predecessors (Korablinov and Braslavski, 2020; Kaffee et al., 2023), and also many construction pipelines are inefficient regarding human labor, as they do not utilize modern tools that could reduce human work, such as assisting Large Language Models (LLM). LLMs have opened new opportunities for assisting human annotators, especially in low-resource languages where the range of pre-trained models is limited (Gilardi et al., 2023; Kuzman et al., 2023).
Consequently, we decided to design, implement, and execute a modern approach to creating KBQA datasets tailored explicitly for the low-resource environment. We selected Wikidata as KG due to its extensive, multilingual coverage and dynamic, open, and free nature (Vrandečić and Krötzsch, 2014). Notably, we did not use any translation, ensuring that the output data sounded natural. Moreover, an advantageous byproduct of this pipeline was the concurrent development of machine read-
10978
ing comprehension (MRC) and information retrieval (IR) datasets, requiring no extra human annotation. MRC is essential for AI to understand and analyze texts like a human reader (Rajpurkar et al., 2016; Kwiatkowski et al., 2019), while IR is crucial for efficiently extracting relevant information from vast databases (Nguyen et al., 2017; Thakur et al., 2021).
We summarize our contributions as follows:
• We introduce the PUGG 1 dataset, which encompasses three tasks — KBQA, MRC, and IR2. This dataset features natural factoid questions in Polish and stands out as the first Polish KBQA resource 3. To address a range of complexities, we have enriched the dataset by complementing natural questions with simpler, template-based questions.
• We propose a semi-automated dataset construction pipeline designed for low-resource environments. The pipeline results in the creation of KBQA, MRC, and IR datasets while drastically reducing the labor of human annotators. Accompanying this pipeline is a comprehensive implementation 4. Moreover we share insightful findings and detailed statistics obtained from the PUGG dataset construction using the pipeline. These provide valuable resources for future developers of datasets. Additionally, we developed few utility custom methods, e.g. for entity linking, that are useful in diverse contexts.
• We provide an evaluation of baseline models, thereby establishing benchmarks for future research using the PUGG dataset.

heading: 2 Related Work
text: KBQA Existing KBQA datasets have been comprehensively studied and compared in works done by Korablinov and Braslavski (2020) and Jiang and Usbeck (2022). A significant finding is the lack of a Polish KBQA dataset. Most KBQA datasets are primarily in English, with exceptions like the Chinese NLPCC-KBQA (Duan and Tang, 2018),
1The name "PUGG" refers to "Pirate Pugg", a fictional character from "The Sixth Sally" of "The Cyberiad" by Stanisław Lem. Pirate Pugg is depicted as being obsessed with gathering information.
2https://huggingface.co/datasets/clarin-pl/ PUGG
3The dataset license: CC BY-SA 4.0 4https://github.com/CLARIN-PL/PUGG
Russian RuBQ (Korablinov and Braslavski, 2020), the multilingual QALD (Perevalov et al., 2022) and MCWQ (Cui et al., 2022) (both not including Polish). The closest dataset resembling a KBQA task in Polish is the multilingual MKQA (Longpre et al., 2021), where approximately 42% of its 10,000 questions are answerable by Wikidata entities. However, MKQA cannot be classified as a proper KBQA dataset due to the lack of annotated topic entities.
The study by Korablinov and Braslavski (2020) outlines the various question generation techniques used in existing KBQA datasets. For generating natural questions in our research, we adopted a question generation technique based on query suggestion, initially introduced by Berant et al. (2013). This technique is effective for acquiring natural factoid questions likely to be posed to a QA system, similar to the approaches used in datasets like NQ (Kwiatkowski et al., 2019) and WikiQA (Yang et al., 2015), which were built from questions asked to search engines. For template-based questions, our approach involved creating questions from predefined reasoning templates, a standard method in many KBQA datasets (Bordes et al., 2015; Su et al., 2016; Dubey et al., 2019). Several KBQA datasets used crowdsourced paraphrasing for question diversification (Talmor and Berant, 2018; Su et al., 2016; Dubey et al., 2019). In contrast, our approach only automates this process by incorporating humans during final verification.
IR Many valuable resources for Information Retrieval in the Polish language were recently created. The BEIR-PL (Wojtasik et al., 2024) benchmark was proposed as an automatic machine translation of the BEIR (Thakur et al., 2021) benchmark. This popular zero-shot retrieval benchmark was originally only for the English language. The MQUPQA (Rybak, 2023) dataset is a composition of multiple already existing Polish and multilingual datasets, like CzyWiesz (Marcińczuk et al., 2013), MKQA (Longpre et al., 2021). Additionally, the MQUPQA dataset incorporates other automatic methods for question and answer generation, such as utilizing the generative capabilities of the GPT-3 model (Brown et al., 2020) or employing templates inspired by the structure of Wikipedia. The PolEval (Łukasz Kobyliński et al., 2023) competition featured a passage retrieval task. It comprised three datasets from various domains: Wikipedia-based, e-commerce shop FAQ, and legal questions. Cur-
rently, a Polish Information Retrieval Benchmark (PIRB) (Dadas et al., 2024) provides a platform to evaluate models across various datasets. The models referred to in this benchmark represent the current state-of-the-art in Polish IR.
MRC QA datasets often have a close relationship with IR datasets. The CzyWiesz dataset is based on the Did you know? section of Wikipedia, with provided answers and also relevant articles. Another example is the PolQA (Rybak et al., 2022) dataset, which is comprised of general questions from quiz shows annotated with relevant passages from Wikipedia. The PoQuAD (Tuora et al., 2023) dataset is structured around questions manually annotated to correspond with the best articles on Wikipedia, mirroring the methodology of the SQuAD (Rajpurkar et al., 2016) dataset. Contrastively, our dataset consists of naturally occurring questions, which are afterward annotated to relevant articles.
3 Definitions
A common element in the tasks of KBQA, MRC, and IR is the textual question q. We denote the set of questions as Q. Despite query being common in the field of IR, we use question and query interchangeably, as our dataset’s queries take the form of questions.
KBQA We denote KG as a multi-relational heterogeneous graph G = (E ,R, T ), composed of three elements: a set of entities E , a set of relation predicates R, and a set of triples (facts) T . Each triplet (h, r, t) ∈ T indicates a relation predicate r between two entities, a head entity h and a tail
entity t, where h, t ∈ E and r ∈ R (Hamilton et al., 2017). In the KBQA task, a textual question q and associated topic entities Eq ⊂ E are given. The objective is to retrieve answer entities Aq ⊂ E that satisfy the question based on facts in the G. Therefore, we denote KBQA dataset as DKBQA = {(q, Eq,Aq)}. MRC MRC aims to answer a textual question q based on a given text passage pq. We denote MRC dataset as DMRC = {(q, pq, aq)}, where aq is the answer extracted from pq.
IR The IR task focuses on finding a passage p from a large corpus relevant to a query q. The corpus C is defined as a set of passages, i.e., C = {p1, p2, ..., pn}. The IR dataset is denoted as DIR = {(q, pq)}, where pq ∈ C denotes a passage that is relevant to the query q.

heading: 4 Construction Pipeline
text: This section introduces the construction pipeline for the PUGG dataset, specifically designed to create a dataset with natural and factoid questions in a semi-automated manner. This approach significantly reduces the workload of human annotators. We outline the pipeline’s fundamental design, presented in Figure 1, emphasizing its adaptability to various environments. While this part focuses on the general framework, specific implementation details, such as the models and algorithms used, will be discussed in Section 5.
Question Formulation The initial step of our pipeline involves acquiring a variety of natural factoid questions. We initiated our process using existing datasets to minimize the need for manual
annotation. From previously existing QA datasets, we extract question prefixes ranging from basic (’who...’, ’when...’) to more specific (’which Canadian athlete...’, ’which theater co-created...’). Then, the gathered prefixes are completed to formulate a set of questions. We can employ various methods, including rule-based approaches and language models (Das et al., 2021), and for natural questions, we can also integrate external services.
At this stage, we have a collection of question candidates q′, as some of which may be incorrect. These inaccuracies are not a concern at this point, as they will be filtered out during the human verification process, detailed in Section 4.
Passage Construction The next phase involves text passages retrieval to answer the formulated questions. We use a data source with referenced graph entities, which in our case is Wikipedia. To find relevant articles for each question, various retrieval techniques can be employed, such as dense retrieval (Reimers and Gurevych, 2019) with additional reranking. Once relevant articles are identified, they are segmented into smaller passages and reranked to prioritize passages most likely to contain an answer.
All passages constructed in this phase are added to the passage corpus C needed for the IR task.
Textual Answers, Answer Entities We select the most accurate passage as candidate passage p′q, and we apply a QA model, such as LLM or pre-trained extractive model, to tag a span of passage denoting a candidate textual answer a′q. Such textual answers contain hyperlinks to other articles associated with specific Wikidata entities. We extract these entities and build a set of candidate answer entities A′q.
Topic Entities The subsequent step in our pipeline is performing an entity linking process to identify and link the KG entities mentioned in the questions. We refer to them as candidate topic entities E ′q.
Human Verification To this point, we have acquired all necessary data to construct the KBQA, MRC, and IR datasets: questions q′ accompanied by a passage p′q, textual answer a ′ q, answer entities A′q, and topic entities E ′q. All these elements are obtained through fully automated processes. While automation significantly reduces the need for human labor, it is not entirely error-proof. To ensure
the high quality of our dataset, we implement a human verification process. The detailed procedure of this human verification is depicted in Figure 2. During this process, candidate elements q′, p′q, a ′ q, A′q, and E ′q undergo verification. This leads to the final elements q, pq, aq, Aq, and Eq, respectively. The final setsAq ⊆ A′q and Eq ⊆ E ′q indicate that the validated entities are subsets of their initial candidate sets. Note that the verification procedure (Figure 2) consists of multiple conditions, which may result in the datasets varying in size. This is reflected in the relationship |DIR| ≥ |DMRC | ≥ |DKBQA|.
Template-based KBQA While the proposed pipeline generates natural questions, we also created template-based questions to enrich our dataset. We wanted to provide a broader training and evaluation platform by offering a more schematic and straightforward set of questions, ensuring a reasoning path between topic and answer entities. The template-based questions are also beneficial for semantic parsing-based KBQA methods (Lan et al., 2021).
Figure 3 depicts the procedure of creating template-based questions. We create SPARQL templates paired with corresponding natural language questions, representing specific reasoning paths
in the KG. We specify potential entities and relations to be used within these templates. We insert these entities and relations into the natural language template to construct questions. Then, we run the corresponding SPARQL queries to retrieve answer entities.
At this stage, the formulated questions might sound unnatural, especially in inflected languages like Polish. We use two strategies to address this: word inflection and question paraphrasing. We can automate the inflection process using NLP tools like spaCy (Honnibal et al., 2020) or LLMs. We also use LLMs to paraphrase the questions for greater diversity and complexity. Given the automation of these processes, we ensure the meaningfulness of all questions through human verification.

heading: 5 Pipeline Execution
text: This section delves into the specific implementation of the construction pipeline for the PUGG dataset, as previously outlined in a general framework in Section 4. Our implementation was adapted for Polish NLP resources, which face challenges like limited task-specific pre-trained models and lower performance than English.
Question Formulation In implementing our question acquisition step, we utilized two Polish datasets, CzyWiesz (Marcińczuk et al., 2013) and PoQuAD (Tuora et al., 2023). Question prefixes were extracted either by taking the first {1, 2, 3} tokens from each question or by extracting text up to the first occurrence of a named entity. We employed three NER models: pl_core_news_sm, pl_core_news_lg from Spacy (Honnibal et al., 2020), and WikiNEuRal (Tedeschi et al., 2021). Each of these models provided a unique perspective in identifying named entities, thereby contributing to the variety of the prefixes. To formulate natural questions from these prefixes, we followed previous studies (Berant et al., 2013; Rybin et al., 2021) and used the Google Suggest API.
Passage Construction We followed established methodologies from prior research (Kwiatkowski et al., 2019) and employed the Google Search Engine 5 to retrieve Wikipedia articles relevant to each question. We processed the top 10 search results using the API, focusing on Wikipedia entries. Questions without a Wikipedia article in the top 10 results were discarded. The text and interarticle references of these Wikipedia articles were then obtained using the Wikipedia API6. The retrieved articles were segmented into shorter passages using a sliding window approach, with a window length of 120 words and a step size of 60 words. We ranked these passages for each question according to their relevance. This was achieved by leveraging the PyGaggle (Pradeep et al., 2023) library with the multilingual reranker model unicamp-dl/mt5-3B-mmarco-en-pt (Bonifacio et al., 2021).
Textual Answers, Answer Entities For textual answer tagging, we employed GPT-3.5-turbo 7 (Brown et al., 2020) with an originally designed prompt, detailed in Appendix A. Due to the model’s generative nature and tendency to alter or paraphrase the original text, we developed a custom method to extract tagged segments accurately. This method is described in Appendix A. As previously described, candidate answer entities were directly referenced in the text, allowing for straightforward extraction.
Topic Entities Implementing the entity linking step presented several challenges, primarily due to the lack of robust tools or models for entity linking in the Polish language. Our testing of multilingual models like mGENRE (De Cao et al., 2022) and adapted for Polish BLINK (Wu et al., 2020)
5https://developers.google.com/custom-search/ v1/overview
6https://pl.wikipedia.org/w/api.php 7https://platform.openai.com/docs/models/
overview
yielded unsatisfactory results, particularly for short contexts such as individual questions. Additionally, given the planned human verification stage, a method with high recall was desired. To address these challenges, we developed a heuristic method tailored to our requirements and the available resources. It leverages the Wikipedia search engine to identify potential entities for single words, combined words and identified named entities. Additionally, it collects entities referenced in the retrieved pages and utilizes title similarity measures to ensure the relevance of identified entities to the question. More details on the method can be found in Appendix B.
Human Verification The general procedure for human verification is illustrated in Figure 2. We implemented this by dividing it into two distinct stages. The first stage focused on identifying two aspects: questions with correctly assigned passages and questions where the textual answers within these passages were accurately tagged. The second stage of human verification had two parts: annotators marked the correct answer entities and then identified the correct topic entities. All annotators were employed in Poland and fluent in Polish. They were familiar with the Polish culture and social context. Appendix C presents more details about annotation procedures and guidelines.
Template-based KBQA The developed templates are detailed in Appendix D.1. It is important to note that while our template-based KBQA dataset contains fewer templates compared to other datasets, ours are more general. This is achieved by injecting not only entities but also relations into the templates, enhancing their diversity. We used entities from Wikipedia’s Vital Articles Level 4 8 and 173 manually selected relations. Any entities lacking a Polish label were excluded. Given the vast number of possible inputs (entities and relations) for the templates and that most of them will not yield answers, random input selection was not feasible. Therefore, we divided the process into two steps, each involving the execution of a SPARQL query. First, we gathered potential sets of inputs, and then, we selected some of these sets to retrieve answers. We also utilized the specified inputs to create questions using natural language templates.
Then, we inflected and paraphrased the constructed questions using the GPT-3.5-turbo model
8https://en.wikipedia.org/wiki/Wikipedia: Vital_articles
(Brown et al., 2020). Following this, we filtered out examples without high similarity to their original form based on the longest common sequence analysis. One annotator verified the questions. Similarly to the natural questions, the annotator was employed in Poland, fluent in Polish, and familiar with Polish culture and social context. The statistics of the verification can be found in Appendix D.1.
Outcome The execution of our pipeline resulted in the creation of the PUGG dataset, featuring three tasks: KBQA (natural and template-based), MRC, and IR. Statistics for each dataset are presented in Table 1. The detailed statistics of the pipeline steps, unique entities and relations in the dataset, and the distribution of examples across different template types are available in E. Due to the utilized sliding window approach in passage construction, all passages from corpus C that overlapped with any of pc were removed. As Wikidata is a vast KG and using it for research can be inconvenient, we provide sampled versions of the KG: Wikidata1H and Wikidata2H. These are subgraphs created by traversing 1 or 2 relations from each answer and topic entity, representing two different levels of data complexity.

heading: 6 Experimental Setup
text: In this section, we outline the evaluation methodology used to assess the performance of baseline models on the PUGG dataset.
KBQA For the KBQA baseline, we evaluated the performance of KAPING (Baek et al., 2023), a zero-shot framework that leverages an LLM for retrieving answer entities. We slightly modified
the knowledge retriever module by incorporating a step that retrieves a subgraph of the KG by traversing n edges, regardless of their direction, from the topic entities. Our preliminary experiments demonstrated enhanced performance of the modification, showcasing improvements in both accuracy and processing speed. Subsequently, we follow the original procedure, which involves retrieving k triples based on their textual embeddings. For embedding purposes, we utilized the mmlw-retrievalroberta-large retrieval model (Dadas et al., 2024). We employed gpt-3-turbo as the LLM, prompted with tailored queries as detailed in Appendix F. The hyperparameters were selected empirically, setting k = 40 and choosing n to be 3 for Wikidata1H and 2 for Wikidata2H. As a metric, we employed accuracy, which measures the proportion of answers included in the LLM’s response for each question. It is calculated as follows:
Accuracy = 1
|Q|
|Q|∑
i=1
num of incl. answersi |Ai|
While Baek et al. (2023) also used accuracy, we refined it by calculating the correct answer proportion per example and excluding entities’ aliases, providing a more realistic measure of KBQA efficacy.
MRC For the MRC task, we selected models commonly used for the extractive question answering task. We trained and evaluated HerBERT (Mroczkowski et al., 2021) models in an extractive fashion alongside a generative approach using the plT5 (Chrabrowa et al., 2022) models. Models were trained for 10 epochs and evaluated with SQuAD metrics (Rajpurkar et al., 2016). Exact match measures the percentage of predictions that exactly match the gold answer. The F1 metric measures the average token overlap between the prediction and ground truth answer, where both the prediction and answer are treated as a bag of tokens.
IR Recently, IR has gained significant interest within the Polish research community, and many models have been developed and are open to the research community. These models have already been pre-trained on large datasets, which is why we did not fine-tune them to our dataset. The silver retriever (Rybak and Ogrodniczuk, 2023) model was trained on the MAUPQA dataset. We also
evaluated E5 (Wang et al., 2024) multilingual embedding models, which were trained on contrastive objectives on large weakly-labeled text pairs and afterward fine-tuned on existing datasets and are performing very well on Polish texts. The MMLW retrieval models (Dadas et al., 2024) were trained on a parallel corpus with Polish-English text pairs with a bge-large-en (Xiao et al., 2023) teacher model and are currently on the top of the PIRB leaderboard. We also provide results of well-established BM25 (Robertson and Zaragoza, 2009) baseline with Morfologik 9 plugin in Elasticsearch.
Additionally, we evaluated reranker models, focusing on those developed in the BEIR-PL benchmark and recent models that have appeared on the PIRB leaderboard. Those models were trained by Dadas and Grębowiec, 2024 with knowledge distillation from mT5-13B model introduced in the mMARCO publication (Bonifacio et al., 2021). For reranking, we employed the BM25 retrieval algorithm to select the top 100 passages for subsequent analysis. Finally, we provide a score of the combination of the best retriever and reranker, namely multilingual-e5-large retriever and polishreranker-large-ranknet reranker, to evaluate currently the best IR pipeline available. We calculated the well-established metrics for the IR task: MRR@k, NDCG@k, Recall@k (Thakur et al., 2021; Wojtasik et al., 2024).

heading: 7 Results and Discussion
text: KBQA The summarized results are presented in Table 2. For natural and template-based questions, utilizing KG significantly improves accuracy. The overall accuracy is not high, indicating the challenging nature of the newly introduced PUGG dataset. This complexity highlights its potential as a valuable resource for advancing research and development in the field of KBQA. As expected, reasoning over 1-hop (1H) KG was easier than over 2-hop (2H) KG, reflecting the increased complexity of KG. There is a clear gap in efficacy between natural and template-based questions. That was expected, as template-based questions were designed to be easier. Interestingly, they benefit more from the use of KG than the natural ones. We think that it can be caused by their schematic construction mechanism. Moreover, our pipeline for natural questions does not ensure the existence of appropriate reasoning
9https://github.com/allegro/ elasticsearch-analysis-morfologik
paths in the graph, which could also cause lower efficacy.
MRC The results of the MRC baselines, as presented in Table 3, suggest that extractive models excel in identifying exact matches within the text. On the other hand, large generative models have demonstrated a capacity to achieve a high degree of general answer overlap, as reflected by their F1 scores. Compared to the baseline results disclosed in the PoQuAD publication (Tuora et al., 2023), which reported exact match and F1 scores of 66.22 and 81.39, the current results suggest that the dataset constitutes a greater challenge for the models.
IR The scores presented in Table 4 reveal that the dataset poses a significant challenge for the lexical BM25 approach. The questions have limited lexical overlap; therefore, this method is ineffective. Nonetheless, current dense retrieval models are exhibiting high performance. Surprisingly, the mmlw-retrieval-roberta-large model, despite being currently ranked at the top of the PIRB benchmark, still falls behind the multilingual-e5-large model. This suggests that the dataset is a valuable resource for assessment and should be included in the PIRB benchmark in the future. The reranker models improved the BM25 rankings significantly, and combining a dense retriever with a reranker has achieved remarkably high scores across all metrics.

heading: 8 Limitations and Future Work
text: This section outlines the limitations of our study and potential directions for future work. (1) The natural questions are open domain, focused on location and time, and are created and answered from the Polish cultural, political, and historical perspective. (2) The pipeline for natural questions may sometimes miss certain answer entities. This is because not all answers are present or explicitly referenced in the textual answer. (3) Some of the KBQA natural questions might not have corresponding facts in the KG, as our pipeline does not guarantee the existence of an appropriate reasoning path between topic and answer entities. However, as Wikidata is continuously updated and expanded, this limitation may diminish in the future. (4) The questions might contain grammatical imperfections or mental shortcuts yet remain understandable. (5) Automated annotation with LLM led to variability in the precision of tagged answers in the MRC task due to the absence of specific tagging guidelines. (6) Our study examined a limited number of baseline models. Future evaluations could, in particular, include open-source LLMs like Llama (Touvron et al., 2023) for MRC and KBQA tasks, as well as models that reason directly over the KG structure, such as PullNet (Sun et al., 2019), for KBQA task. (7) While our focus was on standard tasks, we acknowledge the potential for exploring additional tasks using the PUGG dataset. These tasks include entity linking, subgraph retrieval, relation extraction, question type classification, and question generation.

heading: 9 Conclusion
text: To address the significant resource gap for lowresource languages, our work introduces the PUGG dataset, the first Polish KBQA dataset, which also encompasses MRC and IR tasks. It consists of natural and template-based factoid questions. The dataset is the outcome of our proposed semiautomated construction pipeline, designed for lowresource environments. Leveraging modern tools like LLMs as annotation assistants have significantly reduced the need for human labor. Additionally, we developed few utility methods, such as entity linking, which are useful in various contexts. The PUGG dataset and our pipeline’s comprehensive implementation, findings, and detailed statistics from the PUGG dataset construction provide valuable insights for future research. Further-
more, the evaluation of baseline models on this dataset reveals its challenging nature, underscoring its potential to advance the field and contribute to developing more robust QA systems.

heading: 10 Ethical considerations
text: The process of dataset creation using LLMs and pre-existing datasets entails the potential risk of inheriting biases from both the models and the original data sources. To address this concern, a pipeline could incorporate multiple LLMs and diverse datasets as a mitigation strategy.
We used sources with a low risk of containing private data or offensive content. However, during the human verification process, we further ensured that the dataset did not include such data. As mentioned in Section 5, all annotators were employed in Poland and were fluent in Polish. They were familiar with the Polish culture and social context.

heading: 11 Acknowledgments
text: This work was supported by Polish Ministry of Education and Science under the programme: "Support for the participation of Polish scientific teams in international research infrastructure projects", agreement number 2024/WK/01 and project of the Minister of Digitization No. 1/WI/DBiI/202 (PLLUM). This work was also partially funded by the European Union under the Horizon Europe grant OMINO – Overcoming Multilevel INformation Overload (grant number 101086321, http://ominoproject.eu) co-financed with funds from the Polish Ministry of Education and Science under the programme entitled International Co-Financed Projects, grant no. 573977.

heading: A Textual Answers Tagging
text: The designed prompt is presented in Table 5. The annotated spans were extracted from the LLM’s responses using lemmatization and longest common sequence analysis.

heading: B Topic Entity Linking
text: The designed entity linking method primarily relies on the Wikipedia search engine, title similarity, and information about the neighborhood of the question.
The Wikipedia search engine is accessed via the MediaWiki API 10. This search system identifies page titles or content that match a given textual query. Title similarity is measured by assessing the similarity of provided texts, utilizing both the longest common sequence and the longest common prefix approaches. To construct the neighborhood of the question, we retrieved Wikipedia pages from the top 10 Google search results and then extracted the first five links from each of these articles. These results are then used to determine whether the entity found by the algorithm belongs to such a neighborhood. It is important to note that, in this context, the neighborhood of the question is not associated with the KG.
As the output, we expect four types of entities: exact entities, neighborhood entities, named entities, and combined entities. Detailed information
10https://www.mediawiki.org/wiki/API:Search
on this process can be found in the pseudocode provided in Algorithm 1. For tokenization, lemmatization, and NER, we used the SpaCy tool (Honnibal et al., 2020) with the pl_core_news_lg model.

heading: C Human Verification

heading: C.1 First Stage
text: To ensure high-quality data, the annotation team included annotators and a super-annotator. The process involved: (1) initial guideline preparation, (2) a full review of annotator decisions reviewed by the super-annotator, and (3) a targeted review
Algorithm 1 Entity Linking Method

heading: Input:
text: Q - input question. Constants: L← [noun, adjective, proper noun, unknown] T ← tokenize_to_words(Q) N ← named_entities(Q) Output: Eexact - set of entities closely matching the title of Wikipedia pages Enbhd - set of entities not precisely matching Wikipedia titles but belonging to the question neighborhood Enamed - set of named entities belonging to the question neighborhood Ecomb - set of entities formed by combining two or more words

heading: Algorithm:
text: for each t ∈ T do
if pos(t) ∈ L then res← search_wikipedia(t) l← lemma(t) Eexact ← high_similarity(res, l)
for each n ∈ N do res← search_wikipedia(n) Enamed ← in_neighborhood(res) for each t ∈ T do if pos(t) in L then
res← search_wikipedia(n) Enbhd ← in_neighborhood(res)
for each t ∈ T do if pos(t) == ’noun’ then
R← get_nouns(children(t)) A← get_adjectives(children(t)) Rq ← R× [t] Aq ← A× [t] for each q ∈ Rq ∪Aq do
res← search_wikipedia(q) Ecomb ← in_neighborhood(res)
of problematic examples by the super-annotator. This process refined the guidelines and focused on resolving ambiguities in annotations. Examples with improperly formulated questions or lacking information for accurate answers were rejected, especially those with significant grammatical or lexical errors that made them incomprehensible. Technically, this step involved flagging documents in the Inforex system (Marcińczuk et al., 2017), with the following set of flags: (1) correct: indicates both the question and answer are correct in the passage. (2) incorrect question: indicates the question is formulated incorrectly. (3) incorrect passage: indicates the passage does not answer the question. (4) incorrect fragment: indicates the answer is located elsewhere in the passage.

heading: C.2 Second Stage
text: Two annotators carried out this stage. To facilitate a consistent and measurable approach, we separated 10% of the examples as common for both annotators, while the rest were individually assigned. These shared examples served as a basis for calculating annotation metrics and ensuring reliability and consistency in the annotation process. Annotating the correct answers was a straightforward task. However, the annotation of topic entities presented more complexity. As Rosales-Méndez et al. (2018) have pointed out, there is no consensus on the concept of an entity and what entity linking should link to, as it varies greatly depending on the application. Due to the absence of universally acknowledged guidelines, we defined a topic entity as a source entity from which the reasoning method should begin its process. In cases where annotators were uncertain about either answer or topic entities, the problematic examples were rejected to maintain the dataset’s quality. The entire second stage of the annotation process was carried out using a spreadsheet application. During the annotation of answer entities and topic entities, we achieved Cohen’s kappa scores of 0.785 and 0.675, with accuracy scores of 0.892 and 0.895, respectively.

heading: D Template-based KBQA

heading: D.1 Templates
text: We have developed 8 templates for schematic question creation, detailed in Table 6. We distinguish the following three general techniques.
N-hop templates retrieve information by traversing N relations from the given entity.
Reverse N-hop templates function similarly but involve traversing in the reverse direction.
The Entity Mask technique enriches questions by referring to the answer without direct mention. For example, instead of naming "Ludwig van Beethoven", we might use "composer".

heading: D.2 Paraphrasing and Inflection Prompts
text: Table 7 presents the prompts for inflecting and paraphrasing questions constructed using the natural language templates.

heading: D.3 Human Verification
text: Inflected and paraphrased questions were verified using the following set of annotation flags: correct, incorrect, and resembling.
Correct implies the semantic meaning of the processed question remains unchanged compared to the original. Incorrect flags a change in semantic meaning. For instance, the original question ’Who is the creator of the web browser?’ paraphrased as ’What material is the web browser created of?’ illustrates this change. It is also worth mentioning that incorrect questions often involve the reversal of relations: Whose doctoral supervisor is Max Perutz? was paraphrased as Who is Max Perutz’s doctoral supervisor?. The fact that LLMs may struggle to understand reverse connections, was also highlighted by Berglund et al. (2023). During annotation, we noticed some question patterns frequently repeated in specific templates like one-hop templates. We labeled these as resembling and excluded them from the final dataset. For example, ’Where was X born?’, was common due to the ’place of birth’ being a prevalent relation for people on Wikidata. The statistics of verification are presented in Table 8.

heading: E Detailed statistics

heading: E.1 Pipeline stages
text: Table 9 summarizes the number of examples processed at each stage of the PUGG construction, both for natural and template-based questions. Each step either increased or reduced the number of examples. This step-by-step analysis could be beneficial for scientists and engineers aiming to execute similar pipelines. It offers a precise estimate of the volume of data necessary at the beginning and the anticipated human labor required during the verification stages. Notably, textual answer tagging and entity verification stages contribute to the
most significant reductions in data volume. The initial steps (gathering questions from existing QA datasets, extracting prefixes, and formulating questions) significantly increased the number of potential examples. This increase was essential for the subsequent stages that reduced questions. The detailed reasons for the reductions are described in Section 4 and 5, however they are summarized in the following points.
Natural Questions (1) Questions for textual answer tagging: questions without any Wikipedia article in the top 10 results from the search engine were discarded. (2) Questions for textual answer tagging: reduced to those where tags generated by the LLM were successfully parsed. (3) Questions with the correct passage: filtered to questions with passages correctly answered the questions. (4) Correct textual answers: filtered to questions with correct textual answers. (5) Questions with verified answer entities: questions without any correct answer entities were discarded. (6) Questions with verified topic entity: questions without any correct topic entities were discarded. (7) KBQA/MRC examples: the final dataset examples differ from those in the corresponding previous steps due to several manual interventions. These include deduplication and manual entity linking.
Template-based Questions (1) After filtering: questions without high similarity to their original form were filtered out. (2) After verification: human verification ensured the meaningfulness of questions.

heading: E.2 Outcome
text: Table 10 provides detailed statistics of PUGG, including unique topics, answers, and relations for both natural and template-based questions. Table 11 shows the distribution of examples across different template types used in the template-based questions.

heading: F KBQA Baseline Prompts
text: We adapted the LLM prompt from KAPING (Baek et al., 2023) by translating and slightly modifying it to emphasize the need for listing entities in their non-inflected form. The adapted prompt is presented in Table 12.

heading: Template name Train Test

heading: Future Work
text: The PUGG dataset and our pipeline’s comprehensive implementation, findings, and detailed statistics from the PUGG dataset construction provide valuable insights for future research. Further-
more, the evaluation of baseline models on this dataset reveals its challenging nature, underscoring its potential to advance the field and contribute to developing more robust QA systems