Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata
Authors: Silei Xu
Shicheng Liu
Theo Culhane
Elizaveta Pertseva
Meng-Hsi Wu
Sina Semnani
Monica Lam
Venue: EMNLP 2023 Main (2023)
Keywords: knowledge base question answering
Wikidata
semantic parsing
large language models

Abstract: While large language models (LLMs) can answer many questions correctly, they can also hallucinate and give wrong answers. Wikidata, with its over 12 billion facts, can be used to ground LLMs to improve their factuality. 

This paper presents WikiWebQuestions, a high-quality question answering benchmark for Wikidata. Ported over from WebQuestions for Freebase, it consists of real-world data with SPARQL annotation. 

This paper presents a few-shot sequence-to-sequence semantic parser for Wikidata. We modify SPARQL to use the unique domain and property names instead of their IDs. We train the parser to use either the results from an entity linker or mentions in the query. We fine-tune LLaMA by adding the few-shot training data to that used to fine-tune Alpaca. 

Our experimental results demonstrate the effectiveness of this methodology, establishing a strong baseline of 76% and 65% answer accuracy in the dev and test sets of WikiWebQuestions, respectively. By pairing our semantic parser with GPT-3, we combine verifiable results with qualified GPT-3 guesses to provide useful answers to 96% of the questions in dev. We also show that our method outperforms the state-of-the-art for the QALD-7 Wikidata dataset by 3.6% in F1 score.

Fine-tuned LLMs Know More, Hallucinate Less
with Few-Shot Sequence-to-Sequenee Semantic Parsing over Wikidata
Silei Xu*
Shicheng Liu" Theo Culhane Elizaveta Pertseva
Meng-Hsi Wu' Sina J. Semnani Monica S. Lam
Computer Science Department, Stanford University Stanford, CA
(silei, shicheng, tculhane, pertseva, sinaj, lam) (cs estanford.edu 'iilly.ai jwu@ailly.ai
Abstract
While large language models (LLMs) can an- swer many questions correctly, they can also hallucinate and give wrong answers. Wikidata, with its over 12 billion facts, can be used to ground LLMs to improve their factuality.
Entity Linker
place@
WikiSP [Semantic Parser)
('A Bronx Tale',
'Q 130705')
This paper presents WikiWebbuuestioss, high- quality question answering benchmark for Wikidata. Ported over from WebQuestions for Freebase, it consists of real-world data with SPARQL annotation.
SELECT DISTINCT MBERE
wd:01130705 wdtafilming location
This paper presents a few-shot sequence-to- sequence semantic parser for Wikidata. We modify SPARQL to use the unique domain and property names instead of their IDs. We train the parser use either the results from entity linker or mentions in the query, fine-tune LLaMA by adding the few-shot training data to that used to fine-tune Alpaca.
WIKIDATA
No Response
Response
From Wikidatto, the fiming location 'A Branx Tale' includes New New York
Our experimental results demonstrate the ef- fectiveness of this methodology, establishing a strong baseline of 76% and 65% answer ac- curacy in the dev and test sets of Wiki Questions, respectively. By pairing seman- tic parser with GPT-3, we combine verifiable results with qualified GPT-3 guesses to pro- vide useful answers to 96% of the questions in dev. We also show that our method outperforms the state-of-the-ar art for the QALD-7 Wikidata dataset by 3.6% in Fl score.
[GPT-3]
GPT-3 guesses that the movie took place in Bronx, New York
An Overview of WikiSP. An entity linker is used to link entities in the user query to their unique ID in Wikidata; e.g. "A Bronx Tale" is linked to entity ID "QI 130705". The query and entity linker outputs are fed to the WikiSP semantic parser to produce modified version of SPARQI, where property IDs (e.g. "P915") are replaced by their unique string identifiers (e.g. "film- ing locatioct"")) applying the query Wikidata fails return result, we default to GPT-3, labeling the result as a GPT-3 guess. Returned answers are presented the context of the query, the swer is acceptable; if not, we also show the guess from GPT-3. Here WikiSP mistakenly uses instead of "narrative location" the user detects the mis- take, thumbs down the answer, and the GPT-3 answer is provided.
Figure I:
1 Introduction
Large language models (LLMs) such GPT-3 can answer open-domain questions without access to external knowledge or any task-specific training examples. However, LLMs are prone to halluci- nate (Bang et al., 2023), while using a convinc- ing and confident tone. This may cause signifi- cant harm as people inereasingly accept LLMs as knowledge source (Goddard, 2023; Weiser, 2023).
tell if the
location"
Equua contribution Codee data, and model are available at ht tttps:/ httpt//tttbt/.t/////// cond wi k dat.a emn1

---

7% Incorrect GPT
4% Incorrect GPT
semantic parsers for large knowledge bases such Wikidata.
6% incomplate GPT
Pretrained with the internet corpora, LLMs are already familiar with the syntax of formal query languages such as SQL. (Hu et al., 2022; Poesia et al., 2022: Li et al., 2023; An et al., 2023; Nan et al., 2023; Arora et al., 2023). When given simple SQL schemas, they can perform zero-shot semantic parsing of simple natural language queries into for- mal queries. Unlike Freebase, the KB used in most of the KBQA semantic parsing research, Wikidata does not have a pre-defined schema, making a much harder problem. has 150K domains, 3K applicable properties, and 107M entities, each of the properties and entities are uniquely identified with PIDs and QIDs, respectively. While zero-shot LLMs can generate SPARQL queries for the easiest and most common questions, they do not know all the PIDs and QIDs, and nor is it possible include them in a prompt.
15% Correct GPT
27% Incomplete GPT
66% Correct GPT
76% Verified from WildSP
GPT-3 only
WikiSP GPT-3
Figure 2: Distribution of correct, incomplete, and incor- rect answers for the WikiWehQuestions dev set, when GPT-3 is used alone and when combined with WikiSP,
In contrast, traditional knowledge base ques- tion answering (KBQA) grounded with given knowledge base. Semantic parsing (SP) has been widely used to tackle this challenging task, where the questions are first parsed into a logical form and then executed to retrieve answers from the knowledge base. It has better interpretability than GPT-3 and other information-rettreww basdd ap- proaches (Dong et al., 2015; Miller et al., 2016; Sun et al., 2018, 2019) where answers are predicted directly.
This paper presents WikiSP. a few-shot sequence-to-scqueeee semantic parser for Wikidata that translates a user query, along with results from an entity linker, directly into SPARQL queries. To handle the 100M+ entities in Wikidata, we train the parser use either the entity linker results mention in the query; to handle the 150K domains and 3K applicable properties, we modify SPARQL to use domain and property names instead of their unique QIDs and PIDs, respectively. We fine-tune a LLaMA (Touvron et al., 2023) with a few-shot training set along with the instructions used to fine- tune Alpaca (Taori et al., 2023).
To handle large knowledge bases, previous SP- based approaches tend to use a multi-stage pipeline of sub-tasks, starting with extracting the relevant subgraph based on entities detected in the ques- tions (Yih et al., 2015; Luo et al., 2018). Such approach struggles with questions that have large search space and fails to understand questions that refer to information missing in the knowledge graph. Having to retrieve the relevant subgraphs to create the logical form conflates query resolution with semantic parsing, rendering classical query optimization inapplicable.
1.2 A New Dataset: WikiWeb
End-to-end seq2seq translation, on the other hand, has mainly been used schemas of rela- tively small relational databases (Yu et al., 2018; Xu et al., 2020a,b) and web APIs (Campagna et al., 2017: Su et al., 2017). To handle large knowledge graphs, recent work proposed retrieving (1) infor- mation on linked entities, (2) exemplary logical forms relevant to the query (Gu et al., 2021; Ye et al., 2022), and (3) schemas as context to seman- tic parsing (Shu et al., 2022). Others use induction iterative methods to generate complex logical forms (Cao et al., 2022b; Gu and Su, 2022).
Most of the widely-used high-quality benchmarks for KBQA are based Freebase (Bollacker al., 2008) which has been shut down since 2015. With outdated knowledge,
hard to compare the re- sults with modern LLMs such as GPT-3, since an- swers have changed over time for most of the ques- tions. Wikidata, despite being the largest and most popular knowledge base nowadays, has very few datasets annotated with SPARQL queries; they are either extremely small (Usbeck et al., 2017) syn- thetic (Saha al., 2018).
We migrated the popular WebQuestionsSP (Yih et al., 2016) benchmark from Freebase to Wiki- data, with updated SPARQL and up-to-date an- swers from the much larger Wikidata.
1.1 Few-Shot Seq2Seq Semantic Parsing
This paper investigates how we can leverage large language models (LLMs) to create seq2seq neural

---

1.3 Complementing Large Language Models Trained on Wikipedia and all of the internet, LLMs answer many questions directly. Unfortunately, the user cannot tell if the answers are correct, thus requiring them to fact-check every answer.
and backing it up with GPT-3 guesses. WikiSP can provide verifiable results for WiliWebQuestions 76% of the time and improves the guesses by GPT- 3, resulting in errors only 4% of the time (Figure 2).
Unlike humans, GPT-3 always sounds definitive even when they are wrong by providing specific and plausible facts. For example, on the question "what is the biggest country in Europe by popula- tion?", GPT-3 answers "Germany", when the an- swer is "Russia". Or, on the question, "where does the name Melbourne come from?" GPT-3 answers "Melbourne comes from the Latin word 'melbur- num' meaning 'blackburn' or 'blackbird'." .", but in reality, Melbourne is named after William Lamb, 2nd Viscount Melbourne. It is not possible to tell when GPT-3's answers are wrong, and every an- swer needs to be fact-checked.
Related Work
2.1 KBQA
The KBQA task aims to make large knowledge bases accessible by natural language. One com- mon approach is semantic parsing where natural language query is translated into formal logical form, which then executed to retrieve an answer from the knowledge base. To handle large KBs, one method is to formulate SP as a multi-staged search problem by retrieving entities and expanding the graphs according to the relationships between their properties and the query (Yih et al., 2015, 2016; Luo et al., 2018). Lan and Jiang (2020) add constraints the staged query graph generation method. Another popular method is seq2seq models obtained by fine-tuning pretrained language models. Das et al. (2021) first find other queries that contain semantically similar subparts, and con- struct new logical form by combining the similar subparts of the found queries. Ye et al. (2022) search over the KB based on predefined rules to de- rive set of candidate logical forms, rank them, and generate the final logical form. Cao et al. (2022b) first generate a "sketch" program and then fill in its arguments. Gu and Su (2022) use dynamic pro- gram induction to generate query structures. Based on a user query, Shu et al. (2022) retrieve entities, example logical forms, and related schema. Unlike FreeBase, Wikidata does not have a fixed schema.
Semantic parsers can be used to complement LLMs as they are interpretable; their results are grounded in Wikidata, which we assume to be cor- rect. It is possible for semantic parsers to misun- derstand a query, but by providing the answer in the context of the query, the user can spot the error.
We propose getting the best of both worlds by answering the question with WikiSP if possible. Otherwise, we report GPT-3's guesses by prefac- ing with: "GPT-3 guesses that" (Figure 1). In this way. the user can have full confidence with the answers from the former, while also benefiting from the latter. It is easier for users to fact-check answer than trying to find the answer.
1.4 Contributions
WikiWebhuestions, high-quality semantic parsing dataset for Wikidata, migrated from the popular WebQuestions dataset for Freebase.
Another approach to KBQA is based on graph retrieval (Dong et al., 2015; Miller et al., 2016; Sun et al., 2018, 2019; Mavromatis and Karypis, 2022; Sen al., 2021: Vivona and Hassani, 2019; Verga et al., 2021). It predicts the answers directly within the subgraph extracted based on the topic entity in the question. Yu et al. (2023) combine seman- tic parsing with retrieval and achieve the state-of- the-art on the WebQuestionsSP dataset (Yih et al., 2016). However, retrieval-baas methods cannot handle entire categories of questions, such as ques- tions with no available answer and questions like "the tallest mountain" where no entities are men- tioned by name They have poor interpretability and do not support query optimization.
WikiSP, few-shot Seq2Seq semantic parser by fine-tuning LLaMA with few shot training set. We improve the learnability of SPARQL queries by replacing the IDs of properties and domains with their unique names; we tolerate errors in entity linking by accepting mentions in the queries as entities. We establish a first, strong baseline of 76% and 65% answer accuracy for the dev set and test set of our new WikiWebQuestions benchmark, respectively. We also demonstrate that our method surpasses the state of the art for QALD-7 wikidata set by 3.6% in F1 score.
We improve GPT-3's trustworthiness by first returning interpretable results from semantic parser

---

2.2
KBQA Benchmarks
and then combine that information with embed- dings computed over every entity in the knowledge base to predict which entity the mention is most likely to be referring to. Prior to ReFinED, the state-of-the-art was ELQ (Li et al., 2020). They similarly generate embeddings for each entity in the knowledge base, and then use the predicted mentions of entities combined with these predicted embeddings to generate likely entities.
Most of the early KBQA benchmarks are based on Freebase (Berant et al., 2013; Yih et al., 2016; Tal- mor and Berant, 2018). Recently, new benchmarks have been created for Wikidata (Cao et al., 2022a; Saha al., 2019). However, these benchmarks are created using rule-based synthesis or paraphrases, which are easier for semantic parsers. CSQA col- lects Souman-writtee questions for single triples and constructs complex questions using fixed rules with very limited natural language variety (Saha et al., 2019). KQA Pro first synthesizes queries with canonical natural language and then crowdsources human paraphrases (Cao et al., 2022a). Campagna et al. (2019) show that a model can achieve signifi- cantly higher accuracy over paraphrased data com- pared to real-world data even for untrained queries. Thus, we base our WikiWebQuestions dataset on WebQuestionsSP (Yih et al., 2016), where data are collected from real-world users using the Google Suggest API.
Semantic Parsing for Wikidata
Wikidata is the largest public knowledge base with over 12 billion facts represented by subject- predicate-object triples using 100+ million entities and 10,000 properties. 3,000 of the properties are useful for answering natural language questions, whereas the rest are used link data in Wikidata with external library catalogs and databuse IDs. Entities and properties are given unique identi- fiers, QIDs and PIDs, respectively. For example, the fact that Joe Biden is the president of the US can be represented where P39 is the PID for property position held, Q6279 and Q11696 are QIDs for Joe Biden and the president of the United States, respectively.
triple (Q6279, P39, Q11696),
2.3
LLMs for Semantic Parsing
Shin et al. (2021) show the promise of few-shot prompting LLMs for semantic parsing. They use constrained decoding to enforce the syntax of the formal language, and achieve comparable results with a smaller fine-tuned BART model (Lewis et al., 2020) on datasets with small database schemas. Rubin al. (2022) fine-tune a small retriever to obtain the most relevant few-shot examples to use for each input. Niu prompted Codex model to break down the natural language input to make the task easier for smaller semantic parser. LLMs have also been applied to semantic parsing on relational databases (Hu et al., 2022; Poesia et al., 2022; Li et al., 2023; An et al.. 2023; Nan et al., 2023; Arora et al., 2023). The schemas used in these projects are very small when compared to Wikidata.
3.1 Query Format
Unlike relational databases and Freebase, Wikidata has no predefined domains or types, Any entity can have an arbitrary set of properties. However, even though Wikidata is property- all named entities have or more instance of properties to some domain entity; domain entities are organized into a hierarchy with the subclass of property.
(2023) use a few-shot
Note that the names of domain entities and prop- ertics are unique. Non-domain entities, on the other hand, can be ambiguous. For example, "Lincoln" can refer to the president, a car brand, a sparrow, an aircraft, and many different cities.
We posit that it is impossible for LLMs to memo- rize the QIDs and PIDs for domains and properties. We modify the format of SPARQL queries to use the more mnemonic property name, instead of its PID. Similarly, we use entity names for domains. For example, the original SPARQL for the query "What car models does GM make?" is
2.4 Entity Linking
Entity linking involves finding the named entities in query, and linking them to the corresponding entities in the knowledge graph so that the query can be exccuted using the proper entities ref- erence points. The current state-of of-the-art entity linking model on the WebQuestionsSP dataset is ReFinED (Ayoola et al., 2022). They use bidirec- tional transformer on the query to predict the most likely mentions of named entities within a query,
SELECT DISTINCT 2x WHERE
2x wdt:.31/wwd wd:Q3231690
2x wdt:P176 wd:Q81965.
This says that we are seeking x, where x is transitively either an instance of (wdt:311)

---

subclass of (wdt:P279) of an automobile model (wd:Q3231690), and z has General Mo- tors (wd:Q81965) as the manufacturer (wdr:P176). Note wdt is the prefix for Wikidata property, and wd is for Wikidata entity.
the semantic parser produces the semantic parse of T in our modified SPARQL format.
For the example above, the SOTA ReFinED en- tity linker (Ayoola et al., 2022) returns {Generaa Motors, Q81965)). Unfortunately, it misses the entity automobile model (Q3231690), term not usually considered be an entity.
With our modification, the query becomes:
SELECT DISTINCT ?x WHERE {
?x wdtrinstanc_ood wodtinssane__ofivdrsubbbba of/wddrsubclass_of wdaautomobllee model.
3.2.2 Recovering from Entity Linker Errors
?x watuaaufrr wd:Q81965.)
We want our semantic parser to be able to recover from mistakes by an entity linker. That is, the semantic parser should use entity linking when it is helpful, but it should still try to predict the right logical form when the linker fails.
For non-domain entity QIDs, we also accept string in lieu of a QID in case of entity linking errors. At inference time, we use simple heuristics to resolve the string to a QID before applying the query. For example, "wd:Q81965" in the query may be replaced with "wd:GM". See Section 3.2.2 for more details.
The semantic parser is trained to accept, along with the user query, an optional set of potentially useful QIDs from the entity linker. We include samples where some of the supplied linked entities are not used in the gold answer, as well as samples where there are missing linked entities. For the latter, we use mentions the original query in lieu of the QIDs. At inference time, we use the men- tions to look up the QIDs Wikidata. If multiple matches exist, the most popular entity is returned. An example is shown in Appendix A.
Normally, refrain from changing standard query notations since LLMs have been pretrained on them. However, we posit that learning this new syntax is much easier than learning the PIDs and QIDs. Our experimentation with few-shot prompt- ing suggests that LLMs can easily adjust to this format.
3.2 Entity Linking
With the above example where the entity linker misses "automobile model", the semantic parser is likely to predict "car model" by copying from the user query. We search "automobile model" among aliases in domains to find the correct QID. This design allows the model to potentially recover from entity-linking failures.
Linking entities for WikiiebQuestions is particu- larly difficult. First, since the dataset is collected from real- worl-world questions without prompting the users for more information, users tend refer to their entities of interest without using their full names. Second, the questions are generally short with very limited context, making it harder to disambiguate among entities with similar names. Lastly, many QIDs in Wikidata are used to repre- sent terms not generally known "named entities", For example, domain entities are often ignored by entity linker models, in "What is the biggest country in Europe by population?", both "country" (Q6256) and "Europe" (Q46) are required to con- struct the correct SPARQL, but entity linkers only provide "Europe" and ignore "country".
WikiWebQuestions (WWQ) Dataset
Despite being the most popular large knowledge base for long time, existing benchmarks Wiki- data with labeled SPARQL queries are unfortu- nately either small or of low quality. On the other hand, benchmarks over the deprecated Free- base still dominate the KBQA research with better- quality data. For example, the WebQuestions (Yih et al., 2015) dataset was collected by using Google Search API instead of human paraphrasing or syn- thesis. As a result, it is much more natural and truly reflects the real-world questions users may ask. This dataset is later annotated with SPARQL over Freebase, named WebQuestionSSP (Yih al., 2016). Examples with no legitimate SPARQL retrieve answers from Freebase are dropped. In total, WebQuestionsSP consists of 3098 examples in the training set and 1639 in the test set.
3.2.1 Semantic Parsing with Entity Linking To handle ambiguous entities, we use an entity linker to first find the domain names and QIDs of the entities mentioned in the text. We train semantic parser that accepts users' input along with the results produced by the entity linker.
Formally, given a user input T, and a set of entity linker results (e, q). where e is the name (default label) Wikidata gives to an entity and is its QID,

---

We migrated WebQuestionsSP, the best collec- tion of natural language questions over a general knowledge graph, from Freebase to Wikidata, with the help of an automatic tool we developed, based on Google's s entity mapping² and Wikidata's re- lation mapping'. About 60% of the dataset was automatically converted. One of the authors of this paper, who did not participate in model tuning, manually converted those instances that failed to convert automatically.
5.1 Entity Linking
We use ReFinED (Ayoola et al., 2022) for entity linking, which is the current state of the art for WebQuestionsSP. As discussed before, Wikidata treats many common terms such as "country" as named entities and assigns them QIDs. To fine-tune ReFinED to learn such terms, add the question and entity pairs from the training of WikiWeb- Questions the data used train ReFinED's s ques- tions model.
We run 10 epochs of finetuning using the default hyperparameters suggested by Ayoola et al. (2022). For each identified entity, we provide the mention in the original utterance, the QID, as well as its domain in plain text. The information is appended to the utterance before being fed into the neural semantic parsing model.
4.1 Migrating WebQuestionSSP to Wikidata
Here are the major decisions we made in migrating WebQuestionsSP dataset to Wikidata. While much bigger, Wikidata does not necessarily contain all the information available in Freebase. For example, it lacks countries' trade partners, hence we drop all such questions from the WebQuestionsSP dataset. If multiple paths can lead to the correct answer, we choose the path that provides the most com- plete answers and has the best availability among entities in the same domain. For example, when asking for books written by an author X, we can either search for books whose author is or find notable works of X that are books. While the latter is more efficient, the property notable works is not always available for all authors and it often does not provide complete list. Thus, we annotate such examples using the former representation.
5.2 The WikiSP Semantie Parser
We prepare the training data with entities provided by fine-tuned ReFinED. Comparing with the gold entities, ReFinED provides extra entities in 215 cases, while missing at least one entity in 137 cases. When ReFinED failed to produce the correct en- tities, we replace the missing QIDs in the logical form with the corresponding mention of the entity in the question. During evaluation, if a mention of an entity is predicted by the model, we look up the QID using the Wikidata "whsearchentities" API 4 We fine-tune LLaMA with 7B parameters be- cause it has been shown to perform well despite its relatively small size (Touvron et al., 2023). We include the Alpaca (Taori et al., 2023) instruction following data, which was derived using the self- instruct (Wang et al., 2023) method, in our training data. The training data samples in WikiWebQues- tion start with the following instruction: "Given a Wikidata query with resolved entities, generate the corresponding SPARQL. Use property names instead of PIDs.". We concatenate the resolved entities and the user utterance together as input. We up-sample the WikiWebQuestion fewshot set 5 times and train for 3 epochs using 2e-5 learning rate and 0.03 warmup ratio.
We also cleaned up the original dataset. The dataset contained questions like "who does Ronald- inho play for now in 2011?". We drop the appended year as it conflicts with "now" in the utterance, and it would refer to the live information in Wikidata.
In total, we dropped 9% of the examples from WebQuestionsSP and created a training, dev, and test set of 2431, 454, and 1431 samples, respec- tively. Given that Wikidata has 100 million entities and 3,000 useful properties for answering ques- tions, the training data set is woefully inadequate and can be considered as a "fewshot" training set at best.
5 Implementation
5.3 Executing Queries on Wikidata
This section discusses the implementation details of the entity linker and the WikiSP semantic parser.
SPARQL queries are used to retrieve answers from the Wikidata SPARQL endpointS, Since Wikidata
hhttps://evvelopers: http:///eeeoooerrr.ggeeeeeo wow.:e...ooooec.ooooo com/ freebase
"hstps://www wwwp:wwwwwwwiiiaiaaaaaaaaaaaaaaa http::/wwwwwiiiaaaaaa.oo////apa/ org/w/api.ghp?
aetiorswbsearchentttiee
Phttps: //www////wwwwwww www.wwikkiaee diiidtt..gg////
"https:///wwl http://wwwwwwwwwak../.c wtt.daaata....vviee.o
wi tkkidattawikiiPojjeet httpoo/eetteeeeeeeeeaaaaaaaeeeee
Wi :lkkidaata:sPARO__qqeee

---

EM
F1
EM
F1
WikiSP (ours)
65.5
71.9
WikiSP (ours)
75.6
76.9
No Entity Linking
66.5
67.6
Table I: Results of WikiSP on the WWQ test set.
No mentions, trained with ReFinED
73.3
75.0
No mentions, trained with Oracle entities
72.2
73.4
PIDs and QIDs for properties & domains
73.6
74.7
is actively being updated, the gold SPARQL can be easily re-executed to acquire up-to-date answers, allowing the benchmark compare with forthcom- ing iterations of large language models.
Table 2: Ablation results of WikiSP on the WWQ dev set.
mention, alternative QID. Our ablation study on this feature tested two training strategies: ReFinED. The entity linker tuples are produced by fine-tuned ReFinED, which may be missing entities in the gold target. The data show that generating unseen QIDs is needed for missing entities.
6 Experiments
In this section, we evaluate WikiSP on WikiWeb- Questions and demonstrate how it can be used to complement large language models such GPT-3,
6.1 Semantic Parser Results
Oracle. The entity linker tuples are exactly all the entities used in the gold. The model would only encounter missing QIDs at test time when ReFinED fails to generate all the necessary QIDs.
We evaluate our model with two different answer accuracy metrics: (1) exact match (EM): the per- centage of examples where the answers of the pre- dicted SPARQL exactly match the gold answers, and (2) Macro Fl score (FI); the average Fl score for answers of each example. The evaluation re- sults are shown in Table 1. Our approach achieves 65.5% exact match accuracy and 71.9% Fl score the WWQ dataset.
The answer accuracy of the model using en- tity linked tuples from ReFinED ("No mentions, trained with ReFinED" in Table 2) lags by 2.3% when compared against our best model. The model using Oracle ("No mentions, trained with Oracle entities" in Table 2) lags by 3.4%. These results indicate that allowing mentions is useful for recov- ering from entity linking errors.
As reference, the current state-of-the-art result on the original WebQuestionsSP dataset for Free- base is 78.8% Fl (Yu et al., 2023). The result was obtained with a combination of semantic parsing and retrieval. The Wiki Wibhueetuesssooss dataset is slightly different, as discussed above. More signifi- cantly, unlike Freebase, Wikidata does not have fixed schema and ours is end-to-end, seq2seq semantic parser.
6.2.3 Names vs. IDs for Properties & Domains Our logical form replaces PIDs with property names, and domain-entity QIDs with the domain names. Here we evaluate the effectiveness of this query format. We compare our approach with the original SPARQL where all properties and entities are represented with PIDs and QIDs. Our ablation study shows that our representation with property names and domain names improves the answer curacy by 2.0% (Table 2). This shows that LLMs can adapt to changes in query notation with fine- tuning, and it is easier to learn names than remem- bering random IDs. If we did not allow mentions in the predicted logical form, the replacement of QIDs with their names is likely to be more significant.
6.2
Ablation Experiments
6.2.1
Entity Linking
Our first ablation study evaluates the need for entity linking with ReFinED, by replacing it with simply using the LLM to detect entities as mentions. In this experiment, all entity IDs in the training data are replaced by their mentions; during inference, we map the predicted entities to their actual QIDs according to Section 3.2.2.
6.3 Complementing GPT-3
The results show that replacing the neural entity linker with just using mentions reduces the exact match by 9.1% and the Fl score by 9.3%. This suggests that entity linking important.
LLMs like GPT-3 can answer many questions on general knowledge correctly; however, they may also hallucinate. WWQ is representative of popular questions, so we expect GPT-3 to perform well. We use text-davinci-O00 with the temperature set to 0 to evaluate GPT-3's performance on WWQ.
6.2.2 Allowing Mentions as Entities
Our logical form is designed to recover from entity linking errors by allowing entities be specified by
On the dev set of WWQ, GPT-3 answers 66.4%

---

of the questions correctly and provides incomplete answers to 26.5% of the questions. For example, when asked "What does Obama have a degree in?", GPT-3 correctly identifies President Obama's 's po- litical science degree, but fails to mention his law degree. In total, GPT-3 gives wrong answers to 7.1% of the questions.
potentially recover from missing entities, can- not recover from incorrect entities. This is espe- cially common for character roles, as some char- acter roles have different entities for books and movies or even different series of movies. Some- times WikiSP located the correct mention from the question, but the lookup failed. For example, the model located the mention of the event "allied invasion of France" in question "where did the al- lied invasion of France take place?", but failed find the corresponding entity from Wikidata by the name.
For this dev set, we can give definitive answers to 75.6% of the questions with WikiSP (Table 2). For the rest of the questions (24.4%), accounting for the overlap between the GPT-3 and our semantic parser's results, the percentages of guessing cor- rectly, incompletely, and incorrectly are at 15.2%, 5.5%, and 3.7%, respectively (Figure 2).
6.4.3 Errors Beyond Entity Linking
Semantic parsing in Wikidata challenging as there are no predefined schemas, and there are 150K domains and 3K applicable properties, Some representative mistakes include the following: Wrong property (17.1%). 17.1% of the er- rors are caused by predicting the wrong property. Some of the examples require background knowl- edge to parse. For example the answer of the ques- tion "what did martin luther king jr do in his life?" should return the value movement, while the model predicts occupation. Properties are a chal- lenge in Wikidata because as illustrated here which property to predict depends on the entity itself.
In summary, the combination of GPT-3 and WikiSP makes it possible give definitive, cor- rect, and complete answer three quarters of the time for the dev set. Users can also benefit from GPT- 3's guesses the rest of the time at a 3.7% error rate, which is about half of the original error rate.
6.4 Error Analysis
We analyzed the 111 examples in the WWQ dev set where the model failed.
6.4.1
Acceptable Alternative Results (18.0%)
Our analysis shows that 18.0% of the "errors" can actually be deemed to be correct.
Missing domain constraint (5.4%). Another common problem is missing the domain constraint. For example, the model correctly identifies that property shares border with should be used for question "what countries are around Egypt?". How- ever, it does not limit the answer to countries only, thus extra entities are returned.
Reasonable alternate answers (11.7%). In 11.7% of the cases, the model predicts an alterna- tive interpretation to the question and returns a rea- sonable answer that is different from the gold. For example, the gold for question "what did Boudicca do?" uses the position held property, while the model predicts occupation property. Both are con- sidered valid answers to the question.
7 Experiment with QALD-7
Reasonable alternative SPARQL but no an- swer was retrieved (6.3%). In another 6.3% of cases, the model predicts reasonable alterna- tive SPARQL, but the SPARQL returns no answer. Sometimes, since the information for the "correct" property is missing, the question is represented with a similar property. For example, since resi- dence property is missing for Patrick Henry, the gold SPARQL for "where did Patrick Henry live?" uses place of birth instcad, while our model pre- dicts residence.
For another evaluation of WikiSP, we apply our model on Task 4 from QALD-7 (Usbeck et al.. 2017) dataset. QALD-7 is part of the QALD (Ques- tion Answering over Linked Data) which a series of challenges started in 2011 known for their com- plex, manually created questions. It mainly focuses on DBpedia, but QALD-7's Task 4 is engineered for Wikidata. The task includes 100 train examples, which we use to fine-tune our model and 50 test examples. There is no dev set.
We choose QALD-7 as it is manually crafted dataset with complex questions. We avoid datasets built on synthetic or human pannaa--rappraeeed data, such as CSQA (Saha et al., 2018) and KQA-Pro (Cao et al., 2022a). As they have limited natural lan- guage variety between the training and evaluation
6.4.2 Errors in Entity Linking (35.1 ))
The biggest source of errors is entity linking. En- tity linker failed to provide the correct entities in 35.1% 1% of the failed examples. While WikiSP can

---

EM
F1
12%
18% Incorrect
STAGG (Yih et al., 2016)
19.0
GGNN (Sorokin and Gurevych, 2018)
21.3
WDAqua (Diefenbach al., 2017)
40.0
20%
16% Incomplete GP
Verifie
WikiSP (Ours)
38.0
43.6
Incompete
62% Correct
34% Correct GPT
Table Evaluation results of WikiSP on QALD-7 Task 4 and comparison with prior work.
GPT-3 only
WikiSP GPT-3
data, models can get artificially high accuracy. For example, simple BART based model can achieve over 90% accuracy KQA- Pro even without an entity linking module (Cao et al., 2022a).
Figure 3: Distribution of correct, incomplete, and incor- rect answers for the QALD-7 test set, when GPT-3 is used alone and when combined with WikiSP.
The QALD-7 test set provides both the SPARQL queries as well as the answers. To double-check the correctness of the QALD-7 dataset, we applied the 50 gold queries of the test set to Wikidata and found that 4 did not return an answer. We hypothesize that the discrepancy is caused by the change in Wikidata structure/quantity of information. We evaluate WikiSP by comparing the answers where possible, and by comparing the generated SPARQL syntactically otherwise.
additional 34% correct, 16% incomplete, and only 12% wrong.
7.3 Discussion
We did not conduct error analysis on the perfor- mance of QALD-7 it has no dev set. The author evaluating GPT-3 noted that the test of QALD-7 is much more complicated than the training data (of just 100 samples), with most of the queries con- taining multiple properties. This explains the lower accuracy of WikiSP on QALD-7 when compared to WikiWebQuestione which has few-shot training data set with a similar distribution as the test set.
For this experiment, we use the same hyper- parameters and data format as described Sec- tion 5.3. In addition to the training data for WikiSP, we also include the QALD-7 train samples, upsam- pled 20 times.
This result suggests that the performance of WikiSP depends heavily on a good few-shot train- ing data for fine-tuning the LLMs. We hypothesize that we can increase the performance of WikiSP in handling less popular questions with a better, possibly synthesized, training dataset.
7.1 QALD-7 Results
Our model achieves 38% accuracy the QALD-7 dataset and outperforms the Fl score of the state- of-the-art WDAqua (Diefenbach et al., 2017) by 3.6%, as shown in Table Note that WDAqua is based on retrieval, whereas WikiSP is based on sequence-to-sequence semantic parsing. QALD- 7 (Usbeck et al., 2017) reports WDAqua the winner of the leaderboard with 55.2 F1, however the authors of WDAqua reported 40.0 Fl in their papers (Diefenbach et al., 2017).
8 Conclusion
We have created new high-quality benchmark, WikiWebQuestions, for large knowledge- ques- tion answering. The dataset is based on the popular WebQuestionsSP dataset with natural questions, annotated with SPARQL for Wikidata.
7.2 Complementing GPT-3 on QALD-7
We establish first, strong baseline of 65% an- swer accuracy and 72% F1 score for WikiWar Questions. This is achieved by fine-tuning LLaMA with a few-shot training data set using a SPARQL query format modified for semantic parsing.
Similar to WWQ, we also assess the combination GPT with WikiSP on QALD-7 as shown in Fig- ure 3. The GPT model used was "text-davinci-002". Since there is no validation set and the test set is already very small, one of the authors who was not involved in training finetuning the model evaluated GPT-3 the test set.
We show that we can reduce the hallucination of large language models like GPT-3 by grounding with a semantic parser. For the dev set of Wiki- WebQuestions, this combination approach provides useful information for 96% of the questions in the dev set of the benchmark. More importantly, it gen- erates verifiable answers for 76% of the questions.
GPT-3 is fully accurate on 62% of the questions, 20% incomplete, and 18% wrong. With our ap- proach, we can provide 38% verifiably good an- swers from WikiSP; the guesses of GPT-3 get an

---

Limitations
Acknowledgements
While applications of large language models seem to expand every day, this paper mainly focuses on factoid question answering. Long-form text genera- tion, for example, is outside the scope of the experi- ments of this paper, but the methodology described here may be extended to this setting in the future. Even though knowledge bases source of facts, large portion of the knowledge available in digital form (c.g. Wikipedia, news arti- cles, etc.), is not organized into knowledge bases. As such, the results of this paper can be considered complementary to the larger body of fact-checking research based on free text.
This work is supported in part by the National Sci- ence Foundation, the Alfred Sloan Foundation, the Verdant Foundation, Microsoft Azure AI credit, KDDI, JPMorgan Chase, and the Stanford Human- Centered Artificial Intelligence (HAI) Institute. We also thank the reviewers for their valuable com- ments and suggestions.
important
References
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou, 2023. Skill-based few-shot selection for in-context learning.
Our semantic parser can be used to verify an- swers from LLMs. However, this additional round of running the semantic parser and querying Wiki- data increase the response latency, which may be noticeable by end-users of such systems.
Aseem Arora, Shabbirhussain Bhaisaheb, Harshit Nigam, Manasi Patwardhan, Lovekesh Vig, and Gau- tam ShrofT. 2023. Adapt and decompose: Effficient generalization text-to-sql via domain adapted least- to-most prompting.
All of our datasets and experiments ducted for English. Expanding to other languages, while possible (Moradshahi al., 2020) are outside the scope of this work.
con-
Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. ReFinED: An efficient zero-shot-capahle approach to end-to-end entity linking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies: Industry Track, pages Hybrid: Seattle, Washington Online. As- sociation for Computational Linguistics.
Our experiments were performed using GPT-3 (davinci-002) as that was what we had access to when we started the project. Undoubtedly, the later LLMs will produce better results. Nonetheless, the need have verifiable results based on live database accesses will remain.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. Quyet V. Do, Yan Xu, and Pascale Fung. 2023. multitask. multilin- gual, multimodal evaluation of chatgpt on reasoning. ballucination, and interactivity.
Ethical Considerations
LLMs are used by millions of people everyday. We hope that this line of work will help make them more reliable for everyone, mitigating some of their potential downsides, and giving users access to more accurate information. Our use of Wikidata will enable future researchers and developers to connect their systems with a large, diverse and live knowledge graph that is updated every day. We do not anticipate any harm resulting from the methods introduced in this work.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from -anwwer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
We did not crowdsource any datasets for this paper, as the questions are converted from a previ- ous dataset and all the re-annotation and analysis is done by the authors.
Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017. Al- mond: The architecture of an open, crowdsourced. programmable virtual assistant. In Proceedings of the 26th International Conference on World Wide Web WWW 17, pages 341-350, New York, New York, USA. ACM Press.
To conduct experiments in this paper, used an estimated total of 60 NC96ads-A100 GPU hours on Microsoft Azure. Each finetuning experiment takes roughly 3 hours, and we conducted roughly 20 experiments to arrive at the results in this paper.
privacy-

---

Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Lunguage Design and Implementation, PI.DI 2019, page 394-410, New York, NY, USA. Association for Computing Machinery.
the 29th International Conference on Computational Linguistics, pages 1718-17311 Gyeongju. Republic of Korea, International Committee on Computational Linguistics.
Yushi Hu, Chia-Hsuan Lec, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627-2643, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie. Yutong Xiang. Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022a. KQA pro: A dataset with ex- plicit compositional programs for complex question answering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6101-6119, Dublin, Ireland. Association for Computational Linguistics.
Yunshi Lan and Jing Jiang. 2020. Query graph gen- eration for answering multi-hop complex questions from knowledge bases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 969-974, Online. Association for Computational Linguistics.
Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2022b. Program transfer for answering complex questions over knowledge bases. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8128 -840, Dublin, Ireland. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeutlemoyer. 2020. BART: Denoising sequence-to-sseqenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya God- bole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case- based reasoning for natural language queries over knowledge bases. Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing. pages 9594-9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433-6441, Online. Association for Computational Linguistics.
Dennis Diefenbach, Kamal Singh, and Pierre Maret. 2017. Wdaqua-coreO: A question answering compo- nent for the research community. In Semantic Web Evaluation Challenge, pages 84-89. Springer.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang. Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao. Ruiying Geng. Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can Iim already serve as a database interface? big bench for large-scale database grounded text-to-sqls.
Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Ques- tion answering over Freebase with multi-column con- volutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 260-269, Beijing, China. Association for Computational Linguistics.
Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny 7hu. 2018. Knowledge base question answering via encod- ing of complex query graphs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2185-2194, Brussels. Belgium, Association for Computational Linguistics.
Jerome Goddard. 2023. Hallucinations in chatgpt: A cautionary tale for biomedical researchers. The Amer- ican Journal of Medicine.
Costas Mavromatis and George Karypis. 2022. ReaRev: Adaptive reasoning for question answering over knowledge graphs. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2447-2458, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler. Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond i.i.d.: Three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021. ACM.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly read- ing documents. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language
Yu Gu and Yu Su. 2022. ArcaneQA: Dynamic program induction and contextualized encoding for knowl- edge base question answering. In Proceedings of

---

Processing, pages 1400-1409, Austin, Texas. Associ- ation for Computational Linguistics.
Van Durme. 2021. Constrained language models yield shotshot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mehrad Moradshahi, Giovanni Campagna, Sina Sem- nani, Silei Xu, and Monica Lam. 2020. Localizing open-ontology QA semantic parsers in a day using machine translation. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 5970-5983, Online. As- sociation for Computational Linguistics.
Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma, Yuzhong Qu, and Chin- Yew Lin. 2022. TIARA: Multi-grained retrieval for robust question answering over large knowledge base. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8108-8121, Abu Dhabi, United Arab Emirates, Association for Computational Linguistics.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing few-shot text-to- sql capabilitics of large language models: A study on prompt design strategies.
Daniil Sorokin and Tryna Gurevych. 2018. Modeling se- mantics with gated graph neural networks for knowl- edge base question answering. In Proceedings of the 27th International Conference on Computational Lin- guisties, pages 1306-3317, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yilin Niu, Fei Huang. Wei Liu, Jianwei Cui, Bin Wang, and Minlie Huang. 2023. Bridging the Gap between Synthetic and Natural Questions via Sentence De- composition for Semantic Parsing. Transactions of the Association for Computational Linguistics, 11:367-383.
Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encar- nacion. 2017. Building natural language interfaces to web apis. In Proceedings of the 2017 ACM on Con- ference Information and Knowledge Management, pages 177-186.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gul- wani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In The Tenth In- lernational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net.
Haitian Sun, Tania Bedrax-Weist and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNL.P-JCCNIP) pages 2380- 2390, Hong Kong. China. Association for Computa- tional Linguistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning retrieve prompts for in-context learning. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguistics.
Amrita Saha, Ghulam Ahmed Ansari, Abhishek Laddha. Karthik Sankaranarayanan. and Soumen Chakrabarti. 2019. Complex program induction for querying knowledge bases in the absence of gold programs. Transactions of the Association for Computational Linguistics, 7:185-200.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Com- plex sequential question answering: Towards lear- ing to converse over linked question answer pairs with a knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Alon Talmor and Jonathan Berant. 2018 The web knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 641-651, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Priyanka Sen, Armin Oliya, and Amir Saffari. 2021. Expanding end-to-end question answering on differ- entiable knowledge graphs with intersection. ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 8805 8812, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pro-
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instruction-followingg llama model. https:/ htsts//gittbbb http://ttb.c../tttssoaaaaoaaaaae wttp///ee/tttttaaaaeee stanford paca.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
graph generation: Question answering with knowl- edge base. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing ( (Volume I: Long Papers), pages 1321-1331 Beijing, China. Association for Computational Linguistics.
Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017. 7th open challenge ques- tion answering over linked data ( In Semantic web evaluation challenge, pages 59-69. Springer.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming- Wei Chang, and Jina Suh. 2016. The value of se- mantic parse labeling for knowledge base question answering. Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206 Berlin, Germany. Association for Computational Linguis- tics.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021. Adaptable and interpretable neural MemoryOver symbolic knowledge. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3678-3691, Online. Association for Computational Linguistics.
Donghan Yu, Sheng Zhang, Patrick Ng. Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, and Bing Xiang. 2023. De- caf: Joint decoding of answers and logical forms for question answering over knowledge bases. In ICLR 2023.
Salvatore Vivona and Kaveh Hassani. 2019, Relational graph representation learning for open-domain ques- tion answering.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: large-scale human- labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instrue Aligning language models with self-generated instructions. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages I 13884-13508, Toronto, Canada. Association for Computational Linguistics.
Benjamin Weiser. 2023. Here's what happens when your lawyer uses chatgpt. The New York Times.
Silei Xu, Giovanni Campagna, Jian Li, and Monica Lam. 2020a. Schema2qa: High-quality and low-cost q&a agents for the structured web. In Proceedings of the 29th ACM International Conference on Infor- mation & Knowledge Management, CIKM '20, page 1685-1694, New York, NY, USA. Association for Computing Machinery.
Silei Xu, Sina Semnani, Giovanni Campagna, and Mon- ica Lam. 2020b. AutoQA: From databases to QA semantic parsers with only synthetic training data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 422-434, Online. Association for Computa- tional Linguistics.
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022. RNG-KBQA: Generation augmented iterative ranking for knowledge base ques- tion answering, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6032-6043, Dublin, Ireland. Association for Computational Lin- guistics.
Wen-tau Yih, Ming-Wo Wei Chang. Xiaodong He, and Jian- feng Gao. 2015. Semantic parsing via staged query

---

A Examples of Recovering from Entity Linking Errors
Here, we illustrate our proposal of using entity mentions to recover from entity linking errors. In the training set, have the following example:
Query: What year did giants win the world series?
Original Gold SPARQL:
SELECT DISTINCT WHERE
Ty wdttsports_sse seartsssssoonnnee ofr.oooleeeeooroooopppeee
wd:Q265538:
wit:winner wd:Q308966; wdt:point in time ?x.
Gold Entity linker result:
World Series (QID Q265538), San Francisco Giants (QID Q308966);
ReFinED result:
San Francisco Giants (QID Q308966);
Here, the ReFinED entity linker model fails to identify the "World Series" entity. Our proposal of mentions gives the semantic parser a chance to recover from entity linker failures. To train the parser generate mentions, our training includes samples like this:
Query: what year did giants win the world series?
ReFinED result:
San Francisco Giants (QID Q308966);
Gold target:
SELECT DISTINCT 2x WHERE
ly wdt:sports seasoneneene of league_ competition:
wd:world series;
wdt:winner wd:Q308966;
wdt:point in time ?x.
The gold query mentions "world_ serress At in- ference time, our heuristics use the predicted men- tion to look up the actual Wikidata entity. For ex- ample, if wd:worlddserieee is predicted at inference time, our heuristics maps it back to wd:Q265538.


Reviews:
[{"review_id": "LL6YNU5oAC", "reviewer": "Reviewer_1ArN", "summary": "", "questions": "No examples or citations for claims made in first paragraph (this could be a footnote, “as reported in news media …”, or as part of a Figure 1).\n\n(Line 39). What is this “alarm” based on? In which contexts do people “rely on LLMs as a knowledge source”? What are the high-stakes scenarios, for example?\n\n(Lines 70-71; Line 78): Citations needed. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n> Not so helpful Figure 1. The figure shows the end results, but a more impactful Figure 1 would illustrate better more exactly what the problem is and what the plan is to address it.\n\n\nThank you for pointing this issue out. \n\nWe will add a diagram to illustrate the user interaction flow to Fig. 1:\n```\nQuery → WikiSP (Semantic parser) → SPARQL → Wikidata → Answer\nif Answer is null, Query →  GPT-3 → qualified answer\nelse displays answer \n    if user downvotes answer, Query → GPT-3 → qualified answer\n```\n\n1. Example with correct semantic parse\n\n    - User: \"where was anne hathaway born?\"\n\n    - Predicted SPARQL: `SELECT DISTINCT ?x WHERE { wd:Q36301 wdt:place_of_birth ?x. }`\n\n    - Our agent: \"From Wikidata, the place of birth of Anne Hathaway is New York City.\"\n\n2. Example with incorrect semantic parse\n\n    - User: \"where did bronx tale take place?\"\n\n    - Predicted SPARQL: `SELECT DISTINCT ?x WHERE { wd:Q1130705 wdt:filming_location ?x. }` *(where the correct SPARQL should use property narrative location instead of filming location)*\n\n    - Our agent: \"From Wikidata, the filming location of ‘A Bronx Tale’ includes New Jersey and New York City.\"\n\n    - User: downvote this response *(since they notice that the agent is answering the wrong question)*.\n\n    - Our agent: \"Here’s GPT-3’s unverified answer: The movie took place in the Bronx, New York.\"\n\n> (Section 3.2.2) Recovering from NED errors. I had to reread this section a few times and still wasn’t sure of some things. Including examples in the Appendix would be very helpful.\n\nThank you for this valuable suggestion. We will include more concrete examples in the appendix. For instance, in the training set, we have the following example:\n- Query: \"what year did giants win the world series?\"\n- Gold SPARQL is: \n```\nSELECT DISTINCT ?x WHERE { \n  ?y wdt:sports_season_of_league_or_competition wd:Q265538; \n  wdt:winner wd:Q308966; \n  wdt:point_in_time ?x. }\n```\n- Gold NED result: \"World Series with QID Q265538; \n                             San Francisco Giants with QID Q308966;\"\n- Refined NED result: \"San Francisco Giants with QID Q308966;\" \n\n*(Here, the refined NED linker fails to identify the “World Series” entity.)*\n\nOur proposal of mentions gives the semantic parser a chance to recover the missing QID. \nTo train the parser to generate mentions, our training includes samples like this:\n\n- Query: \"what year did giants win the world series?\"\n\n- NED result: \"San Francisco Giants with QID Q308966;\"\n\n- Gold target: \n```\nSELECT DISTINCT ?x WHERE { \n  ?y wdt:sports_season_of_league_or_competition wd:world_series; \n  wdt:winner wd:Q308966;\n  wdt:point_in_time ?x. } \n```\nThe gold query mentions `world_series`. At inference time, our heuristics use the predicted mention to look up the actual Wikidata entity. For example, if `wd:world_series` is predicted at inference time, our heuristics maps it back to `Q265538`. \n\n> Missing information about the dev set. Since this is the main data discussed in the paper, it would be very helpful to have some statistics about it. Is the dev set very diverse in terms of types of questions, relations in answers, complexity of queries, and the number of answers that are possible? This would help better understand evaluation, for example how the researchers verified GPT-3 answers. No examples of generated SPARQL compared with gold queries in the dev set.\n\nThank you for pointing this out. Our dataset was derived from WebQuestionSP, for which the statistics have been reported in prior work, e.g., Table 1 of [Gu et al. 2021](https://arxiv.org/abs/2011.07743). Certain natural language related statistics were also reported by the original authors of WebQuestion ([Berant et al. 2013]( https://aclanthology.org/D13-1160/ )) and WebQuestionSP ([Yih et al. 2016]( https://aclanthology.org/P16-2033/ )). Consequently, we share a similar set of characteristics, with minor differences originating from our conversion procedure (Section 4.1). We will make sure we cite these references. \n\n> (Section 4) Missing details about how the dataset was made. Consider discussing an evaluation of the automatic conversion of the dataset (section 4), through an analysis of the “60%” of the dataset that was automatically converted well (line 424). Also consider discussing the reliability of one person in converting the rest (which is about ~2000 examples) How long did that take? How could we verify the quality of that manual conversion? (4. 1) Porting dataset to Wikidata from Freebase: How to decide if a question is not covered by Wikidata? How reliable is that procedure (lines 431-436)?\n\nTo convert Freebase entities and properties to their Wikidata equivalents, our automatic conversion tool uses the mappings provided by Google and Wikidata. However, the mappings do not cover all the entities and properties. Additionally, the automatically converted SPARQL queries may not yield the expected results, as the information might be represented using a different property with similar semantics in Wikidata. Thus, the automatic tool can only convert 60% of the original dataset. \n\nIt took approximately 4 weeks to complete manual annotation by a programmer working full time. To improve the reliability of manual conversion, we also developed a tool for the human annotator with automatic validation. This tool first checks the syntax of the SPARQL, and then executes it to retrieve answers from Wikidata. The retrieved answers are shown to the annotator along with the original answer in the WebQuestions dataset. If the answers are different, the annotator will revisit the question to double-check if the SPARQL is correct.\n\nDespite being a much larger knowledge base, Wikidata is not a superset of Freebase. For example, flight information such as \"which airport to fly into Rome?\" is not available in Wikidata, but can be answered by Freebase. We drop examples from WebQuestions if and only if the question cannot be answered by Wikidata. \n\n> (Section 6.2) The demonstration of existing tools as part of the pipeline, specifically the use an entity linker, appears well founded, but there is no evidence given to support its inclusion. Why didn’t the ablation study also include a step where the entity linking stage is completely removed? This could be the secret sauce, so I’d be interested in a comparison with input as just raw text.\n\nThank you for this valuable suggestion, we will carry out an ablation experiment in the final paper.\n\nMany related semantic parsing over KBQA tasks use an entity linker step: [Gu et al. 2021]( https://arxiv.org/abs/2011.07743 ) (the need for an entity linker for a large-scale KBQA system is discussed under Section 3.2) and [Chen et al. 2021]( https://aclanthology.org/2021.acl-demo.39/ ).\n\nIn our dev set, there are many cases where translating entity mentions to their QIDs is challenging. Common examples include acronyms (e.g. \"what does time warner own?\", where time warner refers to WarnerMedia) and contextual omissions (e.g. \"who was darth vader in episode 3?\", where episode 3 refers to \"Star Wars: Episode III – Revenge of the Sith\", \"who are the two state senators of georgia?\" where georgia refers to the state). These examples illustrate the importance of a neural NED. Generally, the need for an entity linker stage is also needed when dealing with tail entities and a customized knowledge base.\n\n> Missing future directions. What do your results suggest for the next iteration of the research cycle?\n\nThank you for the opportunity to discuss the next iteration of our research! Our future direction is based on an additional experiment we performed for Reviewer 3 on the QALD-7 ([Usbeck et al. 2017]( https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf )) dataset. We will first discuss the results of the experiment to demonstrate how our findings motivated future research. \n\n**Why we choose QALD7 and not other datasets**: In this paper, we focus on question answering on Wikidata and we avoid datasets built on synthetic data or human paraphrase data, such as CSQA ([Saha et al. 2018]( https://ojs.aaai.org/index.php/AAAI/article/view/11332 ))  and KQA-Pro ([Cao et al. 2022]( https://aclanthology.org/2022.acl-long.422/ )). For synthetic or human paraphrase data, due to the extremely limited natural language variety shared between the training and evaluation data, models can get artificially high accuracy. For example, a simple BART model can achieve over 90% accuracy on KQA-Pro even without an entity linking module. \n\nThus to add another dataset as per request of Reviewer 3 we chose QALD-7. This dataset is part of the QALD (Question Answering over Linked Data) challenges, a popular series of challenges started in 2011 known for their complex, manually created questions. It mainly focuses on DBpedia, but QALD-7 's Task 4 is engineered for Wikidata. The task includes 100 train examples, which we use to finetune our model, and 50 test examples for evaluations. \n \n**Experimental results**: The QALD-7 test set contains both SPARQL queries as well as the answers. To double-check the correctness of the QALD-7 dataset, we applied the 50 gold queries of the test set to Wikidata and found that 4 did not return an answer. We hypothesize that the discrepancy is caused by the change in Wikidata structure/quantity of information. Thus the results below are reported only on the 46 remaining. Note that the prior work we are comparing to evaluates on all 50. \n\nFor this experiment, we use the same hyperparameters and data format as described in Section 5.2. In addition to the training data for WikiSP, we also include the QALD-7 train examples, upsampled by 20 times.\n\nQALD-7 ([Usbeck et al. 2017]( https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf )) reports that the leader WDAqua gets an F1 score of 0.552, but the paper of WDAqua  ([Diefenbach et al. 2017]( https://link.springer.com/chapter/10.1007/978-3-319-69146-6_8 )) reports 0.40. The runner-up reports an F1 score of 0.427. Our model achieves an F1 score of 0.47. Note that WDAqua is based on retrieval, whereas WIkiSP is based on semantic parsing. (The differences between retrieval and semantic parsing are discussed in Section 2.1) \n\nOne of the authors who was not involved in training or fine-tuning the model evaluated GPT-3 on the test set. GPT-3 is fully accurate on 63% of the questions, 20% incomplete, 16% wrong. WikiSP gets 41% of the questions correct.  So we can provide 41% verifiably good answers; the guesses of GPT-3 get an additional 33% correct, 15% incomplete, and 11% wrong. It was noted that the test set is significantly more complicated than the training data (of just 100 samples), with most of the queries containing multiple properties, thus explaining the lower accuracy. (We do not report a detailed error analysis because there is no dev set).\n\n**Future direction**: The results suggest that an important future direction is to improve the few-shot training data by including more complex queries.\n\n> No examples or citations for claims made in first paragraph (this could be a footnote, “as reported in news media …”, or as part of a Figure 1). (Line 39). What is this “alarm” based on? In which contexts do people “rely on LLMs as a knowledge source”? What are the high-stakes scenarios, for example? (Lines 70-71; Line 78): Citations needed.\n\nThank you for pointing these issues out! Here are some reports and citations for the high-risk scenarios discussed in paragraph 1:\n\n- [Weiser. 2023]( https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html )\n- [Lanyado. 2023]( https://vulcan.io/blog/ai-hallucinations-package-risk )\n- [Goddard. 2023]( https://www.amjmed.com/article/S0002-9343(23)00401-1/fulltext )\n\nHere are citations for lines 70-71, \"Suffering from limited training data compared to the massive knowledgebase, seq-2-seq approach is outperformed by approaches using subgraph extraction\":\n\n- [Yin et al. 2021]( https://www.sciencedirect.com/science/article/pii/S0167739X20330752?via%3Dihub )\n- [Gu et al. 2021]( https://arxiv.org/abs/2011.07743 )\n- [Banerjee et al. 2022]( https://dl.acm.org/doi/10.1145/3477495.3531841 )\n\nCitation for Line 78, \"Pretrained with internet corpora, LLMs are already familiar with formal query languages such as SQL ([Li et al. 2023]( https://arxiv.org/abs/2305.03111 )) and SPARQL ([Lubiana. 2023]( https://www.wisecube.ai/blog/sparql-queries-gpts-and-large-language-models-where-are-we-currently/ )).\"\n\n### References\n\n[1] Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases]( https://arxiv.org/abs/2011.07743 ). *Proceedings of the Web Conference 2021.*\n\n[2] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic Parsing on Freebase from Question-Answer Pairs]( https://aclanthology.org/D13-1160/ ). *Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.*\n\n[3] Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. [The Value of Semantic Parse Labeling for Knowledge Base Question Answering]( https://aclanthology.org/P16-2033/ ). *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). *\n\n[4] Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin, Jian-Guang Lou, and Feng Jiang. 2021. [ReTraCk: A Flexible and Efficient Framework for Knowledge Base Question Answering]( https://aclanthology.org/2021.acl-demo.39/ ). *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations*\n\n[5] Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017. [7th Open Challenge on Question Answering over LinkedData (QALD-7)]( https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf ).\n\n[6] Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. [Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph]( https://ojs.aaai.org/index.php/AAAI/article/view/11332 ). *Proceedings of the AAAI Conference on Artificial Intelligence, 2018. *\n\n[7] Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022. [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base]( https://aclanthology.org/2022.acl-long.422/ ). *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).*\n\n[8] Dennis Diefenbach, Kamal Singh, and Pierre Maret. 2017. [WDAqua-core0: A Question Answering Component for the Research Community]( https://link.springer.com/chapter/10.1007/978-3-319-69146-6_8 ). *SemWebEval 2017. Communications in Computer and Information Science, vol 769.*\n\n[9] Benjamin Weiser. 2023. [Here’s What Happens When Your Lawyer Uses ChatGPT]( https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html ). *The New York Times.*\n\n[10] Bar Lanyado. 2023. [Can you trust ChatGPT’s package recommendations?]( https://vulcan.io/blog/ai-hallucinations-package-risk ). *vulcan.io.*\n\n[11] Jerome Goddard. 2023. [Hallucinations in ChatGPT: A Cautionary Tale for Biomedical Researchers](https://www.amjmed.com/article/S0002-9343(23)00401-1/fulltext). *Commentary in The American Journal of Medicine.*\n\n[12] Xiaoyu Yin, Dagmar Gromann, and Sebastian Rudolph. 2021. [Neural machine translating from natural language to sparql](https://www.sciencedirect.com/science/article/pii/S0167739X20330752?via%3Dihub). *Future Generation Computer Systems, 117:510–519.*\n\n[13] Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck, and Chris Biemann. 2022. [Modern baselines for sparql semantic parsing.](https://dl.acm.org/doi/10.1145/3477495.3531841). *In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2260–2265, New York, NY, USA. Association for Computing Machinery.*\n\n[14] Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://arxiv.org/abs/2305.03111). \n\n[15] Tiago Lubiana. 2023. [Sparql Queries, Gpts and Large Language Models – Where Are We Currently?]( www.wisecube.ai/blog/sparql-queries-gpts-and-large-language-models-where-are-we-currently ) *Wisecube AI*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths\n\n- Novel means of fact-checking to improve trustworthiness in a LLM. Using semantic parsing as a step for factuality checking is a great idea and suggests hypotheses that are readily testable.\n\n- Revamped dataset. Making a dataset of SPARQL queries has many challenges no matter its size, and the researchers here present one that is fairly large and appears challenging (although that is hard to determine without any examples given or other data available). The quality of the dataset (half automatically converted well with the remaining half being manually fixed) may be demonstrated in the results of the task, which suggest that this LLM can be fine-tuned to generate valid SPARQL queries for 72% of the examples in the dev set.\n\n- An analysis that provides insight into SPARQL generation using LLMs by both considering the data, the prompts, and additional tools to add to the pipeline (entity linking). ", "reasons_to_reject": "Weaknesses:\n\n- Not so helpful Figure 1. The figure shows the end results, but a more impactful Figure 1 would illustrate better more exactly what the problem is and what the plan is to address it.\n\n- (Section 3.2.2) Recovering from NED errors. I had to reread this section a few times and still wasn’t sure of some things. Including examples in the Appendix would be very helpful.\n\n- Missing information about the dev set. \n  - Since this is the main data discussed in the paper, it would be very helpful to have some statistics about it. \n  - Is the dev set very diverse in terms of types of questions, relations in answers, complexity of queries, and the number of answers that are possible? This would help better understand evaluation, for example how the researchers verified GPT-3 answers. \n  - No examples of generated SPARQL compared with gold queries in the dev set.\n\n- (Section 4) Missing details about how the dataset was made. \n  - Consider discussing an evaluation of the automatic conversion of the dataset (section 4), through an analysis of the “60%” of the dataset that was automatically converted well (line 424). Also consider discussing the reliability of one person in converting the rest (which is about ~2000 examples) How long did that take? How could we verify the quality of that manual conversion?\n\n  - (4. 1) Porting dataset to Wikidata from Freebase: How to decide if a question is not covered by Wikidata? How reliable is that procedure (lines 431-436)?\n\n- (Section 6.2) The demonstration of existing tools as part of the pipeline, specifically the use an entity linker, appears well founded, but there is no evidence given to support its inclusion. Why didn’t the ablation study also include a step where the entity linking stage is completely removed? This could be the secret sauce, so I’d be interested in a comparison with input as just raw text. \n\n- Missing future directions. What do your results suggest for the next iteration of the research cycle?", "questions_to_authors": "No examples or citations for claims made in first paragraph (this could be a footnote, “as reported in news media …”, or as part of a Figure 1).\n\n(Line 39). What is this “alarm” based on? In which contexts do people “rely on LLMs as a knowledge source”? What are the high-stakes scenarios, for example?\n\n(Lines 70-71; Line 78): Citations needed. ", "paper_topic_and_main_contributions": "Summary: For the task of knowledge base question answering (KBQA), the researchers study a means of increasing user confidence in an answer suggested by a LLM for a given question. The problem is that LLMs hallucinate answers, a problem that leads to users having to verify the factuality of all generated data. How could we get reassurance about an answer to a question? The first step is to generate a SPARQL query using a fine-tuned language model, retrieve the corresponding answer in a KB, and present that to the user alongside the LLM’s guess as to the answer to the original query. \n\nTo make a dataset to study this problem, researcher adapt an earlier version of web questions paired with executable queries originally made for Freebase. The researchers use this new dataset, WikiWebQuestions, for few-shot sequence-2-sequence SPARQL parsing using LLaMA. For encoding, the researchers test using a named entity disambiguator to identify entities first, appending entity mentions to the natural language query. For decoding, they test generating natural language strings for properties and entities in the generated SPARQL query rather than QIDs and PIDs, with the assumption that the LLM generations using natural language strings will be better formed (more likely executable). Results are reported with error analyses.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0U4ka8it6y", "reviewer": "Reviewer_CHSV", "summary": "", "questions": "1. Are constrained decoding techniques like PICARD (Scholak2021, https://github.com/ServiceNow/picard) considered for decoding the SPARQL queries?\n\n2. I am not sure how the backoff to GPT3 system would work? When is model not confident enough to backoff to GPT3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n> WebQuestionsSP was proposed in 2016 and the answers for that also would have changed (line 104-107)? How does the proposed dataset ensure that the answers are still relevant/correct.\n\nThank you for this question. We conducted a thorough conversion from Freebase to Wikidata to make sure that the answers are correct and relevant.\n\nTo convert Freebase entities and properties to their Wikidata equivalents, our automatic conversion tool uses the mappings provided by Google and Wikidata. However, the mappings do not cover all the entities and properties. Additionally, the automatically converted SPARQL queries may not yield the expected results, as the information might be represented using a different property with similar semantics in Wikidata. Thus, the automatic tool can only convert 60% of the original dataset. \n\nIt took approximately 4 weeks to complete manual annotation by a programmer working full time. To improve the reliability of manual conversion, we also developed a tool for the human annotator with automatic validation. This tool first checks the syntax of the SPARQL and then executes it to retrieve answers from Wikidata. The retrieved answers are shown to the annotator along with the original answer in the WebQuestions dataset. If the answers are different, the annotator will revisit the question to double-check if the SPARQL is correct.\n\nDespite being a much larger knowledge base, Wikidata is not a superset of Freebase. For example, flight information such as \"which airport to fly into Rome?\" is not available in Wikidata, but can be answered by Freebase.  We drop examples from WebQuestions if and only if the question cannot be answered by Wikidata. \n\n> Are constrained decoding techniques like PICARD (Scholak2021, https://github.com/ServiceNow/picard) considered for decoding the SPARQL queries?\n\nThank you for suggesting this approach.\n\nFrom our error analysis, we did not observe any SPARQL syntax error in the model predictions. We did, however, encounter a few cases of imaginative properties. Unfortunately, with the Wikidata semantic graph, we do not know what properties are available for a specific entity, unlike databases with fixed schemas.\n\n> I am not sure how the backoff to GPT3 system would work? When is model not confident enough to backoff to GPT3?\n\nThank you for this question. Indeed, we should have clarified how such a system works.\n\nWe will add a diagram to illustrate the flow of our system to Fig. 1:\n\n```\nQuery → WikiSP (Semantic parser) → SPARQL → Wikidata → Answer    (1)\nif Answer is null, Query →  GPT-3 → Qualified answer             (2) \nelse displays answer                                             (3) \n    if user downvotes answer, Query → GPT-3 → Qualified answer   (4)   \n```\n- (1) WikiSP parses user query to SPARQL, which is then used to fetch result from Wikidata;\n\n- (2) If Wikidata returns no result, this means that we may have gotten the parse wrong, or Wikidata has no answer, we use GPT-3 to return a qualified answer to the user;\n\n- (3) If there is a result, the result is displayed within the full context of the SPARQL query;\n\n- (4) The user can \"downvote\" our response, e.g. if there is a mismatch between the user’s intended query and the semantic parser’s prediction. In such cases, we follow up with GPT’s response, by saying that: \"Here’s GPT-3’s unverified answer:  …\"\n\n1. Example with correct semantic parse\n\n    - User: \"where was anne hathaway born?\"\n\n    - Predicted SPARQL: `SELECT DISTINCT ?x WHERE { wd:Q36301 wdt:place_of_birth ?x. }`\n\n    - Our agent: \"From Wikidata, the place of birth of Anne Hathaway is New York City.\"\n\n2. Example with incorrect semantic parse\n\n    - User: \"where did bronx tale take place?\"\n\n    - Predicted SPARQL: `SELECT DISTINCT ?x WHERE { wd:Q1130705 wdt:filming_location ?x. }` *(where the correct SPARQL should use property narrative location instead of filming location)*\n\n    - Our agent: \"From Wikidata, the filming location of ‘A Bronx Tale’ includes New Jersey and New York City.\"\n\n    - User: downvote this response *(since they notice that the agent is answering the wrong question)*.\n\n    - Our agent: \"Here’s GPT-3’s unverified answer: The movie took place in the Bronx, New York.\"\n\n> Can you cite sources for \"This may cause significant harm as people increasingly accept LLMs as a knowledge source.\" (line 38-39)\n\nCertainly! Here are some sources:\n\n[1] Benjamin Weiser. 2023. [Here’s What Happens When Your Lawyer Uses ChatGPT]( https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html ). *The New York Times.*\n\n[2] Bar Lanyado. 2023. [Can you trust ChatGPT’s package recommendations?]( https://vulcan.io/blog/ai-hallucinations-package-risk ). *vulcan.io*.\n\n[3] Jerome Goddard. 2023. [Hallucinations in ChatGPT: A Cautionary Tale for Biomedical Researchers]( https://www.amjmed.com/article/S0002-9343(23)00401-1/fulltext ). *Commentary in The American Journal of Medicine.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. WWQ dataset is a useful resource to the community facilitating integration of large KBs with LLMs.\n\n2. The proposed idea of using string representations instead of Wikidata IDs is well-founded and empirically justified.", "reasons_to_reject": "1. WebQuestionsSP was proposed in 2016 and the answers for that also would have changed (line 104-107)? How does the proposed dataset ensure that the answers are still relevant/correct.", "questions_to_authors": "1. Are constrained decoding techniques like PICARD (Scholak2021, https://github.com/ServiceNow/picard) considered for decoding the SPARQL queries?\n\n2. I am not sure how the backoff to GPT3 system would work? When is model not confident enough to backoff to GPT3?", "paper_topic_and_main_contributions": "1. The paper introduces a novel semantic parser for Wikidata, accompanied by the creation of a new dataset called WikiWebQuestions. This dataset contains semantic parses over Wikidata, providing a valuable resource for training and evaluating semantic parsers in the context of Wikidata.\n\n2. The proposed semantic parser, WikiSP, is developed by fine-tuning LLama with Alpaca instruction tuning set and modified SPARQL queries. Notably, the SPARQL queries have their ids replaced by names, and the parser is further augmented with named-entity disambiguation (NED) results, which are combined with user queries.\n\n3. The paper presents an extensive evaluation on their proposed WikiWebQuestions dataset, offering valuable insights into the performance of the semantic parser. By highlighting different categories of errors, the evaluation provides a roadmap for future research and the development of improved techniques in semantic parsing for Wikidata.\n\n4. In addition to WikiSP, the paper introduces a shallow integration of WikiSP with GPT3. The semantic parser initially responds to user queries, and its outputs are subsequently supplemented and backed up with responses from GPT3, providing a two-step approach to generate more comprehensive and accurate answers.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Can you cite sources for \"This may cause significant harm as people increasingly accept LLMs as a knowledge source.\" (line 38-39)", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper reads well."}, "extra_text": {}}, {"review_id": "NnoK25U6cJ", "reviewer": "Reviewer_2RKA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n> This paper only conducts experiments on a single dataset (self-collected), but there are many existing KBQA datasets (e.g., WebQuestions, GrailQA) in the literature. Conducting experiments on popular KBQA datasets can make it better understand the effectiveness of the proposed method.\n\n> This paper does not compare with other KBQA methods in the literature.\n\nWe focus on question answering on Wikidata and we avoid datasets built on synthetic data or human paraphrase data, such as CSQA ([Saha et al. 2018]( https://ojs.aaai.org/index.php/AAAI/article/view/11332 ))  and KQA-Pro ([Cao et al. 2022]( https://aclanthology.org/2022.acl-long.422/ )). For synthetic or human paraphrase data, due to the extremely limited natural language variety shared between the training and evaluation data, models can get artificially high accuracy. For example, a simple BART model can achieve over 90% accuracy on KQA-Pro even without an entity linking module. \n\nTo better evaluate the effectiveness of our method as proposed by the reviews we also evaluated our model on QALD-7 ([Usbeck et al. 2017]( https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf )). This dataset is part of the QALD (Question Answering over Linked Data) challenges, a popular series of challenges started in 2011 known for their complex, manually created questions. It mainly focuses on DBpedia, but QALD-7 's Task 4 is engineered for Wikidata. The task includes 100 train examples, which we use to finetune our model, and 50 test examples for evaluations. \n \nThe QALD-7 test set contains both SPARQL queries as well as the answers. To double-check the correctness of the QALD-7 dataset, we applied the 50 gold queries of the test set to Wikidata and found that 4 did not return an answer. We hypothesize that the discrepancy is caused by the change in Wikidata structure/quantity of information. Thus the results below are reported only on the 46 remaining. Note that the prior work we are comparing to evaluates on all 50. \n\nFor this experiment, we use the same hyperparameters and data format as described in Section 5.2. In addition to the training data for WikiSP, we also include the QALD-7 train examples, upsampled by 20 times.\n\nQALD-7 ([Usbeck et al. 2017]( https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf )) reports that the leader WDAqua gets an F1 score of 0.552, but the paper of WDAqua ([Diefenbach et al. 2017]( https://link.springer.com/chapter/10.1007/978-3-319-69146-6_8 ))  reports 0.40. The runner-up reports an F1 score of 0.427. Our model achieves an F1 score of 0.47. Note that WDAqua is based on retrieval, whereas WIkiSP is based on semantic parsing. No comparable results were reported on semantic parsing. (The differences between retrieval and semantic parsing are discussed in Section 2.1) \n\nOne of the authors who was not involved in training or fine-tuning the model evaluated GPT-3 on the test set. GPT-3 is fully accurate on 63% of the questions, 20% incomplete, 16% wrong. WikiSP gets 41% of the questions correct.  So we can provide 41% verifiably good answers; the guesses of GPT-3 get an additional 33% correct, 15% incomplete, and 11% wrong. It was noted that the test set is significantly more complicated than the training data (of just 100 samples), with most of the queries containing multiple properties, thus explaining the lower accuracy. (We do not report a detailed error analysis because there is no dev set).\n\n### References \n\n[1] Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. [Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph](https://ojs.aaai.org/index.php/AAAI/article/view/11332). *Proceedings of the AAAI Conference on Artificial Intelligence, 2018.* \n\n[2] Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022. [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://aclanthology.org/2022.acl-long.422/). *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).*\n\n[3] Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017. [7th Open Challenge on Question Answering over LinkedData (QALD-7)](https://svn.aksw.org/papers/2017/ESWC_2017_QALD/public.pdf).\n\n[4] Dennis Diefenbach, Kamal Singh, and Pierre Maret. 2017. [WDAqua-core0: A Question Answering Component for the Research Community](https://link.springer.com/chapter/10.1007/978-3-319-69146-6_8). *SemWebEval 2017. Communications in Computer and Information Science, vol 769.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include:\n\n1. Introduction of a new dataset, WikiWeb-Questions, which is a high-quality knowledge base question answering benchmark for Wikidata. \n\n2. The paper presents the first effective few-shot sequence-to-sequence semantic parser for Wikidata.\n\n3. The proposed methodology reduces the hallucination of large language models like GPT-3 by grounding it with a semantic parser for Wikidata. ", "reasons_to_reject": "- This paper only conducts experiments on a single dataset (self-collected), but there are many existing KBQA datasets (e.g., WebQuestions, GrailQA) in the literature. Conducting experiments on popular KBQA datasets can make it better understand the effectiveness of the proposed method.\n\n- This paper does not compare with other KBQA methods in the literature. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about improving the factuality of large language models (LLMs) by grounding them in Wikidata. \n\nThe main contributions of the paper are:\n\n1. The creation of the WikiWeb-Questions dataset, which serves as a benchmark for large knowledge-base question answering.\n2. A simple sequence-to-sequence semantic parser over Wikidata.\n3. The pairing of the semantic parser with GPT-3, which provides a combination of verifiable results and qualified guesses that offer useful answers to 96% of the questions in the dev set of the benchmark.\n\nOverall, the paper demonstrates that grounding LLMs with a semantic parser for Wikidata can reduce hallucination and improve the factuality of their answers.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper Topic And Main Contributions:
* Introduction of a novel semantic parser for Wikidata, accompanied by a new dataset called WikiWebQuestions. This dataset contains semantic parses over Wikidata, providing a valuable resource for training and evaluating semantic parsers in the context of Wikidata.
* The proposed semantic parser, WikiSP, is developed by fine-tuning LLama with Alpaca instruction tuning set and modified SPARQL queries. 
* Extensive evaluation on the proposed WikiWebQuestions dataset offers insights into the performance of the semantic parser. By highlighting different categories of errors, the evaluation provides a roadmap for future research and the development of improved techniques in semantic parsing for Wikidata.
* In addition to WikiSP, the paper introduces a shallow integration of WikiSP with GPT3. The semantic parser initially responds to user queries, and its outputs are subsequently supplemented and backed up with responses from GPT3, providing a two-step approach to generate more comprehensive and accurate answers.

Reasons to accept:
* Introduction of a new dataset, WikiWeb-Questions, which is a high-quality knowledge base question answering benchmark for Wikidata.
* The paper presents the first effective few-shot sequence-to-sequence semantic parser for Wikidata.
* The proposed methodology reduces the hallucination of large language models like GPT-3 by grounding it with a semantic parser for Wikidata.

Reasons to reject:
* This paper only conducts experiments on a single dataset (self-collected), but other KBQA datasets exist (e.g., WebQuestions, GrailQA). The authors address this in the rebuttal by adding experiments on QALD-7. They also clarify that they focus on Wikidata.
Decision: Accept-Main