MarkQA: A large scale KBQA dataset with numerical reasoning
Authors: Xiang Huang
Sitao Cheng
Yuheng Bao
Shanshan Huang
Yuzhong Qu
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge base
Question answering
Numerical reasoning

Abstract: While question answering over knowledge bases (KBQA) has shown progress in addressing factoid questions, KBQA with numerical reasoning remains relatively unexplored.  In this paper, we focus on the complex numerical reasoning in KBQA, and propose a new task, NR-KBQA, which necessitates the ability to perform both multi-hop reasoning and numerical reasoning. We also design a logic form in Python format called PyQL to represent the reasoning process of numerical reasoning questions. To facilitate the development of NR-KBQA, we present a large NR-KBQA dataset called MarkQA, which is automatically constructed by a small set of seeds.  Each question in MarkQA is annotated with its corresponding SPARQL query, alongside the step-by-step reasoning path in the QDMR format and PyQL program. Experimental results of some state-of-the-art QA methods performed on the MarkQA dataset show that complex numerical reasoning in KBQA faces great challenges.

MarkQA: A large scale KBQA dataset with numerical reasoning
Xiang Huang, Sitao Cheng, Yuheng Bao, Shanshan Huang, Yuzhong Qu State Key Laboratory for Novel Software Technology, Nanjing University, China Iiianghuang, stcheng, yhbao, shanshan huang) @smail.nju.edu.cn yzqu @nju.edu.cn
Abstract
rant, 2018) and GrailQA (Gu al., 2021), respec- tively, require numerical reasoning, but this part of the questions just focuses on some aggregation operations and lacks complex multi-step numerical reasoning, The remaining only need to match a graph pattern on KB (multi-hop reasoning), with- out the need to perform numerical reasoning. As result, the questions that require complex numer- ical reasoning has been covered by previous datasets (e.g. "How much more VAT do you have to pay to buy the most expensive iPhone 13 in Rus- sia than in Japan?" or "How many times longer is the longest aircraft carrier than the shortest?").
While question answering over knowledge bases (KBQA) has shown progress in address- ing factoid questions, KBQA with numerical reasoning remains relatively unexplored. In this paper, we focus on the complex numer- ical reasoning in KBQA and propose new task, NR-KBQA, which necessitates the abil- ity to perform both multi-hop reasoning and numerical reasoning. We design logic form Python format called PyQL to represent the reasoning process of numerical reasoning questions. To facilitate the development of NR-KBQA, we present a large dataset called MarkQA, which is automatically constructed from a small set of seeds. Each question in MarkQA equipped with its corresponding SPARQL query, alongside the step-by-step soning process in the QDMR format and PyQL program. Experimental results of some state- of-the-art QA methods on the MarkQA show that complex numerical reasoning in KBQA faces great challenges.
In this paper, we propose a new challenging task, NR-KBQA (Knowledge-basee Question Answer- ing with Numerical Reasoning). Different from tra- ditional KBQA which mainly focuses on multi-hop reasoning, NR-KBQA focuses numerical rea- soning and its combination with multi-hop reason- ing. As shown in the left part of Figure 1, multi-hop reasoning needs to match a graph pattern in the KB, the difficulty of which comes from the composition of KB items (entities, relations, and classes). On the other hand, the difficulty of numerical reason- ing comes from the composition of mathematical operators. It is worth noting that computational tasks entail the involvement of multiple operands, while each operand may be obtained by match- ing graph pattern on the KB. Consequently, the combination of multi-hop reasoning and numerical reasoning will lead to a combinatorial explosion of the number of logical form structures, which poses huge obstacle to semantic parsing models.
1 Introduction
Knowledge-basee -ased question answering (KBQA) aims to answer a question over a knowledge base (KB). It has emerged as a user-friendly solution to access the massive structured knowledge in KBs (Lan al., 2021; Shu et al., 2022). Among the tensive knowledge stored in KBs, the quantitative property is the 3rd most popular property on Wiki- data (only less than Wikibaseltem and ExternalID), and there are more than 95M facts with quantitive properties. Given the large amount of quantitative facts stored in KB and the ability to perform precise symbolic reasoning through query languages, it is natural to KBQA a solution to real-world problems that require numerical reasoning.
To support the study of this task, we construct large-scale dataset called MarkQA (CoMplex NumericAl Reasoning over Knowledge Basc Question Answering Dataset) which starts from 1K questions posed by humans and automati- cally scales to 32K examples. Each question in MarkQA is equipped with a question decomposi- tion in the widely used QDMR format and our pro-
shown that existing KBQA datasets are insufficient for numerical reasoning. We find that only 10% and 16.2% of questions in Complex (enplexWWoQuestions(CCW)))) (Talmor and Be-
However,

---

What is the 2021 asset of the company founded by Bill Gates and Paul G. Allen
NR-KBQA
Multi-hop Reasoning
Bil
founded by
333,779M
Numerical Reasoning
Bill Carasa
2017
Microsoft
assets
Microsoft
Poul Alen
250,3129
Paul Allenn
founded
point in time
2021
CLB
BE Ganss
Microsoft
2021 asset of company whose founder was born in Chappaqua and educated at Hamilton College
Poul Alln
Fib
Chappaqua birth placa
585M
Chappaqua
Mor:: Randolih
Netttin
2017
Randbiph
Marc
founded by- Nettlix losal ecsets-
MC
(R.013M)
Hamitton Collage
educalnd
point time
2021
Chappaqua
Man: Randilsh
Nutiita
HC
2021
Legand
Enily node Constrain
Number node
Function node node
During 2017 to 2021, how much more annual increase in total assets is the company founded by Bill Gates and Paul G. Allen than the company whose founder born Chappaqua educated Hamilton College?
Compute
Figure I: An example shows multi-hop reasoning, numerical reasoning, and their combination.
posed corresponding logic form in Python, namely PyQL. The QDMR can be seen as an explicit rea- soning path and regarded as a question decomposi- tion resource. Meanwhile, PyQL. not only acts as a reasoning step but can also be directly transformed into SPARQL query. It offers more human- readable alternative to SPARQL and can casily be extended for further developments. We believe that MarkQA will serve valuable resource to foster further development of KBQA.
sourcing. We refer to this manner as LF-to-NLQ. Though efficient, they only consider sampling LF's from a single connected sub-graph in KB. Conse- quently, the generated question is mostly a factoid question aiming to find an entity entity set that meets the specific conditions, thereby only real- izing multi-hop reasoning. However, many real- world questions require the introduction of com- plex numerical reasoning over multiple connected subgraphs, which can not be covered by the previ- ous construction methods. When multi-hop reason- ing and numerical reasoning intertwine, it is insur- mountable to choose reasonable query patterns by enumerating all possible ones, given that the search space can comprise millions of patterns, of which only a minuseule fraction is deemed reasonable.
The remainder of this paper is organized as fol- lows: Section 2 reviews related work. Section 3 defines the NR-KBQA and introduces PyQL. The construction and analysis of MarkQA are demon- strated in Section 4. Section 5 presents experi- mental results. We conclude our contributions and future work in Section 6.
To ease this problem, we collect some questions as seeds in an NL.Q-to-LF manner, making sure the questions are reasonable. Mcanwhile, we propose general framework called Seeds-to- Forest (SoF) to automatically scale the dataset.
2 Related work
2.1 Knowledge-based question answering
KBQA refers to QA systems that take KB as the underlying knowledge source. The primary focus of KBQA research is on identifying graph patterns in the KB that satisfy the constraints (multi-hop rea- soning), leaving numerical reasoning uncovered. This influences the construction of KBQA datasets. Most KBQA datasets (Trivedi et al., 2017: Talmor and Berant, 2018; Gu et al., 2021) are es- sentially built through the OVERNIGHT (Wang et al., 2015) method or its extensions, consisting of three steps: sample logic forms(LF), convert LF to canonical questions, and paraphrase them into natural language questions (NLQ) through crowd-
2.2 Numerical Reasoning
There are several types of QA tasks involving nu- merical reasoning:
1) Some KBQA datasets (Talmor and Berant, 2018; Gu et al., 2021) consider counting, superla- tives (argmax, argmin), and comparatives (>, >=, <, <=). But their questions only perform the caleu- lation at most once, and the types of operator are limited to just comparison and some aggregations.
2) The most famous Machine Reading Compre- hension (MRC) dataset with numerical reasoning is DROP (Dua et al., 2019). It also lacks com- plex numerical reasoning. As a result, a large por-
thttps://githhie hettp:///gttuub..o///c/////rr/re www////w///a/////////ooo

---

tion of methods of DROP follow Multi-predictor-
ers more comprehensively the support of numerical reasoning and more complex SPARQL grammars. Moreover, while KoPL requires the development of a specific execution engine and can only execute on JSON format KB, PyQL can directly be com- piled to executable SPARQL queries, without the burdensome design of an additional executor and supporting native KB format.
based manner (Dua et al., 2019; Hu al., 2019) which employs multiple predictors to derive dif- ferent types of answers. For example, they use a multi-classifier to find the answer to a counting question as the answer is restricted to integers from 0 to 10. This is a far cry from actually solving numerical reasoning problem.
3) Math Word Problem (MWP), such as MATHQA (Amini et al., 2019), ASDiv (Miao et al., 2020), and LILA (Mishra et al., 2022), require in- tricate complex numerical reasoning. However, MWP is based a virtual scenario, while real- world problems often require to access accurate knowledge. MWP model does not require the abil- ity to query knowledge from knowledge sources, but just the ability to correctly understand the ques- tion itself and reason numerically.
In MarkQA, we provide each question with a QDMR and a PyQL to present the reasoning pro- cess. We believe it would be a valuable resource to improve the interpretability of the Al system.
3 NR-KBQA
In this section, we present the formal definition a question with numerical reasoning (NRQ), which is the base of NR-KBQA Then, we introduce PyQL to showcase the reasoning steps of NRQ.
4) Table QA, such FinQA (Chen al., 2021), may involve multi-step numerical reasoning. Com- pared FinQA, we focus on KBQA tasks in gen- eral rather than the financial domain. Besides, com- pared to Table QA, KBQA has some inherent supe- riority in terms of the amount of knowledge, aggre- gation for large search space, and case of multi-hop multi-domain querying.
3.1 Numerical Reasoning Question
An NRQ is any question, requiring mathemati- cal calculations, such as arithmetic calculation, aggregation, or comparison, to reach the answer. An NRQ essentially consists of the descriptions of value and the computation process. The con- notation of NRQ can be defined recursively in form:
To alleviate the above problems, our dataset re- quires not only the ability to interact with underly- ing knowledge sources but also the ability to per- form complex numerical reasoning.
<NRQ> ::= <Func> <Arg> { <<rg>>
(1)
<Arg> ::= Num| <NRQ> <Des>
(2)
<Des> :: Rel <Var>
<Des> <Des>
(3)
2.3 Interpretable Reasoning Path
<Var> :-= Ent
Num
<Des>
<NRQ>
(4)
When it comes complex reasoning (e.g., multi- hop, numerical, or logical reasoning), it is cru- cial to examine whether a model really deeply un- derstands the underlying groblem-solving process rather than merely producing the answers.
In this grammar, <NRQ> represents the intrin- sic meaning of NRQ, which can be regarded as the outermost function (<Func>) applied to one or more <Arg>. <Arg> corresponds to a constant value (Num), a description of an entity's 's numeri- cal attribute (<Des>), or another <NRQ>. <Des> describes the relationship (Rel) between a vari- able (<Var>) and the entity being deseribed, while the <Var> corresponds to an entity (Ent), a Num, a <Des> or a <NRQ>. Equation 2 and 4 allow for the nesting of numerical and multi-hop reasoning, thereby enabling the representation of complex NRQ.
There are mainly two common representations of reasoning path: question dccomposition in natu- ral language, and formal representation symbolic language. QDMR (Wolfson et al., 2020) is a widely used question decomposition format. decom- poses a question into sequence of reasoning steps, and each step is an intermediate question phrased in natural language. Some works instead choose a more formal symbolic language representation. LILA (Mishra et al., 2022) and Binder (Cheng et al.. 2023) adopt Python to showcase the process of reaching the answer. (Cao et al., 2022) develops a Knowledge oriented Programing Language (KoPL). Different from KoPL, our proposed PyQL consid-
Based this recursive nature, the query graph of an NRQ can be modeled as a tree and that of sub-question
sub-tree. For the example in Figure 1, the right part (in red) a computational tree where each intermediate node is a function

---

node and each leaf a constant value or an attribute value (green node). The attribute value node is acquired by matching& graph pattern in the KB, which the previous datasets focus on and can be seen as a description of a value.
PyQL as structural COT. Besides, the code- style format also benefits LLMs in understand- ing due to their pre-training on code resources.
4 MarkQA
3.2 PyQL
4.1 Dataset Construction
We propose PyQL (Pythonic Query Language for SPARQL). logical form written in Python as reasoning step representation for NRQ. A PyQL is sequence of commands: [c, C23 Cn where & either initializes a PyQL object or calls a function on the object. As shown in the top left of Figure 2, the user should first initialize a PyQL object and sequentially add functions to construct the whole query. Each function represents a reasoning step such as stating the relation between two entities or computing the average. A valid PyQL can directly generate an executable SPARQL query. In detail, PyQL encapsulates various SPARQL syntax ele- ments, such as Basic Graph Patterns, Assignments, Filters, Aggregations, and Subqueries. The detailed function list of PyQL can be found in Appendix H. The main features of PyQL can be summarized as follows:
Our construction comprises 4 steps: Seeds collec- tion, Paraphrasing, Generalization, and Composi- tion, detailed in Figure 2. We extract this process into a general framework and name it Seeds-to- Forest (SoF).
4.1.1 Seeds Collection
As mentioned in 2.1, we collect seed questions an NLQQ-t--LF manner. This allows for a greater diversity of questions, providing the possibility to reach questions with longer reasoning paths while ensuring the meaningfulness of each seed question. Besides, the question is more natural, as they are posed directly by humans, instead of transformed from randomly searched logic forms.
We invite 10 graduate students familiar with KBQA to annotate seed questions. We instruct them to focus on exploring patterns specific to cer- tain relations, thus enhancing the variety and orig- inality of the questions. Annotators are required to annotate at least three questions for each quanti- tative property and each question must involve at least one computational operation. Furthermore, for different questions related to the same rela- tion, their computational structure must be differ- ent, which means that the differences between the two questions can not be merely the replacement of entities. In addition, all the entities and rela- tions in the questions are recorded for logic form annotation purposes.
User-friendly and conciseness. PyQL offers an intuitive and concise approach for query- ing KB by shielding users from unreadable and lengthy database query language. It al- leviates the burden of learning and writing SPARQL and effectively reduces the entry bar- rier for the community in utilizing SPARQL. It also makes sure that the generated SPARQL is grammatically correct and uniformly for- matted. Specifically, in MarkQA, the aver- age token length of PyQL is only 60.6% of SPARQL, and the grammar errors when using PyQL as output is half of those of SPARQL.
We then invite six graduate students familiar with QDMR, PyQL, and SPARQL to annotate the QDMR and PyQL for each seed question. We in- struct annotators to first write down the QDMR (some sub-questions) of each question, then anno- tate the PyQL for each sub-question. The SPARQL of each question is automatically generated from PyQL and the annotators need to make sure that each SPARQL. is executable with a unique answer. An annotated example and an auxiliary annotation system page can be found in Appendix D and E. For each seed question, we ask another three annotators to check if is meaningful and can be posed in the real world. We only keep the questions receiving over approvals, resulting in 10. 10.19%
Step-by-Step reasoning path. PyQL, in a symbolic manner, shows the transparent rea- soning pathway of a question. Compared to SPARQL or S-expression, which is presented as whole query and is difficult to parse or decompose, PyQL exhibits how to construct a ef-
query step by step. PyQL also serves ficient supervision signal, with our experiment showing up to a 19% performance improve- ment. With the prevalence of the Large Lan- guage Model (LLM) and Chain-of-Thougee (CoT) (Wei et al., 2023), it is feasible to use

---

example(Question Loglc Form)
Paraphasing
During 2017 2521, how much more annual incresse
(: Durig 2017 2021, much more onnual increase in assets [Morossfff [bietla]?
[Microson] than (Nettlo)
Zerom 2017 to 2121, much more dowe (blorossst tatal aoaads
your then [Netiicc??
-Pyyau)
122400 "M17_2077 2017)
Generate more examples
0.0dd
0.00d 122365
1200711
P2428"7N 2017)
2021)
Composition
panaphrase, replace untilies generalization
Generalization
0.001.
quandby
2021 222211
2017"; (MS_incresse")
buadd www.suunnRyyA... G02287 P5423' 'Ve 2017
RAMIL www.dyy 2o2000
www.s/aaaaoooooooooooooo P2AS Wh. 201 20112
b_sdt__onnjyeeeeWW "NN_2017" __noresse))
a.adh__brnoyeeeppt 2017) (eor_s""")
2017
2525122
hace Cummay Roudan tit 2047 2011)
your sap") NNF_aruusl)
2:d0t_bnndssut (M5 arnuof TNF jaw.unuunu "onswer")
bndddd incresse.
R- Crase
tbsdded by- Microot
Padd C.
Semple some canditates to composite
Cardicanes:
Final example(Question and Logic Form)
www.heroorrrtooccoooo 1H QSE24 wi Peoele Ppp (o000))
G048
During 2017 to 2921. how much more annuoll increase in socal assets the company founded by Bill Gabes and
C007011
G240
03084
04R1211
Paul G. Allen than company whose founder was born
2021
2330
2000
2021
PyQLO
sade luccn "compla 1P112"
(wil))
Sample examples
aads compmmm 10162005"
[e____sd"""" comphh""
445t wo "aa0o" Naades 2017 ind
Horli Sorealler Handdigh
Ruai du by ooeea" "ooeadt 2010 h0nn
lacst 1960" 10111
Pandis
oeer Whe "nooeht 2010 z0ne luad cour
quarthty mompp/"""""./..///a PP403",
2017
vol FRat
a.dd quanthy" P2400,
P2408" .N. asspts 2017 2017)
add vil JusiTe pad "997" 03113011)
Figure 2: The feeds-to-Fores, (SoF) data construction framework. We first collect some seed questions and corresponding logic forms. And then generate more examples by paraphrasing, generalization, and composition.
dropped. For QDMR and PyQL., we ask another two annotators to check the correctness indepen- dently and correct all error cases. In the end, we collect total of 950 seed questions for 318 quanti- tative properties.
seed questions, we introduce some perturbations to avoid the model learning the bias from literals. We also collect the alias of the entity through the skos:altl.bbel. To ensure the quality and unambigu- ous nature of aliases, we have removed aliases that can be associated with multiple entities simultane- ously, as well as aliases that are identical to the labels of other entities.
4.1.2 Paraphrasing
We perform question paraphrasing through GPT- 3.5-turbo increase the surface-level diversity of the questions. The prompts and paraphrased re- sult can be found in Appendix F. Given a question, we enclose the mentioned entities in parentheses and anonymize them as A/B/C. Then, we ask the model to paraphrase the anonymized question, but when outputting, it should restore the specific en- tity labels based on the correspondence between A/B/C and the entity labels. In this way, we can obtain question paraphrases in a more controlled manner, and the model's output directly usable. For each question, we instruct GPT-3.5-turbo to generate 10 paraphrases that preserve the original question's 's semantics.
For each seed question, we sample up to 30 gen- eralized questions. We employ the new entity's 's label alias and its QID to replace the original in the question and logic form, respectively. As a consequence, we obtain about 20k examples.
4.1.4 Composition
So far, our emphasis has primarily been on explor- ing numerical reasoning. In real-world scenarios, numerical and multi-hop reasoning often arise to- gether. In this step, we incorporate multi-hop rea- soning into MarkQA by converting entities in the questions into descriptions. This is done through two steps: Sub-graph Sampling and Naturalization.
We sample 1000 paraphrases of 100 questions and find only 4.3% with minor problems, so we consider the quality to be basically acceptable and obtain 9,366 paraphrases.
The aim of Sub-graph Sampling is to sample a sub-graph centering around the target entity for de- scription. We consider structures within two hops, where each variable can be restricted by at most two triple patterns, resulting in six types of structures (detailed in Appendix J). These structures cover most structure patterns of previous complex KBQA datasets (e.g. LC-QuAD, CWQ, GrailQA). During sampling, we ensure that the sub-gra bub-graph is suffi- cient (able to uniquely identify the entity) and non-
4.1.3 Generalization
In this step, we extract the logic form templates for each seed question by masking entities with vari- ables. We execute the templates to acquire more entities that satisfy the SPARQL to generate more examples. In addition, for numerical literals in the

---

Dataset
Size
NRQ
Canonical LF
Struct
SPT-OP
Avg NS
CWQ (Talmor and Berant, 2018)
34,689
10.0%
6236
56
0.30
LC-QuAD 2.0 (Dubey et al., 2019)
30,226
8.9%
200
20
6
0.09
GrailQA (Gu et al., 2021)
51,100
16.2%
4330
22
8
0.16
KQA Pro (Cao al., 2022)
117.970
45.1%
45563
418
6
0.57
MarkQA
31,902
100.0%
12410
1054
15
2.51
Table 1: Detail statistics MarkQA with other datasets. NRQ represents the percentage of questions that require numerical reasoning. Canonical LF means the number of distinct SPARQL templates, which is done by masking all entities, and types. Struct means the number distinet skeleton structures of SPARQL, acquired by masking all entities, types, variables, and relations. We also took into consideration that different query graphs may be isomorphic heterogeneous graphs due to the different ordering of triples. SPT-OP refers to the number of numerical operations supported. Avg NS means the average number of computational operations.
redundant (without extra triple patterns). To guaran- tee the meaningfulness of the sub-graph, we collect high-quality relations from existing datasets, in- cluding SimpleQuestions, CWQ, and GrailQA. For Freebase relations, we transform them into wiki- dataIDs using Wikidata's official mapping page. We calculate their distribution as a base to select triple patterns and incorporate some smoothing techniques to balance the sampling probability and alleviate the long-tail phenomena.
the difficulty of numerical reasoning (Avg NS) and supports more comprehensive operators (SPT-OP). Combined with multi-hop reasoning, the template of our query far exceeds others (Canonical LF). This results more diverse questions that have not been included in previous datasets. Considering the query structure (Struc), MarkQA shows a great diversity compared to others. This is intuitive, as the combination of KB graph patterns and compu- tational graphs produces a combinatorial explosion. We provide other detailed statistics in Appendix B.
For Naturalization, we prompt GPT-3.5-turbo transform the sub-graphs into natural language descriptions. To prevent information leakage, we mask the target entity and the inner entity as a spe- cial token and instruct GPT-3.5-turf which one to describe. In other words, it should not output the label of the target entity or the internal entity variable. Otherwise, it is not a real two-hop prob- lem, since the internal variable is already leaked. can be found in Appendix G. With our well-designed prompt, GPT-3.5-turbo excels at the task of converting sub-graphs into nat- ural language descriptions. We sample 100 (sub- graph, description) pairs and find 93 acceptable. In this step, 50% of questions from 4.1.3 have one or two entities replaced with descriptions. Each ques- tion generate most 2 new questions through this step. Finally, we obtain 31,902 examples.
4.2.2 Operators Analysis
The question of MarkQA considers 15 types of operators. including 5 arithmetic operations (addi- tion, subtraction, multiply, divide, absolute value), aggregation operations (count, argmin, argmax, average, summation), and 5 comparative opera- tions(>, <, >= <=, =). With their combination, MarkQA supports most of the questions that re- quire numerical reasoning. The percentage of ques- tions that have at least one arithmetic, aggregation, or comparative operation are 88.1% %, 24.8%, and 34.6% 34.6%, respectively. There are 71.20% of questions with different operators and the average number of operators per question is 2.51.
The prompt
4.2.3 Answer Analysis
The answers of MarkQA are all unique, which may manifest as a numerical value, an entity, or a boolean value. When presented with questions that produce collections of entities, we transform them into ones with unique answers (such as the size of the entity set or the maximum value of one of their attributes). The percentage of answers in the datasct that are a number, an entity, and a boolean value are 72.9%, 17.9%, and 9. %, respectively.
4.2
Dataset Analysis
4.2.1
Statistics
Our MarkQA consists of 31,902 examples. A de- tailed comparison with existing datasets is shown in Table 1. Compared with existing datasets, MarkQA has quite lot more questions that require numerical reasoning (NRQ). In addition to being more numerous, MarkQA is superior in terms of

---

Methods
Output
Overall
LLD
Compositional
Zero-shot
SPARQL
34.24
70.05
53.71
6.32
T5-base
PyQL
40.70
78.32
63.10
10.39
SPARQL
38.68
78.32
63.58
6.07
GMT
PyQL
43.63
82.10 10
68.33
11.71
SPARQL
37.19
76.82
57.37
7.01
QDTQA
PyQL
42.57
84.59
70.89
7.01
Table 2: QA performance (%) on test set of MarkQA.
4.2.4 Quality
logic forms, adapt T5-base and two representative SP meth- ods MarkQA: (1) T5-base (Raffel et al., 2020) language model, and we model the QA task as a sequencce-l-ssqueenn task. We concat ques- tion, entity linking, and relation linking results as input. (2) GMT (Hu et al., 2022) uses a multi-task learning framework to refine the retrieved linking results along with generating target logical forms. (3) QDTQA (Huang et al., 2023) improves KBQA performance by incorporating question decomposi- tion information. For each model, we report the per- formance of two formats as output, i.e. SPARQL and PyQL.
baseline models. Concretely,
We invited three workers to evaluate the final dataset. Each example can be accepted if receives over 2 approvals. In detail, we randomly sample 100 examples from the test set of MarkQA to ex- amine their questions, QDMR, and PyQL. We find that all samples have fluent questions. 5 questions are ambiguous or not meaningful, which means they are not expected in real-world questions. The QDMR or PyQL of 8 questions is problematic. In total, 89 out of 100 examples acceptable.
5 Experiment
5.1 Experimental Setup
Our training/validatiionnest sets contain about 70%/20%/10%% of the data, corresponding to 22,352, 6,334 and 3,216 instances, respectively. Following (Gu et al., 2021), we evaluate the gen- eralizability of MarkQA by three levels (i.e. I.I.D, Compositional, and Zero-shot). For the validation and set, the proportion of the three levels of generalization remains the same as the origi- nal paper (50% for Zero-shot, 25% for Composi- tional. 25% for I.L.D.. Given that the answer of MarkQA are unique, we consider accuracy (ACC) the evaluation metric.
5.3 Main Results
Table 2 summarizes the evaluation results on MarkQA. Compared with existing complex KBQA datasets, MarkQA is more challenging given that the performance of the state-of-the-art method drops dramatically. After involving PyQL., the over- all performance improves significantly, where the relative increase ranges from 12.80% to 18.87%. It is easy to grasp since PyQL is more concise repre- sentation and the average token length of PyQL is only 60.6% of SPARQL. Besides, the grammar of PyQL is simpler than SPARQL, resulting in fewer grammar errors. For 95% of the questions, the top when adopt-
5.2 Baselines
generated results are grammarly ing PyQL as output, while this metric drops to 90% with SPARQL as the output, which indicates the superiority of PyQL.
There are mainly two mainstream methods in KBQA, namely Information Retrieval (IR) meth- ods and Semantic Parsing (SP) methods. IR meth- ods retrieve subgraph from KB and rank candi- date entities in the subgraph to reach the answer. They only handle questions whose answer is the entity in the sub-graph, unable answer questions that require further reasoning (computa- tion) of the nodes in multiple connected subgraphs. Therefore, we only consider SP methods, which parse natural language questions into executable
The result also shows that Compositional and Zero-shot settings are challenging. We observe a maximum of 19.4% relative performance gap be- tween the I.I.D and Compositional setting. Mean- while, the Zero-shot setting is the most challenging among the three levels. It not only includes unseen schema items but also naturally contains unseen combinations of schema items which the Composi-

---

Methods
Over.
LLD
Comp.
Zero.
Type
Over.
L.D
Comp.
Zero
All
40.7
78.3
63.1
10,4
T5-base
40.7
78.3
63.1
10.4
NR
42.0
85.4
64.3
12.9
w/ gold E
46.5
88.3
72.7
12.1
NR and MR
38.5
65.5
61.8
5.1
w/ gold R
479
79.2
65.5
23.1
NR and MR(1)
43.3
70.1
70.2
6.5
w/ gold ER
57.6
89.8
76.1
31.9
NR and MR(2)
28.7
55.6
44.8
2.3
Table 3: Detailed analysis of T5-base with PyQL as output. w/ gold E or R means we use golden entity or relation linking results. Over., Comp., and Zero. stands for Overall, Compositional, and Zero-shot, respectively.
Table 4: Performance different types of questions on T5 (PyQL). NR and MR mean numerical reasoning and multi-hop reasoning, respectively. MR(1) and MR(2) mean one-hop and two-hop reasoning. respectively.
tional level focuses on. Moreover, some relations may have an entirely new query structure (masking all entities, types, variables, and relations) even after masking the relations. We also analyze the av- erage PyQL length of the three settings and find no obvious differences among the three levels (from 7.4 to 7.8), indicating that the difficulty is not pri- marily brought about by the longer reasoning path.
that the inclusion of multi-hop reasoning further in- creases the question's 's difficulty. Rows 4/5 presents the performance of questions that require numeri- cal reasoning and one-hop/two-hop reasoning, spectively. The questions with two-hop reasoning exhibit significantly lower performance than the ones with one-hop reasoning.
Conclusion
5.4 Analysis
In summary, our contributions follows:
Performance with golden linking We perform oracle experiment find out why all models fail to achieve a satisfying performance and give an up- per bound of these models on MarkQA. As shown in Table 3, given the golden linking results, the performance increases significantly on all settings, especially the Zero-shot setting with a 21-point in- crease. This meets the expectation as the Zero-shot setting additionally has unseen relations. However, even with golden linking results, the performance of all models is still poor. In specific, the perfor- mance of the Zero-shot setting is far behind the Compositional setting. demonstrates that linking is indeed a challenge, but not the only challenge.
We propose new task NR-KBQA which ne- cessitates the ability perform both multi- hop reasoning and numerical reasoning.
2.
We devise PyQL, domain-specifit language to portray the step-by-step reasoning process. Aside from providing convenience for annota- tion, it also has superiority when serving as a supervised signal for complex reasoning.
3. We construct a large-scale dataset, namely MarkQA, with 31,902 examples which is auto- matically scaled from 1K real questions. Each question is equipped with a QDMR, PyQL, and SPARQL.
Performance of different types of questions Table 4 shows the performance of different types of questions. Row I represents the performance of all questions. Row 2 showcases the performance of subset of questions that only requires numerical rea- soning. By excluding newly generated questions in the Composition stage, this subset provides a measure of the inherent difficulty of numerical rea- soning itself. The performance only sees a slight increase of 1.3%, indicating that numerical rea- soning is indeed challenging and that the dataset's primary difficulty stems from this aspect. Row displays the performance of questions that require both numerical and multi-hop reasoning. This sub- set of data is complementary to the data in Row 2. The performance drops by 2.2%, suggesting
4. We migrate two state-of-the-arr baselines from Freebase to Wikidata. Experimental results show that NR-KBQA faces great challenges.
Overall, we for the first time explore and discuss numerical reasoning in KBQA from task formula- tion, reasoning paths representation, dataset con- struction, and baseline implementation. We believe this work should prompt the research in this area.
Limitations
The major limitations of the work include: (a) The seed question is manually annotated. It would be more efficient to automatically collect

---

human-asked questions for constructing a large- scale datasct. (b) Our dataset construction process includes some synthetic steps, such as replacing the entity with a description, which may have resulted in some rigid expressions. (C) PyQL, for now, is SPARQL-oriented. It will be more universal after being generalized to other query languages, such SQL, Lamb and others.
Human Language Technologies Proceedings of the Conference, 1 1:2368-2378.
Mohnish Dubey, Debayan Banerjee, Abdelrahman Ab- delkawi, and Jens Lehmann. 2019. Le-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In The Semantic Web Web-ISWC 2019, pages 69-78, Cham. Springer International Publishing.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler. Percy Liang. Xifeng Yan, and Yu Su. 2021. Beyond I.I.D.: Three levels of generalization for question answer- ing on knowledge bases. The Web Conference 2021 Proceedings of the World Wide Web Conference, WWW 2021, 2021:3477-3388.
Acknowledgements
This work supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62072224. The authors would like to thank all the participants of this work and anonymous reviewers.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dong- sheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-HJCNLP), pages 1596-1606, Hong Kong. China. Association for Com- putational Linguistics.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koneel-Kodziorski, Yejin Choi, and Hannanch Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-hased for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). pages 2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics.
Xixin Hu, Xuan Wu, Yiheng Shu, and Yuzhong Qu. 2022. Logical form generation via multi-task learn- ing for complex question answering over knowledge bases. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 1687- 1696, Gyeongju. Republic of Korea, International Committee on Computational Linguistics
Xiang Huang, Sitao Cheng, Yiheng Shu, Yuheng Bao, and Yuzhong Qu 2023. Question decomposition tree for answering complex questions over knowl- edge bases. Proceedings of the AAAI Conference on Artificial Intelligence, 11924442221
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nic, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022. KQA pro: A dataset with explicit compositional programs for complex question an- swering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pages 1101-6119, Dublin, Ireland. Association for Compu- tational Linguistics.
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang. Wayne Xin Zhao, and Ji-Rong Wen. 2021. A sur- vey on complex knowledge base question answering: Methods, challenges and solutions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4483-4491. International Joint Conferences on Artificial Intelli- gence Organization. Survey Track.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hap Hao Huang, Bryan Routledge, and William Yang Wang. 2021. Finqa: A dataset of nu- merical reasoning over financial data. Proceedings of EMNLP 2021.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yin Yeh Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online. Association for Computational Linguistics.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni. Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer. Noah A. Smith, and Tao Yu. 2023. Binding Lan- guage Models in Symbolic Languages. ICIR, pages 1-27.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. LILA: A unified bench- mark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 5807-5832, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019, Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. NAACL HLT 2019- - 2019 Conference of the North American Chap- ler of the Association for Computational Linguistics:

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. T5: Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
2020. BREAK down: A question understanding benchmark. TACL.
Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria- Esther Vidal. 2020. Falcon 2.0: An entity and rela- tion linking tool over wikidata. Proceedings of the 29th ACM International Conference on Informa- tion amp; Knowledge Management, CIKM '20, page 3141-3144 -148. New York, NY, USA. Association for Computing Machinery.
Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma. Yuzhong Qu, and Chin- Yew Lin. 2022. TIARA: Multi-graine
retrieval for robust question answering over large knowledge base. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8108-8121, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as a snowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- fional Linguistics; Human Language Technologies, Volume 1 (Long Papers), pages 641-651, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. Le-quad: A corpus for complex question answering over knowledge graphs. In International Semantic Web Conference, pages 210-218. Springer.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume I: Long Pupers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le. and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning large language models.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the EMNLP: System Demonstru- tions, pages 38-45, Online. Association for Compu- tational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoay Goldberg, Daniel Deutch, and Jonathan Berant.

---

A Numerical Properties Selection
Reasoning type
Prop
Arithmetic
88.
We first select the Numerical Properties(num with more than 100 statements or qualifiers recorded in Wikidata (of January 23, 2023). This leaves us with 377 n.m_p. This is to avoid se- lecting rare or long-tail numerical attributes since they are likely to be uncommon and difficult to comprehend. Additionally, a numerical attribute has too few records, it becomes challenging to gen- erate sufficient number of questions during the generalization process, leading to data imbalance issues. Then, manually remove quiring high-level domain knowledge (e.g. redshit, pKa, longitude of ascending node). Each removed num_p is checked by two annotators. If both anno- tators agree that it is hard to ask meaningful NRQ for this num_p __ then this num_pis dropped. Finally,
Aggregation
24.8
Comparison
34.6
Arithmetic and Aggregation
14.3
Arithmetic and Comparison
28.6
Aggregation and Comparison
7.9
Arithmetic and Aggregation and Comparison
3.1
Table 7: Propertion of questions with di different merical reasoning type and their combination.
Type
Number of NRQ
Proportion
num _ rc-
NR
20,745
64.4%
NR and MR
11,468
35.6%
NR and MR(1)
7,606
23.6%
NR and MR(2)
3,862
12.0%
Table 8: Distribution of different types of question. NR and MR mean numerical reasoning and multi-hop rea- soning, respectively. MR(1) and MR(2) mean one-hop and two-hop reasoning, respectively.
we retain 318 num
B Data Analysis
Here we provide more statistical details of MarkQA. Tabel shows the number of distinct en- tities, properties, and answers covered by MarkQA, along with the average PyQL and question length. Table 6 presents the distribution of different numer- ical reasoning types among the three generalization levels. Tabel 8 shows the distribution of questions with different levels of the combination of numeri- cal reasoning and multi-hop reasoning.
Experimental Details
C.1 Knowledge Base
We utilize Wikkidaa-2023--1--1 dump the knowledge base. This dump consists of 21 bil- lion facts and takes 1.8TB uncompressed on disk, which is far larger than another popular used KB, i.e. Freebase with 8 billion facts. To lower the bar- rier of using Wikidata for KBQA, we elaborately tailor the Wikidata dump to get subset that con- tains only QA-related facts. We remove the facts that include but are not limited to articles, refer- ences, multilingual, multimedia, tables, and others useless for QA. This leaves with about 5 bil- lion facts which the same order of magnitude as Freebase.
Measurement
value
# distinct entity
19,342
# distinct property
582
# distinct quantitative property
318
distinct answer
9,677
Avg PyQL len
7.39
Avg question length(tokens)
25.6
C.2 Implementation Details
Table Key statistics for MarkQA
For all baselines, For entity linking we use open-source linking tools ELQ. to retrieve candidate entities, the correspond- ing scores, and the mentions. For relation linking, we use Falcon 2 (Sakor et al., 2020).
the same linking results.
Type
LIID
Comp.
Zero.
Over.
Arithmetic
24.4
25.2
50.4
100.0
Aggregation
22.3
23.9
53.7
100.0
For GMT and QDTQA, we keep their original training settings. For T5-base,
Comparison
27.8
33.9
38.3
100.0
adopt the im- plementation based on Hugging Face (Wolf et al., 2020), with Adafactor optimizer. The learning rates of all three models are set to 5e-5. All models are trained for 30 epochs. For evaluation, the beam size 25. All models are trained and evaluated on
Table 6: Distribution(%) of different levels with dif- ferent numerical reasoning types. Note that there are overlaps among the three reasoning types (the sum of the percentages may not equal to 1).

---

Nvidia RTX3090 GPU,
C.3
Error Analysis
We sample 100 error cases from GMT model in PyQL format and summarize the errors into the following categories: (1) Wrong reasoning path (28%): We manually analyze the percentage of questions that predicate wrong reasoning path regardless of the linking result. (2) Entity linking error (42%) and Relation linking error (26%)) We compare the KB item in the predicted PyQL with the golden ones and find that 68% of questions have least one wrong item. (3) Grammar error (4%): This means the generated PyQL is grammal- ically incorrect and cannot be compiled to generate executable SPARQL.

---

D
Annotation Example
Questio Content
Among all (radio belecopee[184386s how many times the masimum focal langth compared to the average?
Question 4145351889
Non-numerical
Entities Q164356
SELECT
SELECT DISTINCT
Tradio telescape, wotF31/idt22/9 wd.3184356.
Tracio telescope, 1 PiP2151 Istatement, radio, telescope, focal, 1.
Intatement redio telescope focal, ParR2151 Pvllue redio telescope focal Ivolue redio, telescope, focal. 1 wwkosseeuunniiyyeeeoont Pradio, telescoge, focal.
ORDER BY DESC//radio telescope, focal 1)
LIMI
SOLECT (MVG(hradio, telescope focal, 2) Paug radio telescope, focal Tradio jelescope, 2 F3t wd:Q184356.
Tradio, telescope 2 pP2151 Nitatmmett radio, telescope, focal
Tutatement redio teleucope focal Pac12151 Pualun st redio Jelencope focal Toalae, radio telesoppe focal 2 vnkkanequeritkknuuun Tredio, telescope focal
BIND (Tradio tetesioptesecppeeaeo local "Tvgg selescope Socal) AS Tanswer)
Reasaning steps
telescapees
PyQL0
wadel type appe.comtrain(0188455 "redio, telescope.
lingth
add quantiyyraaii jelescope_11 /P2151) Yadio telescope focal (1)
What the masimum velue 42
maxt radio selescope focal_
selescopes
b- PyQLO
type, constraing
telescope 21
longth badd quantiyy'rade radio telescope_21 1P2151; tradio www.scapeeeecceecccc focal
What the average of #5
avgl' telescope focal
focal)
PyQLO
selescope, 'wgg telescope focal'), lanswer")
cadd sub queryla, b)
prinetc spargl)
submitt
cancol
Figure 3: An annotation example of a seed question our annotation system. Annotators are required to annotate the question, the IDD of KB item (entities, numeral properties, non-numeral properties). SPARQL, and reasoning steps (QDMR and PyQ.).

---

Auxiliary Annotation System
Property information
Input
P1872
BACK
ifier
Basic information
ID
Label
Description
Also known as
As
As
property qualifier
rumbars
players, plaers. minimum, players, minimam number
players
players game
Distribution of subject types
Entiyy type
Number Top example
Minecraft, Eattle Wesneth, Smash Ultimate
91
board gume)
chess, Monopoly, Thum Taois, backgummon, Escape adventures contract bridge, Secret litler, Uno, Wonders skat
0142/1draad gamet
70
Distribution of unit
Unit
Number
Top example
1248
Factorio, Minecraft. The Battle Wesneth, Super Smazh
player
Bohnerspiel
human
Totopoly
filters
encity tope
min
maN one
Q1880eede game)
31400oo gamee)
player
Q1422 Alcard garme)
human
Examples
Subject
Subject label
Subject number
Subject
Statement
other statements
036417864
Among Us
736
asports diackeling
quantity statement
E2 soldt
link
P111d/quantio P2347gllumnion)
cisck
Q104223925
video game
quantity unit time statement linnk
ERDE7/social
media fni cwars
stoo) P22041price)
055004330
Dying Stay
game
quantity unit time statement
paoediunte soldt
link
pubrsidate sizel
Figure An example of our auxiliary annotation system. Given quantitive property, the system provides comprehensive information including its label, aliases, description, subject type distributions, unit distributions of attribute values, and at most 2000 entities that have this property. Each entity is also provided with other quantitive properties the entity has to help the annotator ask questions with multiple quantity properties.

---

F Prompt Used in paraphrasing
Prompt used in paraphrasing
Give 10 rewrites with the same meaning for each sentence following. Try to keep ALL the content within [] with the symbol in your output. []'s corresponding content is marked after the sentence. Use their meanings, but do not show them in rewrites.
For example: [[A]:[MDDonald']] You know that [A] stands for McDonald's, but use [A] instead of McDonald's in rewrites.
So DO NOT contain []'s corresponding content in your output.
for example:
Among [A], [B] and [C], is the one with the largest load capacity also the most expensive to build? { [A]:MS Allure of the Seas [B]:Wonder of the Scas [C]:MS Oasis of the Scas possible rewrite: Which of the three ships, [A]. [B], or [C], has the highest cost of construction, and is also the one with the largest load capacity?
DO NOT contain []'s corresponding content in your rewrites!! Output the output corresponding to each sentence in strict accordance with the following sample format: sentence 1:
rewrite 1:
rewrile 2:
rewrite
rewrite
rewrite
rewrite
rewrite
rewrite
rewrite
rewrite
Table Prompt used in paraphrasing. We request the model to perform paraphrasing task provide an example. Given question, enclose the mentioned entities in parentheses and anonymize them A/B/C. This step is taken to prevent the model from omitting or tampering with the entity labels during the paraphrasing process. Then, we ask the model paraphrase the anonymized question, but when outputting, should restore the specific entity labels based on the correspondence hetween A/B/C and the entity labels. In this way, we can obtain question paraphrases in a more controlled manner, and the model's output is directly usable.

---

G Prompt Used in Composition
Prompt used in composition
Given some wikidata subgraphs centering on a center wikidata entity called [topic_entit]]] you need to get descriptions for each "opic_entity! base on their subgraph.
The input contains some subgraphs denoted Eeperttcs("Proprrri M in the end.
by *subgraph 1M with meanings of
each "subgraph contains:
[topic_entity] the name of topic entity.
[lype]: the most frequent wikidata type of topic csoity.(some abstact entity may not have a type, then there is no [type] field.)
[triples]: some triples centered on [topic_entity: A triples looks like <subject, property, object>.
The solution this task is: For each *subgraph i*, use the semantics of its [triples] to get the description [topic_entity]]] You need include the meaning of all triples in the description.
description an appositive sentence of hopic__ntityy]] but contain the label of the topic entity. To generate description.......
describing how write description]
ATTENTION:
1. PAY SPECIAL attention directional properties ends with prepositions has inverse property...... 2. The descriptions should be short possible(less than 6 tokens disregarding the entity label), but also need to be diverse and fluid possible. You may rewrite the mcaning of properties. [other atlentions]
Examples of input and your output: <input>
*subgraph 0#:f [topic entity]: Resurrection [type]:literary work [triples]<<<[topic entity], writer, Leo <[topic_entiyy], *Property_Moe Meanyngsen (author:main creator(s) of a written work) (genre:creative work's genre or an artist's field of work (P101))
Tolstoy>, <[topic entity], genre, political fiction))
<endofinput>
[other examples]
<output>
Solvel:
*subgraph_0 0* a subgraph about Resurrection. Base on the meaning of triples,
Answer: |*subgraphh0%:10 [t: @political fiction @ book written by @Leo Tolstoy @]
[other Solutions]
<endofoutput>
<endofprompt>
Now process following questions with subgraphs. Output strictly follows the example format.
Table 10: Prompt used in composition.

---

Function List of PyQL
PyQI.function
Brief description
Basic graph pattern add fact
Add triple pattern.
add_quate
Add triple pattern with quantity relation.
add add_ qua__quatty____ with qualifier
Add triple pattern with quantity relation which is constrained by qualifier.
add add_ hud__quaity__uie btr_bby__guaaa qualifier
Add qualifier with quantity relation which is constraint of triple pattern.
add_ wtt_tyye_cossrrie
Add triple pattern with type constrain.
add filter
Add comparative constrain (>, <, etc).
add_ bind add assignment
Bind the result of expression to variable
Bind some variables/entitics to variablc
add_sub_query
Add sub_query into current query
Arithmetic
add
Addition of some numbers/variables. We also provide ceil and floor versions for add, sub, mul, and div.
sub
The difference between two numbers/variables. The product two numbers/variables.
mul
div
The division of two numbers/variables.
abs
The absolute value of a number/variable.
Aggreggation
add max max
Calculate the maximum value of given variable. Support to return the m-th latest to n-th largest item.
add min
Calculate the minimum value of given variable. Calculate the average value of given variable.
add_avg _av
add sum
Calculate the summation value of given variable. Count the occurrences of matching triples or variables. Calculating the ranking of a variable in list of variables.
add_count
add rank
Boolean
add_compare
Whether two arguments meet a condition (>, etc).
Other
add_sub_query
Add sub-queries to the outer query.
Table 11: A brief introduction of PyQL's functions.

---

I PyQL API implementation example
class PyQl O):
def iinn Lintt__iii (self):
self aarplleppaterreee
self aggregation=[]
self sub_query=[]
self head=" self-head="
self answer"s
def add_ wad__type_consteeee conetccosttrin(seef,, type_id:str, new_var:str):
wdt:P33/wwt::2279 wd:" type_id + ".")
def mod_cconttself,com htt.::bbb:sttt.e..arrrsssss group_ab]=None):
if :roup_obj!=None:
self www.aaggtgoo..gaooeegg apppndd(pp GROUP BY *tgroup_obj)
selheaet_head( SELECT (DISTINCT +count_obj+') AS "tnen_vart') '+' '+group_obj)
else:
self. self.set_head('SEE 'SLECTT (DISNICII 'tcount_obj=') AS "tee_vvrr') ')
def add_ (self, max_obj:str,
objj''''.offseeeeeea 1imit=1)):
self.
www.aagggatio.aaaee.... www.naddppnddnnnnneeee ("ORDER DESC(" max_obj + ")")
if limit!=None"
self self.
www.gggggtion.apeeeeeeee ("LINI "+str(limit))
if offset!=0:
self self.aggeen self. (rrtunnn.s
wwgregation. apppeiiii "+str(offset))
def add_ bind(self, equation:str, var_name:str):
self.add. wawd_trrplee.patemnt BBID( equation var_name
)
self. set.seettssseeear__raaae
def add (slff compare_obl Union[str, float], operator:str, Hmmpare_obj:Uniion float))
float]))
self.add_triple tut..aad_.ippee.patteeeeFfL ')."
str(compare_objj) '+ operator +' str(compare_obj2)
def add assi assaasiignnnn assignment (self, var list: war_s new new_var:str):
self @elf.add_triplee tripedd_riplleeeeteene ("Valees Values **new_var*' ('+' ..joi((va_lisst+*Y)
def coddccoppreeseef, obj1:Union[sti float], op:str, obj2:Union[str,flobd op "
self. belf.add add bind( + obj1
obj2 "TRUE", "FALSE")', "?answer")
self ("sEEEE ?answer")
def add_sub_ ssl_uurr(sself.xuu..eeeeeee
for sub_q in sub_query:
sub_q sparql
self wtt..uubuuurr..ppenddddll.
@property
def (serq)
self self.triple, triple
patlle__aatern_tett
self sub_query_text = ueff___ccostruutssu__uueery
self.aggregation_text cossrs__oostuuuteegggggaaaaaa
if not self head_already head: SELECT DISTINCT '+self.answe
sparql_temp head self. self.sub_quer bubb_query_text self. triple_pattern_tek 'AnJAn' self.aggregation_tex return spargl_temp
head {An'
Figure 5: The implementation of PyQL APIs. For the sake of brevity and better understanding, have omitted some technical details.

---

J Six Structure Considered in Composition
One triple constrain target entity
Description: the birthplace of Rudolf Riedlbauch
Subgraph: <Rudolf Riedlbauch, place of birth, [target_ entttt]] (larget_entity is Dyšina)
Two triple constrain target entity
Description: the house located in Turtle Bay and designed by Perkins Eastman Subgraph: <[targtt_entity] architect, Perkins Eastman>,
[target_entity location, Turtle Bay> (larget_entity] is Turkish House)
One triple constrain target entity, one triple constrain inner variable Description: the place where the child of Llewelyn Lloyd died Subgraph: <[target_entity), child, variabble-
<[inner_ vvaribl_, place of death, [target_entity)> ([target_entity is Llewelyn Lloyd,
is Charles John Andersson)
Two triple constrain target entity, one triple constrain inner variable
Description:
the video game publisher whose founder was educated Shanghai Jiao Tong University
Subgraph: <miHoYo, instance of, video game publisher>
<miHoYo, founded by, variabbaa
<[inner variable], educated ,Shanghai Jiao Tong University>
([argett_entity is miHoYo, [inner_variable is Cai Haoyu)
One triple constrain target entity, two triple constrain inner variable Deseription:
the dircetor the film reccived Academy Award for Best Picture and composed by Kris Bowers babgraph:<(inner_variie director,ttarget___...... [targeteentity)>
<[inner_variablle variable], award received, film received Academy Award for Best Picture> variarr_arrablee]__ composer, Kris Bowers>
(Itarget_entiee entity] Peter Farrelly, variabl Green Book)
Two triple constrain target entity, two triple constrain inner variable
Description: the World Cup Golden Ball winner who participated in the World Cup in Qatar Subgraph: <[target_entity] award received, World Cup Golden Ball>
[targettentityl, participant in, <eriar_/araalee>>>> <[inner_variable], country, Qatar>
sports scason of league or competition, FIFA World Cup> ([target_entity] is Lionel Messi, l variable] is 2022 FIFA World Cup)
Table 12: Six structures considered in composition.


Reviews:
[{"review_id": "REb7JFWMkO", "reviewer": "Reviewer_6fJ7", "summary": "", "questions": "1. What portion of the dataset has entities replaced with 2-hop connections? If we discard these, how much of the dataset remains?\n2. With gold Entity and Relation linking, what are the new numbers for any baseline you can compute on the portion of the dataset remaining? (Can you recompute table 4 last row with the reduced dataset?)\n\nBased on these numbers, I'm willing to raise my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n### Q1: What portion of the dataset has entities replaced with 2-hop connections? If we discard these, how much of the dataset remains?\nHere are some statistics of MarkQA.\n\n| Question Type | Number of Question |  Proportion |\n|---|---|---|\n| Questions only require numerical reasoning | 20745  | 64.4%  |\n| Questions with one-hop connections | 7606  |   23.6%  |\n| Questions with two-hop connections | 3862  | 12.0%  |\n\nDiscarding the questions with two-hop connections, there remain 28351 questions accounting for 88.0%.  \n\n### Q2: With gold Entity and Relation linking, what are the new numbers for any baseline you can compute on the portion of the dataset remaining? (Can you recompute Table 4 last row with the reduced dataset?)\n\nHere lists the performance of the T5_PyQL after dropping the questions with two-hop connections:\n\n| Methods  |  Overall |  I.I.D |  Compositional  |  Zero-shot  |\n|---|---|---|---|---|\n| T5-base  |  45.5  |  87.5  |  71.8  |  11.7  |\n|  w/ gold E  |  49.6  |  94.8  |  77.0  |  13.7  |\n|  w/ gold R  |  56.0  |  89.2  |  75.3  |  30.0  |\n|  w/ gold ER  |  61.2  |  95.9  |  81.1  |  34.1  |\n\nCompared with the original, the performance of gold ER does not increase significantly (from 59.6 to 61.2) after discarding the two-hop questions. It indicates that the difficulty is primarily brought by numerical reasoning instead of multi-hop reasoning (To some extent, multi-hop reasoning has somehow been solved. The F1 score can achieve 94% on another multi-hop KBQA dataset, ComplexWebQuestion, by a T5 model given golden linking results (Hu et al., 2022).).\n\n> Hu, X., Wu, X., Shu, Y., & Qu, Y. (2022, October). Logical form generation via multi-task learning for complex question answering over knowledge bases. In Proceedings of the 29th International Conference on Computational Linguistics(pp. 1687-1696).\n\n&nbsp;\n\n**In addition to the questions you raised, please allow us to elaborate some necessary clarifications on other aspects.**\n\n&nbsp;\n\n\n### 1. The overly complex question is unrealistic.\nIt is worth noting that the difficulty of the example in Figure 1 stems from three factors: \n\n- Complex numerical reasoning, \n\n- Two-hop reasoning,\n\n- Two entities were replaced with different subgraphs.\n\nIf a question does not meet all three conditions at the same time, it is likely to be considered acceptable. \n\nIn our dataset, we have controlled the token length of the question to avoid the occurrence of overly complex and impractical questions.\nTherefore, a question that satisfies all three conditions is very rare (less than 1%) and this kind of question can be considered as representing the most complex structures in MarkQA.\nHowever, inevitably, some overly complex questions may still arise.\n\nThe aim we posting this example is to better illustrate how numerical reasoning and multi-hop reasoning are intertwined, instead of encouraging extremely hard questions. We agree that it would be more readable to choose another more realistic question as an example.\n\nAnother consideration is that, though difficult (and a little impractical), it is still a question that humans can solve.  Including a small subset of extremely hard questions enhances the robustness and compositional generalization capability of the model.\n\n### 2. Lack of details about the numerical properties selected and more examples of numerical complexity not present in prior datasets.\n\nThanks for your suggestions, we promise to add selection strategies of numerical properties in the appendix of the revised version and include more representative questions in the introduction.\n\n#### Details about the selection of numerical properties(num_p).\nWe first select the num_p with more than 100 statements or qualifiers recorded in Wikidata (of January 13, 2023). This leaves us with 377 num_p. \nThen, we manually remove some num_p requiring high-level domain knowledge (e.g. redshit, pKa, longitude of ascending node). Each removed num_p  is checked by two annotators. If both annotators agree that it is hard to ask meaningful NRQ for this num_p, then this num_p is dropped. Finally, we retain 318 num_p.\n\n#### More examples of numerical complexity are not present in prior datasets.\nIn terms of the number of operations, previous datasets involve numerical reasoning at most once (their #NRQ is nearly equal to #Avg NS in Table 1). And in terms of the kind of operations, they only consider the Superlative, COUNT, and Comparison. Their question is like:\n\n- Which lake has the largest catchment area? (GrailQA, requires 1 superlative operation) \n-  How many popular musics are influenced by the band which has ISNI 0000 0001 2369 4269? (KQA pro, requires 1 count operation)\n-  Which player started his career after 2004 and was the Cleveland Browns draft ? (CWQ, requires 1 comparison operation)\n\nHowever, many questions that are common in the real world are not supported by the previous dataset. Our dataset encompasses a broader range of numerical questions, such as:\n-  How many of Japan's largest sports stadiums could be filled with the number of new COVID-19 infections in Japan in 2023? (Require 3 operations: 1 summation, 1 superlative, and 1 division)\n-  How many times longer is the longest aircraft carrier than the shortest?  (Require 3 operations: 2 superlatives and 1 division)\n- How much more VAT do you have to pay to buy the most expensive iPhone 13 in Russia than in Japan? (Require in total of 4 calculations: 1 superlative, 2 multiplies, and 1 subtraction)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They offer a strong motivation in that other datasets or baselines offer only a limited evaluation of numerical questions in the KBQA setting, and offer a concrete resource that could be very useful for advancing the field in this direction. The paper is written well.", "reasons_to_reject": "1. Key weakness, which is arguably subjective : I believe replacing any entity in the (already numerical question) with a 2-hop subgraph is adding complexity for complexity's sake, while simply being unrealistic.\nTake the question in Fig 1 for example : \"During 2017 to 2021, how much more annual increase in total assets is the company founded by Bill Gates and Paul G Allen than the company whose founder was born in Chappaqua and educated at Hamilton College\"? This does not strike me as a question an actual human would ask. \n\nInstead, you could simply ask the two sub-questions in any of the existing systems in literature, get Microsoft and Netflix respectively, then ask \"During 2017 to 2021, how much more annual increase in total assets is Microsoft than Netflix?\". I also think a 1-hop replacement directly in the question is reasonable. (\"... total assets is company founded by Bill Gates than ...\")\n\nWhile I recognize this may seem arbitrary, there is a reason we don't see a lot of 4 or 5-hop questions in the general KBQA literature - it's unrealistic. To be clear, I think it's great that this work adds certain *numerical* complexities such as subtraction etc. But I'm not in favor of adding pointless complexity, and I'm current not sure how much of the poor performance of the baseline is being driven by such.\n\n2. Lack of details about the numerical properties selected and more examples of numerical complexity not present in prior datasets would be useful.", "questions_to_authors": "1. What portion of the dataset has entities replaced with 2-hop connections? If we discard these, how much of the dataset remains?\n2. With gold Entity and Relation linking, what are the new numbers for any baseline you can compute on the portion of the dataset remaining? (Can you recompute table 4 last row with the reduced dataset?)\n\nBased on these numbers, I'm willing to raise my score.", "paper_topic_and_main_contributions": "This work proposes a new task NR-KBQA which focuses on complex numerical reasoning in the KBQA setting. To this end, they introduce a new dataset called MarkQA, composed of 1K 'seed' questions naturally created by human annotators, and scaled up to 32K examples using a mixture of automatic and verification processes. They also introduce an intermediate expression using PyQL, which aims to simplify the creation of a SPARQL query for those unfamiliar with it, and can be deterministically converted to such.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lCt8X3f48z", "reviewer": "Reviewer_PuQT", "summary": "", "questions": "Question A: Why don’t you consider this work as a resource paper? You mentioned in the introduction that no numerical reasoning questions exist for KBQA, but Table 1 shows other datasets that contain NRQ. Did you mean 100% only NRQ dataset? But then, can you frame your contribution as ‘you are introducing NRQ for KBQA’ as you mentioned in the introduction? This is contradictory and very confusing. Also, jeopardizing contribution claims.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Why don’t you consider this work as a resource paper?\nWe acknowledge that our main contribution is the MarkQA dataset, but we consider our work as a Question Answering track paper for 2 reasons:\n\n1. In addition to the contribution of the dataset resource, this is the first work that thoroughly analyzes and discusses numerical reasoning questions over knowledge bases. To elaborate, we first formally model the task. To explore how to tackle this task, we discuss and provide reasoning paths as further resources. Also, we implement some baselines and showcase the usefulness of the reasoning paths.\n\n2.  PyQL has shown its effectiveness for QA systems. This can also be seen as an improvement to the existing systems. What's more, in addition to NR-KBQA, we hope to showcase that representations like PyQL, designed through this scenario, can help advance the development of interpretable reasoning. More details can be referred to in Q3. \n\n###　Q2: Can you frame your contribution as ‘you are introducing NRQ for KBQA’?\nPrevious datasets, indeed, contain some questions with simple numerical reasoning. \n\nHowever, as we mentioned in the Introduction and Related Works, these datasets were not primarily focused on addressing numerical reasoning. \nThe numerical questions in current datasets are far from being solved, both in terms of the number of numerical questions and the scenarios that their questions can support. \nNone of them have provided a comprehensive resource for research from problem formulation, dataset resource, reasoning path representation, and baseline implementation. \n\nTo promote the research in this area, we for the first time formulate this task and provide a high-quality, fully-annotated, and large-scale dataset.\nBased on your feedback, we will rephrase this statement to \"We explore and discuss this issue for the first time.\" in the revised version to reduce controversy.\n\n### Q3: PyQL is a very focused/narrow contribution.\n\nRegarding the significance of PyQL, its contributions to the community can be summarized as follows:\n\n1. A transparent reasoning path. For a deep learning model, it is unknown how it obtains the output even though the answer is correct. If a reasoning path is offered, a model can directly learn how the answer is derived, improving the unexplainability.\n\n2. Moreover, PyQL is a supervision signal, which is also a well-structured resource of reasoning, as the model can learn more efficiently with the symbolic form. For humans, an NRQ can be solved by a combination of retrieving and calculation steps. Given these steps, how to tackle a reasoning problem can be learned strategically. In addition, with the rise of CoT in LLMs, PyQL can be put into the few-shot prompts as a step-by-step reasoning demonstration. Also, the CoT itself is a supervision signal for the auto-regressive model and an interpretable path for human beings.\n\n3. More readable, lower the threshold. Speaking of a SQL-like query, it is always time-consuming to write and revise when involving complex numerical operations like some aggregations. PyQL is more readable than SPARQL and can be converted to SPARQL non-destructively.  This provides a convenient resource for the KBQA community to write, check, and generate SPARQL. As mentioned in the Experiment, the average length of PyQL is only 57% of that of SPARQL.  After using PyQL, performance gains range from 33% to 69%.\n\nIt is also worth noting that PyQL is not a specific design for our dataset. It supports most of the commonly used SPARQL grammar, including basic graph pattern, aggregation, sub-query, arithmetic, filter, and boolean. This is general enough to model most of the questions in previous datasets, not just ours. Besides, it is easy to extend the functionality through secondary development. \n\nOur aspiration of PyQL is to serve as a transparent reasoning pathway in the KBQA area, which can be used as a reasoning resource, not limited to a narrow usage for annotation or learning simplicity on our dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tFocuses on more challenging questions – NRQ in KBQA setting.", "reasons_to_reject": "•\tMain contribution claim of the paper is not clear. Please refer to questions section. \n\n•\tI wonder this paper should be in ‘resource’ track. Otherwise, only remaining contribution is PyQL format and its effect of NRQ, which is a very focused/narrow contribution.", "questions_to_authors": "Question A: Why don’t you consider this work as a resource paper? You mentioned in the introduction that no numerical reasoning questions exist for KBQA, but Table 1 shows other datasets that contain NRQ. Did you mean 100% only NRQ dataset? But then, can you frame your contribution as ‘you are introducing NRQ for KBQA’ as you mentioned in the introduction? This is contradictory and very confusing. Also, jeopardizing contribution claims.", "paper_topic_and_main_contributions": "The paper presents a KBQA dataset with the main focus of numerical reasoning (and also multi-hop). Authors argue that existing KBQA datasets do not contain sufficient numerical reasoning questions and hence there is a clear void in this space. Hence, they introduce MarkQA, a dataset that contains 100% numerical reasoning questions and answers.\n\nPaper also presents PyQL, which is a logical form written in Python language. Authors claim that PyQL makes it easier for users to write compared to more formal and traditional SPARQL query language.  A PyQL can generate executable SPARQL queries and they have showed in evaluation, it is easier to generate through the learning algorithms.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ISf5OosgUR", "reviewer": "Reviewer_Y7yn", "summary": "", "questions": "How do you obtain paraphrases in 4.1.2 Paraphrasing?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: How do you obtain paraphrases in 4.1.2 Paraphrasing?\nParaphrasing is obtained by gpt-3.5-turbo, we design a prompt to obtain paraphrase for each question. Details can be found in Appendix C. \n\nWe request the model to perform a paraphrasing task and provide an example. \n\nGiven a question, we enclose the mentioned entities in parentheses and anonymize them as A/B/C. This step is taken to prevent the model from omitting or tampering with the entity labels during the paraphrasing process.                                        \n\nThen, we ask the model to paraphrase the anonymized question, but when outputting, it should restore the specific entity labels based on the correspondence between A/B/C and the entity labels.\n\nIn this way, we can obtain question paraphrases in a more controlled manner, and the model's output is directly usable.\n\n\n### Q2:  The experiments are not comprehensive and lack the ablation of numerical parts.\nThanks for your suggestions, we promise to add the ablation study of the numerical part in the revised version.\nHere is a detailed analysis of numerical reasoning and multi-hop reasoning (The model is T5_PyQL). \n\n| Question type  |  Overall |  I.I.D |  Compositional  |  Zero-shot  |\n|---|---|---|---|---|\n|  All     |    43.7  |  84.1  |  67.9  |  10.9  |\n|  Only require numerical reasoning   | 45.4  |  92.4  |  69.6  |  13.3   |\n|  Both require numerical reasoning and multi-hop reasoning |  40.8  |  68.8  |  66.2  |  5.8  |\n|  Both require numerical reasoning and one-hop reasoning  |   45.7  |  74.3  |  75.2  |  6.5  |\n|  Both require numerical reasoning and two-hop reasoning  |  30.7  |  57.1  |  47.8  |  4.3 |\n  \n\nRow 1 represents the overall performance of all questions. \n\nRow 2 showcases the performance of a subset of questions that **only** require numerical reasoning. By excluding newly generated questions in the composition stage, this subset provides a measure of the inherent difficulty of numerical reasoning itself. The performance only saw a slight increase of 1.7%, indicating that numerical reasoning is indeed challenging and that the dataset's primary difficulty stems from this aspect. \n\nRow 3 displays the performance of questions that require **both** numerical reasoning and multi-hop reasoning. This subset of data is complementary to the data in Row 2. The performance dropped by 2.9%, suggesting that the inclusion of multi-hop reasoning further increases the question's difficulty. \n\nRows 4/5 present the performance of questions that require numerical reasoning and one-hop/two-hop reasoning, respectively.\n\n&nbsp;\n\nWe also add another baseline (T5-large_PyQL). The corresponding accuracy is shown below.\n\n| Method  |  Overall |  I.I.D |  Compositional  |  Zero-shot  |\n|---|---|---|---|---|\n| T5-Large | 46.26 | 87.80 |  78.95 | 8.52 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Numerical reasoning is a challenging and important problem in NLP. The authors propose a useful dataset that focuses on numerical reasoning, which can be a valuable testbed for future work.", "reasons_to_reject": "1. Missing limitations. \n\n2. A clear formulation of a sample in MarkQA should be described in a paragraph, referring to Figure 2 alone is not enough.\n\n4. The experiments are not comprehensive. More baselines should be presented as a data resource paper. There are no ablation analyses for the numerical reasoning part.", "questions_to_authors": "How do you obtain paraphrases in 4.1.2 Paraphrasing?", "paper_topic_and_main_contributions": "In this paper, the authors propose a new challenging task, NR-KBQA, which focuses on numerical reasoning and its combination with multi-hop reasoning. \n\nSpecifically, they construct a large-scale dataset called MarkQA of 32K examples each with  QDMR  question decomposition and corresponding logic form in Python, i.e., PyQL.\n\nPyQL provides a step-by-step reasoning process and can directly be compiled to executable SPARQL queries.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a KBQA dataset with the main focus of numerical reasoning (and also multi-hop). Authors showed limits of existing resources and propose a novel resource to address this issue.
Decision: Accept-Main