Best of Both Worlds: Towards Improving Temporal Knowledge Base Question Answering via Targeted Fact Extraction
Authors: Nithish Kannen
Udit Sharma
Sumit Neelam
Dinesh Khandelwal
Shajith Ikbal
Hima Karanam
L Venkata Subramaniam
Venue: EMNLP 2023 Main (2023)
Keywords: Temporal KBQA
Fact Extraction
Semantic Parsing
Questional Answering
TL;DR: A novel combination of KBQA and text corpus based QA, where a targeted temporal fact extraction technique is used to assist a KBQA system wherever it fails

Abstract: Temporal question answering (QA) is a special category of complex question answering task that requires reasoning over facts asserting time intervals of events. Previous works have predominately relied on Knowledge Base Question Answering (KBQA) for temporal QA. One of the major challenges faced by these systems is their inability to retrieve all relevant facts due to factors such as incomplete KB and entity/relation linking errors. A failure to fetch even a single fact will block KBQA from computing the answer. Such cases of KB incompleteness are even more profound in the temporal context. To address this issue, we explore an interesting direction where a targeted temporal fact extraction technique is used to assist KBQA whenever it fails to retrieve temporal facts from the KB. We model the extraction problem as an open-domain question answering task using off-the-shelf language models. This way, we target to extract from textual resources those facts that failed to get retrieved from the KB. Experimental results on two temporal QA benchmarks show promising ~30% & ~10% relative improvements in answer accuracies without any additional training cost.

Best of Both Worlds: Towards Improving Temporal Knowledge Base Question Answering via Targeted Fact Extraction
Nithish Kannen St Udit Sharma", Sumit Neelam', Dinesh Khandelwal
Shajith Ikbal", Hima Karanam"*, L Venkata Subramaniam
IBM Research, India
Amazon Alexa AI, UK
Abstract
Question
Who was the President of the United States during WW2?
Temporal question answering (QA) is a spe- cial category of complex question answering task that requires reasoning over facts assert- ing time intervals of events. Previous works have predominately relied on Knowledge Base Question Answering (KBQA) for temporal QA. One of the major challenges faced by these systems is their inability to retrieve all rele- vant facts due to factors such as incomplete KB and entity/relation linking errors (Patidar al., 2022). A failure to fetch even single fact will block KBQA from computing the an- swer. Such cases of KB incompleteness are even more profound in the temporal context. To address this issue, we explore an interest- ing direction where targeted temporal fact extraction technique is used to assist KBQA whenever it fails to retrieve temporal facts from the KB. We model the extraction problem an open-domain question answering task using off- the-shelf language models. This way, we target extract from textual resources those facts that failed to get retrieved from the KB. Experimen- tal results on two temporal QA benchmarks show promising ~30% & -00 relative im- provements answer accuracies without any additional training cost.
AMR
(h haaeeorg-rolee
:ARGO (a amr-unknown]
:RRG1 (c/ country
:ame (n name
:op1 "United"
:ap2 "States"))
:RG (p president)
:time [w / war
(n2 / name
cop1 "WW2")))
-expression
À have-org- 91(h, "United States", "president") interval(hi, h) A Interval(ti, "WW2") A overiap/hl, ti)
Main-A
have-arg-role 91/h, "United States", "president") interval(hi,h) h)
Aux-A
Interval(ti, "WWz')
Figure I: Example question, its AMR and X-expressions
NLP and Semantic Web communities have shown immense interest in this problem lately. Work in the NLP community has primarily focused on using textual data as a knowledge source, and typically uses deep-neural models to represent knowledge and for reasoning. While those approaches achieve impressive accuracies (Zhang et al., 2021, 2020), they are typically limited in their reasoning capa- bilities. Although reasoning is starting to receive attention in NLP community (Wei al., 2023), rea- soning over large amounts of unstructured knowl- edge in text is still a challenge. Work in the Se- mantic Web community has primarily focused on Knowledge Base Question Answering (KBQA), where a KB is used to retrieve facts. While better- equipped for complex reasoning (Bhutani et al., 2020; Wu et al., 2021), they typically suffer from incomplete knowledge (Patidar et al., 2022). In this paper, we present a novel combination of successful elements from past approaches, i.e., KB-based and text-corpus based, for complex QA, in a way to overcome their individual limitations.
1 Introduction
Complex Question Answering involves the integra- tion of multiple facts identified and extracted from disjoint pieces of information (Vakulenko al., 2019; Saxena et al., 2020; Fu et al., 2020: Neelam et al., 2022a). Two critical components in systems trying to achieve this are: 1) Knowledge Source- to retrieve relevant facts, and 2) Reasoning to reason over relevant facts to derive the final answer. Both
'hhis work was done when Nithish Kannen was at IBM Research, India as Summer Intern. At that time he was also an (integrated
"emedorr'smmstrsss s) Electrical Engineering Student at IIT Kharagpur, India.
"Correspondence e-mails: nithishkannende @ gmail.com, sha- jmoha @ in ww..iibcomm...mmm hkaranam @ in. ibm.com

---

Our combination strategy is to use targeted extrac- tion technique that would assist KBQA whenever it fails. KBQA failure when it fails to retrieve relevant facts needed to answer the question from the KB because of reasons such as incomplete KB and inaccurate entity/relation linking. Identifica- tion of these specific points of KBQA failure is critical step in our approach, which is enabled by decomposition of the question's logical represen- tation obtained by semantic parsing. The targeted extraction technique compensates for those KB fail- ures by extracting facts from textual resources in an open-domain QA fashion. Concretely, we make effective utilization of the KB (reliable but not ex- haustive) and the textual resources (vast but noisy), essentially combining the best of both worlds.
2 Our Approach
Figure 2 shows a block diagram of our proposed approach, with two groups of modules: 1) Upper line of KBQA pipeline modules, and 2) Lower line of Extraction pipeline modules for targeted fact extraction. Our strategy to bank on the KBQA pipeline, owing to its reliability, and trigger Extrac- tion pipeline only to aid KBQA when it fails.
2.1 KBQA Pipeline
For KBQA pipeline, we adopt a modular design as in (Neelam et al., 2022a: Kapanipathi et al., 2021), with two high-level modules: (1) Question Under- standing to derive logical semantic representation of the NL question and to further decompose it. (2) KB Linking and Answering to ground the Entity and Relation mentions in the logical representation onto the KB. Both these modules in our approach are similar to (Neelam et al., 2022a) except for the event specific decomposition described next.
In this work, we focus on temporal questions. They additionally involve reasoning over temporal facts, i.e., assertions on points and intervals in time of events'. For example, to answer Who was the President of the United States during World War 2?, we need know the temporal facts of both World War 2 and the list of all US presidents.
We use X-expression constructed from Abstract Meaning Representation (AMR)(Banamnn 2013) for logical semantic representation of the questions (Neclam et al., 2022a). Figure 1 gives an illustration of NL question, its AMR and X- expression. As shown, the X-expression compactly represents the mentions of events in the question as its sub-components, facts about those events needed from the knowledge source, and the reason- ing steps needed to derive the final answer. We de- compose X-expression into two components to help localize the points of KB failures: 1) Main-X: Part of the X-expression related to the unknown vari- able, i.e., main event being questioned. In Figure 1, is the unknown variable, whose value if found is the answer. 2) Aux-X: part of the X-expression not related to the unknown variable, but related to the other events in the question as shown in Figure 1. This part adds temporal constraints on the answer candidates. We use a simple rule-based approach to perform this decomposition, where unknown vari- able is used as anchor segregate the respective components. We provide additional details in A.
et al.,
The main contributions of our work are: 1) A novel combination of KBQA and lextual-extraction for temporal question answering. 2) X-calculus based semantic representation to identify KBQA gaps. 3) An targeted extraction of temporal facts from textual resources using off-thc-shelf models. We show that it is possible to achieve significant improvements on two temporal QA benchmarks even without any task-specific training and helieve the promising initial results will foster research in this direction.
QA style approach for
Related Work: GRAFT-Net (Sun et al., 2018) and PullNet (Sun et al., 2019; Xiong et al., 2019) uti- lize both KB and textual resources but do not ad- dress temporal reasoning. Unlike approaches using end-to-end neural models, we adopt modular ap- proach as in (Neelam et al., 2022a) because of the flexibility it offers to combine textual extraction with the KBQA, with additional benefits such as interpretability and easy domain-udaptation An- other main difference in our textual extraction ap- proach is the usage of LMs off-the aff--he-shelles, hence bypassing the need for domain-specific training and datasets that are hard to obtain. A detailed literature review is presented in A.6.
2.2 Targeted Temporal Fact Extraction
We know KBQA failure has occurred when the complete lambda expression fails to return an an- swer from the KB. The first critical step in our approach to localize points of KBQA failure within the lambda expression. For this, we exam- ine answers derived independently for the aux-A
'In this paper, we consider entities and facts (represented as triples in KBs) with associated time intervals as events. For example, World War 2, (start time: 1939, end time: (945))

---

Knowledge Base Question Answering (K8QA)
NL Question
Question Understanding
KB Linking and Answering
Final Answer
AMR
À Expression
Reformation
Entiiy/ Relation linking
SPARCL
À Decomposition
main-AA, a
temporal fact (aux)
main answer candidates
Targeted Extraction Temporal Fact from Text
Temporal Reasoner
Dacument
Passage Ranker
RCQA Style Fact Extraction
A-to-NL
Retriever / Ranker
Completion
(temporal facts (main &ux))
Figure 2: An illustration of the proposed approach. Upper line of modules correspond to the KBQA pipeline, while lower line of modules are related to targeted fact extraction from textual resources.
and main-À À using the KBQA pipeline in order to classify the failure into one of the below cases?:
We take that part of the main X-expression that would fetch the answer candidates from the KB (leaving out the temporal fact specific components), and pass that onto the KBQA pipeline. For each answer candidate obtained, extract the temporal information via the extraction pipeline as in A.3.2. Temporal Reasoning. Upon successful gather- ing of all the temporal facts (i.e., time intervals), either from the KB from text, ral Reasoning module select the final answer from among the answer candidates of the main-A. For example in Figure 1, X-expression component overlap(hi, Li) corresponds to the temporal reason- ing step. which essentially represents the overlap between the time intervals tí (of Ww2) and hi (of answer candidates of the main-A). Candidates that comply with the reasoning condition chosen the final answer. Other temporal reasoning cate- gories handled include before, after, now etc.. Al- gorithm describes our overall flow sequence.
Aux Failure: Temporal fact corresponding to the temporal constraint (aux-X) ) missing the KB.
For example, in Figure 1, when time interval of World War is missing in the KB.
Main Failure. Temporal fact(s) corresponding to the unknown variable is missing in the KB. For example, in Figure 1, when time interval of Franklin D. Roosevelt (the US president during WW2) as president in office is missing.
Tempo-
The goal now is to extract the identified missing facts from textual resources using the extraction pipeline (described next). In this way, we assist KBQA using targeted fact extraction, which has several advantages: 1) A focused extraction from text (guided by KB) makes textual fact extraction, which usually noisy, reliable. 2) The comple- menting strengths textual resources and KBs are leveraged to overcome their individual limitations. Aux Failure. Answer for aux-À is time interval, i.c. composed of time intervals of all the events part of aux-À, that imposes temporal constraints the answer candidates. We extract the KB missing fact through the extraction pipeline and construct reformed >-expression, which is simply replacing the auxiliary part of the original X-expression with the extracted time interval. This will enable answer derivation without fetching facts related to aux-À from KB. We show an example of this in 1.3.1.
2.3 Extraction pipeline
There are scenarios where may look to extract facts from textual resources: 1) KBQA fails for aux-À because of Linking failure, 2) KBQA fails for aux-À because missing temporal fact, and 3) KBQA fails for main-A because of missing tem- poral fact. We pose extraction as an open-domain QA problem where: 1) for each missing fact, the corresponding NI. query is generated, 2) top-k pas- sages relevant to the NL query retrieved from textual resources, and Reading Comprehension QA (RCQA) style answer derivation is performed by treating NI query as the question and top-k re- trieved passages as the context. We show that even a naive pipeline with pre-trained LMs as the back-
Main Failure: Here we attempt to extract missing facts corresponding to main-X. However, we do not fully rely on textual resources for main-X, because it represents event with unknown variable.
Note that there could also be combination of failures. We use the flow sequence in Algorithm to handle all cases.

---

bone demonstrates significant improvements and argue that an improved fooainn-rained, pipeline will only further boost performance.
2) KB+ TemporalText extracting temporal facts only from text and the remaining facts from KB, and 3) Open-Domain QA- A state-of-the-art open- domain- system called RAG (Lewis et al., 2020) that answers purely from text. RAG consists of a retriever based on Dense Passage retrieval (DPR) and generator based on BART jointly trained. We refer the readers to the original paper for more details this baseline.
À to NL. For example, for the aux- in Figure 1, the equivalent NL query is When was Ww2?. We use simple rules to convert À of the missing facts into its corresponding NL query. Since we deal with temporal facts, all the queries start with When, and we add was or did depending on whether the event being considered entity-based or triple-based. Document retrieval. For each NL query, we per- form document retrieval in 2 steps: first, get list of relevant documents and then choose top-k scored list of passages from them. (1) Entities of the question are extracted using BLINK (Wu et al., 2019) and Wikipedia pages of all the enti- ties are collected allowing minimal lexical varia- tions?. We also use NL text query generated from X-expression to search on MediaWiki API (2) We use Siamese-BEi BERT networks (Reimers and Gurevych, 2019) to rank passages within the re- trieved documents. The Bi-Encoder picks out top- 50 relevant passages based on passage-query simi- larity and the Cross-Encoder re-ranks them. We use public checkpoints trained on the MS-MARCO dataset (Bajaj et al., 2018).
3.1 Implementation Details
Our system pipeline is implemented using Flow Compiler" (Chakravarti et al., 2019) that stitches together the gRPC services of the individual mod- ules. X-expressions are defined using ANTLR grammer. SPARQL queries are run public Wiki- data end point". We reuse the KBQA pipeline im- plementation of (Neelam et al.. 2022b).
Results and Discussion
Table 1 shows performance comparison of our approach (KB+Tex)) against the baselines on TempQA-WD WD dataset. KB+Text achieves an im- provement of 0.095 in F1 score (~30% relative improvement) over Only-KB, demonstrating the ef- fectiveness of our approach in making a targeted utilization of the textual resources to assist KBQA, Comparison to KB+ TemporalTex illustrates the reliability of facts obtained from KB, whenever available. Performance of Open-Domain QA is inferior to Only-KB, illustrating the superior rea- soning capability of KBQA systems.
QA based fact extraction. We use top-3 ranked passages context derive answer for the NL query in Reading Comprehension QA (RCQA) style. We use BERT trained on the SQUAD (Ra- jpurkar al., 2016)) dataset as the QA model. Note that RCQA style extraction gives one answer, which is sufficient for point-in-time extraction. For time intervals (start and end times) we further take the sentence identified by the RCQA model and use its AMR tree obtain time interval by examining sibling nodes to that containing the RCQA answer.
System
Precision
Recall
F1
Only-KB
0.320
0.329
0.321
KB+TemporalText
0.224
0.227
0.212
Open-Domaie QA (RAG)
0.291
0.240
0.252
KB+Text (proposed)
0.423
0.434
0.416
Table 1: Performance comparison TempQA Tem.QA-WD.
3 Experimental Setup
We used Wikidata as KB and Wikipedia textual resource, and evaluate on two aspects: 1) Tempo- ral QA performance of the overall system and 2) Targeted temporal fact extraction performance. We experiment with 2 datasets: 1) TempQA-WD (Nee- lam et al., 2022b) with 839 questions and 2) Time- Questions (Jia et al., 2021) (Train-9708, Test-3237)
Since extraction pipeline is a critical component in our system, we also evaluate its independent accuracy using a small set of 3709 NL queries (added in supplementary material) generated from our system for facts existing in KB. The extraction pipeline performs better than a statc-of the- open domain QA system as shown in Table 3.
Baselines: 1) Only-KB- answers only from KBQA,
Due discrepancies discussed in A.4, we could not use our KBQA pipeline to evaluate on Time- Questions dataset. Alternatively, we used the end-
hhups://pypi.co "tugs://byp.oogggooee///uzzzzzz org/ppojjec////uuyywwyy/////////
whtpp://www.sseet.nee/ /hanss//hugggnggtccc.o//
httpss///uurrywwiiiittttoo

---

Question: Who was the President during Dawes Plan?
A-expression: À a. have-org-role-Sh (h, "president") interval (hi, h)
KBQA
type its2, "Dawes", "plan") intervel (p2i, "Dawes") overlap(bl p2i)
Is Answerable from NBQA?:No
KBQA Issue: Auxiliary Fact Missing (temporal information of Dawes Plan)
Missing Fact Extraction
Retrieved textual info:
The Dawes Plan proposed by the Dawes Committee, chaired by Charles G. Dawes a plan 1924 that successfully resolved the issue of World War reparations that Germany had to pay.
Auxiliary temporal fact: 1924
Reformed A-expression: X a. interval (hi, h) have-org-role-91 (h, a, "president,""
interval (p22, date (id-mmm-1224]] overlap (hi, p21)
Answer with reformed A-expression (from KB): Calvin Coolidge
Figure 3: Illustration of working example showing the KBQA failure occurring due missing auxiliary fact that is substituted by temporal fact extraction and finally reforming the lambda expression by hard-coding the missing fact.
Error
Incorrect extraction
KBQA
Parsing enor
Miscellaneous
Count
21
9
12
Table 2: Error analysis on 50 randomly selected test instances incorrectly answered by our system.
System
Precision
Recall
missing temporal fact '1924' is extracted from text
Open- momn-Domain QA (RAG)
0.051
0.048
0.049
and the X-expression is reformed.
KB+Text Extraction
0.171
0.163
0.165
Code and experimental setup used for our ex- periments at
Table 3: Evaluation of Extraction pipeline.
wc//ree//man//arrgeeed extrri/ttttrttteeeeetttttttooo
System
EXAQT
EXAQT+
to-end trained EXAQT (Jia et al., 2021) system as the baseline KBQA system that was purpose- built for the specific dataset. With this, we then built equivalent of our proposed approach called EXAQT+TexI, by running our Extraction pipeline wherever EXAQT fails answer. The performance of EXA@T+Text is better for all threshold values with improvement reaching upto 10% as shown in Table 4 This demonstrates that our approach of targeted extraction backing up KBQA is resilient to changes in underlying KBQA. In order to benefit future work, we conduct an егrог analysis by man- ually examining 50 randomly selected test samples from TempQA-W2 that were incorrectly answered by out proposed system. We classify the error into following types: 1) Incorrect extraction: the text pipeline extracted incorrect facts, 2) KBQA error: KBQA produced erroneous answers after missing fact was correctly extracted from text, 3) Parsing error: incorrect parsing of the question, 4) Mis- cellaneous: consists of miscellancous issues like lexical and date representation variations. Table 2 shows the results of the experiment and further highlights the scope of improvement in the extrac- tion pipeline. Figure 3 is an illustration of how a KB Aux- Fux-Failure handled in our pipeline. The
Threshold]
R
0.1
0.396
0.574
0.435
0,406
0.586
0.446
0.3
0.446
0.551
0.465
0,464
0.570
0.483
0.5
0.466
0.520
0.469
0.496
0.553
0.499
0.7
0.468
0.490
0.460
0,505
0,531
0.497
0.9
0.458
0.451
0.440
0.506
0.502
0.486
Table 4: Performance TimeQuestions test set.
Conclusion
In this paper, we propose an approach to combine the knowledge resources of KB (structured) and text (unstructured) for temporal QA by using tex- tual resources to aid wherever KBQA fails. In our approach, semantic representation of the question as X-expression is used to represent the set of facts and reasoning required to answer a question. Those facts that failed to get fetched from KBQA are ex- tracted from the textual resources using RCQA style fact extraction. This way, we perform tar- geted extraction of temporal facts to compensate for KBQA failures. The results of experimental evaluation on two temporal QA benchmark show the effectiveness of our approach even without any additional training cost. We additionally conduct error analysis to highlight the scope of improve- ment in order to guide future work.

---

6 Limitations
for sembanking. In Proceedings of the 7th Linguis- tic Annotation Workshop and Interoperability with Discourse, LAW-ID@ ACI. 2013, August 8-9, 2013, Sofia, Bulgaria, pages 178-186. The Association for Computer Linguistics.
The proposed approach focuses only on tempo- ral reasoning and work is required to generalize it across other reasoning types. In this work we addressed temporal QA as we identified missing KB facts as major pain point in the temporal context (adding temporal facts to KBs has started gaining momentum only in recent times). The pro- posed approach will have to be modified handle more complex questions that can feature multiple aux-À (temporal constraints). Our À decomposi- tion algorithm that is hand-crafted works well for temporal questions, but may have to be tested for other reasoning tasks with appropriate modifica- tions. It would be interesting to come up with learning-based methods for the same. The Extrac- tion pipeline is triggered only when KBQA fails to return answer. However KBQA approaches can also produce wrong answers and we can poten- tially develop methods to predict such errors. To aid future research, we provide a detailed error analysis in Appendix highlighting areas that re- quire improvement. Our Extraction pipeline has a large scope of improvement. The document re- trieval approach is purely based on lexical overlap and considering query semantics would improve it. Our QA and ranker L.Ms can be finetuned for the domain fo-fo Alternatively, training end- to-end neural extraction modules remains to be investigated. Performance metrics used to evalu- ate KBQA systems do not account for mismatches arising because of lexical variations or date rep- resentation variations. Using better metric can provide a better picture of the overall performance.
Nikita Bhutami, Xinyi Zheng, Kun Qian, Yunyao Li, and H. Jagadish. 2020. Answering complex questions by combining information from curated and extracted knowledge bases. In Proceedings of the First Work- shop on Natural Language Interfaces, pages 1-10, Online. Association for Computational Linguistics.
Rishav Chakravarti, Cezar Pendus, Andrzej Sakrajda, Anthony Ferritto, Lin Pan, Michael Glass, Vittorio Castelli, J William Murdock, Radu Florian, Salim Roukos, and Avi Sil. 2019. CFO: A framework for building production NLP systems. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA System Demonstrations, pages 31-36, Hong Kong, China. Association for Computa- tional Linguisties.
Tarcísio Souza Costa, Simon Gottschalk, and Elena Demidova. 2020. Event-qa: A dataset for event- centric question answering over knowledge graphs. CoRR, abs/2004.11811.
Bin Fu, Yunqi Qiu, Chengguang Tang, Yang Li. Haiyang Yu, and Jian Sun. 2020. A survey complex question answering over knowledge base: Recent advances and challenges. CORR, abs/2007.13061
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018a. Tem- pquestions: A benchmark for temporal question an- swering. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon, France, April 23-27, 2018, pages 1067-1062. ACM.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen. and Gerhard Weikum. 2018b. TEQUILA: temporal question answering over knowl- edge bases. In Proceedings of the 27th ACM Inter- national Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22- 26, 2018, pages 1807-1810. ACM.
References
Abdalghani Abujabal, Mohamed Yahya, Mirek Riede- wald, and Gerhard Weikum. 2017. Automated tem- plate generation for question answering over knowl- edge graphs. In Proceedings of the 26th International Conference World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1191-1200, ACM.
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM international conference on informa- tion & knowledge management, pages 792-802.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra. Tri Nguyen, Mir Rosenberg. Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018 Ms marco: A human gener- ated machine reading comprehension dataset.
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Rav- ishankar, Salim Roukos, Alexander G. Gray. Ramón Fernandez Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfi Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunyao Li, Francois P. S. Luus
Laura Banarescu, Claire Bonial, Shu Cai,, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation

---

Ndivhuwo Makondo, Nandana Mihinddukulasorriya. Tahira Naseem, Sumit Neelam, Lucian Popa, Re- vanth Gangi Reddy, Ryan Riegel, Gaetano Rossiello, Udit Sharma, G. P. Shrivatsa Bhargav, and Mo Yu. 2021. Leveraging abstract meaning representation for knowledge base question answering. In Find- ings of the Association for Computational Linguis- tics: ACL/IJCNLP 2021. Online Event, August 1-6, 2021, volume ACL/IICNLP 2021 of Findings ACL, pages 3884-3894, Association for Computational Linguistics.
2022. Do i have the knowledge to answer? investi- gating answerability of knowledge base questions.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguisties.
Nils Reimers and Iryna Gurevych. 2019, Sentence- bert: Sentence embeddings using siamese bertsnetworks. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNIP 2019, Hong Kong, China, November 3-7, 2019, pages 3980-33 3990. Association for Computational Linguisties.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kütttle, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Autrieval-augmentee generation for wwwwledge-intensivy NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Apoorv Saxena, Soumen Chakrabarti, and Partha Taluk- dar. 2021. Question answering over temporal knowl- edge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 6663-6676, Online. Association for Computational Linguistics.
Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Ioannidis, Soji Adeshina, Phillip R. Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2021. Tempoqr: Temporal question reason- ing over knowledge graphs.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge hase embeddings. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 4498- 4507, Online. Association for Computational Lin- guistics.
Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nan- dana Mihindukulasooriya, Young-Suk Lee, Santosh Srivastava. Cezar Pendus, Saswati Dana, Dinesh Garg. Achille Fokoue, G P Shrivatsa Bhargav, Di- nesh Khandelwal, Srinivas Ravishankar, Sairam Gu- rajada, Maria Chang, Rosario Uceda-Sosa, Salim Roukos, Alexander Gray, Guilherme LimaRyan Riegel, Francois Luus, and I. Venkata Subramaniam. 2021. Sygma: System for generalizable modular question answering overknowledge bases.
Jiaxin Shi, Shulin Cao, Liangming Pan, Yutong Xiang, Lei Hou, Juanzi Li, Hanwang Zhang, and Bin He. 2020. Kqa pro: large-scale dataset with inter- pretable programs and accurate sparqls for complex question answering over knowledge base.
Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nan- dana Mihinduukulasooryy,, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, Saswati Dana, Dinesh Garg, Achille Fokoue, G Shrivatsa Bhargav. Dinesh Khandelwal, Srinivas Ravishankar, Sairam Gurajada, Maria Chang, Rosario Uceda-Sosa, Salim Roukos, Alexander Gray, Guilherme LimaRyan Riegel, Fran- cois Luus and L Venkata Subramaniam. 2022a. Sygma: System for generalizable modular question answering overknowledge bases. In Findings of the Empirical Methods in Natural Language Processing: EMNLP 2022.
Haitian Sun, Tania Bedrax- and William W. Co- hen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text, In Proceedings of the 2019 Conference on Empiri- cal Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2380-2399 Association for Computational Linguistics.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Co- hen. 2018 Open domain question answering using early fusion of knowledge bases and text. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 4231-4242. Association for Computational Linguistics.
Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nan- dana Mihindukklasoorriy,, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, ct al. 2022b. A benchmark for generalizable and interpretable temporal ques- tion answering over knowledge hases. arXiv preprint arXiv:2201.0579.
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. Lc-quad: A corpus for complex question answering over knowledge graphs. In Proceedings of the 16th International Semantic Web Conference (ISWC), pages 210-218. Springer.
Mayur Patidar, Avinash Singh, Prayushi Faldu, Lovekesh Vig, Indrajit Bhattacharya, and Mausam.

---

Svitlana Vakulenko, Javier David Fernandez Garcia, Axel Polleres, Maarten de Rijke, and Michael Cochez. 2019. Message passing for complex question an- swering over knowledge graphs. In Proceedings of the 28th ACM International Conference on Informa- tion and Knowledge Management, CIKM 2019, Bei- jing, China, November 3-7, 2019 pages 1431-1440. ACM.
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021. Retrospective reader for machine reading comprehen- sion. In Thirty- Fiffh AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thine Conference on Innovative Applications of Artificial Intelligence, TAAI 2021, The Eleventh Symposium on Educational Advances Artificial Intelligence, FAAI 2021, Vir- tual Event, February 2-9, 2021, pages 4556-14514. AAAI Press.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2023. Chain-of-thought prompting clic- its reasoning in large language models.
Tomer Wolfson, Mor Geva, Ankit Gupta, Yoav Gold- berg, Matt Gardner, Daniel Deutch, and Jonathan Berant. 2020. Break down: A question understand- ing benchmark. Trans. Assoc. Comput. Linguistics, 8:183-198.
Dekun Wu, Nana Nosirova, Hui Jiang, and Mingbin Xu. 2019. general fofe-net framework for simple and effective question answering over knowledge bases. CoRR, abs/1903.115 12356.
Peiyun Wu, Yunjie Wu, Linjuan Wu, Xiaowang Zhang, Zhiyong Feng. 2021. Modeling global semantics for question answering over knowledge bases. CoRR, abs/2101.01510
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo. and William Yang Wang. 2019. Improving ques- tion answering over incomplete kbs with knowledge- aware reader. In Proceedings of the 57th Confer- ence of the Association for Computational Linguis- fics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages Association for Computational Linguisties.
Wen-tau tau Yih, Ming- Wei Chang, Xiaodong He, and Jian- feng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowl- edge base. Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference Natu- ral Language Processing (Volume Long Papers), pages 1321-1331, Beijing, China. Association for Computational Linguistics.
Luke S. Zettlemoyer and Michael Collins. 2012. Learn- ing to map sentences to logical form: Structured classification with probabilistic categorial grammars.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020. Sg-ner net: Syntax-guides machine reading comprehension. In The Thirty- Fourth AAAI Conference on Artificial In- telligence, AAAI 2020, The -ecoodd Innova- tive Applications of Artificial Intelligence Conference. IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9636-9643. AAAI Press.

---

A Appendix
A.1.2 KB Linking and Answering
This is essentially a step to ground the Entity and Relation mentions of X-expression to the KB, i.e., map the elements of A-expression onto the corre- sponding KB elements. Relation mentions are the predicates (for example, have-org-role in Figure 1) and entity mentions the arguments (for exam- ple, United States, president, and Ww2 in Figure 1). The goal of linking is to map, for example Ww2 a node in KB corresponding to World War 2 (Wiki- data id wd:Q362). After linking, we generate corre- sponding SPARQL queries that when executed on the KB endpoint would fetch the intended KB facts. Our approach to linking and SPARQL generation is similar to that of (Neelam et al., 2022a).
A.1 Details of KBQA pipeline (Neelam et al., 2021)
Here we describe the modules in the KBQA pipeline that is adopted from (Neelam et al., 2021).
A.1.1 Question Understanding
The goal of Question Understanding to 1) transform NL questions into corresponding X- expressions that logically represent the set of event- specific facts needed from the KB and the reason- ing needed to be performed to derive the answer and 2) further perform event-specific decomposi- tion. We use method in (Neelam et al., 2022a) construct A-expressions of the questions from their AMR (Abstract Meaning Representation) (Ba- narescu et al., 2013). AMR encodes meaning of the sentence into a rooted directed acyclic graph where nodes and edges represent concepts and re- lations respectively. Such representation is useful because event-specific decomposition of the ques- tion is represented to some extent as sub- sub-paths and sub-graphs in the AMR graph. Figure 1 shows an illustration of AMR and X-expression for ex- ample question. This example illustrates how X- expression compactly represents, the mentions of events in the question (as its sub-components), facts about those events (that need to be fetched from the knowledge source), and the reasoning steps (that need be performed derive the final answer). X-expression constructed for the question is fur- ther processed to decompase into components:
A.2 Targeted Extraction Algorithm
Algorithm 1 Algorithm for the overall approach
illustrated in Figure and its flow sequence.
lambdo Get Lambla (quesstion pusslist- Gel K.K.BA Answer(lamebda) if ans Jist is empty then
main, = Decompose_ KBQA Failure
list CetK B.nnuerr(aazz)
if ans list empty then
Aux Failure
fact Extract From Text(aur)
our aur_facct fortt= orrmed_lambdl Re Reforml.ambloo fact, lambda)
for later use
ans_list
(reformedd CetK BAnswer (re formed lombdaa) aasslist not empty then return
Ans Found
end
ans list = GetK Answer(main)
candidate_facta =
for candidate in oms oms_list list do
1. Main-X: Part of the X-expression related to the unknown variable, i.e., main event being ques- tioned. For example in Figure 1, is the unknown variable, whose value if found is answer to the question.
fact= = Extract_Frrom FFrom_Tenttcmdiiiiie
candidate__octe candddale..aatt..ppeend/ act, candidate)) end for ans
Rempora//ccoomer/canlittttt fateaaanddddtteeetdaddteeee facts, ouxt act)
return else
Ans Found
2. Aux-X: part of the X-expression not related to the unknown variable, but related to the rest of the events mentioned in the question. This part serves the purpose of adding temporal constraint to the candidate answer values for the unknown variable.
return end if
b Ans Found, No missing fact in KB
A.3 Details of Targeted Extraction from Text
We use rule based approach to perform de- composition, that simply uses unknown variable anchor to segregate the respective components. Note that the decomposed X-expressions play a crit- ical role in our approach to identify the points of failures of the KBQA pipeline and to further decide on the use of Extraction pipeline.
Our goal is to use textual resources to assist KBQA failures, which can happen for two reasons: 1. Linking failure when KB linking step fails successfully map mentions in the X-expression the corresponding KB entities and relations. For example, in Figure when mention Ww2 fails to get linked to the World War node in the KB.

---

2. Missing facts- KBs are known be incomplete. and hence may fail fetch a specific fact, simply because is not present in the KB. For example, if temporal information corresponding to World War is not present in the KB, attempt to fetch time interval corresponding to X-expression part interval(li, "Ww2") would fail.
À a. have-org-role-91(0, a, "United States", "pres- ident")
and pass that onto the KBQA pipeline. Then for each answer candidate obtained we try to extract time interval from the textual resource. For exam- ple, if Franklin D. Roosenveli is one of the answer candidates, we try to extract the time interval of Franklin D. Roosenveli being the US president from the textual resources.
X-expression specifies all facts that need to be fetched from the KB. A failure fetch even single fact would block KBQA from computing the final answer. To handle failures we need to know the specific facts that failed to get fetched from the KB, so that we can look for them in the textual resources. For this purpose, we categorize KBQA failure as below, based on the decomposed X-expressions where failure happens into Aux Fail- and Main Failure. We present the flow se- quence Algorithm 1.
Note that we resort to extraction from textual resources only for those facts that failed to get fetched from the KB. We believe this approach of targeted extraction from the text is likely be more accurate than unrestricted extraction, because we are looking to extract facts with set of known variables and only one unknown variable.
A.4 Discrepancies in Evaluating on TimeQuestions
A.3.1 Aux Failure
We could not directly evaluate our approach on TimeQuestions because few discrepancies ob- severd in that datasets such as, invalid answers (with change in time) and incorrect porting of the answers from old datasets without verifying valid- ity on Wikidata. It is because TimeQuestions (Jia et al., 2021) is built from older datasets on different KBs, by mapping only the final answer entities onto the corresponding Wikidata entities and it seems while mapping the validity of the answer is not verified carefully. Hence, we resort to using EX- AQT (Jia et al., 2021) built on TimeQuestions as the base KBQA pipeline and build an equivalent of our proposed approach called EXAQT+Tex for comparison.
This issue arises due to missing temporal fact cor- responding to the aux-À in Upon successful ex- traction of time interval for aux-À, from textual resources, we construct a reformed X-expression from the original X-expression by simply replacing auxiliary part with the time interval of aux-À. For example, X-expression in Figure will be reformed
A a. rave-org-role--11h,, a, "United States", "president") A interval(hi, h) A overlap(hi, (interval 19rt::999-9900001,, interval_ end: 1945- (9-02)).
Note that this reformed X-expression does not contain aux-À. Its NL equivalent is Who was the President of the United States during period from 1st September 1939 2nd September 1945? Thus if reformed X-expression passed onto the KBQA pipeline (instead original X-expression), it should result the same answer, but without the need to fetch facts related to aux-À À from the KB.
A.5 Implementation Details
Our system pipeline implemented using Flow Compiler (Chakravarti et al., 2019) that stitches together the gRPC services of the individual mod- ules. X-expressions are defined using ANTLR grammer. SPARQL queries are run public Wiki- data end point 10 We reuse the KBQA pipeline implementation of (Neelam al., 2022b).
A.3.2 Main Failure
For example, in Figure 1 main-A corresponds to the list of all the US presidents and their time intervals in office as the president. We make an assumption that we can always get the list of all answer can- didates from the KB itself, but may need to look into textual resources only for temporal fact about them. For example, Figure we take that part of the main X-expression that would fetch the answer candidates from the KB (leaving out the temporal fact specific components), i.e.,
A.6 Related Work
Although Complex KBQA has been an active re- search topic (Vakulenko et al., 2019; Saxena et al., 2020; Wu et al., 2021; Shi et al., 2020), there has been very limited research focused Temporal
htttp:://query.wwkidaaaa

---

KBQA. Temporal Questions require identification of time intervals of events and temporal reasoning.
ing AMR (Abstract Meaning Representation) (Ba- nareseu et al., 2013) and X-calculus (Zettlemoyer and Collins, 2012) to get logical representations of the questions. This enables decomposition of the questions at semantic level and is likely robust to linguistic variations as well.
A.6.1 Temporal KBQA Datasets:
TempQuestions (Jia et al., 20) one of the first publicly available temporal KBQA dataset con- sisting of 1271 questions. However, this dataset was annotated over FreeBase, which is no longer maintained and was officially discontinued in 2014. SYGMA (Neelam et al., 2022a) introduced sub- set of TempQuestions that can be answered over wikidata called TempQA-WD. TimeQuestions((ia al., 2021) is another temporal QA dataset that is curated from existing QA datasets and map- ping the answers to Wikidata. We use TempQA- WD, and TimeQuestions data sets to evaluate our approach. CRONQUESTION (Saxena et al.. 2021) is another temporal KBQA dataset that uses its own KB drawn from Wikidata. Event-QA dataset (Costa et al., 2020) is based on Event-KG, curated from DBpedia, Wikidata and YAGO. Since these datasets are generated in a template based manner using existing facts from the KBs, they do not represent the real world challenge of incom- plate KBs. One of the main goals of our approach is to handle the issue of incomplete KBs.
A.7 KB + Text for QA
There have been past work exploring effective- ness of using KB and text resources for complex QA (Sun et al., 2018; Xiong et al., 2019). How- ever, none of them address the temporal context addressed in this work. Prior work using a com- bination of KB and text have largely been based on end-to-end neural models. GRAFT-Net (Sun et al., 2018) constructs a sub-graph from KB and text corpora using an early fusion technique. The task of QA is then reduced to binary classification over the nodes of this sub-graph. PullNet (Sun et al., 2019) proposes to build sub-graph through an iterative Process(Xiong et al., 2019), utilise a graph-attention based KB reader and knowledge- aware text reader.
All these methods are based on end-to-end neu- ral models that require large amount of training data and offer little interpretability, which is essen- tial to evaluate intermediate stages of complex QA systems. Additionally, labeling large amounts of data for KBQA is hard (Trivedi et al., 2017). In this work, we extend modular approach described in (Neelam et al., 2022a), additionally incorporating it with targeted extraction pipeline. We made this choice this particular approach integrates multi- ple, reusable modules that are pre-trained for their specific individual tasks (semantic parsers, entity and relational linkers, rankers and re-rankers and reading comprehension model) thus offering inter- pretability and flexibility for optimal combination of textual extraction with KBQA. Additionally, this does not require a large amount of domain-specific training data.
A.6.2 Temporal KBQA Systems:
TEQUILA (Jia et al., 2b) is one of the first at- tempts to address temporal question answering over KBs. It used an existing KBQA engine (Abujabal et al., 2017) answer individual sub-questions and perform temporal reasoning over the answers to derive the final answer. SYGMA (Neelam et al., 2022a) is another system that works on a Wikidata and uses X-expressions to represent the facts and their temporal reasoning operators. EXAQT (Jia et al.. 2021) is another temporal KBQA system that uses entity and temporal embeddings to gen- erate final answers. TempoQR (Mavromatis et al., 2021) is another system that tries to improve on top of CRONQA (Saxena et al., 2021) and intro- duces temporal KB-completion aspect into tem- poral Questions answering. We did not consider these two systems as they work on curated sub- set of wikidata which has all the temporal facts to answer the given dataset. TEQUILA uses pre- specified set of temporal signals (10 signal words) decompose questions into sub-questions sen- tence level in rule-based manner. Instead, we follow the approach similar to SYGMA that use a sophisticated semantic parsing approach involy-
A.7.1 Question Decomposition
Our work uses a form of logical query decompo- sition, based on X-expression of the NL question, to help effectively combine the KB with the text resourees. Some of the past work in the literature on QA have also explored question decomposition. BREAK IT down (Wolfson et al., 2020) a popu- lar benchmark data that captures complex question as an ordered list of tasks, that when executed sequence will derive the final answer. It introduced

---

question decomposition meaning representation (QDMR) to represent decomposed questions in an intermediate form resembling SQL. TEQUILA (Jia et al., 2018b) used temporal signal (words) based question decomposition to turn natural language questions into sub questions. STAG (Yih et al., 2015) defines core inferential chain and constraints which are analogous to the main and aux defined in our work. However, it's important note that STAGG doesn't 't execute explicit decomposition of the lambda in the manner we do.

---

Question: Who the President during the Dawes Plan?
KBQA
A-expression: À a. have-or:-role-20 (h, a, "president") interval thi, h) type (tp2, "Dawes", "plax") interval (p21, "Dawez") overlap (hi, p21)
Is Answerable from KBQA?:No No
KBQA Issue: Auxiliary Fact Missing (temporal information of Dawes Plan)
Missing, Fact Extruction
The Dawes Plan proposed by the Dawes Comuitteee, chaired by Charles G. Dawes was plan in 1924 that successfully resolved the issuc of World War I reparations that Germany had to pay.
Retrieved textual info:
Auxiliary temporal fact: 1924
Reformed e-expression: À a. interval (hi, h) rove-org-role-ll (z, a, "president;Pi
Interval ip2i, date ("dd-mm- ""dd-mm-1924") overlap (hi, p21)
Answer with reformed A-expression (from KB): Calvin Coolidge
Figure 4: Illustration of working example showing the KBQA failure occurring due to missing auxiliary fact that is substituted by temporal fact extraction and finally reforming the lambda expression by hard-coding the missing fact.
Example
Missing auxiliary fact
Question: What was Franklin Roosevelt's position during World War Il before pearl harbor?
Ground Truth Answer: President of the United States
Answered from KB?: False
Lambda lambda a. position-01(p2, "Franklin Roosevelt", a) interval(p2i, p2) A war(w, "World War II") A interval(wi. w) interval(bi, "Pearl Harbor") before(wi, bi) interval(wi, w) overlap(p2i, wi)
Aux lambda lambda wi. war(w, "World War II") A interval(wi, w) A interval(bi, "Pearl Harbor")
before(wi, bi) A interval(wi, w)
Main lambda lambda a. interval(p2i, p2) A position-01(p2, "Franklin Roosevelt", a)
Is Auxiliary answered from KB?: False
Auxiliary relevant passages (text extraction):
1) "Japan's attack Pearl Harbor took place on December 7, 1941. The U.S. military suffered 18 ships damaged or sunk, and 2,400 people were killed. Its most significant consequence was the entrance of the United States into World War II. The US had previously been officially neutral but subsequently entered the Pacific War, the Battle of the Atlantic and the European theatre of war. Following the attack, the US interned 120,000 Japanese Americans, 11,000 German Americans, and 3,000 Italian Americans.".
2) "On the morning of December 7, 1941, the Japanese struck the U.S. naval base at Pearl Harbor with a surprise attack, knocking out the main American battleship fleet and killing 2,403 American servicemen and civilians. Scholars have all rejected the conspiracy thesis that Roosevelt, or any other high government officials, knew in advance about the Japanese attack on Pearl Harbor. The Japanese had kept their secrets closely guarded, and while senior American officials were aware that war was imminent, they did expect attack on Pearl Harbor."
Auxiliary answer fact: "Japan's 's attack on Pearl Harbor took place on December 7, 1941."
Auxiliary interval: 1941
Reformed lambda: lambda a. interval(p2i, p2) A Opsi,,on-01(p2, Franklin Roosevelt, a) A inter- val date(7-12-194)) A overlap(p2i, wi) System Answer: President of the United States
Table 5: Examples of System Answers

---

Question: Who was Governor of Arkansas when Deewangee released?
A-expression: a. have-arg-role-91 (h, a, "Arkansas", "governar") interval (hi,
KBQA
h] release-01 (r, "Deewangee") Cinterval (ri, =) overlap (hi, xi)
Is Answerable from KBQA?:
KBQA Issue: Main Fact Missing (temporal information of answer candidate(s))
Main Candidates from KB: Bill Clintun, Jim Guy Tucker, Mike Huckabee, Mike Beebe
Auxiliary temporal constraint: 2002 (overlap)
Fact Extraction
Targeted extraction of temporal fact from text: Bill Clinton (1983 1992), Jim Guy Tucker (1992, 1996),
Mike Huckabee (19%6-2007) Mike Beebe (2007 2015)
Temporal Reasoning: Candidate Temporal Interval Overlap (2002)
Answer: Mike Huckabee
Figure 5: Illustration of a working example showing the KBQA failure occurring due to missing main fact. Temporal information for each the extracted main (answer) candidates are extracted from textual resources followed by temporal reasoning to retrieve valid main answer.
Example
Missing main fact
Question: Who was the first man the moon in 1969? Ground Truth Answer: Neil Armstrong
Answered from KB?: False
Lambda argmin(lambda m. man(mu, m, "moon"), lambda lambda mi. interval(di, date("dd- mm-1969")) A overlap(mi, di) interval(mi, mu), 0, I)
Aux lambda
Main lambda : lambda m. (mt,rval(mi, mu) A man(mu, m, "moon")"
Is Auxiliary answered from KB?: False
Main relevant passages (text extraction): 1) "On July 20, 1969, Armstrong and Apollo 11 Lunar Module LM pilot Buzz Aldrin became the first people to land the Moon, and the next day they spent two and a half hours outside the Lunar Module Eagle spacecraft while Michael Collins remained in lunar orbit in the Apollo Command Module Columbia. When Armstrong first stepped onto the lunar surface, he famously said: That's one small step for [a] man, one giant leap for mankind.It was broadcast live to an estimated 530 million viewers worldwide. Apollo 11 effectively proved US victory in the Space Race, by fulfilling a national goal proposed 1961 by President John F. Kennedy öf landing a man on the Moon and returning him safely to the Earthbefore the end of the decade. Along with Collins and Aldrin, Armstrong was awarded the Presidential Medal of Freedom by President Richard Nixon. President Jimmy Carter presented him with the Congressional Space Medal of Honor in 1978, and Armstrong and his former crewmates received Congressional Gold Medal in 2009."
2) "Apollo 11 July 16201324, 1969 was the spaceflight that first landed humans on the Moon. Commander Neil Armstrong and lunar module pilot Buzz Aldrin formed the American crew that landed the Apollo Lunar Module Eagle on July 20, 1969, at 20:17 UTC. Armstrong became the first person to step onto the lunar surface six hours and 39 minutes later on July 21 at 02:56 UTC: Aldrin joined him 19 minutes later. They spent about two and a quarter hours together outside the spacecraft, and collected 47.5 pounds 21.5 kg of lunar material to bring back to Earth. Command module pilot Michael Collins flew the Command Module Columbia alone in lunar orbit while they the Moon's 's surface. Anmstrong and Aldrin spent 21 hours, 36 minutes the lunar surface, site they had named Tranquility Base upon landing, before lifting off rejoin Columbia in lunar orbit."
"On July 20, 1969, Armstrong and Apollo 11 Lunar Module LM pilot Buzz Aldrin became the first people to land on the Moon"
Main answer fact:
System Answer: Neil Armstrong, Buzz Aldrin
Table 6: Examples of System Answers

---

Question
System
Ground
System
Comments
Path
Answer
Answer
Who won best actor when Al- fred Junge won best art direc- tion?
Aux Failure
Ronald
Yul Bryn-
Top Retrieved Fact: In addition the same producer, director and star, the first two films in the trilogy had the same cinemalographer F. A. Freddie Young, composer Miki Rufzsa, art director Al- fred Junge and costume designer Roger Furse. The costumes for this film were executed by Elizabeth Haffenden. In 1955, she would take over from Furse as costume designer for the final film in the trilogy, Quentin Durward. Alfred Junge remained as art director. Issue: The issue here is incorrect Auxiliary fact extraction by the ectraction pipeline. As can be seen, irrelevant passage has been detected as the fact containing Aux- iliary interval. Hence the extracted Aux- iliary fact was 1957 as opposed to the correct date 1947. As result the re- formed lambda consisted of erroneous temporal constraint due to to which the KBQA pipeline returned an incorrect an- swer as expected.
(text extrac-
Colman
ner
tion)
Which team did Wayne Rooney play for before joining Manch- ester United?
Aux Failure (text extrac- tion)
Everton F.C.
England
Top Retrieved Fact: Wayne Mark Rooney born 24 October 1985 an En- glish professional football manager and former player. He is the manager of EFL Championship club Derby County, for whom he previously served as in- terim player-manager. He spent much of his playing career forward while also being used in various midfield roles. Widely considered be one of the best players of his generation, Rooney the record goalscorer for both the England national team and Manchester United.Rooncy joined the Everton youth team at the age of nine and made his professional debut for the club in 2002 at the age of 16. He spent two seasons the Merseyside club, before moving Manchester United for 325.6 million in the 2004 summer transfer window where he won 16 trophies and became the only English player, alongside team- mate Michael Carrick, to win the Pre- mier League, FA Cup, UEFA Champi- ons League, League Cup. UEFA Europa League, and FIFA Club World Cup.

---

Issue: The Auxiliary fact in this case is correctly detected: 2004. However when the reformed lambda with the aux- iliary fact hard-coded
evaluated on the KB, an adjacent entity also comply- ing with the constraints was picked as the final answer. In this case Wayne Rooney was also a player of England be- fore (as well as after) he joined Manch- ester United. However the answer ex- pected was his previous club: Everton F.C. This is due to an inherent issue with the KBQA pipeline.
When did the dustrial revolu- tion in Europe began?
Fail- 1791 ure (text extraction)
Main
late 18th
Top Retrieved Fact: Industrial growth Capitalism has been dominant in the Western world since the end of feudal- ism. From Britain, it gradually spread throughout Europe. The Industrial Revo- lution started in Europe, specifically the United Kingdom, in the late 18th cen- tury, and the 19th century saw Western Europe industrialise. Economies were disrupted by World War but by the be- ginning of World War II they had recov- ered and were having to compete with the growing economic strength of the United States. World War II, again, dam- aged much of Europe's industries. Issue: The textual resource does not contain the exact date mention of the missing tem- poral fact. So the targeted extraction pipeline retrieves the most appropriate sequence of tokens "late 18th century" from the text as the temporal fact.
century
Who is the Gov- emnor of Ari- zona in 2009?
Main Fail- Jan
Jan
Top Retrieved Fact: Governor Jan Brewer assumed office in 2009 after Janet Napolitano had her nomination by Barack Obama for Secretary of Home- land Security confirmed by the United States Senate. Arizona has had four male governors, more than any other state. Issue: The top retrieved relevant fact had explicit mention of Jan Brewer assuming office in 2009. However, it lacks explicit mention of Jane Napoli- tano's end date. The end date being 2009 had be implicitly reasoned out which not trivial for the text based fact ex- tractor. Hence the overlapping dates office (2009) of the two Governors was missed out.
(text Brewer & Janet
Brewer
extraction)
Napoli-
tano
Table 7: Detailed error analysis of a few incorrectly answered questions


Reviews:
[{"review_id": "eryqoj9fiC", "reviewer": "Reviewer_T8H6", "summary": "", "questions": "1. Why work at the level of the lambda expression when deciding when to call the information extraction pipeline? It would have been possible to work with SPARQL directly and identify missing information. This would avoid having to run the lambda-to-SPARQL conversion again.\n2. What is the decision rule for defining auxiliary and main lambda expressions? I believe this distinction is similar to the choice made in STAGG [1].\n3. Line 232: Please clarify what you mean by entity based vs triple based,\n4. Sec 3: Does RAG use the same retrieval components as the information extraction pipeline?\n4. Line 304: It is not clear from the method description what the thresholds for EXAQT correspond to. Please describe the system more clearly.\n5. App A.4: It is not clear why using EXAQT solves the apparent discrepancy in the annotation for TimeQuestions dataset.\n\n[1] Yih, Wen-tau  and  Chang, Ming-Wei  and  He, Xiaodong  and  Gao, Jianfeng. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In ACL\n\nPOST REBUTTAL:\nAuthors have agreed to include more details in the final version of the paper. Their responses have addressed my concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback and comments. We appreciate that you find the approach “intuitive” and agree our work “establishes the claim”. We have responded to your questions and concerns in detail below. We are happy to follow-up in further discussion if any concerns remain.\n\n**RR1:** “The paper would have been complete if it included some statistcs of the different failure modes…” \n\nThe SYGMA KBQA system (https://aclanthology.org/2022.findings-emnlp.284/) that we build upon has reported an ablation study on how it performs with ground truth supplied at each module. This study reveals that entity and relation linking collectively contribute to a higher performance decrease than parsing errors and lambda generation errors. Our primary emphasis lies in scenarios where the text pipeline effectively mitigates KBQA blockages, as opposed to an individual focus on each module.\n\n Following your suggestions we tried to quantify the different error types faced by our proposed (KB + Text) whenever text pipeline was fired (incomplete KB).\n\nWe randomly sampled 50 test points where the model outputs had errors and classified them into error categories as below:\n\n| Error - > | Incorrect extraction | KBQA error | Parsing error | Miscellaneous |\n| --------- | -------------------- | ---------- | ------------- | ------------ |\n| Count     | 21                   | 9          | 8             | 12           |\n\nIncorrect extraction - The text pipeline extracted incorrect facts\n\nKBQA error - KBQA still produced erroneous answers after missing fact was correctly extracted from text\n\nParsing - Incorrect Parsing of the question \n\nMiscellaneous - Questions that don’t fall into any of the categories. Some examples of miscellaneous issues include lexical and date representation variations. Other ambiguities like multiple entities overlapping with the auxiliary date also exist (please refer to example 4 in Table 6 in Appendix)\n\nWe shall add an analysis to the camera-ready version and discuss scope of improvement leveraging the additional page available.\n\n\n**RR2:** “Authors use different backbone KBQA systems for different evaluation datasets but the need for this is not clear (more in questions)....”\n\nThank you for pointing out this concern. We have provided an explanation for the issue in Appendix A.4. We reiterate the reason – the same KBQA system could not be used to evaluate TimeQuestions because that dataset is built from older datasets on different KBs. We observed that the validity of answers are not verified carefully while the answer entities are mapped to corresponding Wikidata entities. In light of these considerations, instead of adapting a single KBQA system to various datasets, we opted to utilize a KBQA system that was **purpose-built** for the specific dataset.  Additionally, this choice allowed us to effectively demonstrate that our proposed methodology is **resilient** to changes in the underlying KBQA system. We establish that our approach can be seamlessly adapted to different KBQA systems; and as you rightly point, the only requirement is that the underlying KBQA system has *intermediate representations*. We acknowledge that this explanation is crucial and should be moved into the main sections rather than the Appendix, which we will do in the camera-ready version.\n\n**Q1:** Why work at the level of the lambda expression when deciding when to call the information extraction pipeline? It would have been possible to work with SPARQL directly and identify missing information. This would avoid having to run the lambda-to-SPARQL conversion again.\n\n\nThanks for this interesting question. We work at the level of Lambda expression because it is agnostic to underlying KB. There are notable advantages to working with Lambda expressions at this level: i) By operating at the Lambda expression level, we effectively prevent the propagation of Entity and Relationship Linking errors into the extraction pipeline. ii) Working with Lambda expressions alleviates the need to address varying representations used by different knowledge bases. This is particularly relevant when dealing with modules such as Lambda to Natural Language (NL) conversion. Given these factors, we work at the Lambda level for decomposition and subsequent localization of KBQA failures instead of other positions in our pipeline.\n\n**Q2.** What is the decision rule for defining auxiliary and main lambda expressions? I believe this distinction is similar to the choice made in STAGG [1].\n\n\nAs stated in line 144, our strategy to decompose the $\\lambda$ expression is based on the unknown variable and it’s dependents. All terms not related to the unknown variable are considered as additional constraints or auxiliary terms,  collectively forming what we refer to as the auxiliary $\\lambda$. In practice, as noted in line 144, we use a straightforward rule-based decomposition that involves  utilizing the unknown variable as anchor to segregate the respective components. We further provide some examples in Figure 1, Table 4 and Table 5 (in Appendix)\nAs you rightly highlight, STAGG defines *core inferential chain* and *constraints* which are analogous to the main and aux $\\lambda$ defined in our work. However, it's important to note that STAGG doesn't execute explicit decomposition of the lambda in the manner we do. We concur with your suggestion to cite this paper and elaborate on the distinctions in the camera-ready version.\n\n\n**Q3.** Line 232: Please clarify what you mean by entity based vs triple based,\n\nA triple- based has {subject,predicate,object} along with its timeline whereas entity-based has a single event (entity) along with its timeline. As stated in line 232, we use this distinction while converting a lambda expression to natural language form. Below we provide an example:\n \n**Triple Based**: $\\lambda$ di. direct-01(d, \"Deepak Sareen\", “Gaddaar”) ^ interval(di, d) -> When did Deepak Sareen direct Gaddaar?\n\n**Entity Based**: $\\lambda$ wi. interval(wi, \"World War 2\") -> When was World War 2?\n\nWe will add this detail to the camera-ready version.\n\n**Q4:** Sec 3: Does RAG use the same retrieval components as the information extraction pipeline?\n\nNo, RAG has different components. More specifically, RAG (https://arxiv.org/abs/2005.11401) consists of a retriever based on Dense Passage retrieval (DPR) and a generator based on BART jointly trained. The purpose of including it was to compare the performance of our system with other state-of-the-art Open-Domain QA baselines. We further demonstrate in Table 2, the strengths of our KB guided extraction (even without specific training) over the RAG framework.\n\n\n**Q5:** Line 304: It is not clear from the method description what the thresholds for EXAQT correspond to. Please describe the system more clearly.\n\nEXAQT (https://arxiv.org/abs/2109.08935) assigns a confidence score to each of the answers it generates. A higher score means a higher probability of it being correct. We provided limited explanations of previous works due to space constraints. We can add these details to the camera-ready paper.\n\n\n**Q6:** App A.4: It is not clear why using EXAQT solves the apparent discrepancy in the annotation for TimeQuestions dataset.\n\nEXAQT may not solve the annotation issue fully for our purpose. Since EXAQT embeddings were built on incorrect annotations (which are incorrect for the current time) its model is able to work reasonably well on that incorrect data. In our case, since we wanted to measure the benefit of our text based extraction, we used EXAQT as it is and measured the relative improvement that text based extraction models is adding over the baseline. This is not expected to solve the annotation issue fully as those can be corrected only through re-annotation of the data or refreshment of the answers for the current time. Please also refer to RR2 for further details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Authors show that supplementing missing KB information from text corpora is an effective way to improve KBQA performance\n- The approach is intuitive and establishes the claim with off-the-shelf components. The only requirement is that the underlying KBQA system has intermediate representations that can be used to identify", "reasons_to_reject": "- The paper would have been complete if it included some statics of the different failure modes: parsing failures (cannot be fixed by the proposed approach) and execution failures (missing edges that can be fixed by the proposed approach)\n  - Since TempQA-WD provides annotations for SPARQL and intermediate nodes in the reasoning, the authors could have used this data to quantify both error types and highlight the biggest room for improvement\n  - POST REBUTTAL: Authors provided the requested error breakdown, which I think will help future work decide what aspect to focus on.\n- Authors use different backbone KBQA systems for different evaluation datasets but the need for this is not clear (more in questions) \n  - POST REBUTTAL: Authors addressed this concern. The different backbones were chosen as competitive baselines for each of the datasets and demonstrate the flexibility of the proposed approach.\n\nPOST REBUTTAL:\nMy concerns are sufficiently addressed.", "questions_to_authors": "1. Why work at the level of the lambda expression when deciding when to call the information extraction pipeline? It would have been possible to work with SPARQL directly and identify missing information. This would avoid having to run the lambda-to-SPARQL conversion again.\n2. What is the decision rule for defining auxiliary and main lambda expressions? I believe this distinction is similar to the choice made in STAGG [1].\n3. Line 232: Please clarify what you mean by entity based vs triple based,\n4. Sec 3: Does RAG use the same retrieval components as the information extraction pipeline?\n4. Line 304: It is not clear from the method description what the thresholds for EXAQT correspond to. Please describe the system more clearly.\n5. App A.4: It is not clear why using EXAQT solves the apparent discrepancy in the annotation for TimeQuestions dataset.\n\n[1] Yih, Wen-tau  and  Chang, Ming-Wei  and  He, Xiaodong  and  Gao, Jianfeng. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In ACL\n\nPOST REBUTTAL:\nAuthors have agreed to include more details in the final version of the paper. Their responses have addressed my concerns.", "paper_topic_and_main_contributions": "The authors note that due to missing temporal relations in incomplete KBs, knowledge base question answering (KBQA) systems fail to get the correct answers even when the question is parsed correctly. The paper proposes to supplement incomplete knowledge bases with temporal event information extracted from a text corpus. \n\nApproach:\n- Use an off-the-shelf KBQA system to parse natural language questions into lambda expressions and executable SPARQL\n- If the execution fails, divide the lambda expression into auxiliary constraint and main expression. Missing information in each part gets handled differently but with the same extraction pipeline\n- The information extraction pipeline consists of a lambda expression verbalizer (to convert lambda into an NL question), an off-the-shelf retriever, and an off-the-shelf reading comprehension model\n- Missing temporal information from the extraction step is plugged back into the lambda expression and it is executed\n\nResults:\n- Using off-the-shelf pipeline components (that are not tuned for the evaluation datasets), the paper shows that supplementing information from text when KB execution fails leads to improvements\n- Authors note that falling back to text corpora only when KB-only approach fails is necessary for good performance. This provides a trade-off between incomplete but precise KB information and vast but noisy text corpus", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q3ggeY01HI", "reviewer": "Reviewer_r8QC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for recognising that the work is potentially “valuable for the research community” as well as agreeing that we “clearly present the problem and the proposed solution”. We appreciate that you are encouraged by our method’s “significant relative improvements in accuracy without additional training.” and are happy to note that you \"recommend accepting the paper\". We hope that given your positive impression of our work, you would reconsider your scores if your concerns are justly addressed. We gladly take all your suggestions to improve the paper and are happy to follow-up in further discussion if any concerns remain.\nPlease find below our clarifications/answers to your concerns:\n\n**RR1:** The goal of this article is to answer temporary questions, but the proposed method is aimed at all types of problems. Therefore, why not directly address all problems in KBQA (time related problems are only one of them)\n\nThis is a great question, and we indeed agree that the proposed method is *generic* and may potentially be used to address other issues within KBQA systems (provided the KBQA systems use intermediate representations that can be leveraged to localize failures). \nWe rely on the KBQA pipeline and propose to aid the system when there is a missing fact. Cases of missing facts are more **profound in the temporal context**  as adding temporal information to KBs only started taking place recently. \nCurrently, our text extraction pipeline performs a targeted extraction of simple temporal facts and our $\\lambda$\\-to-NL system is tailored  to design simple temporal questions. We would need to come up with more general modules to address other domains.\nWe would like to point out that temporal KBQA in itself is a huge domain with significant research interest and KBQA innovations proposed to improve over previous works. Our work extends this literature with a new direction of effectively aiding KBQA systems with textual resources. We believe **missing temporal facts** in KBQA is an important pain-point that we address in this work.\nGiven the positive improvements we observed and highlighted in this work for temporal QA, we do have plans to investigate the broader applicability of this technique to other domains in future.\n\n\n**RR2:** The method in this article mainly utilizes the advantages of KB and Text in accuracy and coverage, respectively, to improve the accuracy of answering complex questions. In that case, it should be compared with more kb text hybrid QA methods.\n\nPlease note the detailed literature review in Appendix A.7. While there exist end-to-end neural modules that integrate both KB and text for addressing complex QA, we did not come across any hybrid methodologies that provide benchmark results on temporal QA datasets like TempQA-WD and TimeQuestions. We build on top of a state-of-the-art modular KBQA system (https://aclanthology.org/2022.findings-emnlp.284/) which offers more *interpretability* than end-to-end systems and requires *less training data* which is often hard to obtain. We show our approach is **agnostic** to underlying KBs and KBQA pipeline, which makes our work relevant even when KBQA systems improve over time. Also, we request you to kindly provide us with some references for papers which we might have missed unknowingly. We assure you to cite them as part of our camera-ready draft.\n\n**RR3:** In the era of LLM and chatGPT, what are the advantages of such work and methods? It is recommended that the author discuss them or conduct sampling experiments.\n\nWe understand your concern. However, we’d like to point out that our approach attempts to aid Knowledge Base Question Answering Systems (KBQA) which are different from pure text-based approaches using LLMs. Studies have shown that even the huge LLMs lack complex reasoning abilities [1] and KBQA is still the go-to strategy in the literature for multi-hop/complex reasoning.  Our targeted fact extraction approach indeed uses LLMs (smaller size) off-the-shelf to aid KBQA systems.\nNotably, we were able to demonstrate positive improvements with smaller LLMS **without task-specific training**. Additionally, we propose a general approach that is **agnostic** to the underlying KBs and the LLMs used to experiment in our paper (with its applicability extending to other KBQA systems, as illustrated in Table 3.) This approach, we believe, will not only hold relevance in the context of **advancing KBQA systems**, but also **benefit from the development of bigger and more specialized LLMs**. As shown in Table 2, the extraction pipeline has huge scopes of improvement. We would like to reiterate that our novelty lies in presenting a novel combination of text and KB systems overcoming their individual limitations. We believe the promising initial results will spur research in this direction of optimally combining these resources. In this regard, we do not intend to compete against models on the scale of 175B, such as ChatGPT, and assure to discuss these implications in our final version.\n\n\n[1] A New Concept of Knowledge based Question Answering (KBQA) System for Multi-hop Reasoning. NAACL 2022.\n\n\n**RR4:** The readability of Figure 1 is not good, for example, some symbols (such as ^ in Figure 1) are not clear enough. Meanwhile, Figure 1 does not demonstrate the process and characteristics of the method proposed in this article. The readability of Figure 2 is not good, it is recommended to use a real example to introduce the complete process.\n\n\nThanks for pointing out this important concern. We had to provide concise images to stick to the page limit during submission. We will make sure to provide more elaborate and readable figures in the camera-ready version, leveraging the extra page available. We will consider your valuable recommendations for improving the figure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1）The authors clearly present the problem and their proposed solution.\n2）The experimental evaluation on two temporal OA benchmarks showcases the effectiveness of the proposed approach, demonstrating significant relative improvements in answer accuracies without requiring additional training.", "reasons_to_reject": "1) The goal of this article is to answer temporary questions, but the proposed method is aimed at all types of problems. Therefore, why not directly address all problems in KBQA (time related problems are only one of them).\n2) The method in this article mainly utilizes the advantages of KB and Text in accuracy and coverage, respectively, to improve the accuracy of answering complex questions. In that case, it should be compared with more kb text hybrid QA methods. \n3) In the era of LLM and chatGPT, what are the advantages of such work and methods? It is recommended that the author discuss them or conduct sampling experiments.\n4) The readability of Figure 1 is not good, for example, some symbols (such as ^ in Figure 1) are not clear enough. Meanwhile, Figure 1 does not demonstrate the process and characteristics of the method proposed in this article. The readability of Figure 2 is not good, it is recommended to use a real example to introduce the complete process.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper delves into the complex task of Temporal Question Answering (QA), which involves reasoning over facts that assert time intervals of events. While previous approaches have mainly relied on Knowledge Base Question Answering (KBOA) for temporal QA, they often encounter challenges in retrieving all relevant facts due to incomplete knowledge bases and errors in entity/relation linking. As a result, blocked reasoning can occur even with the absence of a single critical fact, and this problem is further compounded in the temporal context.\n\nTo address these limitations, the paper presents a novel approach that combines targeted temporal fact extraction with Knowledge Base Question Answering. By doing so, the authors demonstrate promising enhancements in answer accuracies, making their work valuable for the research community. Based on the results and findings, I recommend accepting the paper for publication. However, I encourage the authors to address the suggested improvements and conduct a more comprehensive analysis to further solidify the paper's impact and relevance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ttTBwIzqS5", "reviewer": "Reviewer_LLSL", "summary": "", "questions": "A: why top-3? Have you experimented with other values?\nB: line 271, \"Open-domain QA\": give more details about this system. For example, how does the system retrieve documents?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in giving us your valuable feedback. We are happy with your positive impression of our work. Please find our clarifications/answers to your queries below. We are happy to follow-up in further discussion if any concerns remain.\n\n**RR1:** I miss more information about the results when the KBQA fails and the QA system assists it. More in detail, I miss some results about the number of failures, their type, and the success of the QA system in these situations. I think it is essential to include this analysis of the results. I am concerned about the space limitations, but I think this analysis is more important than some details in Section 2.1, which could be moved to the Appendix (or you could use the additional page given for the camera-ready version.)\n\nThank you for expressing this concern. We indeed agree with you that a statistical failure analysis is important and will surely take your recommendations for the camera-ready version. Kindly note that we use SYGMA and EXAQT as black box KBQA systems while performing these experiments, hence we did not get into the exact details of the failures as its local to those systems. The SYGMA paper (https://aclanthology.org/2022.findings-emnlp.284/) that we build upon has reported a detailed ablation study of their system failures.\n\nFollowing your suggestions we tried to quantify the different error types faced by our proposed (KB + Text) system whenever text pipeline was fired (incomplete KB facts) as below. \n\nWe randomly sampled 50 test points where the model outputs had incorrect answers and classified them into error categories as below.\n\n| Error - > | Incorrect extraction | KBQA error | Parsing error | Miscellaneous |\n| --------- | -------------------- | ---------- | ------------- | ------------ |\n| Count     | 21                   | 9          | 8             | 12           |\n\nIncorrect extraction - The text pipeline extracted incorrect facts\n\nKBQA error - KBQA still produced erroneous answers after missing fact was correctly extracted from text\n\nParsing - Incorrect Parsing of the question \n\nMiscellaneous - Questions that don’t fall into any of the categories. Some examples of miscellaneous issues include lexical and date representation variations. Other ambiguities like multiple entities overlapping with the auxiliary date also exist (please refer to example 4 in Table 6)\n\nWe will add a detailed failure analysis and discuss scope of improvement in the camera-ready version.\n\n**Q1. A:** why top-3? Have you experimented with other values? B: line 271, \"Open-domain QA\": give more details about this system. For example, how does the system retrieve documents?\n\nWe found that the top-3 normally gave us a good boost, beyond that improvements were very minimal. We use the term Open-Domain-QA to refer the RAG (https://arxiv.org/abs/2005.11401) baseline. RAG composes a retriever based on Dense Passage retrieval (DPR)  and a generator based on BART PLM jointly trained without any direct supervision. We omitted details about Open-Domain QA baseline due to space constraints and will add more details in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper reviews and justifies the proposed method based on completing KBQA results with text-based QA.\n- The evaluation is complete and well-defined.\n- The authors compare their system with relevant baselines, obtaining an improvement in performance.", "reasons_to_reject": "- I miss more information about the results when the KBQA fails and the QA system assists it. More in detail, I miss some results about the number of failures, their type, and the success of the QA system in these situations. I think it is essential to include this analysis of the results. I am concerned about the space limitations, but I think this analysis is more important than some details in Section 2.1, which could be moved to the Appendix (or you could use the additional page given for the camera-ready version.)", "questions_to_authors": "A: why top-3? Have you experimented with other values?\nB: line 271, \"Open-domain QA\": give more details about this system. For example, how does the system retrieve documents?", "paper_topic_and_main_contributions": "The paper proposes a method for completing KBQA using text-based QA systems. The authors design a system that detects failures by retrieving facts from a KB and launches a text-retrieval + QA system able to extract those facts. The authors evaluate their approach on temporally restricted questions.\n\n The main contributions are:\n- a method for combining a KBQA system with a text-based QA system\n- evaluation of the proposed system with other relevant models (only KBQA, only text-based QA, KBQA using text-based QA for temporal extraction)\n- evaluation of different datasets", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I appreciate the inclusion of this summary of related work and the extended version in the Appendix. If the paper is accepted, I recommend including related work, extended and as an individual section, using the additional page."}}]

Author rebuttal:

Meta-review: This paper presents a solution for handling missing information in knowledge bases while answering questions over them (KBQA). Particularly when natural language queries translated into SPARQL fail to execute, parts of the query are translated back to natural language and an off-the-shelf textual QA system is used to find the missing information. The reviewers all agreed that the approach is well-motivated and that the paper is well-written. The authors addressed the reviewers' concern regarding missing error analysis. One outstanding concern is regarding the specificity of the approach to handle missing _temporal_ information alone, and the authors argues that this kind of information represents a significant bottleneck in KBQA. Overall, I think this is a strong paper.
Decision: Accept-Main