Question Answering as Programming for Solving Time-Sensitive Questions
Authors: Xinyu Zhu
Cheng Yang
Bei Chen
Siheng Li
Jian-Guang Lou
Yujiu Yang
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
reasoning
question answering
TL;DR: We apply LLMs to represent both the question and context as codes and obtain answer through programming to solve time-sensitive questions.

Abstract: Question answering plays a pivotal role in human daily life because it involves our acquisition of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer can be completely different when the time constraint in the question changes. Recently, Large Language Models (LLMs) have shown remarkable intelligence in question answering, while our experiments reveal that the aforementioned problems still pose a significant challenge to existing LLMs. This can be attributed to the LLMs' inability to perform rigorous reasoning based on surface-level text semantics. To overcome this limitation, rather than requiring LLMs to directly answer the question, we propose a novel approach where we reframe the $\textbf{Q}$uestion $\textbf{A}$nswering task $\textbf{a}$s $\textbf{P}$rogramming ($\textbf{QAaP}$). Concretely, by leveraging modern LLMs' superior capability in understanding both natural language and programming language, we endeavor to harness LLMs to represent diversely expressed text as well-structured code and select the best matching answer from multiple candidates through programming. We evaluate our QAaP framework on several time-sensitive question answering datasets and achieve decent improvement, up to $14.5$% over strong baselines.

Question Answering as Programming for Solving Time-Sensitive Questions
Xinyu Zhu S+
Cheng Yang
Bei Chen 9
Siheng Li
Jian-Guang Lou
Yujiu Yang
Shenzhen International Graduate School, Tsinghua University
Microsoft
zhuxy21@mails tsinghua, edu.cn cn
yang. yuj.uujjuuueee wwwees..tiinngie tsinghua. edu.cn
(bei. chen,
Abstract
Question: Which pasition Crispin Blunt Feb
Question answering plays pivotal role in hu- man daily life because involves our acquisi- tion of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer be completely different when the time constraint in the ques- tion changes. Recently, Large Language Mod- els (LLMs) have shown remarkable intelligence in question answering, while our experiments reveal that the aforementioned problems still pose significant challenge to existing LLMss. This can be attributed to the LLMs' inability to perform rigorous reasoning based on surface- level text semantics. To overcome this limita- tion, rather than requiring LLMs to directly an- swer the question, we propose a novel approach where we reframe the Question Answering task as Programming (QAaP). Concretely, by lever- aging modern LLMs' superior capability in un- derstanding both natural language and program- ming language, we endeavor to harness LLMs to represent diversely expressed text as well- structured code and select the best matching answer from multiple candidates through pro- gramming. We evaluate our QAaP framework on several time-sensitive question answering datasets and achieve decent improvement, up to 14.5% over strong baselines."
Wikipedia Passage:
Crispin Blunt, Crispin Jeremy Rupert Blunt born 15 July 1960 is British Convervat be Party polltccian..
Member Parliament May 2010 to September 2012 he was the Parliamentary Under-Secretary of State for Prisons and Youth Justice within the Ministry of Justice Blunt first entered the House of Commons the 1997
Reigate since and from
(4) With and Without access to external knowledge (e.g. Wikipedia), read rely knowledge answer givan question directly.
Question
LLMs
Answer
External
Knowledge
(b) Represent question, LLMs' internal knowledge and external knowledge (if available) code, then oblain answer through programming, (QAAP)
Represent
Question
Answer
all as codes
External
Knowledge
Check
Match
Figure 1: An example of time-sensitive factual ques- tion from TimeQA (Chen et al.. 2021): (a) illustrates the conventional process of question answering with LLMs, and (b) presents the proposed approach QAaP. The temporal information is colored with light blue and the potential answers are green.
the current president of the United States?". While the answer may appear apparent, its validity de- pends entirely on the specific point in time when the question is posed. This realization underscores the importance of recognizing and accounting for these temporal constraints in our quest for reliable and meaningful information.
1 Introduction
In our pursuit of knowledge and understanding, we often rely on factual questions to uncover the truth about the world around us. However, a critical aspect that is often overlooked is the presence of underlying temporal constraints within these ques- tions. Time, an important dimension in the physical world, emerges as a ubiquitous and formidable con- straint that can significantly impact the accuracy and relevance of the answer. Consider, for instance, the seemingly straightforward question: "Who is
Recently. LLMs have exhibited excellent intel- ligence in question answering and challenged the dominance of traditional search engines. Neverthe- less, despite their impressive performance, relying solely on LLMs for question answering still faces many shortcomings and limitations. As illustrated in Fig. 1 (a), current approaches to utilizing LLMs for question answering can be primarily catego- rized into two types: (I) relying on the internal knowledge of LLMs to generate direct answers,
Work done during intersship at Microsoft. !Our codes and data are available at https://gitte com/TianHong Com/TianHoon2ZY//aap

---

such as Chain-of-Thouur (CoT) prompt (Wei et al., 2022) and (2) presenting LLMs with retrieved doc- uments for reading and subsequent answering, such as ReAct (Yao et al., 2022). However, there are several challenges in answering time-sensitive fac- tual questions with LLMs: 1) LLMs are insensi- tive to numbers (Nye et al., 2022) and struggle to comprehend the sequential, overlapping and in- clusive relationships between dates, which can be attributed their nature as probabilistic models and their inability to perform rigorous reasoning as symbolic systems; 2) it is difficult for LLMs to directly locate the relevant information and provide the correct answer, as relevant facts may often be scattered across long document and expressed in diverse ways, as result, simply searching for keywords in the question relying on retrievers may fail to identify the relevant paragraphs: 3) is very likely that the answer provided by LLMs fails to satisfy the constraint and hard to verify the correctness of the answer since they are given in natural language.
formation is represented as codes, can easily construct two functions Check and Match re- duce hallucination and ensure accuracy.
With our approach, we move beyond the tradi- tional paradigm of using LLMs to directly gen- erate an answer or read given context and then provide answer, avoiding the inability to ver- ify whether the answer satisfies the constraints set out in the question. Furthermore, by storing in- termediate information in code, we overcome the length limit of the model input, thus empowering LLMs to read through the passages and uncover the eligible answer concealed within the lengthy documents. Experimental evaluation on multiple time-sensitive question-answee datasets shows that our approach consistently outperforms strong baselines and approximates the performance of supervised methods. Notably, we achieve up 14.5% 10.5% and 8.6% absolute improvements on TimeQA, TempQuestions and TimeQuestions over state-of-the-art few-shot methods.
In summary, our contributions are as follows. Our experimental results reveal that LLMs cannot accurately answer factual questions with time constraints, even induced with a Chain-of-Thought prompt.
In this work, we endeavor to bridge the gap be- tween existing approaches and the specific chal- lenge posed by time-sensitive factual questions that require both rich world knowledge and intricate rea- soning. To tackle the aforementioned challenges, we seek to harness LLMs' strong ability in natural language and programming language understand- ing to reframe the Question Answering task as Programming (QAaP), depicted in Fig. (b).
We propose a novel reasoning approach for solving the aforementioned problem with LLMs, called QAaP, that reframes the question-answering task as programming by representing question and context code, then obtaining the answer through program- ming, as opposed asking LLMs to directly provide the answer.
Specifically, our method consists of two phases: R.eeresent all as codes, since LLMs cannot rigor- ously reason by themselves, we endeavor to harness their strong coding ability to transform the ques- tion and context into well-structured codes. We first Parse the given question into python dict, then Extract relevant information from the pro- vided context and store these items in a python list. This allows comprehensive gathering and organiz- ing of relevant information dispersed throughout the documents. Besides, the collected code format information helps to facilitate the subsequent pro- cessing and avoid reasoning based on surface-level text semantics, 2. Chooose answer through program- ming, due to the notorious hallucination inherent in LLMs, it is necessary to check if the extracted contents are faithful to the corresponding context, Furthermore, as there may be multiple potential answers, we need to reason out the matching answer to the question. Since all the obtained in-
We demonstrate the superiority of QAaP com- pared to other few-shot methods on several datascts, which achieves up to 14.5% absolute improvement on TimeQA over strong base- lines and surpasses the previous supervised SoTA on TempQuestions.
Related Work
2.1 LLMs augmented with tools
Recently, the rapid advancement of LLMs has brought great progress to the field of natural lan- guage processing (Brown et al., 2020; Hoffmman et al., 2022; Chowdhery al., 2022). However, relying solely on LLMs' own capabilities is still limited, considering that human intelligence lies in the use of tools, many works have explored aug- menting LLMs with tools (Schick et al., 2023; Mi-

---

alon et al., 2023). Cobbe et al. (2021) introduces an extra calculator to LLMs for effectively solv- ing math word problems, and Gao et al. (2022a) applies retriever to obtain evidence for verifying the truthfulness of contents generated by LLMs. Similarly, Gou et al. (2023) employs various kinds of tools to correct the outputs of LLMs.
et al., 2023) and symbolic reasoning (Zhou et al., 2022) tasks. In this work, we mainly solve time- sensitive factual questions, which are more like a cross of the former tasks. Although the content that the question asks may only relate to common- sense and world knowledge, it contains strict sym- bolic constraint, which is time, and understanding temporal relationships requires advanced reasoning ability.
There have also been some works utilizing LLMs' coding ability to offload the solution to an external solver (Gao et al., 2022b; Chen et al.. 2022; Lyu et al., 2023). However, there are several differences between those code-prompt works and ours: 1) prior works mostly were solving mathe- matical and symbolic problems, while in this work we mainly focus on answering the factual question that requires temporal reasoning; 2) prior works only utilized LLMs' own knowledge for problem solving, while we propose to apply LLMs' coding ability to represent both LLMs' internal knowledge and external knowledge (e.g., Wikipedia articles) same-format codes, which enables us to casily enhance LLMS with different sources of knowl- edge, and also facilitates desired processing: 3) prior works did not explore verifying the correct- ness of LLMs--enerated codes, while we propose to incorporate Check and Match steps to mitigate LLMs' hallucination and ensure accuracy.
2.3 Temporal reasoning
There are many works that focus on temporal rea- soning before the era of LLMs. Those methods are mainly based on KBQA systems (Talukdar et al., 2012; Jia et al., 20; Saxena et al., 2021) or MRC models (Zaheer et al., 2020; Izacard and Grave, 2021; Zhang and Yamana, 2022; Yang et al., 2022; Zhang al., 2023b). There are many aspects in which our work differs from these earlier studies: 1) our method is few-shot, while they are mainly supervised; 2) we only use a simple Wikipedia search engine, while KBQA requires high-quality annotated knowledge bases; 3) we are able to verify if the constraint is satisfied, while MRC methods cannot. Nevertheless, these supervised methods are very strong and we include their results in the experiments.
2.2 Reasoning with LLMs
Method
Reasoning ability is a hallmark of human intelli- gence, which allows adapting from limited data and accomplishing unseen sophisticated tasks. Nye et al. (2022) shows that letting language model output intermediate steps improves performance. Zhou et al. (2022) introduces Least-to-Most prompt that instructs LLMs to decompose a problem into multiple sub-problems and solve them one by one. Wei et al. (2022) proposes Chain-of- Chain-of-Thought (CoT) prompting, which induces LLMs to reason step by step and greatly boosts L.MS' reasoning ability. Following works try to improve CoT in many ways, including reducing human efforts in exemplar con- struction (Kojima al., 2022; Zhang al., 2022), improving faithfulness of reasoning process (Lyu et al., 2023), code style CoT (Gao et al., 2022b; Chen et al., 2022) and allowing LLMs to interact with outside environment (Yao et al., 2022).
3.1 Task definition
In this work, we focus solving time-sensitive factual questions with LLMs. The questions pre- dominantly fall into four kinds of Wh-questions: 'What', 'Which', 'Where' and 'Who'. These types of questions seek specific information related to entities, events, and temporal aspects, forming the backbone of many real-world information-ssekkng scenarios.
As shown in the top left of Fig. 2, given a factual question Q "Salomón Rondón played for which team in Mar 2019?", which contains a time constraint Qt "in Mar 2019", LLMs need to provide the appropriate answer A "Newcastle United", which best matches the constraint Qe set out in Q. We use K, and Ke to refer to models' internal knowledge and external knowledge (e.g. Wikipedia) respectively. The context C presented to model can come from either K; or K2 as shown in the left of Fig. 2. We use EI to refer to the ex- tracted information.
All above works have showcased LLMs' remark- able capabilities in solving a wide range of complex reasoning tasks, including commonsense reasoning (Wei et al., 2022), mathematical reasoning (Zhu

---

(a] Represent all as codes
(b) Choose answer through programming
Question
Query answer key
Constraint
Salamdn Rondon played for which
Parse
query -f [aaapet" "Sekendn Hondon",
"time ("stan" deterimet2215,, 3.1)
Mar 2019?
"aajer"
"end" danehmeep2011 3,30
"Smme's Wen' denetmmennrs fend': (S005inc22055 311
F'udjeat: Candidates
SESACY Joy- "adjor"
Intermal Knowledge Venezuelan Salomón Rondon
Match
"reation"
"ooject": "Newcaste
professional footooller who plays 2s a strizer for Premier league club Newcaativ United, in August 2015, Rondon joined Newcastle United
Extracted Information EI
"Umee: (star" da0stmee2008
Item (rudjent "Sacnnnn Ronslee". "plavfor for:
dotetimes2015, 7,
Match Score: 0.09
loan
"okjeer" dutctimonaaiis ved't N
l'sujject:; "Salamón Rondōn"
External Knowledge K.
"Nowcastla United"
Item 2 "udjjet" "Exionon Rondon":
Context
Extract
II
On August 2018 Rondón joined
Trlation "play lor".
Match Score: 0.001
Newcastle United swap, Deight hoading the apposite direction.
loan
"rajer" Nenecvitle United,
['bujjett;; "Selomón flonddnn
Check
"relation" ppla
('sdaset's Item
"obgect": "Dallan Professional F.C..
On 19 July 2019, Rondón signed with Dalian Professional EC then Dallen Yitang| Chinase Super
"Uime': ('strr" danotime/2019 "end" datertime 14)
"waaton" play fre".
"atgout: DDlion rolessional
"mme" cant dewn mepors, 7, "ooaf Hone II
Match Score: 0
Context
first match
CSKA,
Item
Rondon scored his first goal for the Mossow team and who give the best CSKA
I'sudjet": Selemdn tondon".
"rabaton": ylay for".
Answer: Newcastic United
"obeor": "Arona
plaper
"ond" (20 311
The whole framework of QAaP. Relevant and irrelevant temporal information is highlighted in light blue and blue, respectively. The correct answer is green; otherwise, is red. Text related to potential answers bold.
Figure 2:
3.2 Represent all as codes
the extracted information is presented in code form, we can easily construct two functions Check and Match to verify the faithfulness of extraction and find the best matching answer.
Parse. We prompt LLMs to parse the given ques- tion Q into a python dictionary variable named query q. which contains four keys: subject, rela- tion, object, time and their corresponding value r, and t. In addition to the query, we define another variable answer_key to specify where the final answer should be placed at.
Check. We first check if the extracted item is in the same format as query, which ensures the answer is correctly placed at answer_key. and that the ex- tracted time should appear in the corresponding context. When external knowledge Ke is acces- sible, we also check whether the items extracted from LLMs' internal knowledge have shown up in those extracted from Ke because LLMs often generate fake facts.
Extract. Given a text segment from the docu- ment either generated by LLMs or retrieved from Wikipedia as context C₁ we prompt LLMs to ex- tract information related to the question illus- trated in the middle of Fig. 2. Each item of the ex- tracted information FT; = {(5)
N
As exemplified in Fig. 2, the answer Newcas- tle United generated by LLMs also appears in the Wikipedia passage, therefore we keep the extracted item. Note the middle bottom of Fig. 2, the ex- tracted time 2019 of the 4th item does not show up in the corresponding Context 2 and therefore it is removed during the Check step.
j=1 is also represented python dictionary similar to the query and stored in a predefined python list information so as comprehensively gather rel- evant information that appears in different parts of a long document.
Match. In order to choose the best matching an- swer from numerous candidate answers, we employ the intersection over union (IoU) of time between the question Q and the candidate X as the match score, which is defined in Eq. (1), and sort the can- didates according it. When the question specifies both the start Qt. and the end Qt, of the time con- straint, this measurement quantifies the degree of temporal alignment between the question and each
3.3 Choose answer through programming
As mentioned before, L.Mss are not sensitive to numbers and often suffer from hallucinations. For example, LLMs may trickily fill in the time that appeared in the question into the extracted items regardless of the context, resulting in a perfectly temporal matching but incorrect answer. To cope with such a phenomenon, thanks to the fact that

---

candidate. If the time constraint in query only con- tains a start or an end, we choose the inverse of the absolute value of the differences between Qt, and Xt, or Qr and Xt, as match score.
those off-the-shelf retrievers, which we leave for future work.
4.1.2 Baselines
For comparison under a few-shot setting, em- ploy the well-known method CoT (Wei et al., 2022) as the baseline. In addition, we include a recently popular method ReAct (Yao et al., 2022), which incorporates external knowledge and allows LLMs to execute search and lookup actions to interact with the Wikipedia website. Finally, we present results from ofate-of-the-art fine-tuning models for comprehensive comparison.
min Xe,) max (Qr,, X.,) max (Qu,, Xx.) min (Q Xt,)
MS(Q, X)
(1)
Finally, we determine the answer by selecting the candidate that achieves the highest score. The Match step ensures that the chosen answer aligns closely with the time constraint specified by the question, thus guaranteeing the accuracy and rel- evance of the answer. We also discuss applying LLMs for accomplishing the Check and Match functions in Sec. 4.3.4.
4.1.3 Implement details
To fully utilize LLMs' internal knowledge Kü fol- lowing Yu et al. (2023) we elicit K; by prompting LLMs to generate a background document for the given question. Additionally, in order to incorpo- rate external knowledge Ke. we add a search pro- cess to the prompt and apply a Wikipedia search en- gine similar to ReAct. We treat the documents from K, and Ke in the same way. For all experiments. we use gpt-3. 5-turbo as the hackbone model un- less otherwise specified. Details of prompt can be found in Appendix C.
4 Experiments
4.1
Experimental setup
4.1.1 Datasets
We consider several widely-used temporal reason- ing datasets: TimeQA (Chen et al., 2021), Tem- pQuestions (Jia et al., 2018a) and TimeQuestions (Jia et al., 2021).
TimeQA is a time-sensitive question answering dataset curated by extracting time-evolving facts from WikiData and aligning them with Wikipedia pages with the help of human workers. The dataset comprises two subsets with different levels of dif- ficulty. The easy subset contains documents with asked temporal information explicitly stated, while the hard subset requires more advanced reasoning skills as the temporal information is implicit and cannot be retrieved through kcyword searching. TempQuestions and TimeQuestions are two simi- lar temporal QA datasets compiled from a variety of general-purpose KG-QA benchmarks. Ques- tions are divided into four classes: explicit tem- poral, implicit temporal, temporal answer and or- dinal constraints. We choose the explicit subset and remove the questions without any temporal signal. This is because implicit questions require extra steps to uncover the implicit time indicated by the things/events, while we only use a simple Wikipedia search engine and there are so many am- biguous things that have the same name (e.g., song, movie, person name, ...), it is difficult for LLMs to determine which entity the question is referring to. Therefore, strong retrievers are needed to retrieve relevant documents. It worth noting that our method is agnostic to the way of retrieving external documents and can be seamlessly combined with
4.2 Main results
Table I presents the main results on the three tempo- ral reasoning datasets. QAaP consistently demon- strates superior performance across all the datasets under few-shot setting. While CoT performs well on the TempQuestions dataset by relying solely on the LLMs' internal knowledge, it falls short on TimeQA and TimeQuestions. This discrepancy can be attributed to the limited coverage of the LLMs' Kj in memorizing the rich facts of the open world. On the other hand, despite having access external knowledge Ke, ReAct still un- derperforms, indicating its potential shortcomings in locating relevant information from provided doc- uments. Impressively, QAaP outperforms other methods on all the datasets, exhibiting substan- tial improvements. Specifically, achieves a re- markable enhancement of 13.9%, 14.5%, 10.5%, and 8.6% in exact match scores on TimeQA-Easy TimeQA-Hard, TempQuestions, and TimeQues- tions, respectively. The above experimental results clearly demonstrate the effectiveness of QAaP.
The improvement might be explained by two reasons: one is that with QAaP we are able to com- prehensively extract relevant information from the

---

TimeQA-Eas.
TimeQA-Haal Hard
TempQuestions EM
TimeQuestions EM
Backbone
Method
EM
F1
EM
F1
F1
F1
Fine-tune
Previous SoTA
60.5
67.9
46.8
54.6
43.8
44.6
56.8
Few-shot
5pt-r.o
CoT
34.2
21.7
30.4
49.8
28.2
41.2
ReAct
34.3
41.5
25.1
29.4
28.0
33.7
12.6
17.5
QAaP
48.2
58.3
39.6
49,3
60.3
68.1
36.8
46.7
Table 1: Results on temporal reasoning tasks. The previous SoTTA supervised models for TimeQA, TempQuestions and TimeQuestions FiD (Chen et al., 2021), TEQUILA (Jia al., 2018b) and EXAQT (Jia et al., 2021) respectively. The best scores are in bold and the second are underlined.
TimeQA
Check K,
TimeQA
Method
Ke
TempQuestions
TempQuestions
Easy
Hard
Easy
Hard
CoT
x
24.6
21.7
49.8
x
x
32.8
27.6
55.2
CaT-iariant
22.0
18.5
48.8
x
v
45.0
37.0
59.3
QAaP
48.2
39.6
60.3
V
x
39.8
32.1
56.6
V
48.2
39.6
60.3
CoT-variant
48.0
33.7
QAaP"
52.6
46.7
Table 3: Ablation study of checking different aspects.
Table 2: Ablation study of providing enough context in COT. Results are exact match scores. Ke refers to ex- ternal knowledge. indicates given golden paragraphs containing ground truth.
As demonstrated in Table 2, though given the same context, QAaP achieves a much higher ex- act match score compared to directly reading and answering. The performance of the CoT-variant is worse than CoT, which exposes LLMs' limi- tations discriminating the answer that meets the constraint from the other irrelevant ones. Signifi- cant performance improvements are observed when golden paragraphs are provided. This result high- lights the challenges faced by LLMs in accurately identifying the correct answer from lengthy doc- uments. Notably, QAaP surpasses CoT- variant''"' even though the latter directly accesses the golden paragraphs. This finding further underscores the advantage of proposed method.
provided knowledge sources and store it efficiently, which facilitates the subsequent processing steps. The other is that rather than relying solely on LLMs reason over context, QAaP chooses the final answer through programming. Since all the things are represented as codes, QAaP minimizes the need for human intervention to check the extracted con- tents and find the best matching answer.
However, there is still neg-negligible gap pared to the best-supervised models, which vali- dates the difficulty of this task and indicates more efforts are needed in future work.
4.3.2 How much does Check alleviate hallucination?
4.3
Ablation study
4.3.1
Can LLMs find the answer provided enough context?
To examine the effect of Check-in alleviating hal- lucination, we explore it along with two axes: 1) check if the time of extracted items appears in context and 2) check if the items extracted from LLMs' internal knowledge K; are also included in those extracted from external knowledge K..
To investigate if LLMs can answer factual ques- tions with time constraints provided enough con- text, we implement a variant of few-shot CoT by feeding it with the same context used for our ap- proach. Specifically, we present the text segments to them sequentially and ask if they answer the question based the context until they say yes and give the answer. Moreover, we lower the difficulty by using golden paragraphs as context, which con- sists of the paragraph containing the ground truth and three paragraphs before and after it.
As shown in Table 3, compared to checking the extracted time, checking the contents obtained from K, can bring more improvement. This discovery implies that the degree of hallucination is more vere when I.LMs are allowed generate content directly. However, even when conditioned on a given context, LLMs may still make up contents

---

TimeQA
most appropriate answer from multiple candidates. as there are many interfering items. The results indicate that these two steps are complementary and indispensable.
Match
Check
TempQuestions
Easy
Hard
X
34.2
26.7
54,2
48.1
36.4
59.3
Table Ablation study of letting LLMs choose the best answer from extracted candidates.
4.3.4
Are LLMs capable of writing qualified Check and Match functions?
In our previous experiments, the Check and Match functions were manually constructed. However, in this section, we aim to explore the potential of lever- aging LLMs to perform these tasks. Specifically, we provide clear instructions for the Check and Match processes, then we randomly select and con- catenate an instance {Q.q. C, ET) obtained from previous steps to the instruction input prompt, finally present the prompt the LLMs and require them to write the desired functions. The results, as shown in Table demonstrate the feasi- bility of replacing human effort with LLMs. Both ChatGPT and GPT-4 exhibit the capability to com- prehend the task and provide an eligible solution.
TimeQA Easy
Model
TempQuestions
Hard
ChatGPT
47.9
38.6
56.9
GPT-4
48.2
38.5
56.9
Manual
48.2
39.6
60.3
Table 5: Employing LLMss for constructing Check and Match functions. ChatGPT and GPT-4 can be accessed via https://chat... http://chat.ppeai..cco com.
that do not align with the provided context. The ex- perimental results affirm the vital role played by the Check step in mitigating hallucination, shedding light on the substantial presence of hallucination in LLMs. These findings emphasize the importance of verifying the contents generated by LLMs to guarantee accuracy and reliability.
This promising result suggests the potential di- rection for future research to explore how LLMs can self-verify their answers through programming. While LLMs have shown impressive performance in many NL. tasks, ensuring the accuracy and reli- ability of their generated responses remains a chal- lenge. Exploring the integration of these verifica- tion mechanisms within LLMs can lead to more ro- bust and trustworthy question-answering systems.
4.3.3
Can LLMs directly identify the best matching answer the question?
We are curious about whether is trivial for LLMs to find the best matching answer given the parsed query and extracted information. To this end, we re- move the Match step and prompt LLMs to provide the answer through in-context learning.
4.4 Case study
The results are presented in Table 4, it can be ob- served that without the Match step, the exact match score relatively decreased by 8.1% on TimeQA- Hard, which stresses the indispensability of the Match step when the constraint necessitates rigor- ous reasoning to satisfy. In contrast, there is only a slight drop in performance TimeQA-Fasy and TempQuestions. This can be explained by the fact that we did not design very sophisticated Match function for selecting the best answer. Addition- ally, the strong result of directly matching further substantiates the advantages of representing the question and context as codes, which can facili- tate better reasoning for LLMs and significantly improve the accuracy compared to reading then answering methods.
A typical exemplar is presented in Fig. CoT encounters hallucinations and makes up facts that don't 't exist, which exposes its limitation when the question asked is beyond its scope of knowledge. Even given the golden paragraphs, CoT-variant* fails to reason correctly and mistakes Margrete Aamot Øverland the final answer. ReAct tries to find the relevant sentences by looking up keywords, however, neither the year nor the keyword the question appears in the passage. This is one of the challenges we mentioned carlicr, without strong retrievers LLMs are usually unable to solve these questions, resulting in performance upper bounded by external retrievers.
As a comparison, results generated by QAaP are shown at the bottom. We omit the process of Check and Match steps for the simplicity of illustration. First, L.MSs parse the question into a Python die- tionary named query and generate a relevant back- ground document with their internal knowledge.
is worth noting that the Check step plays key role in the final performance when the Match step is absent. Without the Check and Match steps, LLMs face challenges in directly identifying the

---

@verland spouse between May Jun CONTEXT: Amnlf @verland was Norwegian outhor, poet, and cribic.
QUESTION:
Personal life, In 1918 he hed married the singer Hildur Arntzen| | 1888--1557 J. Their marriage wes dissolved in 1939, 1940, he married Bartholine Eufemia Leganger (1903-1995 They separated shortly after, officially divorced in 1945, diverland was married to journalist Margrete Aamot @iverland ( 1913-11778 during June 1945. Bartholine Eufemia Leganger
GOLDEN ANSWER:
COT (WITHOUT CONTEXT):
Arnulf Everland was Norwegian author, poet, and social commentator. was to pianist compuser Harriet Cohen from 1937 untill her death 1967. There is no answer. x
COT-VARIANT
Context: SCONTEXT>>
answer context?
Answer: Margrete Aamot @verland X
REACT:
Thought 1:1 need search Amull fiverland and find out who his spouse was between May 1944 and Jun 1944.
Action Search (Amulf dverland)
Observation 1: Arnulf @verland. Ole Peter Arnulf Øverland 27 April 1889 - March 1968 ) was e Norwegien poet principally
and
poetry which served to inspire the Norwegian resistance movement during the German occupation Norway during World War Il.
who Arnulf diverland's spouse between May 1944 can look up the years between 1944 and 1944, First can look up "1044".
Thought paragraph
maybel
Action Loakup[1544]
Observation 2: No more results.
Thought There no results for "1944", let's look "spouse".
Action 3: Lookupispouse] Observation 3: No more results.
Thought 4: The passage does not tell who Amull Overland's spouse was between May 1944 and Jun 1964,
Action 4; Finish[]
Question Parsing:
ppytho
("aubject"; "Azoulf @vezland", 'relation": "apouse" "oct" Hoow,
babaatt
datetiiee (1964,
"end": datetime (1946,
"object"
Generate background document from Wikipedia to answer the given question: Arnulf @verland was a Norwagian author, poct, and critic. He was married to the author and transistor Berit As from 1945 until death 1968. Extract information relevant to the query:
python
"Arnulf @verland", "relation": "apouse", "object": "Bezit As", "time": (*start"; datatime (1945, "end": datetime (1968, 12, 31131)
information tLon.
Context: <CONTEXT>
Extract information relevant the query:
*> "python
infomationn.o af ('eubject":
@verland", "relation"; "spouse", "objeet";
Asutzen". "tise": (*start"; datatime (1918, 1, 1), "end": (1933,
211)1)
information (faanddi
"Aroult @vazland", "relation": "apouse", "objact": "Bertholine
Sufenia Legangee", "time" ("staat": datetine (1940, 1, 1), "and"; datetine (1945, 12, a1)))) infomation (igendd"""abble Annot @verland", "time": " { ("start": datetimee 6, 12, "end": None)))
"Aroule @vezland", "zelation": "apouze", "object";; "Mazgrate
<Check>>
<Match>>
matching answer: Bartholline Eufomia Leganger.
Figure 3: Examples of correct and incorrect answers were obtained with different methods on TimeQA. Interference answers or wrong parts and time information not related to the question are highlighted in red and blue respectively. Correct answer and relevant time in green and light blue. The text implies potential answers in bold.
Here again, the LLMs hallucinated a non-existing fact that "Arnulf Øverland was married to Berit Ås". After extracting the relevant information, we perform the Check step verify the faithfulness of extraction and the reliability of contents generated by LLMs. As the extracted information is struc- tured as code, we can easily filter the wrong and fake items and therefore mitigate the hallucination, which is proved in Sec. 4.3.2. Finally, we choose the best matching answer through the Match step.
Discussion and Future Work
Our findings reveal the inherent difficulty faced by LLMs in accurately answering seemingly straight- forward factual questions when they involve spe- cific time constraints. This can be attributed to the nature of neural networks. A well-known prob- lem with neural networks that they are black-box models and do not have symbolic reasoning ca- pabilities. Consequently, even the most powerful language models may struggle to perform rigor- ous reasoning. However, in this work, instead of

---

relying solely on LLMs to directly provide the an- swer, we reframe the question-answeriin lask as programming. By leveraging LLMs to represent the question and context as codes, we can easily alleviate the hallucination and improve answer ac- curacy through the subsequent Check and Match processes,
our approach consistently demonstrates superior performance over strong basclines with LLMs. We hope this work can shed light on the future research direction on enhancing LLMs' reasoning ability to tackle real-world questions with various constraints and developing more efficient methods to reduce the hallucinations LLMs frequently encounter.
We hope our work can shed light on future work in exploring how to enhance LLMs' reasoning abil- ity with more advanced means and tools. Addition- ally, we anticipate there will be more human-in-ther loop approaches to effectively reduce the halluci- nation while only requiring minimal labor efforts, for example, incorporating structured knowledge to automate the verification process. Moreover, mainly focus on solving factual questions with time constraints, but many factual questions in real life may contain various types of constraints, such as number, order, location, etc. For different tasks, our method can be casily adapted to deal with other types of constraints, we just need to represent the constraint into an appropriate class in python and define the metric that measures how well is the constraint satisfied in the Match function. Further- more, convenient to cope with different kinds of hallucination by incorporating additional check processes into the Check function. We leave that for future work.
Limitations
The results on multiple datasets demonstrate the effectiveness of our proposed framework QAaP, which can improve the accuracy of LLMs in an- swering time-sensitive factual questions. However, there are still limitations in our work:
1) We do not evaluate our method on other question-answering datasets with different kinds of constraints. The main reason is that there are limited relevant datasets. We also do not include question-ans
datasets that require multi-hop retrieval to collect enough related documents like hotporQA (Yang et al., 2018) since a strong re- triever is needed and the accuracy may be mainly depended on the retriever, but our approach is ag- nostic to the way of introducing external knowledge and can be combined with off-the-shelf retriever seamlessly.
2) When solving questions requiring only com- monsense or world knowledge, QAaP may not be necessary because there is no constraint in the ques- tion that needs rigorous reasoning to satisfy. This limitation can be also found in Table 1 of Lyu et al. (2023) where faithful-CoT does not help on Strate- gyQA dataset (Geva et al., 2021).
We view our effort as the first step towards solv- ing open-domain questions with some kinds of strict constraints and we hope this work can inspire other work to tackle questions with constraints in more general scenarios. promising research di- rection is to enable LLMs to achieve comparable accuracy rates to search engines while maintain- ing the simplicity of interaction, which will greatly boost the practicality of LLMs.
3) We only report experimental results with one backbone model due to the high cost of other OpenAl models. However, further prove the effectiveness of our method, we conduct some small-scale experiments with text-davinci-003 and include the results in Appendix B, which verifies that our approach performs better than other baselines with differ- ent backbone models. We leave exploring other open-souree models (Wang et al., 2022) for future work.
This is mainly
6 Conclusion
In this work we propose a novel approach, QAaP (Question Answering as Programming), to tackle the challenges posed by time-sensitive factual ques- tions. By leveraging LLMs' exceptional abilities in natural language understanding and programming, QAaP can transform diversely expressed text into well-structured codes, enabling LLMs to capture both the desired knowledge and the underlying con- straints, particularly the temporal aspects. Exper- iments demonstrate that existing L.LMs face sig- nificant difficulty in effectively comprehending the temporal constraint stated in the question. While
Ethies Statement
In this work, our proposed approach QAaP can ef- fectively solve factual questions requiring temporal reasoning, however, there still a potential social risk. The main concerning risk our work may be the hallucination of utilized LLMs (Zhou et al.,

---

2021; Ji al., 2023; Zhang et al., 2023a), since the method we develop is aimed answering factual questions, when applied to LLMs that are de- ployed into real production environments, it may provide erroneous and misleading answers to users due to the hallucination happened during the Pars- ing and Extract step. Similar challenges exist in the computer vision field too (He et al., 2023a.b). This can be tackled by designing more sophisticated Check and Match steps, which we also highlight in previous experiments and is proved effective in alle- viating hallucination, thus helping reduce potential risks posed to society.
and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/21 10.14
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2022a. Rarr: Researching and revising what language models say, using language models. arXiv preprint arXiv:2210.08726.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Gra- ham Neubig. 2022b. PAL: program-aided language models. CoRR,
References
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use laptop? A question answering benchmark with implicit reasoning strategies. Trans, Assoc. Comput. Linguistics, 9:346-361.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurlPS.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. 2023, Critic: Large language models can self-correct with tool-interactive critiquing, ArXiv, abs/2305. 1173..
Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang. Yulun Zhang, Zhenhua Guo, and Xiu Li. 2023a. Weakly-supervised concealed ob- ject segmentation with sam-baser pseudo labeling and multi-scale feature grouping. arXiv preprint arXiv:2305.11 11003.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks. CoRR, abs/2211.12588..
Chumming He, Kai Li, Yachao Zhang, Yulun Zhang, Zhenhua Guo, Xiu Li, Martin Danelljan, and Fisher Yu. 2023b. Strategic preys make acute preda- tors: Enhancing camouflaged object detectors by generating camouflaged objects. arXiv preprint arXiv:2308.03166
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive ques- tions. In NeurlPS Datasets and Benchmarks.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanunalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov,
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc. Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. CoRR, 0ats/2203.155555
Gautier Izacard and Edouard Grave 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL. pages 874- 880. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv,

---

Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018a. Tem- pquestions: A benchmark for temporal question an- swering. In WWW WWW (Companion Volume), pages 1057- 1062. ACM
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
Ping Yang, Junjic Wang. Ruyi Gan, Xinyu Zhu, Lin Zhang. Ziwei Wu, Xinyu Gao, Jiaxing Zhang. and Tetsuya Sakai. 2022. Zero-shot learners for natural language understanding via unified multiple choice perspective. In EMNLP, pages 7042-7055 Associa- tion for Computational Linguistics.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy. Jannik Strötgen, and Gerhard Weikum. 2018b. TEQUILA: temporal question answering over knowl- edge bases. In CIKM, pages 18 1807-1810. ACM
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In CIKM, pages 992-802. ACM.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov. and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. In EMNLP, pages 2369-2380. Association for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. CoRR, abs/2205.11916.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.0366
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang. Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. CoRR, abs/2301.13379
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Confer- ence for Learning Representation (ICLR).
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yo Yu, Asli Celikyilmaz. Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey. CoRR, abs/2302.07842.
Manzil Zaheer, Guru Gurugamesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In NewrlPS.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber. David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. Show your work: Scratchpads for interme- diate computation with language models. In Deep Learning for Code Workshop.
Muru Zhang, Ofrr Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023a. How language model hallucinations can snowball. CoRR, abs/2305.
Yuxiang Zhang, Junjie Wang. Xinyu Zhu, Tetsuya Sakai, and Hayato Yamana. 2023b. Ner-to-mre: Named- entity recognition completely solving as machine reading comprehension. CoRR, (bbs/2305.0390.
Apoory Saxena, Soumen Chakrabarti, and Partha P. Talukdar. 2021. Question answering over temporal knowledge graphs. In ACL/JCNLP (1), pages 6663- 6676. Association for Computational Linguisties.
Yuxiang Zhang and Hayato Yamana. 2022. HRCA+: advanced multiple-choice machine reading compre- hension method. In LREC, pages 6059-6068. Euro- pean Language Resources Association.
Timo Schick, Jane Dwivedi-Yes Yu, Roberto Dessi, Roberta Railcanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. CORR, abs/2210.03493..
Partha Pratim Talukdar, Derry Wijaya, and Tom M. Mitchell. 2012. Acquiring temporal constraints be- tween relations. In CIKM, pages 992-1001. ACM.
Chunting Zhou, Graham Neubig. Jiatao Gu, Mona T. Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In ACLAJCNLP (Findings). volume ACL/IJCNPP 2021 of Findings of ACI., pages 1393-1404. Association for Computational Linguistics.
Junjie Wang. Yuxiang Zhang. Lin Zhang, Ping Yang. Xinyu Gao, Ziwei Wu, Xiaoqun Dong. Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang. Xiao- jun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang, 2022. Fengshenbang 1.0: Be- ing the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Olivier Bousquet, Quoe Le, and Ed Huai hsin

---

Chi. 2022. Least-to-mon tost prompting enables com- plex reasoning in large language models. ArXiv, abs/2205.10622
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang. Ruyi Gan, Jiaxing Zhang, and Yu- jiu Yang. 2023. Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers). pages 4471-4485. Toronto, Canada. Association for Computational Linguisties.

---

Dataset
Split
of samples
TimeQA
Easy
2997
TimeQA
Hard
3078
TempQuestions
Explicit
297
TimeQuestions
Explicit
942
Table 6: Dataset statisties.
A Dataset Statistics
Dataset statistics are shown in Table 6.
B Additional Experiments
We conduct experiments with text-davinci-003 and present the results in this section. Due to the high cost of OpenAl APIs, we only sample 100 questions for each dataset with random seed set O. As shown in Table 7, the results confirm the effectiveness our method.
Full Prompts
TimeQA prompt is 3-shots and divided into three parts illustrated in Fig. 4, Fig. 5, Fig. 6. We have included other prompts in the supplementary ma- terials and will open source all of them for further research.

---

TimeQA-Ears
TimeQA Hard
TempQuestions EM
TimeQuestions EM
Backbone
Method
EM
F1
EM
F1
F1
Few-shot
text-davinci
CoT
17.0
26.2
15.0
28.0
39.0
49.6
33.0
44.8
ReAct
30.0
42.7
26.0
37.4
33.0
42.2
22.0
31.6
QAaP
38.0
50.1
38.0
49.4
47.0
51.9
40.0
48.9
Table 7: Results on temporal reasoning tasks with text-davinci- The best scores are in bold and the second are underlined.
Solve a question answering task by first parsing the question, figuring out what the question is asking, representing the query as python dictionary with the key where the answer should be stored specified as either the subject or object, with the initial value of None. Secondly, decide what entities to search to find the answer. Thirdly, generate background document that contains information relevant the question being asked. Then read the generated document and the information that directly explicitly relates as the value the answer key in dictionary. For example, 'XO joined team 2017, ..., in 2019, B team signed contract with XXX", is easy to know that team and A team are mutually exclusive, therefore the termination time of A team is in 2019. Represent the extracted information as dictionary and adding to list. the context does not tell any useful information, extract nothing. examples.
searched passage, step by step
question, place answer
Here
Question:Which school did David Jolly go Question parsing:
Python
query { ("subject"; "David Jolly", "relation": "go to school", "object"; None, "time": ("start"; datetime (1989, 1, 1), "end"; datetime (1989, 1, 31))}
answer key "object"
Search:
pyyhon
entities_ to tt_ssercch ("David Jolly"]
Generate background document from Wikipedia to answer the given question: David Jolly is American politician who served as the U.S. Representative for Florida's 13th congressional district from 2014 to 2017. He graduated from Indian Rocks Christian School 1990.
Extract information relevant the query:
'python information. append (( ("subject"; "David Jolly", "relation": "go to school", "object"; "Indian Rocks Christian School", "time"; ("start:: datetime (1986, 1, "end": datetime (1990, 12, 31) })
David Jolly David Wilson Jolly born October 31, 1972) is an American attorney former lobbyist, and politician who served as the U.S. Representative for Floridas 13th congressional district, based in Pinellas County, from 2014 to 2017. He was subsequently reelected in November 2014, winning 75 percent of the vote, but was unseated in 2016 by former Goveroor Charlie Crist after court-ordered redistricting made his district more Democratic. September 2018, Jolly announced he had left the Republican Party
Context:
Extract information relevant to the query:
There is nothing relevant to the query,
Context: Early life... Jolly was born in Dunedin, Florida, the son of Judith and Lawson Jolly a Baptist pastor He received his B.A. degree from Emory University in 1994 and his J.D degree from the George Mason University School of Law in 2001.
Extract information relevant the query:
python
information append (1 (("ubjject:; "David Jolly", "relation"; "go to school", "object": "Emory University", "time": : ("start"; datatime (1990, 1, 1), "end"; datetime (1994, 12,
31) )))
information. append ((
(("sud((""u ("subject"; "David Jolly", "relation"; "go to school", "object": "George Mason University", "time": ("start": datetime (1995, 1, 1), "end": datetime (2001,
12, 31)
Figure TimeQA prompt part

---

Question:Which position did Crispin Blunt hold in Feb 2011?
Question parsing:
"pyyhonn
query = ("subject": "Crispin Blunt", "relation": "hold position", "object": None,
"time": ("start": datetime (2011, 2, 1), "end": datetime (2011, 2, 28) }}
answer key "object"
Search:
"python
entities_
("Crispin Blunt"]
Generate background document from Wikipedia answer the given question: Crispin Blunt is British Conservative
Party politician who has been the Member of Parliament (MP) for Reigate since 1997. In February 2011, he was appointed as the Parliamentary Under-SSccreary of State for Prisons and Youth Justice in the Ministry of Justice. Extract information relevant to the query: ppython
information, append ("subject": "Crispin Blunt", "relation": "hold position",
"object": "Member of Parliament", "time": ("start": datetime (1997, 1, 1), "end":
None)
"Crispin Blunt", "relation"; "hold position", "object": "Parliamentary Under- Secretary of State for Prisons and Youth Justice", "time": ("start": datetime (2011, 2, "end": None)
information
Context: Crispin Blunt Crispin Jeremy Rupert Blunt born 15 July 1960 ) is a British Conservative Party politician. He has served as Member of Parliament MP for Reigate since 1997, and from May 2010 September 2012 he the Parliamentary Under-Secretary of State Prisons Youth Justice within the Ministry Justice Blunt entered the House of Commons at the 1997 general election, when he replaced the then MP Sir George Gardiner who had been deselected by the Constituency Conservative Association Executive Council and joined the Referendum Party
Extract information relevant to the query:
'python
information append "subject"; "Crispin Blunt", "relation"; "hold position",
"object": "Member of Parliament", "time": ("start": datetime (1997,, 1, 1), "end": [)
information
"Crispin Blunt", "relation"; "object": "Parliamentary Under- Secretary of State for Prisons and Youth Justice", "time": ("start": datetime (2010, 5, 1), "end": datetime (2012, 9, 30)
position",
Figure 5: TimeQA prompt part 2.

---

Question:What's the capital Klamath County, California between Aug and Jun 1855?
Question parsing;
'python
quezy = ("subject"; None, "relation"; "capital of", "abject"; "Klamath County,
California", "time": ("start"; datetime (1854, 8, 1), "end"; datetime (1855, 6, 30) ))
answer_Kee key "subject"
Search:
"python
entities_toe search "Kaammahh County"]
Failed entities:
ppython
failed entiiiiettttteeee ("Klamath County"]
similas_entities entities ("Klamath th County, California", "Klamath County, Falls, Oregon"]
similar
Oregon",
"Klamath
Search:
"python
entities_te search ["Kl Kmathh County, California")
Generate background document from Wikipedia answer the given question: Klamath County, California was county of California from 1851 to 1874, The county seat was Crescent City from 1851 to 1854 and then Yreka from 1854 to 1874. Extract information relevant to the query:
'python
information. append (1 (l"sujjec":: "Crescent City", "relation": : "capital of", "object": "Klamath County, California", "time": ("start"; datetime (1851, 1, 1), "end":
datetime (1854, 12, )1
information. www.aappedddccc (ep"uj(("sujeet"") "Yreka", "relation"; "capital of", "object": "Klamath County, California", "time": ("start": datetime (1854, 1, 1), "end": datetime (1874, 12, })
Context: Klamath County, California Klamath County was county of California from 1851 to 1874 During its existence, the county moved twice and ultimately portions of the territory once had were carved up and added to nearby Trinidad,
countys southwestern coast. 1854 the county Crescent City, because its larger population. But western portion the county was unrepresentative of mining interests in the eastern portion of the county, and so in 1856, the county seat was moved inland, to Orleans Bar, now Orleans 1857, Del Norte County, including Crescent City, was split off from Klamath County In 1874 Klamath County finally abolished, divided between Siskiyou and Humboldt counties. Extract information relevant the query: python
counties. The original county
moved
information. append (1" ("subject": "Trinidad", "relation": "capital of", "object": "Klamath County, California", "time": ("start": datetime (1851, 1, 1), "end": datetime (1853, 12, )1)
information. append (1 (("sujject"; "Crescent City", "relation"; "capital of", "object": "Klamath County, California", "time": ("start"; datetime (1854, 1, 1), "end":
datetime (1855, 12, 31)
information. (f (( ("subjjcc""; "Orleans", "relation"; "capital of", "object"; "Klamath County, California", "time": ("start"; datetime (1856, 1, "end": datetime (1856, 12, 31))))
Figure TimeQA prompt part 3.


Reviews:
[{"review_id": "6l7ZwmuhM5", "reviewer": "Reviewer_PAqi", "summary": "", "questions": "Please check the weaknesses and provide your answers.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, insightful and constructive comments!\n\n### Description of the challenges and necessities\n\nIn response to Question 1, regarding the **challenges** (denote by **C**) and the **necessities** (denote by **N**) of reframing the QA as programming:\n\n- **C1.** As mentioned in L062-065, LLMs struggle to comprehend the sequential, overlapping and inclusive relationships between dates, which can be attributed to their nature as probabilistic models and their inability to perform rigorous reasoning as symbolic systems.\n- **N1.** Though LLMs cannot rigorously reason by themselves, they are able to generate high-quality and executable codes, we endeavor to harness their strong coding ability to transform the question and context into well-structured codes, this avoids rigorous reasoning based on surface-level text semantics and facilitates the subsequent processing and reasoning over the collected code-format information (Check and Match steps).\n- **C2.** As mentioned in L066-073, it is difficult for LLMs to directly locate the relevant information and provide the correct answer, as relevant facts may often be scattered across a long document and expressed in diverse ways, this challenge is also proved by a recent work [1].\n- **N2.** In QAaP, we apply LLMs to transform document segments into same-format python dicts and save them in a python list, this allows a comprehensive gathering of the relevant details dispersed throughout (arbitrary number of) documents.\n- **C3.** As mentioned in L100-105, the answer directly provided by LLMs in natural language is hard to verify and is often erroneous, it is very likely that the answer does not satisfy the constraint and thus incorrect as shown in Table 2, Table 4 and Figure 3.\n- **N3.** Since all the obtained information is represented as codes, we can easily construct two functions Check and Match to reduce hallucination and ensure accuracy as mentioned in L091-099. These two steps are built on the previous steps and play a key role in the final performance as illustrated in Table 3 and 4, without which performance will drop 10+%.\n\nWe will make modifications to make the description of challenges and necessities more clear and coherent in the next version, thanks for the pointer!\n\n### Difference explanation and experimental comparison.\n\nIn response to Question 2,\n\ni) Regarding ***“not so sure if it is suitable to call it “reframe QA task as programming””***, in QAaP we parse the question into a python dict, extract relevant information from documents and represent them as python dicts, the entire problem solving process is based on those codes and the answer is determined by code execution, therefore we call it “reframe QA task as programming”.\n\nii) Regarding ***difference between prior works***, the most similar works are KBQA and MRC methods, there are several differences: 1) our method is few-shot, while they are mainly supervised; 2) we only use a simple Wikipedia search engine, while KBQA require high-quality annotated knowledge bases; 3) we are able to verify if the constraint is satisfied, while MRC methods cannot.\n\niii) Regarding ***experimental comparison to prior works***, in Table 1 the supervised baseline TEQUILA and EXAQT for TempQuestions and TimeQuestions  are KBQA methods, the baseline for TimeQA is a MRC model. All of them are strong baselines that perform answer extraction based on the question and we have recognized that there is still a gap between few-shot methods and supervised methods in L373-376.\n\nWe add clarification of differences between prior works and experimental comparison in the next version, thanks for the suggestion!\n\n[1] **Lost in the Middle: How Language Models Use Long Contexts**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n1.\tThis paper proposes a new way to solve the temporal question answering task. The key of new method is to reframe the QA task as the programming task.\n\n2.\tThis paper evaluates the QAaP framework on multiple time-sensitive QA datasets and achieves significant improvement. And It is great to see many discussions in the paper.", "reasons_to_reject": "Weaknesses:\n\n1.\tIn the introduction, authors mentioned “To tackle this aforementioned challenges, … ability in natural language and programming language …”. But question is also a natural language.  So why do we need to reframe the QA to programming. The description of the challenge is not clear. \n\n2.\tI am still not so sure if it is suitable to call it “reframe QA task as programming”. It is a step of  “question to query” and another step of information extraction based on LLMs. It is more related to reformulate text question to a structured query and then do extraction. There are many papers that use the similar methods. It is necessary to do the comparisons with them here on both the difference explanation and experimental comparison. The provided results in the experiments section are not enough to prove your contribution.", "questions_to_authors": "Please check the weaknesses and provide your answers.", "paper_topic_and_main_contributions": "This paper explores the inability of LLMs to handle rigorous reasoning task like temporal question answering. The key idea is to reframe the QA task as programming task. So the model doesn’t need get the answers directly using the input. Instead, this paper reformulates the QA as programming. In details, authors parse the questions that is same to translate the question to be a query. Meanwhile, this paper prompts LLMs to extract the information based on the query. Finally, the matching step will choose the best answer using a match score. This is similar to ranking problem. In general, the whole idea is simple and clear. The key is to reformulate the question as a well-defined query.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xKlfh7fdoR", "reviewer": "Reviewer_NXcU", "summary": "", "questions": "- Did you try parsing the question into any other structured form other than code to see how it performs compared to the proposed approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, insightful and constructive comments!\n\n### On method may not be useful with more powerful LLMs\n\nRegarding ***“it is possible that with more powerful LLMs, the improvement from the extra parsing step won't be as useful”***, the model we use is gpt-3.5-turbo, which is probably the most powerful LLM except for GPT-4, but our experiments show that it still performs poorly on time-sensitive QA task. GPT-4, the most powerful LLM, also makes simple mistakes as shown in a recent work [1], indicating there is still a long way to go before we can rely on LLMs alone to solve any QA task. We believe that the performance of our method can scale with the power of the backbone model, and this is verified by our additional experimental results below using text-davinci-003.\n\n### Generalization to other QA tasks\n\nIn response to Reason 1, we focus on solving questions with time constraints, however, there is few QA dataset that is specifically constructed to incorporate strict constraints in the questions. Nevertheless, our method can be easily extended to cope with other types of constraints, we just need to represent the constraint into an appropriate class in python (we use datetime.datetime() to represent time), and define the metric that measures how well is the constraint satisfied in the Match function. When there is no constraint, Check and Match steps are not necessary and our method degenerates into a method similar to CoT and ReAct. We’ll add these details in the revision, thanks for the pointer!\n\n### Performance on different LLMs\n\n| Method                   | TimeQA-Easy (EM/F1) | TimeQA-Hard (EM/F1) | TempQuestions (EM/F1) | TimeQuestions (EM/F1) |\n| :----------------------- | :------------------ | :------------------ | :-------------------- | :-------------------- |\n| CoT (text-davinci-003)   | 17.0/26.2           | 15.0/28.0           | 39.0/49.6             | 33.0/44.8             |\n| ReAct (text-davinci-003) | 30.0/42.7           | 26.0/37.4           | 33.0/42.2             | 22.0/31.6             |\n| QAaP (text-davinci-003)  | **38.0/50.1**           | **38.0/49.4**           | **47.5/52.5**             | **40.0/48.9**             |\n\nIn response to Reason 2, due to the high cost of OpenAI api, we only reported results of gpt-3.5-turbo in our previous version. To address the concern raised by Reviewer NXcU, here we did small-scale experiments with text-davinci-003 (random seed set to 0, sampling 100 questions each dataset) and the results confirm the effectiveness of our method. We’ll add the additional results as above in the paper.\n\n### Other types of structured form\n\nIn response to Question1, we did not parse the question into other structured form because LLMs are pre-trained on texts and codes, it is natural to apply them to transform the inputs from natural language to programming language. If the question/relevant information is parsed into other structured form other than code, it may be more complex and require more human efforts to verify its truthfulness and alleviate hallucination, which plays a key role in the final performance as shown in Table 4. We leave applying other types of structured form for future work.\n\n[1] **How Language Model Hallucinations Can Snowball**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors do a thorough analysis of the impact of the proposed approach on one kind of QA datasets (time-sensitive).\n- The approach is tested on multiple (3) datasets with time-constraints and outperforms state-of-the-art (ReAct and CoT).", "reasons_to_reject": "- It is not clear how the proposed method will generalize to other QA tasks with other constraints (or no constraints). \n- It would be helpful to study how the presented approach would perform on different LLMs.", "questions_to_authors": "- Did you try parsing the question into any other structured form other than code to see how it performs compared to the proposed approach?", "paper_topic_and_main_contributions": "The paper proposes a way of prompting LLMs to answer questions by first parsing questions and external knowledge into code (i.e. programming language) prior to prompting the LLM to match the parsed question to the external evidence/knowledge in order to answer the question. The authors compare the proposed method to two state-of-the-art prompting approaches—ReAct and CoT—on three datasets containing time-sensitive questions and beat both baselines. The paper is quite well-written, and easy to follow and the proposed approach shows a good improvement over ReAct and CoT. However, it is unclear how the findings generalize to other LLMs; it is possible that with more powerful LLMs, the improvement from the extra parsing step won't be as useful or that the approach does not generalize as well to other types of constraints (beside time constraints).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sLtiQ3ywcG", "reviewer": "Reviewer_LQzX", "summary": "", "questions": "1. Could you elaborate on not using implicit questions in TempQuestions and TimeQuestions datasets for experiments?\n2. How is the check function implemented? L461 mentions manual construction but no further description.\n3. The wording in sec 3.3 is confusing. In L251-253, the authors say that they check whether items extracted from internal knowledge have shown up in external knowledge. The external and internal knowledge sources are different. What action is taken after making this check and why is it necessary?  ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, insightful and constructive comments!\n\n### On the novelty compared to prior works\n\nIn response to Reason 1, we beg to differ, there are several differences between our work and the prior code-prompt works.\n\ni) Prior code-prompt works mainly utilized the coding ability of LLMs for solving mathematical and symbolic reasoning tasks, while we focus on solving time-sensitive factual questions with this coding ability, therefore the task is quite different.\n\nii) Prior code-prompt works only used LLMs’ own knowledge for problem solving, while we propose to apply LLMs’ coding ability to represent both LLMs’ internal knowledge and external knowledge (e.g., Wikipedia articles) as same-format codes, this enables us to easily enhance LLMs with different sources of knowledge, and also facilitates desired processing.\n\niii) Prior code-prompt works did not explore verifying the correctness of LLMs-generated codes, while we propose to incorporate Check and Match steps to mitigate LLMs’ hallucination and ensure accuracy, our experiments in Table 3 validate the importance of it. Our method requires very little human effort, and we also explore using LLMs for constructing those functions in Sec 4.3.4.\n\nWe will add more clarification of difference between prior works.\n\n### On not using implicit questions\n\nIn response to Reason 2 and Question 1, implicit questions require extra steps to uncover the implicit time indicated by the things/events, while we only used a simple Wikipedia search engine and there are many ambiguous things that have the same name (e.g., song, movie, person name, …), it is difficult for LLMs to determine which entry corresponds to the thing mentioned in the question. Therefore, stronger retrievers are needed to retrieve relevant documents. It is worth noting that our method is agnostic to the way of retrieving external documents and can be seamlessly combined with those off-the-shelf retrievers as mentioned in L333-337. We’ll add these clarification in the revision, thanks for the pointer!\n\n### Implementation of Check function\n\nIn response to Question 2, we described what check in L247-254. Specifically, the logic of the Check function looks like the following pseudo code, we will open source all the codes in the next version.\n\n```python\nimport datetime\nquery = ...\nanswer_key = \"object\"\n...\ndocuments_generated_by_llms = [context1]\ndocuments_from_wiki = [context2, context3, ...]\ninformation_from_llms = [item1]\ninformation_from_wiki = [item2, item3, ...]\n\n# Ensure the extracted items' time appear in the corresponding context, otherwise discard the item\nfor item, context in zip(information_from_llms, documents_generated_by_llms):\n\tCHECK item[\"time\"][\"start\"].year in context or item[\"time\"][\"end\"].year in context\nfor item, context in zip(information_from_wiki, documents_got_from_wiki):\n\tCHECK item[\"time\"][\"start\"].year in context or item[\"time\"][\"end\"].year in context\n\n# Ensure the items extracted from llms generated document appear in those extracted from wikipedia document, otherwise discard the item\nfor item in information_from_llms:\n\tCHECK item[answer_key] in [x[answer_key] for x in information_from_wiki]\n```\n\n### Check internal and external knowledge\n\nIn response to Question 3, internal knowledge refers to documents generated by LLMs, external knowledge refers to documents from Wikipedia, we discard the items extracted from internal knowledge if they do not appear in those extracted from external knowledge, for more details please refer to the code above to see how we implement it. This action is necessary because LLMs often generate erroneous facts, which is investigated in many prior works, one such example is also shown in Figure 3. Check step can mitigate LLMs hallucination and improve the EM score by up to 15.4 as demonstrated in Sec 4.3.2 and Table 3.\n\nWe will add more clarification of difference between prior works and dataset choosing, add more details about how we implement the Check and Match functions, open source our codes and fix typos and in the next version, thanks for the suggestion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors show consistent improvement on three datasets.\n2. It shows an important application of LLMs augmented with tools for temporal reasoning.\n", "reasons_to_reject": "1. The use of the coding ability of LLMs has been demonstrated in prior work [Gao et al. 2022b, Chen et al. 2022, Lyu et al. 2023] for other reasoning tasks, so the novelty is limited\n2. The authors don't test their approach on implicit questions in TempQuestions and TimeQuestions datasets. ", "questions_to_authors": "1. Could you elaborate on not using implicit questions in TempQuestions and TimeQuestions datasets for experiments?\n2. How is the check function implemented? L461 mentions manual construction but no further description.\n3. The wording in sec 3.3 is confusing. In L251-253, the authors say that they check whether items extracted from internal knowledge have shown up in external knowledge. The external and internal knowledge sources are different. What action is taken after making this check and why is it necessary?  ", "paper_topic_and_main_contributions": "This paper introduces a new approach for answering questions with time constraints using LLMs. They propose to reformulate QA as a programming task. The LLM is prompted to generate a python dict. corresponding to the question and also extract some information in a structured form. Further, they use 'Check' and 'Match' functions to choose the correct answer. Results on standard datasets demonstrate the utility of this approach. The ablation studies show the usefulness of different components.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The section headers casing is inconsistent"}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:** The paper proposes a new paradigm in question answering (specific to questions requiring temporal reasoning) by reformulating the QA task as a program and then utilizing the enhanced code-reasoning and understanding capabilities of LLM to solve for the solution. The question and external knowledge are first parsed into a code query, and then the LLM is prompted to match the parsed question to the external evidence/knowledge in order to answer the question. Empirical evaluation on 3 QA datasets with time-sensitive questions shows that the proposed approach beats SOTA prompting baselines (ReAct and CoT) by significant margins.

Overall, all reviewers acknowledged the clear writing of the paper, and the novelty and simplicity of the proposed solution. The claims in the paper are supported well empirically with a solid evaluation performed over 3 datasets and compared to two strong baselines. Most of the concerns raised in the initial reviews were sufficiently addressed by the author response. 

**Recommendations for Improvement:** My suggestion would be to include the additional experiments with other LLMs (text-davinci-003, and possibly more) and elaborating the reason for excluding implicit questions (of TempQuestions and TimeQuestions datasets) in the paper to strengthen it further.
Decision: Accept-Main