TempTabQA: Temporal Question Answering for Semi-Structured Tables
Authors: Vivek Gupta
Pranshu Kandoi
Mahek Bhavesh Vora
Shuo Zhang
Yujie He
Ridho Reinanda
Vivek Srikumar
Venue: EMNLP 2023 Main (2023)
Keywords: Semi-structured
Temporal Reasoning
Inference
Question Answering
New Resource
New Dataset
Wikipedia InfoBox
TL;DR: An novel dataset exploring resoning over temporal facts within entity-centric tables

Abstract: Semi-structured data, such as Infobox tables, often include temporal information about entities, either implicitly or explicitly. Can current
NLP systems reason about such information in semi-structured tables? To tackle this question, we introduce the task of temporal
question answering on semi-structured tables. We present a dataset, TEMPTABQA, which comprises 11,454 question-answer pairs extracted from 1,208 Wikipedia Infobox tables spanning more than 90 distinct domains. Using this dataset, we evaluate several state-of-the-art models for temporal reasoning. We observe that even the top-performing LLMs lag behind human performance by more than 13.5 F1 points. Given these results, our dataset has the potential to serve as a challenging benchmark to improve the temporal reasoning capabilities of NLP models.

TEMPTABQA:
Temporal Question Answering for Semi-Structured Tables
Vivek Gupta 18+ Pranshu K Mahek Bhavesh Vora"l, Shuo Zhang Yujie He', Ridho Reinanda', Vivek Srikumar' 'niiversity of Pennsylvania, 2IIT Guwahati, BBloomberg, 'University of Utah, gvivek @ cis.s..uuee cis.upenn.edu, {k. (k.pranshu, @iitg.ac.in, svivek @cs. @es.utah.edu, {szhang611, 1, yhe247, rreinanda) (acnaanda>bblooneeee @bloomberg..ne
Petya Nedelcheva
Abstract
Personal Information
Country
Bulgaria
Semi-structured data, such as Infobox tables, often include temporal information about en- tities, either implicitly or explicitly. Can cur- rent NLP systems reason about such informa- tion in semi-structured tables? To tackle this question, we introduce the task of temporal question answering on semi-structured tables. We present dataset, TEMPTABQA, which comprises I 11,454 question-answer pairs ex- tracted from 1,208 Wikipedia Infobox tables spanning more than 90 distinet domains. Us- ing this dataset, we evaluate several state-of- the-art models for temporal reasoning. We ob- serve that even the top- pop-perfrriing LLMs lag behind human performance by more than 13.5 F1 points. Given these results, our dataset has the potential to serve as challenging bench- mark improve the temporal reasoning capa- bilities of NLP models.
30, 1983(age
Stara Zagora, Bulgaria
Height
(5 in)
Weight
76 (168
Hondedness
Right
Coach
Orlin Tavetanov
Women's
singles & doubles
(WS August 2007)
Highest ranking
(WD2 2 December 2010)
Current ranking
Retized (In 2018)
Medal record
Women's badminton
Representing Bulgaria
European Chumpionships
Munchester
Women's doubles
2014 Kazan
Women's doubles
Europeun Women's Team Championships
2016 Kazan
Women's is team
2014 Basel
Women's 's team
European Games
2015 Baku
Women's singles
European Junior Champiouships
2001 Spala
Girl' singles
2001 Spala
Girls' doubles
Q1: How many years elapsed between Nedelcheva's 1st and
last silver medals? A1: 6 years
1 Introduction
Q2:
How many bronze medals has Nedelcheva's since
2013? A2: 3
Reasoning about temporal aspects of factual in- formation presents a fundamental challenge for
Q3: Which country did Nedelcheva's represent in the 2015 European Games? A3: Bulgaria
Q4: What was the ranking difference between Nedelcheva's top doubles and singles performances?
contemporary Natural Language Processing (NLP) systems. Factual information related to an entity often evolves over time, and understanding it re- quires understanding the scope of knowledge and temporal intervals. Furthermore, this factual infor- mation is also scattered across semi-structured data several different forms. These forms include both implicit and explicit representations (see lig-- ure for an example). The wide prevalence of these characteristics creates major challenges for NLP models. It requires these models to effectively han- dle changes over time and extract valuable insights from time-dependent data.
5
Figure I: A semi-structured table of women badminton play- ers (source: Wikipedia) along with accompanying temporal questions and their respective answers form TEMPTABQA.
et al., 2020; Chen et al., 2020b) concerning nu- merical aspects of semi-structured tables. They typically examine tables without time-related infor- mation and focus on queries with predominantly non-temporal contexts. Research on temporal as- pects in entity-centric tables, such as Wikipedia Infoboxes, has been limited (Gupta et al., 2020; Neeraja et al., 2021; Kumar et al., 2022). Morales et al. (2016) introduced question answering in entity-centric table context. However, the ques- tions in the dataset are simple and non-temporal. and it's worth noting that the dataset is not open source. Existing studies have only considered a few temporal aspects and addressed a small number of
Previous studies have primarily concentrated on question answering (Pasupat and Liang, 2015; Kr- ishnamurthy et al., 2017) and inference (Gupta
Work done during an internship at Bloomberg, FFqual contributions. &Corresponding authors.

---

time-related factors. Advances in modeling tech- niques. including table pre-training and targeted fine-tuning, have substantially improved reason- ing on semi-structured tables (Müller et al., 2021; Eisenschlos et al., 2020). Moreover, large language models (LLMs) have exhibited impressive perfor- mance across various domains, such as general, finance, and medical, demonstrating their mathe- matical, knowledge-basee and common-sense rea- soning capabilities (Chen al., 2021d; Aly al., 2021; Wang et al., 2021; Lu et al., 2023). However, the effectiveness these models in handling tem- poral aspects remains understudied. Consequently, this paper seeks to address the following research question: "Can modern NLP models effectively rea- son about temporal information in semi-structured tables?"
depth explanations. In contrast, our error analysis shows that models frequently make mistakes, par- ticularly when faced with complex temporal rea- soning questions. Consequently, our dataset serves as a challenging testhed for investigating effective temporal reasoning within semi-structured infor- mation.
Our paper marks significant milestone by pi- oneering the creation of complex temporal ques- tion answering datasets, specifically tailored entity-centric tables. Our primary objective was to introduce a novel challenge addressing intri- cate temporal questions within this context. The TEMPPAABAA dataset not only demands sophisti- cated reasoning but also necessitates firm grasp of temporal common sense, adept handling of arith- metic and numerical aspects. Our work sheds light on the unique temporal specificity of this dataset, setting it apart from existing models. Furthermore, we delves deep into this differentiation, offering a comprehensive array of statisties and analyses that illuminate the multitude of temporal reason- ing challenges posed by the dataset. The findings from above enhance our understanding of temporal reasoning tables and encourage further research. The TEMPTABQA dataset can be accessed at https:://enoor argg/records/10225500 For relevant analysis and modeling scripts refer https:// temptabqa.gi github.io.
To effectively address the above question, we introduce task called temporal question swering on entity-centric semi-structured tables. Figure shows an example. We curate a compre- hensive (covering diverse domains), specialized (temporally aligned), and TEMPTABQA. It consists of 1 11,454 question- answer pairs extracted from 1,208 Wikipedia In- fobox tables across more than 90 domains. Both the tables and questions in TEMPTABQA are compass numerous temporal terms. For example, the Figure table involve multiple dates, age, years, and the corresponding questions also incorporate temporal terms, such as first and last, years (since 2013), ranking, and more. These enhanced tables incorporate temporal information, ensuring that all queries have time-related components. This dataset is the first to explore question question-answericn answering and tem- poral reasoning in semiistructurrd data.
dataset,
Motivation
Dynamic Nature of Information. Tables serve as structured representations that organize and record diverse information types, making them highly useful for studying an entity's timeline. They offer a comprehensive record of events, en- abling clear visualization of the evolution of vari- ous aspects over time. By capturing a chronologi- cal sequence of events, tables allow us analyze the progression of positions, changes in marital status, and the acquísition of awards, serving liable sources for temporal reasoning. Additionally, entity-centrr tables, such as Wikipedia Infoboxes, significantly differ from both unstructured and fully structured data (SQL tables and KGs). These tables possess a semi-structured nature and store data in intricate implicit forms, as discussed in the context of cemi-structured tables (Gupta al., 2020).
We conduct analysis of temporal reasoning chal- lenges in TEMPTARQA, offering both qualita- tive and quantitative insights. Most questions in TEMPTABQA are abstractive and necessitate math- ematical calculations over temporal concepts to arrive at correct answers. The dataset also encom- passes additional test sets to evaluate reasoning in rare domains. Our findings indicate that tem- poral reasoning in TEMPTABQA poses greater challenges compared to non-temporal reasoning in previous tabular datasets. Our assessment of contemporary NLP systems on the TEMPTABQA benchmark exposes their subpar performance in comparison to humans. Humans excel at tempo- ral reasoning, delivering accurate answers and in-
Tables in the Real World. Complex temporal question answering applied to entity-centric semi- structured tables, like Wikipedia Infoboxes, has

---

broad applicability across various fields. In his- torical research and education, helps scholars, historians, and students extract precise historical details, while in financial analysis, empowers analysts with historical financial data for informed investment decisions. In medical research, it aids in accessing historical medical data and clinical trial timelines, and legal professionals use it to review historical legal records. Journalists gain historical context, linguists analyze language dynamies, and businesses optimize supply chains through histor- ical data. Environmental researchers, policy ana- lysts, travelers, software developers, and archivists all benefit from this versatile tool. This underscores the significance of discovering valuable informa- tion within tables across broad spectrum of di- verse fields.
2022), ScquentialQA((QQA) (lyyer et al., 2017), and WIKISQL (Zhong et al., 2017) for question- answering on table are limited terms of both quantity and complexity of temporal questions. Ta- ble 31 show broad comparison across several tab- ular datasets. They fail to cover crucial aspects of temporal reasoning, necessitating the creation of new manually curated dataset that specifically focuses answering temporal questions related to tables. Our dataset, TEMPTABQA, can serve help train and evaluate models, aiding the development of more accurate and robust systems capable effectively reasoning with temporal information in table-based contexts.
3 Our TEMPTARQA Dataset
We create a benchmark for answering temporal questions on entity-centric tables across domains.
Why Temporal Questions Answering? Tem- poral questions require reasoning based on time- related information, falling into two main cate- gories: explicit and implicit: (a.) Explicit tem- poral questions directly involve time-specific de- tails like dates, years, or hours, demanding precise knowledge. For example, in Figure 1, question such as 'When was the Nedelcheva's 's was born?' is an explicit temporal question. (b.) Implicit tem- poral questions rely on temporal terms and rela- tionships that indicate the temporal order con- text of events or entities. These questions may include terms like "rank," " "before," "after," " "pre- decessor," " "successor," and similar expressions. In such cases, the temporal dimension isn't explic- itly stated (e.g.. mention of year not in the table) but must be inferred (or extracted) from the given context with understanding of the relationships be- tween elements. For instance, in Figure 1 'How many Bronze medals did Nedelcheva's won before 2013?' assumes an implicit understanding of the temporal sequence. In essence, addressing tempo- ral questions involves comprehending and manip- ulating time-related information, whether explicit or implicit. This skill is vital in domains spanning historical research to natural language understand- ing, enabling effective reasoning about temporal aspects within data.
3.1 Data Creation
Table Domains Selection. TEMPTABQA is built with Infobox tables from various Wikipedia arti- cles across more than 90 categories. We focus on domains with time-related attributes in tables, particularly articles with Infoboxes containing tem- poral values. We analyze 1,208 Infobox templates! to compile varied domain list, prioritizing entity tables with numerous temporal values like dates and times. Our analysis indicates that longer tables from popular and highly viewed articles contain a higher amount of temporal information, includ- ing an increased presence of temporal terms. As a result, these longer tables are deemed more appro- priate for inclusion in TEMPTARQA."
Annotation Procedure. To generate question- answer pairs from selected tables, we engage Ama- zon Mechanical Turk crowd-workers. MTurk anno- tators draft both the temporal question and answer based on provided table. To ensure clarity, we instruct annotators to write clear, unambiguous, pronoun-free questions with grammatically com- plete answers. We also direct them to avoid yes/no questions and trivial questions that do not require temporal reasoning. This annotation approach en- sures that the questions are challenging enough to evaluate models. We advise annotators to avoid
Why new Table QA dataset? Current datasets such WIRITABLEQUISTTINS (Pasupat and Liang. 2015), SQUALII (Shi et al., 2020), FINQA (Chen et al., 2021d), TAT-QA (Zhu et al., 2021), Hy- BRIDQA (Chen et al., 2020c), FETAQA (Nan et al.,
ttttps://ee.wwkippee orw.a.rggg//aoee.oo wtg//wiki/iikiiediaaa List_ of inffoboxes "We extract tables using BeautifulSoupt and the Wikipedia extraction API.

---

repeating question patterns and instead differ- ent starting tokens like "What," "When." " "Whose," "How," "Where," "Who," " "How many," etc., in or- der to prevent biased evaluation results. We instruct them to incorporate multiple unique rows from the table when formulating questions, including logical aspects that require reasoning to enhance complex- ity and variation. This evaluation approach ensures that the model's ability reason across different parts of the table is assessed, and the questions are not overly basic. We encourage annotators ac- tively create unique questions, avoiding repetition and incorporating linguistic variation. This pro- cess fosters the generation of novel and interesting questions.
Metric
Statistics
Metric
Statistics
#average #questions
22
#unique tables
1208
11454
#question/table
9.5
Table I: TEMPTARQA dataset statistic.
Dataset
#categories
#tables
#QA
train
73
784
7680
dev
67
97
885
head-set
73
202
1851
tail-set
19
125
1038
Table 2: TEMPTABQA dataset splits statistic.
questions, taking into account multiple time-frame reasoning. the presence of single or multiple entities, and the inclusion of mathematical op- erations on temporal aspects. A question is deemed complex if involves at least two si- multaneous temporal reasoning steps from the categorics of before-related, after-related, and in- between/during-relaee related.. Further details regarding these analyses, including mathematical operations such as min, max, count, average, difference, and comparison, can be found in Table
To answer the questions, we asked Turkers to provide brief responses in the form of phrases rather than full sentences. Additional information regarding Turking cost, annotator statisties, bonus and reward criteria, batch splitting, and other de- tails are outlined in the uppendix $D.
Dealing with Annotation Bias. Annotators may rely on personal biases, leading to position, selee- tion, and popularity biases. They might also use repetitive question patterns and similar logical rea- soning. To address these biases, we implemented measures such as: (1) Diverse Table Categories: Including tables from various categories in a sin- gle batch for diversity, with 12 distinct domains and no more than 3 tables from the same domain. (2) Removal of Popular Rows: Excluding frequent keys in entity tables, such as "Year Active," "Born," "Died." etc. (3) Shuffling and Reordering: Address- ing position bias by shuffling table content and re- ordering subheadings like tournament titles, medal tallies, and awards. (4) Mitigating Selection Bias: Lessening selection bias by removing popular sub- sections, such as the "Olympics" section from an athlete table.
Question Type Pereent(%) Question Type Percent(%))
Simple Multiple Entity
57.81
Complex
42.19
47.90
Single Entity
52.10
Table 3: TEMPTABQA questions complexity.
We examine the required temporal intervals, in- cluding before, after, and present, as shown in Ta- ble 4. To categorize questions as current, we use keywords such as "until", "in", ",""" "during", "while", "at the same time", "meanwhile, " "when", "since", and soon. Past questions contained keywords such as "before", "previous to", "prior to", "preceding". and soon., and future questions contained keywords such "after". "following". "successor". "followed by", and soon.
Type
Percent(
Interval
Percent(55 (%)
3.2 TEMPTABQA Statisties and Analysis
fimplicit
63.23
Past
3.08
Dataset. Table presents key metrics, including average row count, total unique tables, total ques- tions, and average questions per table. We provide two test sets instead of one: the Head set with pop- ular frequent domains, and the Tail set with rarer domains. Data split for train, development, head, and tail test sets are shown Table 2.
#explicts
36.76
Future
8. 8.48
forrinal
18.63
Present
66.6 64
Table 4: TEMPTABQA question reasoning type.
In addition, Table 4 also distinguishes between explicit and implicit temporal questions. Explicit questions mention specific time or date, while implicit questions do not mention explicitly such temporal references. We also identified questions that used ordinal words or implied ranking or count- ing operations.
Questions. Table 3 describes the composition and complexity of questions in our dataset. presents the percentage of simple and complex

---

Dataset
Majority Agreement Human Accuracy
Table breaks down answer types by counting examples whose answers are for several entity types: money, person, organization, location, percentage, and product.
Answers.
Dev Head Test
91.56
86.36
86.17 93.62
Tail Test
89.73
86,47
Table 7: Data Validation Statisties, here both metric report the exact match.
Operation #QA|
Operation #QA
Operation #QA
Maximum
402
Sum
312
Count
3564
TeMPTABQA Filtering. We use pre-processing to refine our training set, removing non-temporal and basic questions. The test and development sets undergo manual reviews by NLP experts af- ter initial script-based filtering to maintain qual- ity, focusing complex temporal queries. We correct errors and prioritize questions that demand advanced reasoning, excluding those with direct an- swers or requiring external knowledge. Annotators were instructed to provide clear answers. However, some answers varied in format like "365 days" ver- sus "one year". We made sure evaluation didn't penalize format differences, applying regex rules validated by human checks. For more details filter refer to the appendix $D.
Minimum
377
Average
40
Difference
98
Compare
Table 5: TEMPTABQA questions mathematical operations.
Analysis
Entity
#QA
Entity
#QA
Money
97
Location
411
Entity Type
Person
843
Organization
384
Percentage
44
Product
57
Analysis
Type
#QA
Type
#QA
Count
2130
Ranking
Complexity
Boolean
45
Temporal
4823
Age
1085
Table 6: TEMPTARQA answer entity type and complexity.
Furthermore, we also evaluates answer com- plexity based on types such as count cardi- nal, ranking ordinal, boolean (Yes/No), temporal (Date/Time/Year), and age-related terms.
In the development set, less than 10% of ques- tions, under 7% the Head set, and under 11% the Tail set were ambiguous, as shown Table 7. Under 3% of questions were subjective. The most errors came from complex reasoning, whereas date- time errors were typically year off. Around 82% of the annotated questions reach clear majority consensus, demonstrating high agreement among annotators. For non-consensus questions, another review boosted agreement by 8-10%. By compar- ing the majority and gold answers, human accuracy was found to be 86%, as detailed in Table See appendix $D, table 30 for fine-grained agreement.
3.3 TEMPTABQA Dataset Validation
To ensure answer correctness in TEMPTABQA, we validate the development and two test sets by as- signing three annotators to answer the questions based the table. Annotators are instructed to provide concise yet comprehensive explanations to ensure accuracy and logical reasoning. The given instructions to annotators are as follows: (a.) Use Table Information Only: Annotators are instructed to rely solely on the table information, avoiding external knowledge except for common sense rea- soning. (b.) Clear, Concise, and Unambiguous Answers: Annotators are asked to clear, concise, complete, and unambiguous answers and expla- nations, ensuring accuracy and clarity in the val- idation process. (c.) Avoid Opinions or Assump- tions: To maintain objectivity and accuracy, an- notators are instructed to refrain from including personal opinions or assumptions in their explana- tions. (d.) Exclude Acronyms or Abbreviations: To ensure clarity and avoid confusion, annotators are instructed to avoid using acronyms or abbreviations in their explanations. (e.) Current Date and Year: During the annotation process, we instructed an- notators to consider December 2022 as the current month when answering questions that involve the present moment.
4 Experimental Evaluation
We address the following research questions through our experiments: (a.) Is the new dataset
TEMPTABQA challenging for existing models? (b.) Does finetuning on TEMPTABQA enhance model performance? (c.) Does providing few-shot examples and chain of thought reasoning benefit them? (d.) the performance on the tail domains worse than on the head domains?
We use the following metrics to eval- uate the models: Fl score (F1), Exact Match (EM), Rouge I (R1) and 2 (R2), and Meteor (MET). For evaluation purposes,
Evaluation:
treat December 2022 the current month and year. To ensure models are aware of this, in all experiments, we add new table row CCrrent Date: December, 2022'.

---

Models for Comparison. Since most of the ques- tions in TEMPTABQA require temporal and nu- merical reasoning to answer and are abstractive in nature, we mostly use decoder-only models (ex- cept for BART which are encoder-decoder mod- cls). We consider the following models: (a.) Fine- tuned model: BART-Large, T5-XL, and Flan-TS-- XL, along with smaller versions, all fine-tuned TEMPTABQA. (b.) Zero-shot LLM: T5-XXL,
Model
Size
EM
R1
R2
MET
Head Domain
L
35,51
33.93
35.73
35.67
23,97
T5
XI
35.51
33.93
35.73
35.67
27.07
XXL
38.08
36.77
38.08
38.05
25.86
L
33.81
32.04
33.91
33.87
22.43
Flan-T5
XL
41.80
40.72
41.83
41.8
27.17
XXL
43.29
41.87
43.41
43.40
27.78
LLaMA
2
47.90
40.73
48.36
48.28
33.62
3.5
53.38
49.03
53.64
53.56
39.09
GPT
69.97
65.17
70.24
70.22
50.33
Flan-T5-XXL, LLaMA-2, GPT-3.5 and 4, and PaLM along with smaller versions, without fine- tuning. (c.) Few-shot LLM: Same models as with zero-shot but in few shot settings with three refer- ence examples. (d.) Few-shot LLM with Chain of Thoughts: Similar to the few-shot setup, but with chain-of-thought (Wei et al., 2022) reasoning included with examples."
KG
72.24
68.02
72.98
72.86
52.10
PaIMM
2
69.05
66.82
69.00
68.91
42.32
Human
87.49
86. 17
87.61
87.61
58.87
Tail Domain
L
28.02
25.41
29.01
28.96
18.45
TS
XI.
31.39
28.72
32.39
32.32
20.45
XXL.
30.12
27.65
30.92
30.88
19.82
L
29.06
26.29
29.98
29.92
18.25
Flan-T5
XL
36.54
34.76
37.65
37.58
22.35
XXL
38.68
36.81
39.88
23.53
For additional details on the these models, in- cluding ayperparameter information, plcase refer to the appendix SC.
LLaMA
39.75
31.59
40.68
40.63
327.02
3.5
47.81
43.04
49.22
49.13
33.81
GPT
4
60.54
55.21
62.17
62.15
42.49
KG
4
62.80
57.80
64.18
64.16
43,99
PaLM
2
61.64
58.38
63. 14
63.07
37.23
4.1 Our Findings: Results and Analysis
Human
87.82
86.47
87.97
87.94
37.26
Table 8, 9, 10, show the zero-shot, fine tuned, few-shot w/o chain of thoughts and few-shot with chain of thoughts prompting models performance.
Table 8: Zero Shot Setting.
Flan Maan-T5-25 T5-XXL model, which is even larger in size, in various few-shot scenarios, including chain-of- thought prompting, by impressive margins of 13.79 and 17.18 F1 points. However, when compared the GPT models, particularly few-shot scenarios with chain-of-thoughh prompting, the fine-tuned models fall short by 18.56 and 11.97 on the Head and Tail sets respectively.
TEMPTABQA is Challenging. The dataset presents an challenging task, with all models per- forming significantly worse than the human, refer to Table 8, 9, 10, 11. Even our top-performing model, GPT-4 with Chain of Thought prompting, lags behind humans by 13.19 and 20.61 FI points on the Head and Tail sets, respectively. Addi- tionally, our best fine-tuned model, Flan-TS-XL, trails even further behind, with a margin of 31.75 and 32.58 FI points on the Head and Tail sets. "Furthermore, the GPT model consistently outper- forms other models, such Flan-T5 and T5, in both zero-shot and few-shot settings. Turning ta- bles into knowledge graphs (+KG)* results in the model's superior performance compared conven- tional linearization methods.
Model
Size
EM
R1
R2
MET
Head Domain
38.06
26.72
38.69
38.68
26.58
BART
45.68
34.56
46.35
46.32
29.08
42.37
35.40
42.90
42.84
28.26
49.48
39.03
50.19
50.12
34.07
TS
XI.
52.82
42.16
53.49
53.42
36.42
43.24
37.03
43.85
43.77
28.39
Flan-T5
47.86
39.35
48.49
48.41
29.36
XL
55.74
45.56
56.46
56.41
38.74
Human
87.49
86.17
87.61
87.61
58.87
Tail Domain
Fine-tuning Helps. Our findings, in Table highlight the significant advantages of fine-tuning medium-scale models. Remarkably, fine-tuned Flan- Min-TT-XLL XL models outperform the non-fine-tuned
35.62
24.44
36.74
36.68
24.31
BART
41.99
30.77
43.16
43.08
26.01
36.76
29.89
37.55
37.52
22.97
T5
L
44.45
35.15
45.95
45.75
28,89
XL
51.61
41.19
53.42
53.35
34.61
We didn't include TabT5 due proprietary industry strictions (not published in the open source dataset).
38.20
31.84
39.34
39.28
23.86
Flan-T5
41.83
32.91
43.19
42.99
23.23
"Due to resource limitations we fine-tuning models
XL
55.24
45.08
56.94
56.91
37.11
larger XL size.
Human
87.82
86.47
87.97
87.94
57.26
"Wo use GPT-4 with human in the loop to convert table to
Table 9: Instruction Fine-tune Models.
Knowledge Graph.

---

Few-shot (w CoT) few-shot (w/o CoT) > zero- shot). Tables 10 and 11 shows that seo-shot mod- els outperform their zero-shot counterparts. For instance, GPT-4 shows a gain of 2.0 and 2.23 Fl points on the Head and Tail sets, respectively, in the few-shot version compared to the zero-shot ver- sion. This trend is consistent across models like Flan-T5 and T5, regardless of model size. Notably, larger model sizes (L XL to XXL.) yield im- proved performance. Furthermore, incorporating chain-of-thougle prompting provides an additional boost to the model's performance. Furthermore, linearization outperforms knowledge graphs.
Head vs.Tail domain. Our observations reveal that the tail set posed greater challenges for all mod- els across various settings, while humans achieved similar performance on both sets. Models face greater challenges with tail tables in contrast to head tables. For instance, even the top-performing model, GPT-4, showed a difference of around 9.20 Fl points, performing better on the Head set in zero-shot scenarios. However, this performance gup diminished with few-shot learning and chain- of-thoough reasoning. In few-shot scenarios with chain-of- of-thought prompting, the gap reduced to 7.09 Fl points This phenomenon mainly results from knowledge transfer between less common and widely recognized sports tables. The head tables exhibit many common attributes and pose similar types of questions, in contrast to the rare tables
Model
Size
F1
EM
RI
R2
MET
Head Domain
L
35.79
34. 14
35.89
35.83
23.95
Flan-TS
XL
41.64
40.50
41.70
41.67
27.07
XXL
41.06
39.35
41.23
41.21
26.96
LLaMA
2
53.57
53.57
53.67
53.58
37.95
5
Analysis Breakdown of Performance
3,5
57.35
53.34
57.65
57.54
42.55
GPT
4
71.97
67.07
72.15
72.10
51.60
In our analysis, we examine the results (exact match) of our best model, GPT-4 few-shot with chain of thought, alongside human performance.
+KG
70.48
65.69
70.71
70.66
50.32
PaLM
68.84
66.40
68.96
68.90
42.78
Human
87.49
86.17
87.61
87.61
58.87
Tail Domain
Question Types. We categorize questions based on their types: starting with "what," "where," " "when," "how," or "quantity" (also known as "how many"). The evaluation of the GPT-4 model's per- formance (exact match) compared to humans is
L
29.70
26.78
30.52
30.45
18.88
Flan-T5
XI.
36.48
34.86
37.82
37.75
22.38
XXL
36.48
34.86
37.82
37.75
22.51
LLaMA
2
46.01
37.66
46.70
46.67
31.76
3.5
53.43
49.37
54.24
54.12
39.06
GPT
4
62.77
57.94
64.37
64.34
44.21
+KG
61.99
57.42
63.55
63.52
43.67
presented in Table 12.
PaLM
59.94
57.42
61.46
61.39
36.65
Human
87,82
86.47
87.97
87.94
57.26
Question Type
Head Set
Tail Set
Table 10: Few Shot w/o Chain of Thought Prompting.
#
Human GPT-4
Human GPT-4
what
470
87.23
70.64
326
86.20
64.11
Model
Size
F1 EM
RI
R2
MET
where
22
95.45
90.91
12
83.33
50.00
Head Domain
who
80
88.75
63.75
44
70.45
34.09
when
264
90.87
74.24
102
91.18
69.31
L
35,20
32.88
35.21
35.16
25.32
how many
588
80.27
70.75
378
81.48
67.46
Flan-T5
XL
38.31
35.46
38.52
38.51
25.84
how much
151
81.58
62.91
105
81.90
65.71
XXL
41.95
39.61
42.02
41.94
30.27
LLaMA
2
50.21
44.02
50.45
50.44
35.88
Table 12: Performance comparison Question Types
3.5
62.15
56.13
62,63
62.38
44.63
GPT
4
74.30
68.96
74.49
74.47
53.07
Analysis. Humans consistently outperform the model in all scenarios, with notable performance disparity in the tail domain. The model demon- strates relatively stronger performance in answer- ing "Where" and "How Much" questions compared to other types. However, it faces challenges in tack- ling "What," "Who," and "When" questions, result- ing in lower performance. We observe that humans handle "Where" questions with the least difficulty and struggle the most with "How Many" questions. Conversely, the model encounters significant chal- lenges with "Who" questions and performs rela- tively better with "Where" question types.
+KG
4
72.82
67.37
73.11
73.09
51.95
PaLM
2
68.41
64.07
68.55
68.47
44.38
Human
87.49
86.17
87.61
87.61
58.87
Tail Domain
L
31.22
28.43
31.47
31.34
22.13
Flan-T5
XL
33.12
29.79
34.17
34.14
21.50
XXL.
38.06
34,86
38.81
38.67
27.07
LLaMA
2
46.07
39.26
46.77
46.72
31.63
3.5
55.84
50.05
57.32
57.25
39.60
GPT
4
67.21
61.54
68.66
68.64
47.50
+KG
4
64.67
58.95
66,22
66.19
45.45
PaLM
2
61.56
55.87
62.94
62.81
39.32
Human
87.82
86.47
87.97
87.94
57.26
Table 11: Few Shot with Chain of Thought Prompting.

---

Reasoning Operation. To answer the questions, various analytical reasoning operations are in- volved, such as maximum, minimum, counting, summation, average, difference, and comparison. Table 13 provides a evaluation of the GPT-4 model's performance (exact match) compared to human performance, focusing on these operations.
are more prevalent, with a greater performance dif- ference between the two types observed in the tail set. Notably, humans also struggle more with ex- plicit questions compared to implicit ones, likely due to increased complexity and advanced mathe- matical reasoning requirements. Explicit questions demand deeper understanding and precise reason- ing, explicitly stating specific temporal information, while implicit questions rely more contextual reasoning and inference, allowwng the model leverage broader table information.
Reasoning
Head Set
Tail Set
Operation
#
Human GPT-4
#
Human GPT-4
Maximum
89
95.51
79.78
42
83.34
69.05
Minimum
102
87.25
67.65
86.84
71.05
Counting
603
80.41
73.47
375
82.93
70.41
Summation
44
70.45
52.27
68.42
42.11
Difference
16
62.51
43.75
11
72.72
54.54
Answer Types. We analyze the entity or common noun type of the answer. Answer categories include age (gap or sum), count, monetary terms, ordinal numbers, organization names, percentages, person names, place names, product specifies, temporal en- tities (date, time, day), Boolean (yes/no, true/false. comparison), or unknown (not any specific type). Table 15 presents the model's performance based on the type of answer entity.
Comparison
21
80.95
66.67
69.23
53.85
Table 13: Performance comparison w.r.t Operations.
Analysis. Once again, it is evident that humans consistently outperform the model in all types of operations, particularly in challenging tasks. Fur- thermore, our observations reveal that the model demonstrates relatively stronger performance in analytical reasoning tasks like "maximum" and "counting" compared to other types of tasks. How- ever, it faces significant challenges in tasks such as "minimum," "difference," and "comparison," re- sulting in lower performance levels. Overall, both humans and the model excel in "maximum" tasks while struggling with "difference" and "summa- tion" tasks. Additionally, the model's performance in "minimum" and "comparison" tasks falls short compared to human performance, indicating its limitations in these areas.
Entity Type
Head Set Human GPT-4
Tail Set
#
#
Human GPT-4
Boolcan
2
100
50.00
7
57.14
42.86
Temporal
736
83.7
69.93
411
81.11
64.23
Count
341
83.87
75.37
245
86.94
75.51
Age
133
83.46
56.72
85
91.67
60
Money
17
82.35
64.71
3
66.67
66.67
Percentage
8
62.5
37.5
6
16.67
33.33
Ordinal
6
66.67
50
1
0
0
Place
47
97.87
87.23
24
87.5
62.5
Person
76
89.47
65.79
43
69.77
32.56
Organization
69
89,86
82.61
43
95.35
60.47
Unknown
146
85.62
69.86
107
83.18
60.75
Explicit or Implicit. Our analysis compares the performance of humans and the best model in an- swering explicit and implicit time-related questions. Explicit questions directly mention time and can be found in the table, while implicit questions re- quire inferring the time from the table information. Table 15 showcases the model's performance on both question types.
Table 15: Performance comparison Entity Types.
Analysis. Our analysis reveals that the model struggles with calculating age gaps, boolean, place, and person-related questions, in contrast to count- related questions. Similar to previous findings, both the model and humans perform better on fre- quent head domain tables compared to tail domain tables. However, regardless table type, both hu- mans and the model encounter difficulties with per- centages and ordinals. The model's 's performance is notably weaker in age gap calculations, boolean, place, and person-related questions, while exhibit- ing better performance in count-related questions. Additionally, both humans and the model face chal- lenges with percentages and ordinals across table domains. For GPT-3.5 analysis, refer appendix SA Category-specific analysis based on table do- main is in appendix $B.
Answer Type
Head Set Human GPT-4
Tail Set Human GPT-4
explicit
565
85.31
68.5
296
81,65
57.43
implicit
1018
84.38
71.71
686
87.87
67.64
Table Performance comparison Answer Types.
Analysis. The model demonstrates better perfor- mance in implicit temporal reasoning compared to explicit temporal reasoning. As earlier model strug- gles more with rare and infrequent questions in the tail domain. Implicit temporal reasoning questions

---

6 Comparison with Related Work
Choi, 2021) and cloze-form (Dhingra al., 2022) question answering tasks, as well as event-centric datasets (Ning al., 2018; Wen et al., 2021; Chen et al., 2021a) that explore temporal QA.
Tabular Datasets and Models. Recent stud- ies have explored various NLP tasks on semi- structured tabular data, including tabular natural language inference, fact verification (Chen et al., 2020b; Gupta et al., 2020; Zhang and Balog, 2019), question answering, semantic parsing (Zhang and Balog, 2020; Zhang et al., 2020b; Pasupat and
In terms of modeling, there are temporally tuned language models trained on knowledge-basee ques- tion answering datasets such as CRONKBQA (Sax- ena et al., 2021), TEQUILA (Jia et al., 2018b), EXAQT (Jia al., 2021a), OTR-QA (Shang al., 2021). and TEMPOQR (Mavromatis al., 2021). among others. (Kannen et al., 2022) suggest a targeted approach to extract temporal facts when traditional KBQA methods fail to retrieve them from the knowledge base. Some methods incor- porate temporal aspects during masked language model pre-training (Dhingra et al., 2022; Iv et al., 2022), rather than fine-tuming on downstream NLI tasks. In comparison to prior work, TEMPTAB focuses on temporal question answering specifi- cally on entity-centric tables, while most existing studies address non non-tabular datasets.
Liang, 2015; Krishnamurthy al., 2017; Abbas et al., 2016; Sun et al., 2016; Chen et al., 2020c; Lin et al., 2020; Zayats et al., 2021; Oguz et al., 2020; Chen et al., 2021b; lyyer et al., 2017), and table-to- text generation (Parikh et al., 2020; Li et al., 2021; Nan et al., 2021; Yoran et al., 2021; Chen et al., 2020a).
Various strategies have been proposed to rep- resent Wikipedia relational tables, including Ta- ble2vec (Deng et al., 2019), TAPAS (Herzig et al., 2020), TaBERT (Yin et al., 2020), TabStruc (Zhang et al., 2020a), TABBIE (lida et al., 2021), TabGCN (Pramanick and Bhattacharya, 2021), and RCI (Glass et al., 2021). Pre-training methods have also been studied to improve tabular inference (Yu et al., 2018, 2021; Eisenschlos et al., 2020; Neeraja et al., 2021). Recent shared tasks like SemEval'21 Task 9 (Wang et al., 2021) and FEVEROUS'21 shared task (Aly et al., 2021) have further explored these areas.
Conclusion
In conclusion, this study addresses the effectiveness of current NLP systems in reasoning about tem- poral information in semi-structured data, specif- ically Infobox tables. We introduce the task temporal question answering on semi-structured tables and present the TEMPTABQA dataset, con- sisting of 11,454 question-answer pairs from 1,208 Wikipedia Infobox tables across varied domains. Evaluating state-of-the- -ortt--rrttttttt models on this dataset reveals significant gaps compared human perfor- mance, exceeding 13.5 F1 points. These findings emphasize the need for advancements in temporal reasoning capabilities of NLP models. The TEMPT- ABQA dataset serves as a challenging benchmark to enhance temporal reasoning in NLP models.
In comparision to prior work, TEMPTABQA cen- ters on temporal question answering within entity- centric tables, an untapped domain. While most datasets lean towards non-temporal queries, they seldom address temporal aspects and lack ground- ing in the common sense and the necessary world knowledge. These datasets predominantly empha- size arithmetic reasoning using SQL in structured formats, overlooking the nuanced ceni-structured Infobox-style tables rich in common sense.
Future Directions. From our analysis, we sug- gest future avenues in temporal query answering: (a) Diverse Structures: Expand temporal queries to various table structures, like hybrid composi- tions (text, table, image). (b) Dynamic Queries: Examine evolving tables across consistent time-
Temporal Datasets and Models. Several tem- poral question answering datasets have been intro- duced recently. These include -SNE-VEEST QA (Chen et al., 2021c) and TORQUE (Ning et al., 2020), which are entity-specific reading compre- hension datasets with time-sensitive questions de- rived from Wikipedia paragraphs. TEMPQA-Wi (Neelam al., 2022), CRONQUESTIONS (Saxena al., 2021), and TEMPQUESTIONS (Jia et al.. 20) are question answering datasets focusing on knowledge graph embeddings with temporal links. Additionally, there are open-domain (Zhang and
line. (c) Open Domain Queries: Merge retrieval, extraction, understanding, and temporal reasoning into one framework. (d) Reasoning with LLMs: Tailor large language models (LLMs) for table- specific temporal logic, with more on this in Ap- pendix H. Advanced prompts remain a potential area of exploration.

---

Limitations
so thorough examination is crucial for uses beyond the intended research scope.
First, it focuses solely on entity-centric tables from Wikipedia, excluding non-Infobox tables and other relevant sources, Exploring a broader range of table types would be valuable. Second, despite our ef- forts to ensure unbiased table selection and dataset annotation, inadvertent bias leakage is possible, potentially affecting results.
To ensure fairness, we provided fair wages to MTurk crowd workers and conducted three pilot studies to estimate task completion time accurately, We also plan to release a datashect, full annota- tion template, and other resources for data and model openness. Emphasizing openness enables issue identification and correction, allowing contin- uous improvement based on community feedback. These measures promote transparency and facili- tate further advancements in the field.
Third, due to limited computational capacity, we could only fine-tune models using large sizes (XL), not extra-large (XXL). One idea that could be explore here is using Parameter Efficient Fine tuning (PEFT) (Mangrulkar et al., 2022). Incorpo- rating more open-source Language Models (LLMs) would enhance our understanding of temporal rea- soning capabilities. Lastly, our work primarily tar- gets the English language, while exploring multi- lingual settings would increase applicability and generalizability. These limitations present oppor- tunities for future research and expansion in this field.
References
Faheem Abbas. M. Malik, M. Rashid, and Rizwan 7afar. 2016. Wikiqa - a question answering system wikipedia using freebase, dbpedia and infobox. 2016 Sixth International Conference on Innovative Computing Technology (INTECH), pages 185-193.
Rami Aly. Zhijiang Guo, Michael Sejr Schlichtkrull, James Thome, Andreas Vlachos, Christos Christodoulopoulls, Oana Cocarascu, and Arpit Mittal. 2021. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop Fact Extraction and VERifica- rion (FEVER), pages 1-13, Dominican Republic. Association for Computational Linguistics.
Aknowledgement
The authors express their gratitude Bloomberg's AT Engineering team, particularly Edgar Meij and Prabhanjan Kambadur, for their invaluable feed- back and guidance. We are also thankful for the valuable insights provided by Ellen Riloff, Dan Roth, and the Utah NLP group. Special thanks to Dibyakanti Kumar and Manasvi Kundalia for their contributions. Vivek Gupta acknowledges the support received from Bloomberg's 's Data Science Ph.D. Fellowship and the Cognitive Computation Group at the University of Pennsylvania. This work is partially supported by NSF grants #1801446, #1822877, #2007398 and #2129111. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies of any government agency. Lastly, we extend our appreciation to the reviewing team for their insightful comments.
Muhao Chen, Hongming Zhang, Qiang Ning, Man- ling Li, Heng Ji. Kathleen McKeown, and Dan Roth. 2021a. Event-centric natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics.
Wenhu Chen, Ming-Woi Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021b. Open question answering over tables and text. In In- ternational Conference on Learning Representations.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural lan- guage generation from open-domain tables. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942, Online. Association for Computational Lin- guistics.
Ethics Statement
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for based fact verification. In Interna- tional Conference Learning Representations.
The dataset in this study is designed for research temporal question answering with entity-centric tables. It should be strictly used for research pur- poses, not for other applications. We have dili- gently created the dataset to minimize bias dur- ing table selection and question-ansee answer generation. However, inadvertent bias leakage is still possible,
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021c. A dataset for answering time-sensitive ques- tions. In Thirty-ffthh Conference on Neural Informa- tion Processing Systems Datasets and Benchmurks Truck (Round 2).

---

Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong. Hong Wang. and William Yang Wang. 2020c. Hy- bridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 1026-1036, Online. Association for Computa- tional Linguistics.
Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023. A survey of knowledge enhanced pre-trained language models. IEFE Trans- actions on Knowledge and Data Engineering.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit
Tyyer. 2021. TABBIE: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3446-3456 Online. Association for Computational Linguistics.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting- Hao Huang, Bryan Routledge, and William Yang Wang. 2021d. FinQA: A dataset of mu- merical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Robert Iv, Alexandre Passos, Sameer Singh, and Ming- Wei Chang. 2022, FRUIT: Faithfully reflecting up- dated information in text. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3670-3686, Seattle, United States. Association for Computational Lin- guistics.
Li Deng, Shuo Zhang, and Krisztian Balog. 2019. Ta- ble2vec: Neural word and entity embeddings for ta- ble population and retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR 19. pages 1029-1032. New York, NY, USA. ACM.
Mohit lyyer, Wen-tau Yih, and Ming- Wei Chang. 2017. Scarch-based neural structured learning for sequen- tial question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1821- 1831, Vancouver, Canada. Association for Computa- tional Linguisties.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257- 273.
7hen Jia, Ahdalghani Abujahal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018a. Tem- pquestions: A benchmark for temporal question an- swering. In Companion Proceedings of the The Web Conference 2018 WWW '18, page 1057-1062 Re- public and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281-296, Online. Association for Computational Lin- guistics.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b. TEQUILA: Temporal Question Answering over Knowledge Bases. In Proceedings of the 27th ACM International Conference on Information and Knowl- edge Management, CIKM '18. pages 1807-1810, New York, NY, USA. ACM.
Michael Glass, Mustafa Canim, Alfio Gliozzo, Sa- neem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj. and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1212-1224, Online. Association for Computational Linguistics.
Zhen Jía, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021a. Complex temporal ques- tion answering on knowledge graphs. In Proceedings of the 30th ACM International Conference on Infor- mation & Knowledge Management, pages 792-802.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as remi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics.
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021b. Complex Temporal Ques- tion Answering on Knowledge Graphs, page 792-802. Association for Computing Machinery, New York. NY, USA.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320-4333 Online, Association for Computa- tional Linguistics.
Nithish Kannen, Udit Sharma, Sumit Neelam. Dinesh Khandelwal, Shajith Ikbal, Hima Karanam, and L Venkata Subramaniam. 2022. Targeted extraction of temporal facts from textual resourees for improved temporal question answering over knowledge bases. arXiv preprint arXiv:2203.1 11054.

---

Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gard- ner. 2017. Neural semantic parsing with type con- straints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 1516-1526, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Hailey Schoelkopf. Riley Kong, Xiangru Tang. Mutethia Mutuma, Ben Rosand, Isabel Trindade. Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022. AeT: Free-form table question answering. Trans- actions of the Association for Computational Linguis- rics, 10:35-49.
Dibyakanti Kumar, Vivek Gupta, Soumya Sharma, and Shuo Zhang. 2022. Realistic data augmentation framework for enhancing tabular reasoning. In Find- ings the Association for Computational Linguistics: EMNLP 2022, pages 4411-4429, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xi- angru Tang, Aadit Vyas, Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chem Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open- domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 432 452-447. Online. Association for Computa- tional Linguistics.
Tongliang Li, Lei Fang, Jian-Guang Lou, and Zhoujun Li. 2021. TWT: Table with written text for controlled data-to-text generation. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pages 1244-1254, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Xi Victoria Lin, Richard Sacher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross- domain text-to-SQU SQL semantic parsing. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870 4888,4888, Online. Association for Computational Linguistics.
Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nan- dana Mikindukulasooriya, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, et al. 2022. A benchmark for generalizable and interpretable temporal ques- tion answering over knowledge bases. arXiv preprimt arXiv:2201.05799
Pan Lu, Liang Qiu, Kai-Wei Chang. Ying Nian Wu, Song-Chun Chun Zhu, Tanmay Rajpurohit, Peter Clark. and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations.
J. Neeraja, Vivek Gupta, and Vivek Srikumar. 2021. Incorporating externl knowledge enhance tabular reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2799-2800 Online. Association for Computational Linguistics.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-thee the parameter-efficien fine- tuning methods. https: thub, com/ co//huggiggface/ peft.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering ques- tions. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (FMNLP), pages 158--1122 Online. Association for Computational Linguistics.
Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Ioannidis, Soji Adeshina, Phillip R. Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2021. Tempoqr: Temporal question reason- ing over knowledge graphs.
Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018 CogCompTime: A tool for under- standing time in natural language. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 72-77, Brussels, Belgium. Association for Computational Linguistics.
Alvaro Morales, Varot Premtoon, Cordelia Avery, Sue Felshin, and Boris Katz. 2016. Learning to answer questions from Wikipedia infoboxes. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language P Processing, pages 1930-1935, Austin, Texas. Association for Computational Linguisties.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020. Unified open-domain question answering with struc- tured and unstructured knowledge. arXiv preprint arXiv: 14610.
Thomas Müller, Julian Eisenschlos, and Syrine Krich- ene. 2021. TAPAS at 22022al-2021 task 9: Reason- ing over tables with intermediate pre-training. In Pro- ceedings of the 15th International Workshop on Se- mantic Evaluation (SemEval-20211) pages 423-430, Online. Association for Computational Linguistics.
Ankur Parikh, Xuezhi Wang. Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang. and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryšcinskii,

---

generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLIP), pages 1173-1 186, Online. As- sociation for Computational Linguistics.
Haoyang Wen, Yanru Qu, Heng Ji. Qiang Ning, Ji- awei Han, Avi Sil, Hanghang Tong, and Dan Roth. 2021. Event time extraction and propagation via graph attention networks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 62-73, Online. Asso- ciation for Computational Linguistics.
Panupong Pasupat and Percy Liang. 2015, Composi- tional semantic parsing on semi struturret tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1470- 1480, Beijing, China. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data, In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413 8426, On- line. Association for Computational Linguistics.
Aniket Pramanick and Indrajit Bhattacharya. 2021. Joint learning of representations for web-tables, en- tities and types using graph convolutional network. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1197-1206, Online. Association for Computational Linguistics.
Ori Yoran, Alon Talmor, and Jonathan Berant, 2021. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. arXiv preprini arXiv:2107.07266 Version 1.
Apoorv Saxena, Soumen Chakrabarti, and Partha Taluk- dar. 2021. Question answering over temporal knowl- edge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 6663-6676 6676. Online. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang. Yi Chern Tan, Xinyi Yang. Dragomir Radev, richard socher, and Caiming Xiong. 2021. Gra(pp)a Grammar-augmente aarsinaar-augmeetted pre-training for table semantic parsing. In International Conference on Learning Representations.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma. Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev, 2018. Spider: A large-scale haman-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference Empirical Methods in Natural Lan- guage Processing, pages 3911-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Chao Shang, Peng Qi, Guangtao Wang, Jing Huang, Youzheng Wu, and Bowen Zhou. 2021. Open tem- poral relation extraction for question answering. In 3rd Conference on Automated Knowledge Base Con- struction.
Tianze Shi, Chen Zhao, Jordan
Hal Daumé III, and Lillian Lee. 2020. On the poten- tial of lexico-logical alignments for semantic pars- ing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1849-1864, Online. Association for Computational Linguistics.
Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021. Representations for question answering from documents with tables and text. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2895-2906, Online, Association for Computa- tional Linguistics.
Huan Sun. Hao Ma. Xiaodong He, Scott Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of the compan- ion publication of the 25th international conference on World Wide Web. ACM Association for Comput- ing Machinery.
Hongzhi Zhang, Yingyao Wang. Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020a. Table fact verification with structure-aware trans- former. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1624-1629, Online. Association for Computational Linguistics.
Nancy X. R. Wang. Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021. SemEval- 2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB- FACTS). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 317-326, Online. Association for Computa- tional Linguistics.
Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating lina--inguistic contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Newral Information Processing Systems.
Shuo Zhang and Krisztian Balog. 2019. Auto- completion for data cells in relational tables. In Pro- ceedings of the 28th ACM International Conference

---

on Information and Knowledge Management, CIKM '19, pages 761-770, New York, NY, USA. ACM.
Type
F1
EM
R2 MET
Head
what
470
59.34
57.87
59.43
59.37
40.92
Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol., 112))13:1-13:35.
where
22
88.41
86.36
89.81
89.81
65.5
who
80
61.59
58.75
61.68
61.68
55.28
when
263
69.51
68.06
69.42
69.39
49.79
quantity
588
58.88
56.97
58.91
58.91
45.22
Shuo Zhang, Zhuyun Dai, Krisztian Balog, and Jamie Callan. 2020b. Summarizing and exploring tabular data in conversational search. In Proceedings of the 43rd International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR '20, pages 1537-1540, New York, NY, USA. Association for Computing Machinery.
how
152
50.02
48.03
50.36
50.36
46.65
Tail
what
326
55.67
54.29
56.54
56.42
38.18
where
12
55.26
50.0
55.25
55.25
47.56
who
44
36.31
31.82
36.39
36.39
35.96
when
101
66.12
64.36
66.11
66.11
50.25
quantity
378
57.29
55.29
57.4
57.33
39.76
how
105
56.87
55.24
56.99
56.99
47.83
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451 Florence, Italy. Association for Compu- tational Linguistics.
Table 16: Performance w.r.t Question Type with GPT-3.5 few shots (with chain of thought prompting).
Op.
F1
EM
R1
R2
MET
Head
max
68.74
67.42
68.69
68.69
48.89
min
55.78
53.92
56.19
56.15
47.53
Victor Zhong, Caiming Xiong, and R. Socher. 2017. Seq2sql: Generating structured queries from natu- ral language using reinforcement learning. ArXiv, 0bbs/109.00003.
count
61.11
59.3
61.1
61.1
45.64
sum
36.02
34.09
36.22
36.22
25.59
avg.
100.0
100.0
100.0
100.0
93.75
dif.
45.44
43.75
45.62
45.62
41.15
com.
48.48
47.62
48.64
48.64
35.19
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Seng Chua. 2021. TAT-QA: A question answering benchmark
Tail
max
66.18
64.29
66.81
66.81
47.93
min
52.32
50.0
52.52
52.52
38.45
tent in finance. hybrid of tabular and textual con- Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277-3287, Online. Association for Computational Linguistics.
count
59.74
57.87
59.79
59.72
40.61
sum
43.48
39.47
43.71
43.52
32.36
avg. 34.33
25.0
34.58
33.33
39.66
dif. 56.8
54.55
57.4
57.4
51.08
com. 54.99
53.85
54.83
54.83
35.46
Table 17: Performance w.r. Reasoning Operation with GPT- 3.5 few shots (with chain of thought prompting).
A Analysis Breakdown: GPT-3.5 model
Domain
#
EM Domain
EM
GPT-3,5 Model: Table 16, 17, 18, 19, 20, 21 represent analysis of GPT-3.5 on various aspects such as question type, reasoning operation, explicit and implicit, entity type and category wise (tail and head), respectively.
person
66 59.09
sports
717
56.49
cricket team
18 61.11
history
27
33.33
aircraft
24 66.67
fighter
33.33
finance
31 41.94
court
21
71.43
musician
42 59.52
art
30
83.33
nobel
39 56.41
country
44.44
space
51 49.02
railway
15
80.0
company
21 52.38
website
33.33
B Analysis Breakdown: GPT-4 vs Human
university
18 72.22
monument
77.78
5 40.0
book
86.67
church
15 60.0
leaders
66.67
GPT-4 vs. Human: Table 22, 23, 24, 25 rep- resent analysis of GPT-4 and Human on various aspects such as question type, reasoning operation, explicit and implicit, entity type.
office holders
27 48.15
music
77.27
war
34 38
conflicts
14.29
concert
27 62.96
disaster
55.56
song
18 77.78
movie
66.67
rail line
5 60.0
character
70.37
We also consider what the performance of model comapred to Human examples on particular table/article domain. We consider coarse grained categories for comparison. Table 26, 27 and 28, 29 shows the model head and tail set performance based on table domains for coarse and fine-gained setting.
ships
15 53.33
agency
75.0
board game
48
NET
66.67
NCT
18
72.22
Table 18: Head Set coarse-gained Sategory-Wiss Results with GPT-3.5 few-shot (with Chain of Thought).

---

Type
F1
EM
R2
MET
Type
Tail Domain
Human GPT-4 Hummm GPT-4
Human GPT-4
Human GPT-4
Head
R1
MET
exp.
59.44
56.99
0.6
59.57
48.24
Head Set
imp.
61.08
59.72
0.61
61.1
44.16
85.43
72.5
where
95.95
92.12
95.86
93:42
95.86 93,42
67.45
69.17
Tail
who
91.74
67.47
91.61 67.31
91.61 67.31
78.64
60.99
exp.
52.50
50.0
0.53
52.81
38.64
when
91.56
76.03
91.56 75.97
91.37 75.97
59.84 53.06
imp.
58.51
56.85
0.59
58.76
42.05
quantity
how
82.24
86.93
64.84
72.71
87.45 65.06
81.6 72.7
87.45 65.01
81.6 72.66
49.05 53.58
74.9% 59.31
Taii Set
Table 19: Performance w.rt Implici//xpliict Type with GPT- 3.5 few shots (with chain thought prompting).
what where
88.17
65,91
88.21
67.02
88.14 66.82
55.98 45.32
83.33
55.28
83.33 55.45
83.33 55.45
56.62 47.0
who
75.66
40.08
75.66 40.45
75.66 40.23
67.12 40.22
71.28
93.12
95.12 71.25
65.38 55.12
quantity how
84.39
69.43
84.51 69.55
84.51 69.51
48. 47.19
Type
#
F1
EM
R1
R2
MET
85.41
67.46
85.41 67.78
85.41
67.78
73.16 59.07
Head
Table 22: Comparison between Human and GPT-4 w.r.t.
Temporal
736
65.44
63.32
65.49
65.47
55.4
question type
Count
341
55.91
53.13
55.93
55.93
30.0
Age
133
45.13
43.61
45.09
45.05
24.25
Op. Human GPT-4 Human GPT-4 Human GPTT4
Human GPT-4 MET
Money
17
55.38
52.94
56.25
56.25
54.36
R2
Percentage
8
6.13
0.0
5.36
5.36
6.95
Head Set
Ordinal
6
33.33
33.33
33.33
33.33
20.14
max min
61.62
88.24
70.74
88.24 70.87
88.24
70.66
68.4 59.58
Place
47
69.68
68.09
70.28
70.28
45.74
Person
76
64.37
61.84
64.52
64.52
57.88
sum
78.08
54,33
75,81 54,32
75.81 54,18
43.1 36.88
Organization
69
63.13
60.87
62.28
62.08
52.49
avg.
100.0
100.0
100.0 100.0
100.0
100.0
93.75 93.75
Product
2
100.0
100.0
100.0
100.0
71.88
71.25
46.63
71.25 46.41
71.25 46.41
54,87 39.42
com.
80.95
70.18
82,14 71.24
82.14
71.24
54.37
48.13
Unknown
146
58.78
56.85
59.05
59.0
44.66
Boolean
50.0 2
50.0
50.0
50.0
25.0
Tail Set
max
86.07
70.83
86.96 71,23
Tail
min
91.73
74.51
92.78 74.63
92.78 74.63
62.17
53.76
85:47
72.03
85,59 72.12
85.59 72.07
47.58
48.36
Temporal
412
58.38
55.34
58.56
58.47
52.04
sum
81.35
46.31
$3.47 46.87
83.47 46.68
44.0 33.66
Count
245
58.35
57.96
58.45
58.37
30.32
50.0 58.25
50.0 61.23
50.0 59.31
35.94
60.88
Age
84
61.37
60.71
61.54
61.47
31.92
87.58
Money
3
69.84
66.67
72.46
72.46
65.83
com.
76.92 56.23
76.92 55.98
76.92 55.98
43.75 37.23
Percentage
6
35.0
33.33
37.19
37.19
20.61
Table 23: Comparison between Human and GPT-4 w.r.t.
Ordinal
0.0
0.0
0.0
0.0
0.0
reasoning operations.
Place
24
64.04
62.5
63.89
63.89
41.34
Person
43
34.82
30.23
34.91
34.91
34.62
Type Human GPT-4 Human GPT-4 Homan GPT-4 Human GPT-4
Organization
43
44.85
49.19
47.85
47.85
28.17
RI
MET
Product
7
85.71
85.71
85.71
85.71
42.86
Head Set 86.8 71.4
Unknown
107
54.49
53.27
55,24
55.17
39.14
explicil implicit
87.47 71.37
86.71 71.35
61.25 55.81
Boolean
7
51.03
42.86
51.37
50.27
32.66
86.12 73.16
86.16 73.23
86.1 73.19
56.16 52.64
'Fail
Table 20: Performance w.r.t Entity Type with GPT-3.5 few shots (with chain thought prompting).
explicit implicat
81.65 60.3
82.09
61.02
82.0
60.85
53.43 44.61
87,87
49.68
Table 24: Comparison between Human and GPT-4 w.r.t. type of temporal question
Domain
#
EM
Domain
EM
sports
500
61.2
party
72 52.78
we introduced a warm-up period of 500 steps and applied weight decay at a rate of 0.01 during the optimization phase. We implemented logging and evaluation at every 100-step interval. The learning rate was designated at 2e-5, and gradient accu- mulation process of 8 steps was used to optimize memory resources.
time zone
53.33
holiday
40.62
ships
75
57.33
aircraft
50.0
organization
66.67
disaster
36 41.67
W
war
63
49.21
army
75 42.67
planet
50.0
diseases
32
50.0
Table 21: Tail set course-gained category-Wiss Wise Results with GPT-3.5 few-shot (with Chain of Thought).
Our study further expanded to encompass zero- shot and few-shot experiments. This included, but was not limited to, mhain-of-thougee prompting on cutting-edge models like 3.5 and GPT-4. We delved into various variants of Flan-T5 and T5 mod- els, such large L, XL, and XXL. A thorough anal- ysis was undertaken to compare the performance of these models against human performance bench- marks.
C Models and Hyperparameters Details
In our research, we embarked series of training experiments utilizing several models such as BART, Flan-T5, and T5, each with base, large, and XL variants. Training was conducted over 1-3 epochs, incorporating input sequence lengths from 1024 to 4096 tokens. To foster efficient convergence,

---

Type
Human GPT-4 Human GPT-4 Human GPT-4 F1
Human GPT-4
Type
Head Domsann GPT-
R1
R2
MET
Human
CPT-
Human
Human
GPT-4
Human
GPT-
MET
Head Set
Count
Temporal
Boolaan
K3.87 76.17
100. 50.0
86.74 72.4
56.98 72.46
100.0 50.0
K2.7 76.04
86.91
100.0 50..0
82.7 76.02
72.43
61.16 60.22
40.33 39.27
25.0 31.1
agancy
sater
album
89.58 76.61
73,96 77.75
100.0 $3.85
90:48
73.06
$5.42 76.33
100.0 51.85
90.48 R1:37
73.95
35.02 76.12
100.0 53.85
77.04
71.30
55.65 53.9
55.3 58.05
77.07
Age
84.91 58.21
$4.91 58.21
84.91 58.17
30.99 30.97
aehelese
81:11 76:21
16.11 75.07
16 75.07
16
Money
65.29 66.9
85.29 66.66
66.66
hulminket
67.37
58.50 67.76
88.80 67.76
50.28
Pereentage
62.5 45.19
62.5 45.72
62.5 45.72
36.42 21.33
baschall
90.74 76.42
90.74 76
90.24 76.76
61.88 60 2
Ordinal
66.67 50.0
66.67 50.0
66.67
50.0
28.33 0.0
haskeffell
91 08.98 80.11
91.9R 50
96.07
Place
97.87 87.8
97.57 88.41
97 88.41
60.45 40.67
hranl game
06.3 07.62
96.1
16.1 07.49
66.1 17 72.07
Person
92.62 68.9
68.81
68.81
62.32 38.98
hudy buikder
70.63
58.75 70.78
88.75 70.5
64.75
Organization
91.79
91.79 84.23
20.5 29.68
back
97.07
53.33
83.33 94 04.0
$6.21
Product
100.0 100.0
100.0 100.0
100.0
100.0
71.88 57.09
chaeacter
67.92
88.89
92.59 88.89
76.19
09
92.59 58 99.99
76 19 67
47.68 96.06
61.08 60.85
Unknown
86.51
12.16
85.82
72.62
85.82
72.43
55.2 46,58
christius lender
61.59
55.93 61.63
85.93 61.49
63.95 11.06
Set
chmnh
85.91 71.9
85.03
71.71
85.93 71:31
64 87
Count
Temporal
Boolean
57.14
85.52 67.43
87.1 75.92
S7.47 76.04
$5.61 67.63
54.62
53.52 85.55
87.47 75.97
67.52
28.57 31.1
44.2 39.27
63.4 60.22
company
coutry
concert
war
78.2 74
79.05
77.51
99.42
92.38 78.75
87.60
59,50 99.38
$1 78,6
06
98 93.89 50-38
87.69
91.48 78.64
92:38 78-75
78
06.28
51.71
06.56
$4.01 75.45
Age
91.96 60.57
91.93 60.75
91,93 60.67
45.95 30,97
coun
86
$7.62 86.
87.62 96-17
66.55 68 70.67
Money
03.33
cidket team
84.81
76.67 84.81
7667 54.81
50.18 72.5
Percestage
23.33 35.71
22.22 37.25
22.22 37,25
8.33 21,33
curling
87.5
N7.5 64.92
57.5 64.92
49.46 45.44
Ordinal
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
curent war
85.71 52.98
85.71 $3.83
85.71 52.88
$7.44
Place
87.5 63.75
K7.5 63.64
87.5 63.64
53.05 40.67
cattlqukke
85.48
52.72 84.81
72.4
54.91
$7.96 65.34
Person
75.09 38.69
75.09 39.07
75.09 38.84
65.4 38.98
coomony
54
17.42 53.97
77.42 53.97
52.96 40.35
Organization
95.35
95.35
61.21 39.68
empente
50.48
22.06 51.00
82.96 51.09
58.11 15 16
Unknown
Product
86.38 62.89
100.0 100.0
85:48 63.6
100.0 100.0
85:48 63:18
100.0 100.0
61.06 46.58
50.0 57.09
empire
event
54.06
02
62.35
44.44 53.29
103.0 02.22
95.0
44.44 53.29
100.0 62.22
95.0 17:95
24.85 32.35
77.42 49
69.3
figure slaxing
64
$7.76 64.28
87.76 64.28
Table 25: Comparison between Human and GPT-4 w.r.t.
Gottalllrr
45.28
17.33 4..03
77.38 44.92
48.41 30.64
game
88.3
50.95
80055 58.18
55.53 62.56
entity type
galf
51.85
51.40
91.48 51.85
52.41 39.35
handhall
81.0
$7.18 90.05
87.18 50:95
61.21 59.
Head Demain
lockey
87.6 91.62
87.6 82.64
57.6 52,64
55.94
Human GPT- Human GPT-4
Human GPT4
Human CPT-4 MET
lacrnsse
77.36
50.97 77.27
80197 77.37
45.09 52.11
F1
R1
launchoud
56.07 52.26
56.67 52,05
56.67 52.05
43.19 34.6
muriaal artist
87.06
55 87.6
85.61 87.61
58.94
68.47
agmny
06.6667
77.75
13:96
77.04
13.06 77.04
55.64 77.07
militury confiets
65.13
51
81.48 64.29
46.77
76.33
momumeet
77.69
53.33 77.92
43.35 77:92
64.91 6132
100.0
01.31
100.0
67.50
movic
37.5 77.78
51.5 71.35
57.5 77.38
0011
gime
03.43
80.56
61.51 67.1
musi
98.89
103.0 89
10:0 58. 99.99
74.79
book
83.33
94.04
94.04
58.21 65.41
musicias
75.87
53.33 75.82
93.25 75.82
64.59 58.96
02.50
02.50
60.8
national cricket seam
94.58
59.33 94,44
45.35
54.44
08.02 08102
chunch
85.03
59.57
national Sontball team
03:35
56.11 N1.33
9h11 83.33
18.30 5007
company
92.58 79.05
76.75
78.75
19.25 62.31
nary vesee
66.44
$4.13 07.25
44.13 67.36
66.04
uba
50.0 94.73
90.0 84.07
1000 54.67
55,93 62.9
country
78.61
75.45
all
69.3
79.37
79:17 09:33
51.52 52.42
87.62 19
96.17
68.65 70.67
nobel
78.3
52.00
3E:00 78:26
55 19 57.54
cricktt
76.67
84.81
office hodders
57.04 62.5
57.04 02.93
27.04 02.59
13 54.76
dicseter
70.94
paineer
93.33
102. 23.33
11:0 90:38
62.59 59.4
100.0
67.21
62.10
49.17
person
45.03
94.77 44:15
18:87 44:15
61.00 1102
Splier
44.44
53,29
44.44 53.20
24.85 32.35
polrician
61.67
66.07 14.00
00.07 5162
10.6
77.42 $4.21
53.07
$2.96 40.35
racing
66.49
76,92 16.3
76622 66.3
48.9 47.75
cail
30.0
100.0 90.0
10:0 80.0
79.08 69.88
loader
R5.03
61.65
85.3
63.05 51.06
railway
37.69
V5.33 17.02
45,25 57.62
77.3 75222
militury confiies
53.57
64.20
41.83
right
61.32
96.3
96.3 61.22
57.67
83.33 77.69
77.92
77,92
64.91 61.82
soller
69:23
99.29 09.15
99.29 09.18
44.64 42.9
72.92
46 4h
sclentitt
81.13
90.27 M0.M1
41.37 50.37
57.34 59.38
100.0
100,0
72.74 4868
show
58.33
58.33
17 32
ausician
83.33
75.82
64.59 58.96
96.84 79:35
05.45 75.02
95.0 77.95
65.72
song
72.55
53.33 71.21
93.33 72:72
64.92
national cricket
68.33 84.58
64.44
AR.33 64.44
6K02
space probe
53.09
32.7 53.25
K1.7 52.71
nstinaal Sontall
83.33
53.33
50.07
space prugram
$2.17
55:89 52.70
35.89
K2,05
76.26
#2.05
78.26
57.54
hokees
$7.04
62,83
07.04
62.58
54.76
swimming
96.77 90.62
46.77 50:62
55.87 51.79
person
00.91 50.57
90.64
59.64
40.77
table tenms
73.81
100.0 73.71
100:0 73.91
02.5
line
100.0
100,0
69.68
50.0
calleay
87.69
K7.62
75.52
univerityy
76.1
53,29 74.77
98399 74.27
54.51
K1.22
67.25
86.13 67.26
66.04 63.75
vellesball
37.5 00.16
31.5 00.0
0800
26.3
song
83.33
73.21
64.92 54.85
wetsite
100.0 100.0
space
81.04 52.67
52.93
52.65
58.84 45.72
westling
sports
86.66 71.34
73.45
86.75 71.41
54.92 51.29
university
86.47 61.99
100.0 76.1
74.79
62.51
86.47
8K399
62.51
54.51 53.94
56,02
Table 27: Comparison between Human and GPT-4 w.r.t. fine- grained categorics for Head Set
website
100.0 100,0
100,0
100,0
100.0
79.17
Table 26: Comparison between Human and GPT-4 w.r.t.
Type
Tail Domain GPT- Human
coarse-grain grained categories for Head Set
Human GPT-4 Human F1
GPT-4
Human GPT-4 MRT
R1
R2
airenft
83.33 '0.88
K3.3 700522
83.33
71.52
64.27
55.87
Flan-T5 is an instruction fine-tuned derivative of the T5 language model, purposefully crafted to excel in a wide array of natural language pro- cessing tasks. These tasks include, among oth- ers, text generation, summarization, and transla- tion. With the integration of instruction-based fine- tuning, Flan-T5 boosts its competency in handling zero-shot NLP tasks while also facilitating few- shot in in-context learning. Thanks to its advanced incoder-decoder architecture and attention mech-
army
77.14
55.91
77.68
56.54
77.68
56.32
55.05
47.01
disuster
74.6 44.12
74.6 41.53
74.6 44.32
19.08 29.99
diseases
67.94
70.46 59.9
70.46 59.6
41.08
boliday
76.29 57.37
77.01 57.71
77.01 57.29
48.49 44.26
organization
91.67 71.35
91.67 70.59
91.67
70.59
53.12
70.0
party
86.38 72.89
86.38 73.33
86.38
73.33
62.83 57.61
83.33
83.33 89.33
83.33
83.33
56.25 56.25
ships
84.69 67.61
68.24
84.69
61.13 51.91
spors
91.02 71.96
91.00 72.46
90.98
72.42
55.32 48.08
time zone
87.22 61,85
79.33
62.03
79.33
62.03
64.51 55.11
87.94 59.5
87.62 60.01
87.62
59.93
66.19 49.69
Table 28: Comparison between Human and GPT-4 w.r.t. -narso-grained categories for Tail Sct

---

Type
Dumsin
incorporates an encoder-decoder structure with several layers of self-attention mechanisms and 59-52 50.04
Human
Hummn
uses a shared vocabulary and tokenization scheme, GPT4 Human
Human
F1
R2
MET
amy
77.14 55.91
77.68 56.54
77 68 56.32
5505 47.01
buxing
58. 89 67.97
58.59 67
88.3 67.81
cricke
59.20 70:08
89:30 70.2
89.39 70.2
53.65 44.37
eycling
95.16 62.79
95.06 65:51
95.06 65.51
55.55 11.64
thereby ensuring a consistent representation and efficient processing of text data.
eyclons
74.6 44.12
74.6 44.53
74.6 44.32
4008 29.99
discase
67,94 59.37
70.46 59.9
70.46 59.6
42.88 41,08
n
hockey
90.27 72.07
93.89 75.9
92.27 72.25
93.59 75.04
93.89
92:27 72.25
75.94
$1.05 44.19
$5.96 50.46
We carried out fine-tuning on different variants of the T5 model: TS-Base, T5-Large, and T5-XL. We also conducted zero-shot experiments on T5- Large, T5-XI., and T5-XXI.
huldky
76.29 57.37
77.01 57.71
77.01
57.29
48.49 44.26
erbibor
53.33
83.33
64.27 58.87
glanet
53.53
83.33 X3.35
56.25 56.25
political party
56.38
73.35
62.85 57.61
peay
67.94
87.62 99.03
66.19 49.49
ship
sponts
58.34 99.77
aquash
85.11
50.99
sumo
88.61 64.07
56.57 51.88
GPT-3.5-turbo and GPT-4 are the latest devel- opments in the distinguished GPT series of lan- guage models. GPT-3.5 Turbo is an enhanced vari- ant of GPT-3, boasting approximately 154 billion parameters and demonstrating superior language processing capabilities. It is particularly adept text generation, comprehension, summarization, among other tasks. Conversely, GPT-4 signifies the next step in language modeling with an expected model size of around trillion parameters and im- proved language understanding and generation ca- pacities. These models rely vast pretraining data for superior generalization and exhibit excel- lent performance in both zero-shot and few-shot learning scenarios.
orgnization
01.67
91 70.59
53,12
time wone
57.22
61.83
79.33
62,03
79.33
62.03
64.51 55.11
Table 29; Comparison between Human and GPT-4 w.r.t fine- grained categories for Tail Set
anisms, Flan-T5 efficiently leverages contextual information to generate high-quality output. This capability promotes opportunities for enhanced per- formance and adaptability across various language processing applications.
Our study involved fine-tuning the Flan-T5 model across its different variants: Flan-T5-Bard Flan-T5-Larre, and Flan-T5-XL. Alongside this, we also carried out zero-shot, few-shot (with and without chain of thought prompting) experiments Flan- T5-Large, Flan-T5 -L-L,,, and TX-XXX.
We conducted a suite of experiments on GPT- 3.5 Turbo and GPT-4, focusing on zero-shot and few-shot learning scenarios, examining their per- formance with and without reasoning capabilities.
BART (Bidirectional and RegrRegres Trans- model architecture widely adopted in various natural lan- guage processing tasks. By fusing bidirectional and autoregressive training objectives, BART is ca- pable of exploiting the context of both the input and target sequences. Given that BART features an autoregressive decoder, it can be directly fine-tuned for sequence generation tasks, such as abstractive question answering.
formers) is a robust
Table Representation. Firstly, each table is transformed from HTML into a JSON represen- tation, containing subheadings, rows with keys and their respective values, as well as the table title and category as distinct keys. We employed a lineariza- tion process akin to INFOTABS (Gupta et al., 2020), using delimiters such "tab" or ":" to separate keys, and "newline" or "." for rows. Subsections are partitioned by double "new lines" "#". For instance, in Table the representation is: Title: Petya Nedelcheva # Personal Information Coun- try: Bulgaria; Born: July 30, 1983 (age 38), and soon.
We fine-tuned the BART-Large model, consist- ing of 12 encoder-decooer layers with 440 million parameters, and BART-Base model, comprising encoder-decoder layers and 140 million parameters. The performance analysis and outcomes of these fine-tuned models can be found in the main paper.
T5 (Text-To-Text Transfer Transformer) is a versatile language model urchitecture. Based on the transformer model, T5 is equipped to handle vari- natural language processing tasks. By leverag- ing a "text-to-text" training approach, T5 learns to transform input text into target text, thus enabling it to manage a wide variety of tasks. These tasks include text classification, summarization, trans- lation, and question answering. The T5 model
D Crowdsourcing Details
To construet TEMPTABQA, we divided the task into 80 batches, each consisting of three question- answer pair generations per HITS We assigned
"A Human Intelligence Task, or HIT. a question that needs an answer. A HIT represents a single, self-contained. virtual task that Worker can work on, submit an answer, and collect a reward for completing. HITss are created hy Requester customers in order to he completed by Worker customers.

---

Dataset
Number
Gold/total
units and correct spelling errors during our quality filtering. We prioritize questions involving intricate temporal reasoning and abstract concepts while fil- tering out questions with answers directly present in the question or associated tables. Questions that required external knowledge beyond common sense are also filtered.
3
82/91
Dev
4
3/3
5
1/2
Overall
148/166
3
898/1033
Head
4
20/32
5
23/32
Overall
1627/1821
Our annotators were directed offer concise and pertinent answers. While the majority adhered to the instructions, a few instances deviated, lcad- ing to occasional ambiguities. These ambiguities typically emerged when multiple answer forms con- veyed the same meaning but in different units or formats, such '365 days' '12 months' or 'one year.' We ensured assessment script didn't pe- nalize models or human verifiers for unit or format conversion issues. We established regex rules that encompassed various forms, and these were further validated through human verification across numer- ous samples. Figure 30 shows the exact agreement between across several annotators.
386/468
Tail
26/33
39/49
Overall
999/1155
Table 30: Exact agreement between annotators
each HIT to three distinct annotators, resulting in an average of QA pairs generated per table. The wage for each HIT, which involved generating three question-answar pairs for given table, was set at 0.75 cents based on the average completion time observed during three pilot studies.
All annotators were proficient English speak- ers from countries where English spoken. They possessed master-level qualifications and main- tained a HIT acceptance rate 95% and above. We occasionally rewarded frequent and exceptional annotators with bonus of 3 times the cost of the HIT. To ensure task quality, we implemented tem- porary blocking and rewarding mechanisms for annotators.
E More Examples from TEMPTABQA
Figures 2, 3, 4, 5, 6 show some examples of tabular question answers from TEMPTABQA.
Answering these questions demands from lan- guage models an understanding of temporal rela- tionships to correctly connect time frames to per- tinent events, as well as numerical reasoning to perform calculations, comparisons, and quantita- tive analyses based on temporal data. This includes both basic arithmetic and complex numerical rea- soning like identifying trends or evaluating numeri- cal changes over time.
For verification purposes, each HIT required an- swering three questions and providing a brief expla- nation. An annotator received 0.15 cents per HIT not reached among
for this task. If consensus the initial three annotators, reassigned the HIT another set of three annotators. Here two we start with three pilot study decide the cost of the annotation. Notably, we observed that the top 50 annotators were responsible for annotating approxi- mately 90% of the dataset. This observation aligns with other crowdsourced data annotation projects such as SNLI and MultiNLI.
These questions present challenges for language models due to the multi-faceted nature of the infor- mation required to answer them. First, they demand deep understanding of temporal relationships, en- compassing the ability to interpret and analyze time frames accurately. The language model must effec- tively connect these time frames to specific events, albums, or other relevant entities mentioned in the context.
Validation Details. We employ straightforward
pre-processing scripts to remove non-temporal and basic extractive questions from the training set prior to fine-tuning. For the test and development sets, we enforce rigorous quality control by manually reviewing each Table QA, with input from three experts who are NLP researchers. This process follows the initial automated based filtering and is aimed at ensuring high-quality complex tem- poral questions. Additionally, we address answer
Furthermore, numerical reasoning plays cru- cial role in successfully addressing these questions.
The language model needs to perform calculations, comparisons, and quantitative analysis based on temporal data arrive at the correct answers. This entails not only basic arithmetic operations but also more sophisticated numerical reasoning, such as identifying trends, computing durations, or evaluat-

---

Dataset
Temporal
Entity
Domain
Task
Hybrid
Numerical
Synthetic
Abstractive
INFOTABS
X
Generic
NLI
K
Human
TARFACT
X
X
Generic
NLI
Human
LOGICQA
X
X
Generic
QA
Machine
FETAQA
K
X
Generic
QA
Machine
FINQA
X
X
Finance QA
Human
TATQA
K
X
Finance QA
Human
HYBRIDQA
Generic QA
Human
WIRITABLEQA
Generic QA
Human
SQUALL
Generic QA
Human
WIKISQL
Generic QA
Human
SQA
Generic QA
Human
TEMPTABQL
Genene
QA
Human
Table 31: Comparison of TEMPTABQA with existing standard Tabular Datasets.
Statistie
WIKITABLEQA
HYBRIDQA
SQA
FINQA
TABFACT
FETAQA
SQUALL
# temporal
7807
33713
4634
6774 82.48
11219 37.69
5978
3876
temporal (%)
35.46
48.43
26.40
57.87
34.37
Question Types
explicit
5830
28441
2899
6719
10797
5657
2901
implicit
1977
5272
1735
55
422
321
975
ordinal
1433
5267
276
107
510
754
Temporal Interval
before
1006
1156
173
109
539
after
524
893
159
41
83
259
duration
1249
2520
391
1070
2218
562
567
yes/no
86
6
10
103
91
65
temporal
1783
8588
1845
52
1605
4681
526
Operation Involved
531
1735
185
78
119
115
274
min
597
1272
198
344
219
352
294
count
2313
5789
273
223
597
706
1109
619
965
128
1960
1954
254
316
difference
249
225
2390
4149
60
127
average
124
391
33
1788
2633
38
41
comparison 470
697
184
126
834
38
222
Table 32: Statistics of temporal question present in the existing tabular datasets. Most of the question explicit and involve only numerical reasoning.
ing numerical changes over time.
various question types, answer entity types, rea- soning operations, answer positions, and table do- mains. Both the model and humans demonstrate varying levels of proficiency across different cate- gories. The analysis helps identify weaknesses and areas for improvement in future temporal reasoning models on semi-structured tabular data.
F Further Discussion
Key Findings. Based on our experimental anal- ysis in $4, we conclude that even state-of-the-an art large language models like GPT-4 struggle with temporal question answering on entity-centric ta- bles within TEMPTABQA, despite humans' high performance. Fine-tuning and few-shot learning techniques have a positive impact on the model's performance. The model encounters more difficul- ties in the tail domain, comprising rare occurrences, compared to the head domain with more frequent instances. Techniques involving planations, such as chain of thought prompting, further enhance the model's performance.
ceni-structured Tables. Semi-structured data lies in a realm between raw, unstructured text and rigidly structured content such as Knowledge Gr- pah. This data landscape, where structured frame- works interweave with free-form text, spans the gamut from extensive verbosity like web pages, succinct instances such fact sheets, informa- tion tables, and technical specifications. Unlike databases, this type of data isn't uniformly struc- tured; it can be a heterogencous assortment without preset schemas. Adding to the complexity, explana-
In our breakdown $5, we discovered inconsis- tent performance of the model and humans across

---

VERISIGN
Death Magmetle
Typw
Public
Studi albun by Heraliica
Traded
2008
500
Reconded
May
Industy
Studio
Sou'd City ruos Angeles)
Foundted
April 12, 1995. 28 years
She sg--i efealtss Caliborni
ago-!
Mesiiices HQ (Son Rabbot Culll urii
Founder
James Ditrros
Headiuarters
Virgnss
Gone
Thrcsh mnta Reciyy necal
James Bittoos
[Churman cea)
Lakel
Wor nor Bros.
Revenue
USSEL 42 bilicn (2802)
Prostaoer
Flick Tuuten
Operating Iscome
USSEP4 (1802)
chicoutegyy
income
US30774 malice (2602)
St Anger
Death stag wete
The Apxsilica
Tood annets
V
(2002)
(2008)
Couection (N00)
Total
USB 1.8 billon (2002)
Numeber
927 (022))
Hitallice studio albun ewronsoagy
Q1: What was Verisign's operating income the year the num- ber of employces reached 904? AI: 886 million
(2023)
Sult Cootnct
(2008)
(2014)
Bingles from Eesth Angnesse
Q2: Is net income of Verisign, Inc. was reduced in 2021 compared to 2020? A2: Yes
2006/
Q3: How long did take for Verisign reach over 900 employees? A3: 26 years
Reieaudd: December 15, 2006/1 "iroben Boor Scarre"
Delessdd April 2009
Q4: Who was the CEO in 1995? A4: James Bidzos
Q1: How many singles were released in the same year Death Magnetic was released? A1:
Figure 2: A semi-structured table (source: Wikipedia) along with accompanying temporal questions and their respective answers form TEMPTARQA.
Q2: How many months did it take for Metallica to record
Death Magnetic? A2: 14 Months
Q3: How many years after Death Magnetic did Metallica record another studio album? A3: 8 years
tory text that imparts context isn't It always at hand. Nonetheless, we frequently deduce insights from such diverse and incomplete data, bridging infor- mation gaps based on our expectations about rela- tionships within.
Q4: How many singles were released on the Death Magnetic album in 2008-20099 A4: 3
Figure 3: A with accompanying temporal questions and their respective answers form TEMPTABQA.
table (source: Wikipedia) along
Infobox style, we observe significant distinctions in how information is structured. Wikidata adopts more organized and structured approach, resem- bling a knowledge graph. For example, when deal- ing with person's 's birth details, Wikidata ncatly separates the information into distinct categories like "birth date," "birth place," and "birth name." In semi-structures Infoboxes on the other hand, these details are often combined under single heading, such as "Born." Furthermore, there is noticeable contrast in how relationships are pre- sented. In Wikidata, relationships are systemati- cally categorized. For instance, instead of using a generic "spouse" label, Wikidata provides sep- arate entries for "husband" and "wife," resulting in a more precise representation. In contrast, an Infobox might consolidate such information un- der single "spouse" entry without specifying the gender.
Reasoning Requirements. Navigating semi- structured information necessitates a broad range of reasoning skills. We're tasked with compre- hending makeshift layout composed of elements like text snippets, form fields, or even sub struc- tured components like lists. Querying this data calls for various levels inference, ranging from straightforward lookups e.g. in Figure 1 query- ing Petya born place, to lexical deductions, such as understanding in same table single (WS) and double game (WD) format, junior championship vs. senior events of badminton, to grasping the nature of content within cells, the structure of the various events, the tournament names, tournament years and places, the total and specific medals tally, and the tournament types. Additionally, we might find ourselves aggregating insights across multi- plc rows, such as understanding that Dressage is a non-contact sport in which both genders compete, even conducting intricate reasoning that melds temporal details with general knowledge.
From a Temporal Perspective Temporal details find distinct treatment as well. Wikidata distinctly separates "start date" and "end date," yielding pre- cise timeline information. This stands in contrast Infoboxes, where these details could be condensed into single terms like "service," potentially necessi- tating further interpretation. Wikidata's 's penchant for hierarchy is evident in how complex terms are
Similarity with Knowledge Graph. However, it's important to note that Infoboxes exhibit a high degree of similarity with standard knowledge bases, particularly when compared to Wikidata. Wikidata generally surpasses Infoboxes in terms of compre- hensiveness. When contrasting Wikidata with the

---

Miloš Obrenovié
Reign
23 December 22
(age
September 1860
Nationaity
Predecessor Nexander Karadordevié
Alma mate
M.S. Ricle
Successor
(Obrenovi))
Polyaccriquu Füldirale
Lauianne M.S.
November 1817 25 June 1839
Apptying learning
Sertia)
Successor
naovinated cuassanding
Woreen in Acaderica SNSR
23 April 1815 6 November
2019 king Sun Blet
Transaction Paper Award ICCE
Reborics and Autoraaion
Karadorde
sociey, 2003 Outsanding
Young Person Science
Successor
Himself Prince Serbia)
probably
1783
Sieten National Science
Gomja Dobrinja near Potega
Foundation, 2001 trnovative Teaching Grant: inadl
Ottoman Empire Sertie)
Corporatir, 1999 Fullowohip
Dled
26 September 1950 (aged 80)
Medicus Poundaatoon 1998-60 #7 Scholarship, Gentes Nasional
Belgrade, Sertia, Oitoman Empire
Scientifc career
Flelds
Machine learning, rebotics,
Burial
St. Mark's Church, Belgrade,
physics
Serbia
Q1:
What award did Billard receive the same year she was Nominated Memher SATW? Al: Nominated for Outstanding Women Academies SNSF
Consort
Vukomanovi
lssue
Princess Petria
Prince Obrenovi
Q2: What award did Aude Gemma Billard get when she was years old? A2: King- Sun Fu Best Transactions Paper Award
Prince Mhhalo Obrenowi
Princess Maria
Prince Gabriel
Q3: How many years are between Billard's 's award for The Outstanding Young Person in Science and Innovation and Innovative Teaching Grant? A3: years
House
Obrenovid
Father
Teodor Mihalovié
Vilnja Uralievé
Q4: How many years ago did Aude Gemma Billard was got award for Fellowship Medicus Foundation? A4: 23 Years ago (1999)
Religion
Serbian Orthodox
Q1:
Who was the Prince of Serbia in 1857 before Miloš Obrenović 1? A1: Alexander Karadordević
Q2: How many years before his death did Miloš Obrenović hegin his second reign as the Prince of Serbia? A2: 2 years
Figure 5: semi-structured table (source: Wikipedia) along with accompanying temporal questions and their respective answers form TEMPTARQQ.
Q3: How long after Miloš Obrenovié 's reign as Grand Vožd of Serbia ended did his second reign Prince of Serbia begin? A3: 41 years
In contrast, Infoboxes may not adhere simi- lar systematic categorization, potentially leading ambiguity: Collectively, these instances highlight the structured nature of Wikidata in contrast to the more succinet, semi-structured implicit knowledge of Infoboxes.
Q4: How many years elapsed between reigns of Prince of Serbia for Miloš Obrenović I? A4: 19 years
Q5: Who was the Prince of Serhia in 1840? A: Milan II
Figure 4: A aemi-structured table (source: Wikipedia) along with accompanying temporal questions and their respective answers form TEMPTABQA.
Knowledge Graph. As Infoboxes are highly structured (compared to Web tables), we could translate them to Wikidata and ap- ply existing datasets and algorithms. Despite, this approach holds some promise, it's worth noting that transforming them into a clean and fully structured Wikidata format in itself a challenging task, as highlighted earlier. Nevertheless, it presents an in- teresting opportunity to explore the capabilities of state-of of-the-art language models in achieving this conversion. However, it's important to acknowl- edge that tables not found Wikipedia, such those containing e-commerce attribute values, search grants, medical reports, financial company data, etc., pose their own challenges when it comes to transitioning them into structured knowledge formats like Wikidata.
Entity Table connersion
broken down. For instance, a "government official" could be subcategorized as "president," "prime min- ister," and more. In contrast, Infoboxes might lack this hierarchical clarity, opting for more gener- alized terms. Granular attributes shine in Wiki- data, with individual specifications for attributes like "awards," enabling a detailed breakdown of accolades. Conversely, Infoboxes could consol- idate these attributes, obscuring the specifics of received awards. When it comes to event descrip- tions, Wikidata adopts distinction between "start time" and "end time." leading lucid event elu- cidations. In Infoboxes, these might be captured by a singular term, potentially devoid of tempo- ral context. Lastly, Wikidata's categorization of properties imparts a structured approach to data.

---

Roy Emerson
We also experimented with an approach involv- ing attempted abbleto-paragraph conversion, but it caused models to include unwanted external infor- mation. LLM parametric knowledge lead to out of table unwanted hallucination in the paragraph. The performance variation across these representations were marginal <1% % in the Fl-score, and <0.75% in the exact match.
Full name
Roy Stanley Emerson
Country (sporta)
BMM Australla
Residence
Newport Beach, Califoii,
Born
November 1936 (age 66)
Queensland,
Australia
Height
183 om (6 O ing"
Turned pro
1968 (smaatuu tour from
Retired
1983
Fighht hight-handed seeehaddedd
backhand)
Tennis Hol
1982 (member pageit)
H Future Directions: Other Modeling Techniques
Singles
Career record
1387-416 (77.0%)
Career titles
open ena lated
Based on our observations and discussions, we have identified several promising future directions for enhancing models performance on TEMPT- ABQA:
Highest ranking No. 1,011, Ned Pl
Grand Slam singles results
Australian Open W (1901, 1983, 1995,
1966, 1967)
French Open
(1963, 1987)
Wimbledon
(1964, 1985)
LLM Pre-trained with Temporal Knowl- edge:
Open
(1981, 1994)
Doubles
Explore techniques incorporating tem- poral aspects during pre-training for masked language models (e.g., Dhingra et al. (2022); Iv et al. (2022)). Assess their performance in temporal tabular tasks using auxiliary tasks from temporal aantriinnnnwwringgggg datasets in open domains (Jia al., 2021b), cloze-form, event-centric settings (Dhingra et al., 2022: Chen et al., 2021a; Ning et al., 2018; Wen et al., 2021).
record
Grand Stam doubles
Australian
W
French Open
1964, 1965)
Wimbledon
W (1959, 1961, 1971)
us Open
W (1950, 1960, 1985, 1964)
Team competitions
Devis Cup
W (1950, 1960, 1961, 1962,
1985, 1966,
Q1:
When was the most recent time that Ray Emerson won the Australian Open? A1: 1969
Q2:
How long did Roy Emerson play in the amatcur Ieague before going professional? A2: 15 years
2. Temporal-Aligned Models for Entity- Centric Tabular Data: Utilize temporally tuned language models (e.g., TEQUILA, EXAQT, OTR-QA, TempoQR) tempo- ral knowledge-basce question-answering datasets (e.g., CRONQUESTIONS (Saxena et al., 2021), TEMPQA-W WD (Neelam et al.. 2022)) for answering questions related to temporal events (Jia et al., 2018a,b; Shang et al., 2021; Mavromatis et al., 2021; Saxena et al., 2021; Neelam et al., 2022).
Q3: How old was Roy Emerson when he retired from play- ing professionally? A3: 47 years old
Q4: What year was Roy Emerson inducted into the Interna- tional Tennis Hall of Fame? A4; 1982
Q5: Which Grand Slam tournament did Emerson win the 2nd time he won the Davis Cup team competition? A5: French Open
Figure 6: A semmissructture table (source: Wikipedia) along with accompanying temporal questions and their respective answers form TEMPTABQA.
G Table Representation for LLMs
We experimented with three prompts, each fea- turing detailed instructions similar to those given to human verifiers. These prompts were based on three distinct table representations using dif- ferent delimiters. Our selection process involved choosing the prompt that yielded the best perfor- mance. We present the table input in a linear format, akin to the approach adopted in TABFACT (Chen al., 2020b) and INFOTABS (Gupta al., 2020). Here, we employ distinctive denominator token demarcate rows using ":" and columns using ":". We also explored alternative delimiters such as "I" and "*" as well, the performance was similar.
Integrating External Temporal Knowl- edge: Incorporate knowledge base question- answering datasets like CRONK et al., 2021), nto LLM models during pre- training (e.g.. ERNIE (Zhang et al., 2019), WKLM, KECP, ERICA, DKPLM) or struc- tural adaptation (c.g.. ERNIE-THU, Know- Bert, EaE, JAKET). Explore the use of non-entity temporal relations (e.g.. ERICA. KEPLER, DKPLM, KP-PLM) through pre- training objectives or structural adaptation methods (e.g., FaE, K-adapter, KB-adapter,

---

KLMO, KERM, JointLK, GreaseLM, JAKET, KnowPrompt, OntoPrompt) as described in detail in (Hu al., 2023).
4. Fine-Tuning on Other Temporal Knowl- edge: Investigate benefits of training on syn- thetic and counterfactual temporal data (im- plicit knowledge addition) to enhance model performance, similar to AUTOTNLI (Kumar et al., 2022) and (Eisenschlos et al., 2020). Consider using simple temporal data from un- structured text sources like Time-Sensitive- QA (Chen al., 2021c) and CogComp (Ning al., 2018), or structured text datasets like TempQA-WD, CronQUESTIONS, and TempQuestions (Saxena et al., 2021; Neelam et al., 2022; Jia et al., 2018b; Shang et al., 2021), which feature question-answerne answering over knowledge graph embeddings with temporal links.


Reviews:
[{"review_id": "sWC8E7lpnb", "reviewer": "Reviewer_tNdZ", "summary": "", "questions": "1)The paper could benefit if the authors highlight the real world implications of such a task. \"Can current systems reason about such information in semi-structured tables?\" why is answering this important? What systems can be built using this?\n\n2) Were all the annotated train samples used? Or was there a quality-filtering step before that?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our article and provide valuable feedback.  We appreciate your thoughts and concerns and address them below:\n\nReasons to Reject\n\nQ1. The experimental section lacks complex modelling techniques. The authors have simply used existing LLMs and modelling techniques to compare performance. A novel modelling technique would strengthen the paper.\n\nThe TempTabQA dataset primarily consists of abstractive reasoning questions. Therefore, an ideal model for this task should possess generative capabilities. Existing datasets for tabular questions have predominantly featured extractive queries (See section 2 Motivation), resulting in the prevalence of encoder-focused models for diversity. In our study, we pivot towards state-of-the-art generative models, commonly referred to as Language Model (LLM) variants, encompassing both open-source options like FlanT5, BART, and T5, as well as proprietary alternatives like GPT-3.5 and GPT-4.\n\nAt a high level, we agree with the reviewer that a novel modelling technique such as one involving temporal graphs (e.g. EXAQT, Jia et. al, 2020),  would be a great direction for further work.\n\nOther open-source models either remained unavailable or hadn't gained widespread usage by the time of our paper's submission. It's worth noting that models such as the TabT5 couldn't be included due to proprietary industry restrictions (not open sourced). The primary objective of our paper was to shed light on the temporal question-answering challenge within a tabular context. Our contribution lies in presenting a robust foundational problem and strong baselines for addressing this issue. Moving forward, it would be intriguing to explore alternative models and delve into intricate prompting techniques, as outlined in the potential avenues for future research (as discussed in Section 6).\n\nQ2. The current manuscript only introduces a new dataset with minimal contributions on the modeling or science part. The authors claim that existing modelling approaches are insufficient, however they fail to propose even simple improvements to alleviate some of the shortcomings. I am afraid there isn't enough novelty or scientific contributions in this work to qualify as a research paper at a top-tier conference.\n\nWe respectfully disagree with the assertion that our work lacks novelty or scientific contribution. Quite the contrary, our paper pioneers the creation of complex temporal question answering datasets. Furthermore, it introduces the first complex question answering a dataset tailored to entity-centric tables. In contrast, prior endeavours such as \"Learning to Answer Questions from Wikipedia Infoboxes\" by Morales et al. (2016) (will cite in the introduction section 1) lack the complexity of our questions, and their dataset remains proprietary. Notably, our paper incorporates robust state-of-the-art baselines, leveraging contemporary Language Model Models (LLMs) and revealing their comparative shortcomings in comprehending and reasoning over the TempTabQA dataset.\n\nAs previously mentioned, the primary objective of our paper was to introduce an innovative challenge: tackling intricate temporal questions within the context of entity-centric tables. The TempTabQA dataset demands sophisticated reasoning, incorporating temporal common sense, and adeptly managing arithmetic and numerical aspects. Our work serves to elucidate the distinct temporal specificity of the proposed dataset, setting it apart from existing models. This differentiation and the multitude of temporal reasoning challenges it poses are extensively illuminated in Section 3.2 through a comprehensive array of statistics and analyses. In line with the forward-looking orientation of our work, as outlined in the \"Future Directions\" section (section 6), we believe this study opens up numerous pathways for exploring intricate temporal reasoning on semi-structured tables and other complex structures.\n\nQ3. Some unnecessary details can be moved to the Appendix. For example, the prompts and instructions used to guide the annotator may not be important to the reader. More focus is needed on the novelty.\n\nThanks for the suggestion, we will provide the annotation details to the appendix, thereby streamlining the core content for enhanced readability. Moreover, we recognize the need to reinforce the focus on the novelty of our work. To address this, we will strengthen the motivation surrounding the application of complex temporal reasoning within the domain of entity-centric tables. Doing so will also highlight the significance of the problem at hand and provide a stronger foundation for the originality of our contributions.\n\nQ4. Nitpick: The main sections of the dataset analysis contains very minimal pictorial representation. A lot of tables with numbers makes it hard to read. A few figures and bar graphs could be some good choices to convey the same.\n\nAs recommended by the reviewer, we will shift some results from tabular representation to pictorial form, especially line and bar charts. Thanks for pointing out these presentation issues.\n\nQuestions to Authors\n\nQ1. The paper could benefit if the authors highlighted the real world implications of such a task. \"Can current systems reason about such information in semi-structured tables?\" Why is answering this important? What systems can be built using this?\n\nA1. Complex temporal question answering applied to entity-centric semi-structured tables, like Wikipedia infoboxes, has broad applicability across various fields. In historical research and education, it helps scholars, historians, and students extract precise historical details, while in financial analysis, it empowers analysts with historical financial data for informed investment decisions. In medical research, it aids in accessing historical medical data and clinical trial timelines, and legal professionals use it to review historical legal records. Journalists gain historical context, linguists analyze language dynamics, and businesses optimize supply chains through historical data. Environmental researchers, policy analysts, travelers, software developers, and archivists all benefit from this versatile tool. This underscores its significance in extracting valuable insights from structured tables across diverse domains. We will mention these in the motivation section 2 of the paper in detail.\n\nQ2. Were all the annotated train samples used? Or was there a quality-filtering step before that?\n\nA2. We employ simple scripts to eliminate non-temporal and basic extractive questions from the training set before fine-tuning. For the test and development sets, we uphold stringent quality control, manually scrutinizing each Table QA (with input from three experts) following the initial automated script-based filtering, all aimed at securing high-quality complex temporal questions. Additionally, we rectify answer units and address spelling errors during our quality filtering process. We prioritize questions that involve intricate temporal reasoning and exhibit an abstract nature. We filter questions where the answers are directly stated in either the question itself or the associated tables. We will mention these quality filtering steps briefly in Section 3.3, Dataset Validation, and in more detail in the appendix.\n\nQ3. Some works on temporal QA (both over text and KBs) could be referenced:\n\nA Benchmark for Generalizable and Interpretable Temporal Question Answering Over Knowledge Bases”, in Arxiv, 2022.\nComplex Temporal Question Answering on Knowledge Graphs. Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum\nTargeted Extraction of Temporal Facts from Textual Resources for Improved Temporal Question Answering over Knowledge Bases. Nithish Kannen, Udit Sharma, Sumit Neelam, Dinesh Khandelwal, Shajith Ikbal, Hima Karanam and L. Venkata Subramaniam\n\nMissing References: Thanks for pointing out the missing references, we will add the mentioned paper in the related work.\n\nQ4. Typos Grammar Style And Presentation Improvements: Some figures could be used for representing the data composition. No other major concerns.\n\nAs recommended by the reviewer, we will shift some results from tabular representation to pictorial form, especially line and bar charts. Thanks for pointing out these presentation issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) They introduce a new task and accompany that with a sufficiently large dataset. The new task could be of interest to the NLP community.\n\n2) The paper is well structured and clear. It is easy to read and has sufficient details about the dataset\n\n3) The requirement for such a dataset is well motivated, highlighting the gaps in the existing literature and datasets.\n\n4) A well written section about future directions is encouraging.\n\n5) Detailed breakdown of performance to highlight where exactly the models lack in comparison to humans", "reasons_to_reject": "1) The experimental section lacks complex modelling techniques. The authors have simply used existing LLMs and modelling techniques to compare performance. A novel modelling technique would strengthen the paper.\n\n2) The current manuscript only introduces a new dataset with minimal contributions on the modelling or science part. The authors claim that existing modelling approaches are insufficient, however they fail to propose even simple improvements to alleviate some of the shortcomings. I am afraid there isn't enough novelty or scientific contributions in this work to qualify as a research paper at a top-tier conference.\n\n3) Some unnecessary details can be moved to the Appendix. For example, the prompts and instructions used to guide the annotator may not be important to the reader. More focus is needed on the novelty.\n\n4) Nitpick: The main sections of the dataset analysis contains very minimal pictorial representation. A lot of tables with numbers makes it hard to read. A few figures and bar graphs could be some good choices to convey the same.", "questions_to_authors": "1)The paper could benefit if the authors highlight the real world implications of such a task. \"Can current systems reason about such information in semi-structured tables?\" why is answering this important? What systems can be built using this?\n\n2) Were all the annotated train samples used? Or was there a quality-filtering step before that?", "paper_topic_and_main_contributions": "The paper introduces a new task of temporal question answering over semi-structured tables. For the same, they introduce a new dataset which comprises 11,454 question-answer pairs extracted from 1,208 Wikipedia Infobox tables spanning over 90 distinct domains. They experiment with pre-trained language models on the dataset and report that they lag behind human performance. They provide a detailed analysis breakdown of model performance against humans and additionally highlight the future research directions to guide the community.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some figures could be used for representing the data composition. No other major concerns."}, "extra_text": {"Missing_References": "Some works on temporal QA (both over text and KBs) could be referenced:\n\n1.A Benchmark for Generalizable and Interpretable Temporal Question Answering Over Knowledge Bases”, in Arxiv, 2022.\n\n2. Complex Temporal Question Answering on Knowledge Graphs. Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, Gerhard Weikum\n\n3. Targeted Extraction of Temporal Facts from Textual Resources for Improved Temporal Question Answering over Knowledge Bases. Nithish Kannen, Udit Sharma, Sumit Neelam, Dinesh Khandelwal, Shajith Ikbal, Hima Karanam and L. Venkata Subramaniam, “"}}, {"review_id": "CDeuA5Jxzp", "reviewer": "Reviewer_uQYi", "summary": "", "questions": "A. What is a \"temporal question\"? Since this is an important concept in the paper, it would be nice to have a definition early on.\n\nB. How were GPT models prompted? Was there any randomness in the generation? This is important for reproducibility.\n\nC. Line 497 Future Directions: This paragraph lists research that may be done, but hasn't been done. Why include that in a paper, and dedicate so much space to it?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our article and provide valuable feedback.  We appreciate your thoughts and concerns and address them below:\n\nQ. The task that this paper addresses is not particularly exciting, and adjacent tasks have already been explored\n\nComplex temporal problems must be solved for a variety of crucial applications. It is essential for getting accurate historical insights when conducting historical studies or teaching. It provides financial analysts with historical financial data to help them make informed investment decisions in financial analysis. It grants access to clinical trial timetables and historical medical data for use in medical research. It is utilised by solicitors to study earlier court documents. Through the use of historical data, journalists may enhance their stories, linguists can analyse linguistic patterns, and corporations can improve their supply networks. Notably, this adaptable tool is useful to environmental researchers, policy analysts, travellers, software developers, and archivists. This clearly underlines its importance in gathering insightful information from structured tables spanning several areas. We'll proactively discuss these aspects in detail in the motivation section (Section 2) of the paper. \n\nOur study focuses on complex semi-structured data, notably entity-centric infobox tables. In contrast to earlier investigations that explored temporal reasoning in unstructured contexts such as paragraphs and knowledge graphs, our work focuses on these contexts in complex semi-structured data. This focus has particular difficulties, as described in the section on motivation (Section 2), but it is also very important, as was already mentioned. As indicated in the related study section (Section 7), it is important to note that available datasets for comparable tasks typically comprise straightforward and explicit temporal queries (see Table 31 in Appendix). The dataset statistics in Section 3.2, however, show that our focus is on answering challenging implicit queries. The tables' sophisticated, semi-structured design is what causes this complexity. For a deeper understanding, we intend to supplement the work with a complete comparison, comparable to Section 3.2, presenting the question statistics of various non-temporal tabular datasets.\n\nQ. The paper spends considerable space on research that hasn't been done but may be done some day (starting on Line 497)\nTo emphasize the importance of our suggested datasets for upcoming research in the NLP community, we included a one-paragraph section with directions for future use of TempTabQA. According to earlier and recent ACL recommendations, this is a recommended practice from a research viewpoint.\n\nQuestions For The Authors:\n\nA. What is a \"temporal question\"? Since this is an important concept in the paper, it would be nice to have a definition early on.\n\nWhen we refer to \"temporal questions,\" we are talking about questions that require reasoning based on time-related facts. This can be either explicit, involving dates, years, times, etc., or implicit, using temporal terms like \"rank,\" \"before,\" \"after,\" \"predecessor,\" \"successor,\" and so on. We'll elaborate on this definition along with examples in the introduction section (Section 1) of the paper.\n\nB. How were GPT models prompted? Was there any randomness in the generation? This is important for reproducibility.\n\nIndeed, there was a little randomness in the GPT-3.5 model, while the GPT-4 model displayed almost no randomness. However, variations in performance when altering prompts and regenerating content were very small. We experimented with three prompts, each featuring detailed instructions similar to those given to human verifiers. These prompts were based on three distinct table representations using different delimiters. Our selection process involved choosing the prompt that yielded the best performance. We will elaborate these details in the appendix, along with additional information available on the public GitHub repository that we plan to release upon acceptance. We wish to again emphasise that the performance variation was marginal (< 1% in F1-score and 0.75 in exact match).\n\nC. Line 497 Future Directions: This paragraph lists research that may be done, but hasn't been done. Why include that in a paper, and dedicate so much space to it?\n\nTo emphasise the importance of our suggested datasets for upcoming research, in the NLP community, we included a one-paragraph section with directions for future use of TempTabQA. According to earlier and recent ACL recommendations, this is a recommended practice from a research viewpoint", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows good knowledge of related work\n- The annotation process is sound and high-quality\n- The paper contains a thorough statistical analysis of the new dataset\n- The paper contains a thorough analysis of experimental results, which yields insights into some of the models", "reasons_to_reject": "- The task that this paper addresses is not particularly exciting, and adjacent tasks have already been explored\n- The paper spends considerable space on research that hasn't been done but may be done some day (starting on Line 497)", "questions_to_authors": "A. What is a \"temporal question\"? Since this is an important concept in the paper, it would be nice to have a definition early on.\n\nB. How were GPT models prompted? Was there any randomness in the generation? This is important for reproducibility.\n\nC. Line 497 Future Directions: This paragraph lists research that may be done, but hasn't been done. Why include that in a paper, and dedicate so much space to it?", "paper_topic_and_main_contributions": "This paper introduces the task of answering questions about temporal phenomena, with reference to a semi-structured table. This paper describes how a new dataset was built for this task, by collecting infoboxes from Wikipedia and having crowdworkers write questions about them. The new dataset is analyzed thoroughly. In addition, the paper contains results, thoroughly analyzed as well, with several state-of-the-art NLP models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mDJepS7uw4", "reviewer": "Reviewer_XpmN", "summary": "", "questions": "Q1: Can you adapt state-of-the-art algorithms to your dataset? Which ones and how?\n\nQ2: How is the input of the models formatted?\n\nQ3: More generally, answer the remarks above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our article and provide valuable feedback.  We appreciate your thoughts and concerns and address them below:\n\nReasons to Reject\n\nW1: The comparison with the previous work is very slim and briefly mentioned at the end of the paper. For the dataset, how does it compare to other datasets from a statistical point of view (number of questions, diversity)? Table 31 is in the annex and lacks information.  Besides, the authors quickly evade the question of the knowledge graph question answering.\n\nThe prior work mostly dealt with non-temporal questions on database style tables (WebTableQA, SQUALL, FinQA, TaTQA, and others mentioned in the Related work Section 7). Others include temporal questions on either unstructured data (Time-sensitive QA and others) or  fully  structured, clean data, such as knowledge graphs (CronsQuestions, TempQuestions and others). Our work focuses on entity centric tables such as Wikipedia infoboxes. Entity centric tables such as Wikipedia infoboxes are very different (semi-structured) from both unstructured and complete structured data (sql tables and KG), as described in Motivation Section 2). Entity centric tables are semi-structured in nature and hold data in complex implicit form (see, for example, the discussion on semi-structured tables in Gupta et al, ACL 2020)\n\nSemi-structured data lies in a realm between raw, unstructured text and rigidly structured content. This data landscape, where structured frameworks interweave with free-form text, spans the gamut from extensive verbosity like web pages, to succinct instances such as fact sheets, information tables, and technical specifications. Unlike databases, this type of data isn't uniformly structured; it can be a heterogeneous assortment without preset schemas. Adding to the complexity, explanatory text that imparts context isn't always at hand. Nonetheless, we frequently deduce insights from such diverse and incomplete data, bridging information gaps based on our expectations about relationships within. Navigating semi-structured information necessitates a broad range of reasoning skills. We're tasked with comprehending a makeshift layout composed of elements like text snippets, form fields, or even substructured components like lists. Querying this data calls for various levels of inference, ranging from straightforward lookups (e.g. in athlete table of Petya in paper known born place), to lexical deductions (such as understanding that single  (WS) and double game (WD) format, junior championship vs senior events of badminton), to grasping the nature of content within cells (the structure of event, tournament name, tournament year, medals tally and tournament type). Additionally, we might find ourselves aggregating insights across multiple rows (such as understanding that dressage is a non-contact sport in which both genders compete), or even conducting intricate reasoning that melds temporal details with general knowledge.\n\nMoreover, in response to the reviewer's query, we will present statistics related to the table-specific datasets, and expand Appendix Table 31. In a nutshell, prior datasets primarily consist of non-temporal questions, with the rare temporal questions being explicit and not-grounded in commonsense and general world knowledge essential for real world temporal reasoning. It's worth noting that the scope of reasoning within these datasets is limited, focusing solely on explicit arithmetic temporal reasoning (SQL query oriented) applied to structured SQL-type tables, rather than the more intricate semi-structured infobox-style tables with commonsense and world knowledge.\n\nHowever, infoboxes are highly close to standard knowledge bases, particularly Wikidata (they are automatic conversions between Wikidata and Infoboxes, and Wikidata is generally much more complete).  \n\nWhen comparing Wikidata to the Infobox style, distinct differences arise in how information is organized. Wikidata exhibits a more structured approach, akin to a knowledge graph. \n\nFor instance, when considering birth details of a person, Wikidata neatly segregates information into \"birth date,\" \"birth place,\" and \"birth name.\" Conversely, in semi-structured infoboxes, these details often amalgamate under a single term like \"Born.\" Moreover, the manner in which relationships are presented demonstrates a divergence. In Wikidata, relationships are systematically delineated. For instance, instead of a broad \"spouse\" label, one might encounter separate entries for \"husband\" and \"wife,\" offering clearer insights. In contrast, an infobox might encompass a single \"spouse\" entry without specifying the gender.\n\nTemporal details find distinct treatment as well. Wikidata distinctly separates \"start date\" and \"end date,\" yielding precise timeline information. This stands in contrast to infoboxes, where these details could be condensed into single terms like \"service,\" potentially necessitating further interpretation. Wikidata's penchant for hierarchy is evident in how complex terms are broken down. For instance, a \"government official\" could be subcategorized as \"president,\" \"prime minister,\" and more. In contrast, infoboxes might lack this hierarchical clarity, opting for more generalized terms.\n\nGranular attributes shine in Wikidata, with individual specifications for attributes like \"awards,\" enabling a detailed breakdown of accolades. Conversely, infoboxes could consolidate these attributes, obscuring the specifics of received awards. When it comes to event descriptions, Wikidata adopts a distinction between \"start time\" and \"end time,\" leading to lucid event elucidations. In infoboxes, these might be captured by a singular term, potentially devoid of temporal context.\n\nLastly, Wikidata's categorization of properties imparts a structured approach to data. In contrast, infoboxes may not adhere to a similar systematic categorization, potentially leading to ambiguity. Collectively, these instances highlight the structured nature of Wikidata in contrast to the more succinct, semi-structured implicit knowledge of infoboxes.\n\n\nAs infoboxes are highly structured (compared to Web tables), we could translate them to Wikidata and apply existing datasets and algorithms. \n\nIndeed, this approach holds promise, although it's worth noting that transforming them into a clean and fully structured Wikidata format is in itself a challenging task, as highlighted earlier. Nevertheless, it presents an interesting opportunity to explore the capabilities of state-of-the-art language models in achieving this conversion. However, it's important to acknowledge that tables not found on Wikipedia, such as those containing e-commerce attribute values, research grants, medical reports, financial company data, etc., pose their own challenges when it comes to transitioning them into structured knowledge formats like Wikidata.\n\nFor the baselines, only raw language models are considered, whereas more advanced solutions could have been adopted (also briefly mentioned at the end). Although it is not the core goal of the paper, it could help future works.\n\nCertainly, while there's room for experimentation with more complex prompting approaches, it's essential to note that this falls outside the current scope and objectives of the paper. As a result, we've outlined this avenue for exploration in the section dedicated to future directions.\n\nW2: The authors do not explicitly evaluate the quality of the dataset. In section 3.3, they give good criteria for the evaluation (can we answer only with the table? Is the answer ambiguous? Is it an opinion?). Still, they are not used for the evaluation of the dataset. The human accuracy is about 86% for the exact match. Where do the errors come from (formulation? Hard questions?   = misunderstanding?)\n\nFor the test and development sets, we uphold stringent quality control, manually scrutinizing each Table QA (with input from three experts) following the initial automated script-based filtering, all aimed at securing high-quality complex temporal questions. Additionally, we rectify answer units and address spelling errors during our quality filtering process. We prioritize questions that involve intricate temporal reasoning and exhibit an abstract nature. We filter questions where the answers are directly stated in either the question itself or the associated tables. We will mention these quality filtering steps briefly in Section 3.3, dataset validation, and in more detail in the appendix. \n\nIn certain instances, we encountered questions that demanded external knowledge beyond common sense. These were selectively filtered out. The majority of errors were observed in questions that involvedd unusually complex reasoning. Additionally, many errors stemmed from minor miscalculations in date-time questions, typically off by a year. Ambiguities sometimes arose, such as when multiple answer forms hold the same meaning (e.g., \"365 days,\" \"12 months,\" and \"1 year\"). Furthermore, we make certain that our assessment script does not penalize either the model or the human verifier for any such unit conversion issues. Notably, only a small fraction of questions, namely less than 10% for the development set, less than 7% for the Head set, and less than 11% for the Tail set, were ambiguous and lacked a clear majority, as detailed in Table 8. To provide a broader perspective, we intend to include a cumulative count of majority agreements with respect to varying numbers of annotators in the final paper version.\n\nFurthermore, less than 3% of the questions displayed subjectivity or opinion-based characteristics during the test and development set filtering process. All these nuanced details will be comprehensively covered in the final version of the paper.\n\nW3: More detail on the baseline would be nice. How is the input of the model formatted? This is quite an important point due to the structure of infoboxes. Besides, I wish the authors would have tried to adapt state-of-the-art architecture for (temporal) question answering.\n\nWe experimented with three prompts, each featuring detailed instructions similar to those given to human verifiers. These prompts were based on three distinct table representations using different delimiters. Our selection process involved choosing the prompt that yielded the best performance. We present the table input in a linear format, akin to the approach adopted in TabFact (Chen et al., 2020) and InfoTabS (Gupta et al., 2020). Here, we employ a distinctive denominator token to demarcate rows using “;” and columns using “:”. We explored alternative delimiters like “|” and “#” as well. \n\nWe also experimented with an approach involving the conversion of tables into paragraphs before feeding them to the models. However, this method exhibited the unintended outcome of the models introducing extraneous external information. All these specific details will be elaborated upon in the appendix, along with additional information available on the public GitHub repository that we plan to release upon acceptance. Just to again emphasize that the performance variation was very marginal (< 1% in the  F1-score and 0.75 in the exact match). \n\nMany modern architectural models applied to tabular datasets take an extractive approach, which is generally characterized by non-generative encoder-only models. As an intentional choice, this work focuses on cutting-edge generative large language models, such as FlanT5 and GPT. Furthermore, several of these alternate models are either not publicly available or are proprietary in nature. TabT5, for example, falls into this category and is impossible to fine-tune, resulting in inferior performance datasets when subjected to zero or few-shot testing conditions.\n\nW4: Some analyses in section 5 are sometimes statistically irrelevant. I wish there were p-values. Some sample sizes are too small to draw conclusions (1, 16, 21, 22).\n\nYes, we do agree that some of analysis on the test set have very few samples. We didn’t include this in the main paper due to page limits.  Therefore we plan to provide the analysis for the train set also in the appendix. We will also add p-values wherever possible for tables in section 5. Thanks for pointing this out.\n\nOther points: \n\nO0: The authors could have provided the dataset and code for the baselines on an anonymous link. It is currently impossible to check what the authors said.\n\nWe will release the data and code with the final version. We put an anonymous link to follow the ACL anonymity deadline. \n\nO1: The paper's title is strangely formatted: TEMPTABQA is composed of several words, making it hard to copy-paste it (and parse for tools like Google Scholar).\n\nWe will fix this and put something similar in the final version.\n\nO2: Would it be helpful to include synonym answers for a better exact match score? For example: \"How many years...\" could lead to \"3\", \"3 years\", \"three,\" or \"three years.\" Same thing for dates. Did you have a constraint that the Turks should or should not put the unit?\n\nYou raise a valid point. As previously discussed, we've taken steps to ensure that our evaluation script doesn't unfairly penalize both human annotators and the model for providing answers in different formats. We've crafted regex rules that encompass a variety of such scenarios, and we've further validated this through human verification of numerous samples. Our annotators were instructed to provide succinct and essential information in their answers. While the majority of annotators followed the guidelines diligently, a few instances did deviate despite clear instructions. As part of our final dataset preparation, we plan to refine and include both the cleaned test version and the original test version to address these concerns. \n\nO3: Line 42-44: Research on the temporal aspect was limited.\n\nThanks, we will fix this writing incoherency issues as pointed out by the reviewer.\nO4: Can a question have several correct answers?\nWe request that annotators only write unambiguous questions. However, several format representations for the same answer are possible. For example, 1 year, 365 days, and 12 months can all be valid answers for the same question.\n\nO5: Section 5: cite the papers for the models you use.\n\nWe will fix this writing issue, Thanks for pointing this out.\n\nO6: It is a pity the authors were not able to include bigger models for the finetuned models, like LLaMa or Bloom. I understand the technical difficulties, though. Algorithms like PEFT can help.\n\nWe did not test PEFT, but we will see if we can fine-tune LLaMA on the dataset. Bloom is primarily bilingual; therefore, we didn't use it for our research. Of course, whether we can do so will depend on compute resource availability.\n\nO7: In the table, put the best results in bold and the second best underlined.\n\nGreat suggestion; we would do that.\n\nO8: The abbreviations F1, EM, R1, R2, MET should be introduced somewhere.\n\nThanks for pointing this out; we will add the full form.\n\nO9: Line 354: Is it surprising?\n\nYes, the observation was not surprising, but the huge gap was a novel observation.\n\nO10: Paragraph line 366: The difference between head and tail is slim for finetuned models. Why? This should be reported.\n\nThis phenomenon can be attributed primarily to the knowledge transfer occurring between less common and widely recognized sports tables. These tables share numerous common attributes and exhibit similarity in the types of questions posed. To underscore this observation, we intend to provide specific sport-related results in the appendix. Additionally, we plan to summarise these findings in the main experimental section for greater clarity.\n\nO11: Section 6 is not helpful. It is a kind of conclusion in the middle of the paper that only repeats the previous section. I do not see how the future directions are related to the previous discussions. I would put the section at the end.\n\nWe would move the section 6 to the end of the paper.\n\nO12: What does HIT mean in Section D?\n\nA Human Intelligence Task, or HIT, is a question that needs an answer. A HIT represents a single, self-contained, virtual task that a Worker can work on, submit an answer, and collect a reward for completing. HITs are created by Requester customers in order to be completed by Worker customers.\n\nQuestions For The Authors:\n\nQ1: Can you adapt state-of-the-art algorithms to your dataset? Which ones and how?\n\nWe can adapt the existing generative encoder-decoder and decoder model such as TabT5, LLAMA, Falcon. We can also adapt existing KG based temporal QA model by first converting the table to KG’s. However this could be challenging as already discussed before.\n\nQ2: How is the input of the models formatted?\n\nWe present the table input in a linear format, akin to the approach adopted in TabFact (Chen et al., 2020) and InfoTabS (Gupta et al., 2020). Here, we employ a distinctive denominator token to demarcate rows using “;” and columns using “:”. We explored alternative delimiters like “|” and “#” as well. \n\nWe also experimented with an approach involving the conversion of tables into paragraphs before feeding them to the models. However, this method exhibited the unintended outcome of the models introducing extraneous external information. All these specific details will be elaborated upon in the appendix, along with additional information available on the public GitHub repository that we plan to release upon acceptance.\n\n\nQ3: More generally, answer the remarks above.\n\nPlease see above", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: The dataset is a valuable resource for the community. It is created by humans and comprises a reasonable amount of varied data.\n\nS2: The authors provide an excellent statistical analysis of the diversity of the dataset.\n\nS3: The authors extract interesting insights from the baseline analysis.\n\nS4: The paper is well-written and easy to follow.", "reasons_to_reject": "W1: The comparison with the previous work is very slim and briefly mentioned at the end of the paper. For the dataset, how does it compare to other datasets from a statistical point of view (number of questions, diversity)? Table 31 is in the annex and lacks information. Besides, the authors quickly evade the question of the knowledge graph question answering. However, infoboxes are highly close to standard knowledge bases, particularly Wikidata (they are automatic conversions between Wikidata and Infoboxes, and Wikidata is generally much more complete). As infoboxes are highly structured (compared to Web tables), we could translate them to Wikidata and apply existing datasets and algorithms. For the baselines, only raw language models are considered, whereas more advanced solutions could have been adopted (also briefly mentioned at the end). Although it is not the core goal of the paper, it could help future works.\n\nW2: The authors do not explicitly evaluate the quality of the dataset. In section 3.3, they give good criteria for the evaluation (can we answer only with the table? Is the answer ambiguous? Is it an opinion?). Still, they are not used for the evaluation of the dataset. The human accuracy is about 86% for the exact match. Where do the errors come from (formulation? hard questions? misunderstanding?)\n\nW3: More detail on the baseline would be nice. How is the input of the model formatted? This is quite an important point due to the structure of infoboxes. Besides, I wish the authors would have tried to adapt state-of-the-art architecture for (temporal) question answering.\n\nW4: Some analyses in section 5 are sometimes statistically irrelevant. I wish there were p-values. Some sample sizes are too small to draw conclusions (1, 16, 21, 22).\n\n\nOther points:\nO0: The authors could have provided the dataset and code for the baselines on an anonymous link. It is currently impossible to check what the authors said.\n\nO1: The paper's title is strangely formatted: TEMPTABQA is composed of several words, making it hard to copy-paste it (and parse for tools like Google Scholar).\n\nO2: Would it be helpful to include synonym answers for a better exact match score? For example: \"How many years...\" could lead to \"3\", \"3 years\", \"three,\" or \"three years.\" Same thing for dates. Did you have a constraint that the Turks should or should not put the unit?\n\nO3: Line 42-44: Research on the *temporal aspect* was limited.\n\nO4: Can a question have several correct answers?\n\nO5: Section 5: cite the papers for the models you use.\n\nO6: It is a pity the authors were not able to include bigger models for the finetuned models, like LLaMa or Bloom. I understand the technical difficulties, though. Algorithms like PEFT can help.\n\nO7: In the table, put the best results in bold and the second best underlined.\n\nO8: The abbreviations F1, EM, R1, R2, MET should be introduced somewhere.\n\nO9: Line 354: Is it surprising?\n\nO10: Paragraph line 366: The difference between head and tail is slim for finetuned models. Why? This should be reported.\n\nO11: Section 6 is not helpful. It is a kind of conclusion in the middle of the paper that only repeats the previous section. I do not see how the future directions are related to the previous discussions. I would put the section at the end.\n\nO12: What does HIT mean in Section D?", "questions_to_authors": "Q1: Can you adapt state-of-the-art algorithms to your dataset? Which ones and how?\n\nQ2: How is the input of the models formatted?\n\nQ3: More generally, answer the remarks above.", "paper_topic_and_main_contributions": "In this paper, the authors introduce a new dataset for temporal question answering over Wikipedia's InfoBox. They describe how the dataset is created, give statistics, and run baselines with large language models. Overall, it is a valuable resource for the community, even if some points in the paper, like the evaluation of the dataset and the baselines, can be improved.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Nothing", "Typos_Grammar_Style_and_Presentation_Improvements": "Nothing"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an innovative temporal question-answering task along with a dataset designed for semi-structured tables (info-boxes), which is a well-justified endeavor. The authors' commitment to elaborate more about the motivation behind this task, in response to reviewers' suggestions, is noteworthy. The paper's clarity in presentation and the comprehensive explanation of the data collection process are also notable strengths. Furthermore, the thorough and easily understandable future work section greatly enhances the paper's significance and its contribution to the research community.

I believe that the paper is of high quality. However, I think that if the authors had suggested some modeling modifications to existing methods in order to solve the task, it would have been a great nominee for the best paper award.
Decision: Accept-Main