Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering
Authors: Xin Su
Phillip Howard
Nagib Hakim
Steven Bethard
Venue: EMNLP 2023 Findings (2023)
Keywords: Time-sensitive Question Answering
Temporal Graph Fusion
Temporal Reasoning

Abstract: Answering time-sensitive questions from long documents requires temporal reasoning over the times in questions and documents. An important open question is whether large language models can perform such reasoning solely using a provided text document, or whether they can benefit from additional temporal information extracted using other systems. We address this research question by applying existing temporal information extraction systems to construct temporal graphs of events, times, and temporal relations in questions and documents. We then investigate different approaches for fusing these graphs into Transformer models. Experimental results show that our proposed approach for fusing temporal graphs into input text substantially enhances the temporal reasoning capabilities of Transformer models with or without fine-tuning. Additionally, our proposed method outperforms various graph convolution-based approaches and establishes a new state-of-the-art performance on SituatedQA and three splits of TimeQA.

Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering
Xin Sul.2
Phillip Howard
Nagib Hakim² Steven Bethard'
Uniierriiyy of Arizona
Intel Labs
(xinsu, bethardj@arizona.com edu, {phillip.r. howard, nagib. wawimjeintee.ccom com
Abstract
Context Document
George Washington (Fehnurry 22, 1732 December 14, 1799) was an American political leader. .. ULDK (hundred wards)
Answering time-sensitive questions from long documents requires temporal reasoning over the times questions and documents. An portant open question is whether large language models can perform such reasoning solely us- ing provided text document, whether they can benefit from additional temporal informa- tion extracted using other systems. We address this research question by applying existing tem- poral information extraction systems to con- struct temporal graphs of events, times, and temporal relations in questions and documents. We then investigate different approaches for fusing these graphs into Transformer models. Experimental results show that our proposed approach for fusing temporal graphs into in- put text substantially enhances the temporal reasoning capabilities of Transformer models with or without fine-tuning. Additionally, our proposed method outperforms various graph convolution baasle approaches and establishes a new state-of-the-an performance on Situat- edQA and three splits of TimeQA.
Comtinntal Army on Samuel and John Adams nominsted Washington become Commander in Chief. TLDR (handeed works)
Congress
1775,
Washington farewell to officers Fraunces December 1783 and resigned his commission days later. 1788, Board Visitors of the College of William & Mary decided to re-establish position of Chancellor, elected Washington to the oflice en January 18. TLDR (handred wards)
He started as the president United Sates Jan 1788, 1798, one year that, stepped his presidency position.
QT: What was George Washington's 's position between 1776 Commander-ine
What Gearge Washington' position from 1790 1797?
A: Presidency, Chancellor
Figure 1: Time-sensitive question-ansee answer pairs with context document. The times and answers are in red and blue, respectively.
throughout the context document. Additionally, this example illustrates how changing the time ex- pression in the question may also result in change in the answer: in this case, replacing between 1776 1780 with from 1790 to 1797 changes the an- swer from Commander-in- inChief to Presidency and Chancellor.
1 Introduction
Long- Long-document time-sensitive question answering (Chen et al., 2021) requires temporal reasoning over the events and times in question and an accompanying long context document. Answer- ing such questions is challenging task in natural language processing (NLP) models must com- prehend and interpret the temporal scope of the question as well as the associated temporal infor- mation dispersed throughout the long document. For example, consider the time-sensitive questions about George Washington's 's position provided in Figure 1. The relevant events and temporal infor- mation regarding George Washington's position are scattered across many different sentences in the context document. Since there no one sin- gle text segment containing the answer, the model must integrate and reason over events and times
Though not designed directly for question an- swering, there a substantial amount of research on temporal information extraction (Chambers et al., 2014; Ning et al., 2018a; Zhang and Xue, 2018, 2019; Han et al., 2019; Ning et al., 2019; Vashishtha et al., 2019; Ballesteros et al., 2020: Yao al., 2020; Zhang al., 2022a). Such models can help reveal the structure of the timeline under- lying a document. However, there little existing research on combining such information extraction systems with question answering Transformer mod- els (lzacard and Grave, 2021) to effectively reason over temporal information in long documents.
In this work, we utilize existing temporal infor- mation extraction systems to construct temporal

---

graphs and investigate different fusion methods to inject them into Transformer models.
(2023) use time-aware embeddings reason over temporal knowledge graphs. Similar to our work, Huang et al. (2022) and Shang et al. (2021) an- swer temporal questions on text, but they focus on temporal event ordering questions over short texts rather than time-sensitive questions over long documents. Li et al. (2023) focus on exploring large language models for information extraction in structured temporal contexts. They represent the extracted time-sensitive information in code and then execute Python scripts to derive the an- swers. In contrast, we concentrate on temporal reasoning in the reading comprehension setting, using unstructured long documents to deduce an- swers. This poses more challenges in informa- tion extraction and involves more complex rea- soning, which motivates our integration of exist- ing temporal information extraction systems with transformer-bast language models. The most sim- ilar work to ours is Mathur et al. (2022), which extracts temporal dependency graphs and merges them with Transformer models using learnable at- tention mask weights. We compare directly to this approach, and also explore both graph convolutions and input modifications alternatives to fusing temporal graphs into Transformer models.
We evaluate the effectiivnness of each tempo- ral graph fusion approach on long-document time- sensitive question answering datasets. Our contri- butions are as follows:
1. We introduce simple but novel approach to fuse temporal graphs into the input text of question answering Transformer models.
2. We compare our method with prior ap- proaches such as fusion via graph convolu- tions, and show that our input fusion method outperforms these alternative approaches. 3. We demonstrate that our input fusion approach can be used seamlessly with large language models in an in-context learning setting.
4. We perform detailed error analysis, revealing the efficacy of our method in fixing temporal reasoning errors in Transformer models.
2 Related Work
2.1 Extracting Temporal Graphs
Research extracting temporal graphs from text can be grouped into the extraction of event and time graphs (Chambers al., 2014; Ning al., 2018b), contextualized event graphs (Madaan and Yang. 2021),, and temporal dependency trees and graphs (Zhang and Xue, 2018, 2019; Yao et al., 2020; Ross et al., 2020; Choubey and Huang, 2022; Mathur et al., 2022). Additionally, some prior work has focused on the problem of extracting temporal relations between times and events (Ning al., 2018a, 2019; Vashishtha et al., 2019; Han al., 2019; Ballesteros et al., 2020; Zhang et al., 2022a). The outputs of these temporal relation extraction systems are often used to construct temporal graphs. In this work, we use CAEVO (Chambers et al., 2014) and SUTime (Chang and Manning, 2012) to construct our temporal graphs because they are publicly available (unlike more recent models such the temporal dependency graph parser proposed by Mathur et al. (2022)) and can easily scale to large amounts long documents without requiring additional training.
2.3 Fusing Graphs into Transformer Models
The most common approaches for fusing graphs into Transformer models are graph neural networks (GNN) and self-attention. the GNN-based ap- proach, a GNN is used to encode and learn graph representations which are then fused into the Trans- former model ( (Yang et al., 2019; Feng et al., 2020; Yasunaga et al., 2021: Zhang et al., 2022b). In the self-attention approach, the relations in the graphs are converted into token-to-token relations and are then fused into the self-attention mecha- nism. For example, Wang et al. (2020) uses relative position encoding (Shaw et al., 2018) to encode a database schema graph into the BERT representa- tion. Similarly, Bai et al. (2021) utilize attention masks to fuse syntax trees into Transformer mod- cls. We explore based fusion of temporal graphs into question answering models, compar- ing this approach to the attention-based approach of Mathur et al. (2022), as well as our simpler ap- proach which fuses the temporal graph directly into the Transformer model's input.
2.2 Temporal Question Answering
Jia al. (2018) decomposes questions and ap- plies temporal constraints to allow general question- answering systems to answer knowledge-bol temporal questions. Saxena et al. (2021), Jia et al. (2021), Mavromatis et al. (2021), ) and Sharma et al.

---

Question ?
ereated
before
bade firewell
sfler
Context Document
Insteded by
beting
Regledee by:
Encoding
Graph
Construction
June 1975
before
Question Timee:
1776 1780
aller
1788
Transformer
Temporal Graph
resigned
Figure 3: Temporal graph example.
Gruph Fusion
ERR / x
GNN
tify the events, time expressions, and the temporal relations between them. CAEVO follows the stan- dard convention temporal information extraction that events are only simple actions (typically verbs) with linking of these actions to subjects, objects, and other arguments left to dependency parsers (Pustejovsky et al., 2003; Verhagen et al., 2009, 2010; UzZaman et al., 2013). We add document- event and document-time nodes for each identified event and time, respectively, and add edges be- tween nodes for each identified temporal relation.
Answer
Trowe
Figure 2: Overview of our approach. The ERR method allows for optional fine-tuning of the Transformer model, whereas for the GNN method requires fine- tuning.
3 Method
Our approach applies temporal information extrac- tion systems to construct temporal graphs and then fuses the graphs into pre-trained Transformer mod- els. We consider two fusion methods:
To link the question-ting node to the document- time nodes, we use SUTime? to normalize time expressions to time intervals, and deterministi- cally compute temporal relations between question time and document times edges. For example, given document-time node "the year 2022" and question-tin time node "from 1789 1797" from the question "What was George Washington's position from 1789 to 1797?", the times will be normal- ized to [2022-01-01, 2022-12-31] and [1789-01-01, 1797-12-31] respectively, and the temporal relation between them can then be computed as AFTER. To link the question-time node to document events, for each document-event node, we calcu- late the shortest path in the temporal graph between and the question -iim node and recursively ap- ply standard transitivity rules (see Appendix A. along the path to infer the temporal relation. For example, given path A is BEFORE B and B IN- CLUDES C, and C is BEFORE. An example of constructed temporal graph for Q1 in Figure is illustrated in Figure 3.
1. Explicit edge representation fusion (ERR): simple but novel approach to fuse the graphs into the input text.
2. Graph neural network fusion: GNN is used to fuse the graphs into the token embed- dings or the last hidden layer representations (i.e., contextualized embeddings) of the Trans- former model.
The overall approach is illustrated in Figure 2.
3.1 Graph Construction
Given time-sensitive question and correspond- ing context document, we construct a directed tem- poral graph where events and time expressions are nodes and temporal relations are edges of the type BEFORE, AFTER, INCLUDES, INCLUDED BY, SIMULTANEOUS, or OVERLAP.
infer the relation between A
We extract the single timestamp included in each question, which is either explicitly provided by the dataset (as in SituatedQA (Zhang and Choi, 2021)) or alternatively extracted via simple regular ex- pressions (the regular expressions we use achieve 100% extraction accuracy on TimeQA (Chen et al., 2021)). We add single question-time node to the graph for this time.
3.2 Graph Fusion
For both fusion methods, we concatenate the ques- tion and corresponding context document as an in- put sequence to the Transformer model. For exam- ple, given the question and document from Figure QI, the input is:
For the document, apply CAEVO! to iden-
hhttps: htttp:://nlp.stanfordd stanford, cotu/software//uutii html

---

question: What was George Washington's position between 1776- 1780? context: Congress created the Continental Army on June 14, 1775
where (i) denotes all neighbor nodes that have relation with node i, ci. normalization con- stant that can be learned or manually specified, o is the activation function, Wo is the self-loop weight, and W, is the learnable weights. We refer readers to Schlichtkrull et al. (2018) for more details.
3.2.1 Explicit Edge Representation
In the ERR method, we mark a temporal graph's 's nodes and edges in the input sequence, using <question time> and </question time> to mark the question-time node and relation markers such as <before> and </before> to mark the nodes in the context document and their relations to the question time. Thus, the ERR input for the above example is:
4
Experiments
4.1
Datasets
We evaluate on two time-sensitive question answer- ing datasets:
SSn-iSeni Question Answering (TimeQA; Chen et al., 2021) and SituatedQA (Zhang and Choi, 2021). We briefly describe these two datasets below and provide statistics on each dataset in Table 9 of Appendix A.6.
question: What was George Washing- ton's 's position <question time>between 1776- 1780~/question time>? context: Congress <befoore>creattet<//bfoorz the Continental Army <before>June 14, 1775</before>...
TimeQA The TimeQA dataset is comprised of time-sensitive questions about time-evolving facts paired with long Wikipedia pages as context. The dataset has two non-overlapping splits generated by diverse templates which are referred to as TimeQA Easy and TimeQA Hard, with each split containing 20k question cneestion-answer answer pairs regarding 5.5K time- evolving facts and 70 relations. TimeQA Easy con- tains questions which typically include time expres- sions that are explicitly mentioned in the context document. In contrast, TimeQA Hard has time ex- pressions that are not explicitly specified and there- fore require more advanced temporal reasoning, For example, both questions in Figure 1 are hard questions, but if we replace the time expressions in the questions with In 1788, they will become easy questions. In addition, smaller Human- Human-paraphent Easy and Hard splits are also provided.
This approach aims to make the model learn to attend to parts of the input sequence that may contain answer information. For instance, the model may learn that information related to the answer may be found near markers such as <overlap>, <includes>, <included by>, and <simult taneous>. Additionally, the model may learn that answer-related information may exist be- tween <before> and <after>, even if the answer does not have any nearby temporal information.
3.2.2 GNN-based Fusion
In GNN- fusion, we add <e> and </e> mark- ers around each node, and apply a relational graph convolution (ReIGraphConv; Schlichtkrull et al., 2018) over the marked nodes. ReIGraphConv is a variant of graph convolution (GCN; Kipf and Welling, 2017) that can learn different transforma- tions for different relation types. We employ the ReIGraphConv to encode temporal graph and update the Transformer encoder's token embed- ding layer or last hidden layer representations (i.e., contextualized embeddings). We utilize the Rel- GraphConv in its original form without any modifi- cations.
SituatedQA SituatedQA is an open-domain question answering dataset comprising two sub- sets: Temporal SituatedQA and Geographical Situ- atedQA. We focus on Temporal SituatedQA, which we will hereafter refer to as SituatedQA. Each ques- tion in SituatedQA is accompanied by a temporal annotation that could change the answer to the ques- tion if it is modified. For instance, the question "Which COVID-19 19 vaccines have been authorized for adults in the US as of Jan 2021?" has corre- sponding answer of "Moderna, Pfizer." However, if we change the time to "Apr 10, 2021," 1," the an- swer becomes "Moderna, Pfizer, J&J." As Situat- edQA is re-annotation of a subset of NQ-Open (Kwiatkowski et al., 2019 99; Lee et al., 2019), we use the top 100 passages retrieved from Wikipedia
Formally, given temporal graph G = (V, E). we use representations of the <e> markers from the Transformer model's token embedding layer or the last hidden layer as initial node embeddings. The output of layer 1 for node E is:
11+1 - a Σ
Σ
1
W + (
Crr
(E(()ENT()

---

Method
Model Size
External Tool (Public Avail.)
Add. Data
Knowl, Fusion
BigBird (Chen et al., 2021)
128M
NaturalQ
FiD & Replicated FiD (Chen al., 2021)
220M
NaturalQ
DucTime (Mathur al., 2022)
220M
CAEVO (V), TDGP (X)
NaturalQ
Attention
BigRind MTL (Chen al., 2023a)
128M
TT (V), TR (X)
TriviaQA
Multi-Tase
Longformer MTL (Chen et al., 2023a)
435M
TT (+), TR (X)
TriviaQA
Multi-Task
DPR Query Modified (Zhang and Choi, 2021)
110 M
Question Text
TSM (Cole et al., 2023)
11B
SUTime (V) SUTime (V)
Entire Wiki
Pre- Training
TRR (Ours)
250M
NaturalQ
Input Text
Table 1: Comparison of methods, base models' size, external tools, additional data, and temporal knowledge fusion methods. NaturalQ: Natural Questions, TDGP: Temporal Dependency Graph Parser, TT: BERT-based Temporal Tagger, TR: timestamps retriever.
by FiD (Izacard and Grave, 2021>² for each ques- tion as context documents.
temporal dependency graphs are fused into the attention mechanism of the FiD model, which is fine-tuned on Natural Questions.
Evaluation Metries We use the official evalu- ation methods and metrics provided in the code release of the datasets. For TimeQA, we report the exact match and FI scores. For SituatedQA, report the exact match score.
BigBird + MTL & Longformer + MTL Chen et al. (2023a) inject temporal information into long text question answering systems using multi-task learning (MTL) approach. They train BigBird and Longformer models on time-sensitive question answering tasks, along with three auxiliary temporal-awareness tasks. They explore first fine-tuning on Natural Questions and TriviaQA datasets, finding that TriviaQA results in the best performance. BigBird + MTL is their best model on TimeQA Easy, while Longformer MTL performs best TimeQA Hard.
4.2 Baselines
We compare our models with the previous state-of- the-art on the TimeQA and SituatedQA, which we describe in this section and summarize Table 1. For TimeQA, we compare against:
FiD & BigBird Chen et al. (2021) adapt two long- document question answering models, BigBird (Zaheer et al., 2020) and FiD (Izacard and Grave, 2021), to the TimeQA dataset. Before fine- tuning the models on TimeQA, they also fine- tune on Natural Questions (Kwiatkowski al., 2019b) and TriviaQA (Joshi et al., 2017), find- ing that training on Natural Questions results in the best performance. The best model from Chen et al. (2021) on TimeQA Easy and Hard is FiD, while Bird performs best on the Human- paraphrased TimeQA Easy and Hard splits. Replicated FiD We also report replication of Chen al. (2021)'s FiD model using the code provided by their GitHub repository but even with extensive hyperparameter tuning, we were unable to reproduce their reported performance of the FiD model on TimeQA Easy$.
For SituatedQA, we compare against: DPR + Query Modified Zhang and Choi (2021) adapt retrieval-based model, DPR (Karpukhin et al., 2020), for SituatedQA. They first fine-tune the retriever and reader of DPR on Natural Ques- tions and then further fine-tune on SituatedQA. TSM (Cole et al., 2023) uses SuTime to iden- tify and mask time expressions throughout all Wikipedia documents. They then fine-tune T5- 1.1-XXL (Raffel et al., 2020) to predict the masked time expressions. Finally, they fine-tune the resulting T5-1,1-XXL model on SituatedQA.
4.3 Implementation Details
We use LongT5-bass (Guo et al., 2022), a trans- former sequence-to-sequee model. Our experiments demonstrate that LongT5 outperforms the FiD model (Izacard and Grave, 2020) commonly used long-document question answering tasks. To fairly compare with previous work (Chen al., 2021; Mathur et al., 2022), we pre- pre-train the LongT5 model on Natural Questions (K wiittkowski et al., 2019a) and then fine-tune it
DocTime Tome (Mathur et al., 2022) first uses CAEVO to identify events and time expressions in context documents. Then, custom-trained temporal dependency parser is applied to parse temporal dependency graphs. Finally, the parsed
model, as our base
Phttps: //github.ttttttt com//facebookresserch/ftl "wenhuchen/ Secchuuchhn/mmmm--See Time-Tmmeennn Semn-tensitiveQQA *Other rescarchers have reported the same issue on GitHub

---

TimeQA Easy
TimeQA Hard
TimeQAqp Easy
TimeQAup Hard SituatedQA
RM
F1
EM
EM
FM
BigBird (Chen et ul., 2021)
51.2
60.3
42.4
50.9
47.6
53.7
38.8
45.9
FID (Chen et al., 2021)
60.5
67.9
46.8
54.6
DoeTime (Mathur et al., 2022)
62.4
69.6
48.2
56.4
BigBird TML (Chen al., 2023b)
55.4
63.8
Longformer+ TML (Chen
2023b)
52.4
62.6
48.7
58.5
DPR + Query Modified (Zhang and Choi, 2021)
23.0
TSM (Cole et al., 2023)
24.6
Replicated FiD
55.3
65.1
45.2
54.5
52.7
59.7
39.7
47.0
23.7
LongT5
55.4
64.3
49.9
58.4
53.7
61.6
45.3
52.3
24.7
LSNg LARR
54.2
63.4
49.0
57.6
46.5
55.3
34.3
40.2
27.3
56.9
66.2
54.0
62.0
54.9
62.9
49.3
56.1
29.0
Replicuted FiD (ne-annotuted)
64.0
68.3
LoRgTTERR (re-annotated)
68.8
70.4
Table 2: Performance the test sets. TimeQApe denotes the human-pane caanan-prraprassed splits of TimeQA. The highest- performing model is bolded. Confidence intervals for our results are provided in table 7 in appendix A.3.
TimeQA and SituatedQA, respectively. Ap- pendix A.5 provides other implementation details such as hyperparameters, graph statistics, software versions, and external tool performance.
TimeQA, the LongT3 model fails to outper- form LongT5 without fusion, while the LongT'S model improves over LongT5 on every split and dataset, exhibiting particularly large gains on the Hard splits. On Situated QA, both LongT5ERd erR and LongT3 models improve over the no-fusion LongT5 baseline, with ERR again providing the best performance. The somewhat inconsistent performance of the GNN fusion method across datasets (beneficial on SituatedQA while detrimen- tal on TimeQA) suggests the need for different GNN design for TimeQA, which we leave to future work.
We perform model selection before evaluating on the test sets, exploring different graph subsets with both the ERR and GNN based fusion ap- proaches introduced in Section 3.2. Table 6 in appendix A.2 shows that the best ERR method uses
ttw.-t-------ueetton--iii--i (DT2QT) subgraph and the best GNN method uses the full temporal graph by fusing it into the token em- bedding layer representations of the Transformer model. We hereafter refer to the LongT5 model fused with DT2QT graph using the ERR method LongT5 and the LongT5 model fused with full temporal graph using the GNN method as LongT5 GNN-
To explore the differences between Long ERR and LongT5 models, we analyze 20 randomly sampled examples from TimeQA Hard where LongT5 FRR correct but LongT5 GNN is incorrect. From manual analysis of these 20 examples, all 20 examples share the same pattern: LongT5 fails to capture explicit temporal expressions the context and relate them to the question's time- line, which is crucial for deducing the right an- swer. This suggests that directly embedding pre- computed temporal relations between time nodes into the input is more efficient than implicitly doing so through the GNN, allowing the model utilize them more easily. Table 14 of appendix A.1l shows three the analyzed examples.
4.4 Main Results
We summarize the performance baseline models and those trained with our graph fusion methods in Table 2.
Which baseline models perform best? On TimeQA, our LongT5 model without temporal graph fusion performs better than or equivalent to all other baseline models across every split and met- ric except for the Easy split. The best-performing model reported on TimeQA Easy is DocTime. On SituatedOA, LongT5 with no fusion performs as well as the best-reported results on this dataset.
Does our approach outperform prior work? On TimeQA, the LongT5 model achieves new state-of-the-art three of the four splits, with TimeQA Easy being the exception. On SituatedQA. the LongT5 model achieves new state-of-the- art, outperforming the best reported results on this dataset. Our model excels on datasets that require
Which graph fusion methods perform best? Using L5, we consider both of our ERR and GNN fusion methods described in Section 3.2. On

---

Type
Freq #
Question
Relevant Context
Prediction
Answer
Which team did the player Rivaldo belong to from 1996 1997?
After the Olympics Rivaldo moved Spain as he joined Deportivo Coruña
Deportivo
Deportivo.
La Coruña.
La Liga.
False Negative
What the position of Albert Reynolds from Dec 1992 Nov 1994?
Albert Reynolds politician.. .... Leader of Fiamma Fäil from 1992 1994... He served Teachta Dála (TD) from 1977 to 2002.
an Irish Hanna Häil
Leader of Fi- anna Fäil.
Teachta
Dāla.
Semantic Understanding
3
3 Lucas Papademos was an employee for whom 2010 to 2011?
In 2010 he served as an economic advisor to Greek Minister George Paapandroou.
N/A.
Greek Prime
Minister
George
Pupandreou.
Semantic Understanding & Insufficieut Temporal Information
Which team did the player Miguel Veloso belong from 2001 to 2003?
... Veloso started his football career at S.L. Benfica. but was rejected for being slightly overweight at the time...
Benfica.
N/A.
Semantic Understanding & Temporal Reasoning
What operated LMR 57
Lion was ordered by the Liverpool & Manch- ester Railway October 1837 In 1845 the LMR was absorbed by the Grand Junction Railway (GJR), which turn
London and North West- ern Railway.
Liverpool &
Lion from 1838 1845?
Manchester Railway.
one the constitucats of the London and North Western Railway (LNNWR) year later.
Table 3: Error categories and examples the TimeQA Easy development set. Model predictions are in bold. Underlines denote the correct answers.
more temporal reasoning skills, like TimeQA Hard (where our model achieves 5.8-point higher ex- act match score than DocTime) and SituatedQA (where our model achieves a 4.3-po higher exact match score than TSM).
applicable in the given context but are not included in the gold annotations (as in Table 3 example 2). The remaining errors are primarily related to se- mantic understanding, including the inability to accurately identify answer entities (e.g. identify- ing Greek Prime Minister George Papandreou as an employer in Table 3 example 3), the inability to interpret negation (e.g. in Table 3 example 4, where "rejected" implies that Veloso did not join Benfica), and the inability reason about time with numerical calculations (e.g. "a year later" Table example implies 1846). Addressing the semantic understanding errors may require incor- porating additional entities and their types into the graphs, as well as better processing of negation information and relative times.
Our approach offers further advantages over al- ternative models due to its simplicity, summa- rized in Table 1. The best prior work on TimeQA Easy, DocTime, requires training temporal de- pendency parser on additional data, using CAEVO, and modifying the Transformer model's attention mechanism. The best prior work on SituatedQA, TSM, requires an 11-billion parameter T5 model which is pre-trained on the entirety of Wikipedia. In contrast, our approach only uses SUTime to con- struct graph, requires only minor adjustments to the Transformer model's input, and outperforms prior work using a model with only 250 million parameters.
To better understand the extent of false nega- tives in TimeQA Easy, we re-annotated the 392 test examples where the predictions of the repli- cated FiD model and our Long PRR model are partially correct (i.e., EM 0 and F1 > 0). We then incorporated additional coreferent mentions into the gold label set for these examples. For in- stance, if the original gold answer was "University of California," we added its coreferent mention "University of California, Los Angeles" to the gold answers. We then evaluate both the replicated FiD (the best-performing model we can reproduce) and our Long model on the re-annotated TimeQA Easy split. The last two rows of Table 2 show that while the exact match score for FiD increases by 8.7, the exact match score for our Long ERR
Why does our model not achieve state-of-art per- formance on TimeQA Easy as it does on other splits and datasets? On TimeQA Easy, there is a performance gap between our LongT5pRe model and DocTime. Because the DocTime model has not been released we cannot directly compare with its predicted results. Instead, we randomly select 50 er- rors from LongT5 ERR 's output the TimeQA Easy development set for error analysis. Table shows that most of the errors are false negatives, where the model's predicted answers are typically co-references to the correct answers (as in Table example 1) or additional correct answers that are

---

Model
TimeQA Easy
TineQA Hard
LongTSERR
FiD
Time QA Easy
TimeQA Hard
EM
F1
EM
F1
Correct
Correct
1427
1113
ChatGPT w/ FRR DTTQT Suhgraph
41.6
56.6
25.0
32.0
Correct
Wrong
269
494
43.2
57.2
30.4
38.5
Wrong
Correct
243
260
Wrong
Wrong
1082
1220
Table 4: Performance of ChatGPT on 500 sampled test examples, with and without our ERR method.
Table 5: Comparison the predictions of LongT's the predictions of FiD.
model increases by 11.9. This suggests that our model may be incurring greater penalties for find- ing valid coreferent answers than baseline methods.
TimeQA Clompatiion
TimeUA Exyy TimeiJA Iized
30-
Does our ERR method benefit large language models using in-context learning? We have fo- cused far on temporal graph fusion when fine- tuning models, but large language models such as ChatGPT (OpenAI, 2022) and LLaMA (Touvron et al., 2023) can achieve impressive performance without additional fine-tuning via in-context learn- ing. Therefore, we tested the performance of Chat- GPT (gpt-3.5-turbo) both with and without ERR for fusing the Following previous work (Khattab et al., 2022; Trivedi et al., 2022: Yoran al., 2023) and con- sidering the cost of ChatGPT's commercial API, we randomly sample 500 examples from TimeQA Easy and TimeQA Hard for evaluation. The prompt format for ChatGPT remains the same as the in- put format described in Section 3, except that we concatenate in-context learning few-shot exem- plars and task instructions before the input. We evaluate ChatGPT with and without graph fusion using an 8-shot setting. Examples of prompts are provided Table shows that our ERR graph fusion method improves the performance of ChatGPT on TimeQA Easy and particularly on TimeQA Hard. We note that this improvement is possible because our method can easily integrate with satte-of-the-at large lan- approach to temporal graph
Detta EM (LongTSpss
2000
Context Lengt Tokums)
graph.
Figure 4: Comparison of the performance on different lengths of context documents.
each other's errors on TimeQA Easy (269 vs. 243), Long ERR corrects many more of FiD's errors than the reverse on TimeQA Hard (494 vs. 260). To further analyze these cases, sampled 10 errors from the set where LongT5 ERR correct while FiD was incorrect well the where the FiD was correct while LongTT5 was incorrect. We did this across both TimeQA Easy and TimeQA Hard, totaling 40 examples. Among the 20 exam- ples in which LongT5 ERR was correct and FiD was incorrect, 17 have node markers near the answers in the ERR input sequence, and the most frequent ones are <included by> and <overlap>. The maining 3 examples have unanswerable questions. In the examples in which FiD was correct while LongT5ERe ERR was incorrect, we observe that 13 ex- amples are additional correct answers (i.c., false negatives), while the other 7 examples are semantic understanding errors similar to those discussed pre- viously. These results suggest that our ERR graph fusion approach is providing the model with useful targets for attention which allow it to produce more correct answers.
Appendix A.8. Table
guage models, fusion modifies only the input sequence. Prior work which relies on modifying attention mechanisms or adding graph neural network layers incompatible with this in-context learning setting.
5 Analysis
In this section, we analyze our LongT5TRR model on the TimeQA development set.
How do predictions differ compared to FiD? We compare the predictions LongT5 to the replicated FiD model Table While LongT5 and FiD both correct about the same number of
How does the length of the document affect performance? We compare the performance of LongT5 to the replicated FiD model on var- ious document lengths, as depicted in Figure 4.

---

LongTTERR performs less competitively than FiD on the Easy split for longer documents, This could be attributed to high frequency of false negatives in ERR as discussed previously. Addition- ally, it could be that LongT'ERR is less efficient at string matching on longer documents than FiD. Most of the question times in the Easy split are ex- plicitly mentioned in the context document, which be solved via string matching rather than tem- poral reasoning. However, our LongTSERR model shows substantial improvement on TimeQA Hard, outperforming the FiD model across most lengths.
input-based approach to fusion.
We also did not deeply explore the places where graph neural nerworks failed. For example, the graph convolution over the final layer contextu- alized embeddings using the full temporal graph yielded performance much lower than all other graph convolution variants we explored. We lim- ited our exploration of failures to the more success- ful explicit edge representation models.
Ethics Statement
Wikipedia contains certain biases (Falenska and Çetinoğlu, 2021), and we use data from Wikipedia to train our model, thus are potentially introduc- ing similar biases into our models.
6 Conclusion
In this paper, we compared different methods for fusing temporal graphs into Transformer models for time-sensitive question answering. We found that our ERR method, which fuses the temporal graph into the Transformer model's input, outper- forms GNN-based fusion as well attention-based fusion models. We also showed that, unlike prior work on temporal graph fusion, our approach is compatible with in-context learning and yields im- provements when applied to large language models such as ChatGPT. Our work establishes a promis- ing research direction on fusing structured graphs with the inputs of Transformer models. In future work, we intend use better- pertorrner informa- tion extraction systems construct our temporal graphs, enhance our approach by adding entities and entity type information to the graphs, and ex- tend our method to spatial reasoning.
References
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- BERT: Improving pre-trained transformers with syn- tax trees. In Proceedings of the 16th Conference of the European Chapter the Association for Compu- tational Linguistics: Main Volume, pages 301 1-3020, Online. Association for Computational Linguistics.
Miguel Ballesteros, Rishita Anubhai, Shuai Wang. Nima Pourdamghani. Yogarshi Vyas, Jie Ma. Par- minder Bhatia, Kathleen McKeown, and Yaser Al- Onaizan. 2020. Severing the edge between before and after: Neural architectures for temporal ordering of events. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5412-5417, Online, Association for Computational Linguistics.
Nathanacl Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard, 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273- 284.
Limitations
We use CAEVO and SUTime to construct temporal graphs because of their scalability and availability. Using more accurate neural network-based tempo- ral information extraction tools may provide better temporal graphs but may require domain adaptation and retraining.
Angel X. Chang and Christopher Manning. 2012. SU- Time: A library for recognizing and normalizing time expressions. In Proceedings of the Eighth In- lernational Conference on Language Resources and Evaluation (IRFFC' pages 3735-3740, Istanbul, Turkey, European Language Resources Association (ELRA).
While we did not find graph convolutions to yield successful fusion on TimeQA, we mainly explored variations of such methods proposed by prior work. We also did not explore self-attention based fusion methods, as preliminary experiments with those methods yielded no gains, and Time provided an instance of such methods for compari- son. But there may exist other variations of graph convolution and self-attention based fusion meth- ods beyond those used in prior work that would make such methods more competitive with our
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive ques- tions. In Thirty-fftth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
Ziqiang Chen, Shaojuan WU, Xiaowang Zhang, and Zhiyong Feng. 2023a. Tml: A temporal-aware multi- task learning framework for time-sensitive question answering. In Companion Proceedings of the ACM

---

Web Conference 2023, WWW '23 Companion, page 200-203, New York, NY, USA. Association for Com- puting Machinery.
temporal reading comprehension via precise question understanding. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 375-384, Seattle, United States. Association for Computational Linguistics.
Ziqiang Chen, Shaojuan Wu, Xiaowang Zhang, and Zhiyong Feng. 2023b. Tml: temporal-aware mul- titask learning framework for time-sensitive question answering. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 200-203, New York, NY. USA. Association for Com- puting Machinery.
Gautier Izacard and Edouard Grave 2020. Leveraging passage retrieval with generative models for open domain question answering.
Gautier Izacard and Edouard Grave 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguisticx: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Prafulla Kumar Choubey and Ruihong Huang. 2022. Modeling document-level temporal structures for building temporal dependency graphs. In Proceed- ings of the 2nd Conference of the Saci-Pacific Chap- ter of the Association for Computational Linguistics and the 12th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 357-365, Online only. Association for Compu- tational Linguistics.
Zhen Jia, Abdalghani Abujabal. Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018. Tequila: Temporal question answering over knowledge bases. In Proceedings of the 27th ACM International Con- ference on Information and Knowledge Management, CIKM 118 page I 1807-1810, New York, NY, USA. Association for Computing Machinery.
Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar. 2023. Salient span masking for temporal understanding. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 3052- 3060, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM International Conference on Informa- tion & Knowledge Management, CIKM '21, page 792-802, New York, NY, USA. Association for Com- puting Machinery.
Agnieszka Falenska and Özlem Çetinoğlu. 2021. As- sessing gender bias in Wikipedia: Inequalities in article titles. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 75-85, Online. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers), pages 1601-161.. I, Vancouver, Canada. Association for Computational Linguisties.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi- hop relational reasoning for knowledge-awee tion answering. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309, Online. As- sociation for Computational Linguistics.
ques-
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022. pages 724- 736. Seattle, United States. Association for Compu- tational Linguistics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search- predict: Composing retrieval and language mod- els for knowledge-intensse intensive nlp. arXiv preprint 14X14:2212.14024.
Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint event and temporal relation extraction with shared representations and structured prediction. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLPP pages 434-444, Hong Kong, China. Association for Computational Linguis- tics.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In International Conference on Learning Representations.
Hao Huang, Xiubo Geng, Guodong Long, and Daxin Jiang. 2022. Understand before answer: Improve
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti,

---

Danielle Epstein, Iilia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wa Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019a. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-UCCNL)), pages 6203-6209, Hong Kong, China. Association for Com- putational Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018a. A multi- axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 118--328, Mclbourne, Aus- tralia. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Keleey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019b. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Com- putational Linguistics, 7:453-466.
Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018b. CogCompTime: A tool for under- standing time in natural language. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstra- rions, pages 72-77, Brussels, Belgium. Association for Computational Linguistics.
Kenton Lee, Ming-Wor Wei Chang. and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
OpenAL. 2022. ChatGPT: Optimizing language models for dialogue. OpenAl.
Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng. Shafiq Joty, and Lidong Bing. 2023. Unlocking tem- poral question answering for large language models using code execution.
James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003. The timebank corpus. In Corpus linguistics. volume 2003, page 40. Lancaster, UK.
Aman Madaan and Yiming Yang. 2021. Neural lan- guage modeling for contextualized temporal graph generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 864-881, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 5.511.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguis- tics: System Demonstrations, pages 55 55-60, Balti- more, Maryland. Association for Computational Lin- guistics.
Hayley Ross, Jonathon Cai, and Bonan Min. 2020. Ex- ploring Contextualized Neural Language Models for Temporal Dependency Parsing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 8548-8553, Online. Association for Computational Linguistics.
Apoory Saxena, Soumen Chakrabarti, and Partha Taluk- dar. 2021. Question answering over temporal knowl- edge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 6663-6676, Online, Association for Computational Linguistics.
Puneet Mathur, Vlad Morariu, Verena Kaynig-Fittku, Jiuxiang Gu, Franck Dernoncourt, Quan Tran, Ani Nenkova, Dinesh Manocha, and Rajiv Jain. 2022. DocTime: A document-level temporal dependency graph parser. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 993-1009, Seattle, United States. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem. Rianne van den Berg. Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web, pages 593- 607, Cham. Springer International Publishing.
Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Joannidis, Soji Adeshina, Phillip Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2021. Tempoqr: Temporal question reasoning over knowledge graphs. In AAAI Conference Artificial Intelligence.
Chao Shang, Peng Qi. Guangtao Wang. Jing Huang. Youzheng Wu, and Bowen Zhou. 2021. Open tem- poral relation extraction for question answering. In 3rd Conference on Automated Knowledge Base Con- struction.
Qiang Ning. Sanjay Subramanian, and Dan Roth. 2019. An improved neural baseline for temporal relation

---

Aditya Sharma, Apoorv Saxena, Chitrank Gupta, Mehran Kazemi, Partha Talukdar, and Soumen Chakrabarti. 2023. TwiRGCN: Temporally weighted graph convolution for question answering over tem- poral knowledge graphs. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 2049- 2060, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Meeting of the Association for Computational Lin- guistics, pages 7567-7578, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-are natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Or- leans, Louisiana. Association for Computational Lin- guistics.
An Yang, Quan Wang. Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujjan Li. 2019. Enhanc- ing pre-trained language representations with rich knowledge for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2346- 2357, Florence, Italy. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint wxXiv:230.11397/
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of-thoughh reasoning for knowleege-intensivee multi-step questions. arXiv preprint 221.
Jiarui Yao, Haoling Qiu, Bonan Min, and Nianwen Xuc, 2020. Annotating Temporal Dependency Graphs via Crowdsourcing. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Lnguage Processing (EMNLP), pages 5368-5380, Online. As- sociation for Computational Linguistics.
Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task I: TempEval-3: Evaluat- ing time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Compu- tational Semantics (*SEM). Volume 2: Proceedings of the Seventh International Workshop on Seman- tic Evaluation (SemEval 2013), pages 1-9, Atlanta, Georgia, USA. Association for Computational Lin- guistics.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoming with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023. Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.13007
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained temporal relation extraction. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 2906-2919, Florence, Italy. Asso- ciation for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Mare Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Jessica Moszkowiez, and James Puste- jovsky. 2009. The tempeval challenge: identifying temporal relations in text. Language Resources and Evaluation, 43:161-179.
Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating -stra-linguistic contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mare Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky, 2010. Semeval-2010 task 13: Tempeval-2. In Proceedings of the 5th international workshop on semantic evaluation, pages 57-62.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozoy, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. In Proceedings of the 58th Annual
Shuaicheng Zhang, Qiang Ning, and Lifu Huang. 2022a. Extracting temporal event relation with syntax- graph transformer. In Findings of the Association for Computational Linguistics: NAACL

---

2022, pages 379-390, Seattle, United States, Associ- ation for Computational Linguistics.
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovee. 2022b. GreaseLM: Graph REA- Soning enhanced language models. In International Conference Learning Representations.
Yuchen Zhang and Nianwen Xue. 2018. Neural ranking models for temporal dependency structure parsing. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3339-3349, Brussels, Belgium. Association for Com- putational Linguistics.
Yuchen Zhang and Nianwen Xue. 2019. Acquiring structured temporal representation via crowdsoure- ing: A feasibility study. In Proceedings of the Eighth Joint Conference on Lexical and Computational Se- mantics ( "SEM 2019), pages 178-185, Minneapolis, Minnesota. Association for Computational Linguis- tics.

---

A Appendix
We also consider whether to use all six lations or only three relations, merging (BE- FORE, AFTER, SIMULTANEOUS, INCLUDES, INCLUDED BY, OVERLAP) into (BEFORE, AF- TER, and OVERLAP).
A.1 Transitivity rules
Figure shows the standard temporal transitivity rules we apply to infer relations from paths through the temporal graph.
We used the TimeQA Easy and Hard develop- ment sets for model selections. Table 6 shows the results of model selection. For ERR, merging the relations down to 3 slightly decreases performance on TimeQA Hard, the DT2QT graph is slightly bet- ter for TimeQA Hard, and the DTE2QT graph is slightly better for TimeQA Easy. Since the DT2QT is simpler, we select this model as our ERR model. For GNN, no models outperform L5 alone, regardless of whether the GNN was applied to the token embeddings the final contextualized em- beddings, and whether applied simpler or more complex temporal graphs.
A.2 Model Selection
For the ERR method introduced in Section 3.2.1, we consider a document-timel torliettt-uus-n------- time (DT2QT) subgraph that contains all the edges con- necting document-time nodes to the question-time node, and (DTE2QT) subgraph that builds on the document- time-to-question-ime subgraph by adding all the edges from the document-event nodes to the question-time node. We do not fuse the entire tem- poral graph with ERR because doing so would significantly increase the length of the input. On average, this fusion will add number of tokens equal to twice the number of edges because each edge needs to be represented by two special mark- ers, e.g., <before> 2009 </before>.
tettttt---uu--tiiiiie -iime
A.3 Confidence Intervals
We use the bootstrap resampling method to con- struct 95% confidence intervals the test sets. The results are shown in Table 7.
For the GNN method, we consider both the en- tire temporal graph, and an "all time graph" derived from the DT2QT subgraph, with additional edges for the temporal relations between each time ex- pression node and all other nodes.
A.4 Software Licenses
We train question answering systems to answer English Wikipedia time-sensitive questions. Upon publication, we will release code under the
before
after
includes
included
simultaneous overlup
before after includes included by simultancous
before
X
before
before after
X
after
after
includes
includes
before
after
X
included by
included by
X
includes
included by overlap
simultaneous
overlup
overlap
X
overlap
Figure 5: Transitivity rules.
Model
TimeQA Dev
Easy
Hard
EM
F1
EM
Fl
LongT5
55.2
63.5
48.1
56.4
LongT55 ERR DT2QT Subgraph
56.1
65.1
52.1
60.5
LongT5 ERR DT2QT Subgraph (merge to relations)
56.7
65.3
$1.9
60.1
LongT55 ERR DTE2QT Subgraph
57.0
65.6
50.6
59.0
LongT5 GNN Token Embed + All Time Graph
55.3
64.1
47.8
56.3
LongT55 GNN Token Embed + Temporul Graph
55.5
64.3
48.1
56.8
LongT5 GNN Context Embeed + All Time Graph
11.5
11.5
13.4
13.4
LongT5 GNN Context Embed Temporal Graph
53.9
62.5
48.8
57.6
Table Performance TimeQA development datasets. The highest- highest-performing model bolded. DT2QT Subgraph is the siauumenttime---o-uusttoon subgraph. DTE2QT Subgraph is the subgraph.

---

Models
TimeQA Test
SituatedQA Test
Easy
Hard
HP Easy
HP Hard
EM / F1
EM/FI
EM/FE
EM /
F1
Replicated FiD
55.3 / 65.1
45.2 / 54.5
52.7 59.7
39.7 47.0
153., 56,6 1a3, 66.31
[43.8, 46.61 (58.2,55)
1902, (502,55.21 / [57.466.9
[37.3, 47.2/2.2// [467, 49.4)
LongTS
55.4 64.3
49.9 58.4
53.7 61.6
45.3 52.3
24.7
[54.1, 56.3] 16a.1, 65.11
[18,5,512] [57.2, 59.71
(51,3,559) (59.3, 63.7)
[428. 47.6 1500, S4.)
[23.2,26.4
LoNgTSONN
54.2 63.4
49,0/ 57.6
46.5 55.3
34.3 40.2
27.3
[52.8 5566 [62.1, 64.5)
(47.6, 50.3 [56.4, 58.8)
14.1, 48.8] (SLL,
[32.0 96.3 [300 42.3)
[25.8. 29.3]
LongTSERK
56.9 66.2
54.0 / 62.0
54.9 62.9
49.3 56.1
29.0
[55.5 58.3 [65.,673]
(9.66,5533 (608, 63.2)
(525, (608,
51.5 / (589, 58.1]
30.6,3066
Table 7: Confidence intervals the test sets, generated via bootstrap resampling. HP = Human Parmann-arraphaseed.
MIT license. We list below the licenses of the data, software, and models we used. Our use is consistent with their intended uses.
The total GPU hours of experiments are around 160. We will make our code publicly available upon publication of the paper!!
Stanford CoreNLP: GNU General Public Li- cense Version 3 6.
Graph Statisties The constructed graphs statis- tics are presented in Table 8.
CAEVO, Long-t5-tllobalbbase model, Hug-
gingFace Transformers: Apache License, Ver- sion 2.07
Performance of the External Tools We two tools: SUTime from Stanford CoreNLP and CAEVO. CAEVO's reported precision 0.92 for event-time relations and 0.88 for time-time rela- tions. The accuracy of SUTime in recognizing time expression types and values is 0.96 and 0.82, re- spectively.
TimeQA dataset: BSD 3-Clause License 8 Pytorch: BSD-style license
A.5 Implementation Details
We use SUTime from Stanford CoreNLP V4.5.0 (Manning et al., 2014) to identify and normalize time expressions. We use the T5 implemen- tation from Huggingface Transformers V4.20.1 (Wolf al., 2020) and the LongT5 base model (250 million parameters) with transient-global attention checkpoint -oong-t--gglobll the starting point for training. We follow the code provided in the FiD Github repository to pre- process the Natural Questions dataset. We create new tokens in the tokenizer and model vocabulary for special marker tokens such as <e>. We tune the learning rate 51x10-5 xx 10-5 3 10-5 4x 10-5, x 10 10-5), batch size b (4, 8, 16, 32), and number of ReIGraphConv layers I (1,3, 6) on the development set of TimeQA and use early stop- ping to monitor the exact match metrics. We use ReLU as activation functions in ReIGraphConv layers. We set the hidden state size of ReIGraph- Conv layers to the same as LongT5-Base. All ex- periments are conducted on Nvidia A6000 GPUs.
A.6 Datasets
Table 9 presents the statistics of the datasets used for the experiments. Table 10 shows the input length of the LongT55 model before and after fusing the DT2QT sub-graph using the ERR method.
from Google as
A.7 Temporal Relations
We visualize the temporal relations between the time intervals in Figure
A.8 ChatGPT Prompts
We present the ChatGPT prompts we used in Table Table 11. Considering the overall input length con- straint, for context documents in in-context learn- ing examples, we use sentences containing answers as context for questions that have answers. For questions without answers, we randomly sample a sentence from the original context document as the context.
A.9 GPT-4 Results
www.gnn..ao www.gguu.oorr///// wwt..or//cccnns/ggl-333333 html "ww.. www.appcch..rrg/iccennssss wttp://rcceeeeee////EEEE-22222 2.0
Following the in-context learning setup described in Section 4.4, we replicate the experiment using
/opensoource -at.r//e..rrgp/eoceee//////-aaa
"github. httpp//.w.ooooooecooooo////////e
LICENSE
"https://gi h/tpps::/gitthann thw.bbb.tttttbbo/o www/Inttelaabsm/uu timodal. iiaittive.a//trre/mmnn///uu

---

Dataset
Split
Avg. Nodes
Avg. Edges
In/Out Degree
TimeQA Easy
Train
161
323
2/2
TimeQA Easy
Dev
163
326
2/2
TimeQA Easy
Test
163
324
2/2
TimeQA Hard
Train
160
321
2/2
TAmeQA Hard
Dev
162
325
2/2
TimeQA Hard
Test
323
2/2
TimeQA HP Easy
Train
163
317
TimeQA HP Easy
Test
376
2/2
TimeQA HP Hard
Train
163
317
TimeQA HP Hard
Test
194
376
SituatedQA TEMP
Train
214
213
SituatedQA TEMP
210
209
SituatedQA TEMP
Test
213
212
Table 8: The statistics of constructed graphs. HP = Human-Paraphrased.
Dataset
Split
# Questions
Answerable
# Unanswerable
Avg. # Tokens
TimeQA Easy
Train
14308
12532
1776
2566
TimeQA Easy
Development
3021
2674
347
2635
TimeQA Easy
Test
2997
2613
384
2634
TimeQA Hard
Train
12532
2149
TimeQA Hard
Development
3087
2674
413
2636
TimeQA Hard
Test
3078
2613
465
2636
Human-Paraphaasee Paraphrased TimeQA Easy
Human- Human-Puree Parnphhaad TimeQA Easy
Train
1171
982
189
2907
Test
1171
982
189
2907
Human-Paran Paraphrased TimeQA Hard
Train
1171
982
189
2907
(luman-Paraphinn TimeQA Hard
Test
1171
982
189
2907
SituatedQA TEMP
Train
6009
6009
9923'
SituatedQA TEMP
Dev
3423
3423
9924"
SituatedQA TEMP
Test
2795
2795
0
9922
Table 9: Statistics for the four datasets. Avg. # Tokens is the average number of tokens in the context document. We use the LongTSS model's tokenizer to tokenize context documents. "The length is based on the retrieved top 100 paragraphs from Wikipedia, which document exceeds 10,000 tokens, we truncate the context accordingly.
context document for each question. If the total length of context
the GPT-4 model. However, due to the costs of calling the commercial GPT-4 API, we only ran- domly sampled 100 examples each from TimeQA Easy and TimeQA Hard, resulting in a total of 200 samples for the experiment. The results are shown in Table 12. ERR method achieves similar perfor- mance improvements with GPT-4 as it does with ChatGPT (gpt-3.5-turbo).
A.12 Other Experimented Methods
The following are some of the methods we have tried but yielded lower performance than our main methods reported in Table 6.
1. We tried to use the coreference resolution tools to process the context documents and then construct and fuse temporal graphs based on the processed text, but we found that the coreference resolution preprocessing hurt the performance of the models.
A.10 Additional Error Analysis
2. We tried to fuse constructed temporal graphs into FiD models using relation-awer aware atten- tion (Wang et al., 2020), but we found that the fused FiD models performed almost the same the non-fused FiD models.
Table 13 presents examples, in addition the examples shown in Table
A.11 Analysis of the GNN Fusion Method
3. Our preliminary study found that the perfor- mance of the models can be significantly im- proved if only the most relevant paragraph the context document used as the context.
Table 14 shows examples from TimeQA Hard where LongT5 is correct but LongT5 oNN is in- correct.

---

Dutaset
Tokens Pre-ERR
Tokees Post-ERR
TimeQA
2617
2862
SituatedQA
9923
11515
Table 10: Average LongT5 input length before and after using the ERR method fuse DT2QT subgraphs.
before B
A after
simultancous
B
A
A includes B
A included by B
overlap
A
B
B
A
A
Figure Visualization temporal relations.
W/O Graph Fusion
Instruction: Answer the question based the given context. there no answer, answer answer."
Context: the 1955 general election, Cunningham was as new Ulster Unionist MP for South Antrim. He was delegate to Council Europe Western European Union Parliamentary Assembly from 1956 to 1959.
Question: Which position Knox Cunningham hold before Apr 1956?
Answer: Ulster Unionist
South Antrim
Context: Broughton would go on to teach at Amherst College, Bryn Mawr College (1928-1965) and, later, serve as George I..
Question: Thomas Robert Shannon Broughton was an employee for whom before Jun 1926?
Answer: no answer
W/ Document- Ceetument-Timetto -aneett-Quettoo--- Time Subgraphs Fused Using ERR Method
Instruction:
Answer the question based the given context. there is relations between the times mentioned in the context and the question represented using XML-style tags.
"no answer." The temporal
Context: the <included ludd by> general election, Cunningham was chosen the new Ulster Unionist MP for South Antrim. He was delegate the Council of Europe and Westem European Union Parliamentary Assembly <overlapsfrom 1956 1959</overlaps.
Question: Which position did Knox Cunningham hold <question timeshefore Apr 1956</question time>? Answer: Ulster Unionist MP South Antrim
Context: Broughton would go on teach Amherst College, Bryn Mawr College (caften-1228. Caffeer>192-1165//hfieeee) and, later. serve as George
Thomas Robert Shannon Broughton was an employee whom <question timesbefore 1926</question time>?
Question:
Answer: no answer
Table 11: Examples of ChatGPT prompts.
Model
TimeQA Easy
TimeQA Hard
EM
EM
F1
GPT-4
52.0
62.1
48.0
56.7
ERR DT2QT Subgraph
53.0
64.0
55.0
63.5
Table 12: Performance of GPT-4 on 100 randomly sampled test examples, with and without our ERR method.

---

Type
Freq #
Question
Relevant Context
Prediction
Answer
Where was Richard G. Hovannisian educated from 1954 1958?
Hovannisian received his B.A. in history (1954) from the University of California, Berkeley, and his M.A. in history (1958) and Ph.D. (1966) from University of California, Los Angeles (UCLA).
University Cal- Califor-
University
ifomia, Berkeley
Los
Negative
Angeles
(UCLA)
2 Which team did the player Simone Verdi belong from 2014 2015?
On 18 June 2014, the co-ownership between Torino Milan was renewed for third year, with Verdi leaned Empooli again.
Empoli
Milan
Segenet Kelemu employee for whom from Nov 2013 to Nov 2014?
In 2013, Kelenu joined the Alliance for Green Revolution in Africa (AGRA) as Vice President for Programs for about November 2013, Kelemu became the Di- General International Centre Insect Physiology and Ecology (icipe)...
Alliance for Green
International Centre of
Revolution in Africa
Insect Phys-
iology and Ecology
Semantic Understanding
3
4
Donald Thobega played for which teamn in 2006?
Between 1999 and 2006 be lotal of five cups for the
NIA
Botswana
won
national
Bolswana national foothall teurn.
foothall
team
Which school did Alexan- der Dvorkin go to from 1978?
.nin 1972. became student the Faculty of Russian Language and Liter- ature Moscow Pedagogical Institute ...comqqueenll. in the autumn of 1975. Dvorkin was expelled from the third year the institute
Moscow Podagogical
N/A
Institute
Semantic Understanding & Insufficient Temporal Information
What was the capital of Kolomna from 1929 to Apr 2017!
Within the framework of administrative divi- sions, Kolomna serves as the administrative center of Kolomensky District..
N/A
Kolomensky District
did Ryan Gander work from 2007 to 2015?
...
was made Cocheme Fellow Byam Shaw School of Art, Landon
Table 13: Error categories and examples on the TimeQA Easy development set. Model predictions are in bold. Underlines denote the correct answers.
Thus, we tried to train a cross-encoder-based ranker to rank the paragraphs in the context documents, but found that the pipeline ap- proach using cross- cros-enccddr ranker and LongT5 as a reader was as good as using LongT5 directly generate answers end-to-end.
paragraphs

---

Question: What position did Roland Burris take in May 1989?
LongT5RRR Prediction: Illinois Comptroller
LongT5GNN Prediction:
Illinois Attorney General
Context: In 1978, Burris was the first African-Ameriia elected to statewide office in Illinois, when he was elected Illinois Comptroller. He served in that office until his election Illinois Attorney General in 1990
Annotated explanation: LongT5 prediction of "Illinois Attorney General" is based on the f'at that Burris was elected this position 1990. However, it failed to correctly link the timeline in the question "May 1989" to the context.
Question: Where was Joe Burrow educated in Feh 2015?
LongTTERR Prediction: Ohio State
LongT'SGNX Prediction: Athens High School
Context: Burrow attended Athens High School (201 (-4))-1)110 in The Plains, Ohio After redshirting his first year at Ohio as backup to J. T.. Barrett..
State in 2015, Burrow spent the next two
Annotated explanation: LongT5 LongTSGNNN guess of "Athens High School" is based on Burrow's 's education at Athens High School in The Plains. Ohio. However, it failed to discern how "Feb 2015" in the question aligns with the timeline in the context. As evident, Burrow was at Ohio State in 2015.
Question: Which school did Kyriakos Mitsotakis attend in Feb 1985? LongTTSERR Prediction: Athens College
LongTSGNN Prediction: Harvard University
Context: ...In 1986, he graduated from Athens College. From 1986 to 1990, he attended Harvard University and carned bachelor's 's degree in social studies, receiving the Hoopes Prize
Annotated explanation: LongT5 prediction "Harvard University" is hased on Mitsotakis attending Harvard from 1986 to 1990. Yet, the timeline "Feb 1985" in the question doesn't match the context, From the context, it's clear Mitsotakis graduated from Athens College in 1 1986, implying he was studying there in 1985.
Table 14: Examples where LongTS FRR correct, but LSngTSGNN is incorrect.


Reviews:
[{"review_id": "dtnoxG21Af", "reviewer": "Reviewer_XQnr", "summary": "", "questions": "Please check the three questions in the weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer XQnr,\n\nThank you for your feedback and recognition of our paper's clarity, our temporal graph's generalization ability, our fusion method's compatibility with large language models, and the state-of-the-art performance of our ERR method. We appreciate your insightful comments and have addressed them below.\n\n> The idea to build the temporal graph is attractive in this paper. But the tech contribution of the whole paper is limited. The graph construction is based on the existing systems. \n\nThank you for this feedback. While it is true that our temporal graph construction leverages existing systems, our goal is not to propose new temporal information extraction methods that can be used to construct graphs. Rather, our focus is on introducing an effective approach to construct and fuse graphs for a specific task: time-sensitive question answering, using the existing temporal information extraction systems. \n\nThe novelty of our work is in its (1) investigation of whether fusion of temporal information extraction results can help neural time-sensitive question answering, and (2) investigation of which form of fusion yields the greatest benefits. To the best of our knowledge, there is no prior work investigating similar research questions. Our experiments surprisingly show that a simple but novel form of fusing temporal graphs into the input yields benefits where the popular technique of graph neural networks (GNNs) does not, which provides valuable insights to inform future work.\n\n> the fusing methods have been used in many tasks.\n\nIn this work, we investigated two temporal graph fusion methods, namely ERR and GNN. Our ERR method is a novel and efficient temporal graph fusion approach that has not been proposed previously. It effectively enhances the model's temporal reasoning capabilities and consistently outperforms more complex methods, achieving state-of-the-art (SOTA) performance in multiple scenarios.\n\nRegarding the GNN method, we agree that it has been employed previously to fuse graphs for other tasks. This is actually why we sought to implement it in this work; specifically, our motivation was to use it as a baseline method for comparisons with our newly proposed ERR method. While GNNs have been successfully applied to other tasks, our results are interesting in that they show GNNs are not a good fit for time-sensitive question answering. The fact that ERR outperforms GNNs in our experiments suggests that the investigation of ERR as an alternative to GNNs in other tasks could be a promising direction for future work. \n\n> These two fusing methods have some problems. First, The graph fusion has a bad performance as shown in Table 2. It is not a good candidate to take advantage of the structured temporal graphs.\n\nThank you for this accurate observation regarding the performance of the GNN method. As mentioned previously, our primary objective for the GNN method was to use it as an important baseline for comparison with our ERR method, given its widespread application and its natural fit for graph structure fusion. We agree with your assessment that the GNN method has bad performance, which we highlight in our paper and use as a basis for concluding that ERR is a more efficient method for temporal graph fusion  (lines 415 - 430 and lines 586 - 590).\n\n> Second, the ERR is similar to a data processing step using the temporal graph. Since there is a step to extract the temporal information using other system, it is not supervised that the LongT5 with ERR can get a better performance than the LongT5.\n\nWe appreciate your feedback on our ERR method but believe that characterizing it as a data processing step oversimplifies the complexities of temporal graph fusion, which we address in our work. These complexities include the investigation of how to best use temporal information extraction systems to construct high-quality temporal graphs for time-sensitive questions and which parts of the temporal graph should be fused (please see lines 389 -398 and Appendix A.2 for details). \n\nFurthermore, the simplicity, intuitiveness, and effectiveness of the final graph fusion step are strong advantages of ERR relative to more complicated fusion methods (e.g., GNN). Indeed, EMNLP follows [ACL policy](https://2023.aclweb.org/blog/review-acl23/) in recognizing that \"the goal is to solve the problem, not to solve it in a complex way\" and that \"simpler solutions are in fact preferable, as they are less brittle and easier to deploy in real-world settings.\" \n\nOur findings indicate that the most effective method for improving the temporal reasoning capabilities of transformers via temporal graphs is to incorporate them directly into the model's input. This can be achieved without the need for complex graph modeling techniques, such as those in previous state-of-the-art methods (e.g., DocTime). The effectiveness and simplicity of our approach provide valuable insights to the research community and can inform future work on graph fusion methods.\n\n> The EER method has one limitation. The cost of marking the nodes and edges of temporal graph is high if the input context is long. I am worried it is hard to be extended in large-scale data.\n\nThank you for raising this concern about the scalability of our ERR method. As noted in Section 4.3 (lines 389-398) and Appendix A.2, we only need to fuse the document-time-to-question-time sub-graph of the temporal graph. Therefore, in the main performance tables of our paper, we reported the performance of models that fused this sub-graph. To clarify the input length before and after ERR fusion, we will add the following table to the Appendix of our paper, which shows that using ERR does not significantly increase the length of the input and, therefore, the computational cost.\n\n| Datasets                                       | Avg. # tokens before ERR Fusion | Avg. # tokens after ERR Fusion |\n|------------------------------------------------|------------------------------|-----------------------------|\n| TimeQA (combined all the modes and splits)     | 2617                         | 2862                        |\n| SituatedQA                                     | 9923                         | 11515                       |\n\nAdditionally, the statistical details of all the datasets we evaluated are reported in Table 9 of Appendix A.6 and show that our experiments were conducted using large-scale long-document datasets. For instance, TimeQA Hard comprises over 14,000 training samples and around 3,000 test samples, with an average document length surpassing 2,500 tokens. Similarly, SituatedQA includes 6,009 training samples and approximately 3,000 test samples, with an average document length of around 10,000 tokens.\n\n> The explanation of section 3.2.2 is not clear. What I got is that this method is coming from the Schlichtkrull’s paper. I am not clear if authors propose any modification so that it is more useful for temporal QA task.\n\nThe purpose of Section 3.2.2 is to provide a brief introduction to the GNN model, \"relation graph convolution\", that we used, and to explain how we employed it to fuse the temporal graph. We have made **no modifications** to the structure of this model. We appreciate the reviewer's suggestion and, if our paper is accepted, we will clarify this point in the revised version and consider renaming this section. Additionally, we will include more details about this GNN model in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n1.\tThis paper proposes a way to build a temporal graph based on the temporal extraction method including CAEVO and SUTime. It will extract the importance temporal information from the text. This temporal graph can be a useful knowledge in many tasks.\n\n2.\tTwo fusing methods are proposed in the paper. And they can be used easily with the large language models. The ERR approach outperforms the SOTA method.", "reasons_to_reject": "Weaknesses:\n\n1.\tThe idea to build the temporal graph is attractive in this paper. But the tech contribution of the whole paper is limited. The graph construction is based on the existing systems. And the fusing methods have been used in many tasks. These two fusing methods have some problems. First, The graph fusion has a bad performance as shown in Table 2. It is not a good candidate to take advantage of the structured temporal graphs. Second, the ERR is similar to a data processing step using the temporal graph. Since there is a step to extract the temporal information using other system, it is not supervised that the LongT5 with ERR can get a better performance than the LongT5. \n\n2.\tThe EER method has one limitation. The cost of marking the nodes and edges of temporal graph is high if the input context is long. I am worried it is hard to be extended in large-scale data.\n\n3.\tThe explanation of section 3.2.2 is not clear. What I got is that this method is coming from the Schlichtkrull’s paper. I am not clear if authors propose any modification so that it is more useful for temporal QA task.", "questions_to_authors": "Please check the three questions in the weaknesses.", "paper_topic_and_main_contributions": "This paper designs a method to answer the time-sensitive questions. There are two key steps in this method. The first one is to use temporal extraction system to construct the temporal graphs. The purpose is to extract the event, time and temporal relations in advance and to construct a graph. With this structured temporal information, the second step is to investigate the way of fusing the temporal graph with the transformer models. The whole idea is clear and well explained in the paper.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cU1eJPAwhk", "reviewer": "Reviewer_D8zG", "summary": "", "questions": "- Could you give some more qualitative examples like in Table 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer D8zG,\n\nThank you for your review and the positive remarks on the organization of our paper, the intriguing aspects of our ChatGPT analysis, our evaluation setting, and the state-of-the-art performance of our method. Your mention of the qualitative insights from Table 3 is particularly appreciated. We address the insightful comments and questions you raised in your reviews below.\n\n> There are a lot of missing fields in Table 2, and most baselines have not been directly compared across datasets, so it’s difficult to generalise improvements.\n\nWe appreciate your observation regarding missing values in Table 2. These missing values are associated with results that were directly reported in prior work. In cases where results are missing, this indicates that the prior work did not test on all of the datasets used in our experiments. Hence, values corresponding to those methods and datasets are not available. \n\nNevertheless, for every dataset used in our experiments, we compared our results to all prior comparable work reported on that particular dataset. Our results indicate that our approach consistently outperforms the prior state-of-the-art models in various scenarios. This not only demonstrates that we tested on more datasets compared to prior works, but also further validates that our method can generalize across multiple datasets.\n\nSince our initial paper submission, we also replicated the FiD model on the SituatedQA dataset for further comparison. It has an exact match score of 23.7, which indicates that our LongT5_ERR method still significantly outperformed it. The experimental details are the same as the Replicated FiD baseline described in the paper, and we will add this new result to Table 2 in the final version of our manuscript. We will also update our discussion of Table 2 to clarify that missing values are associated with a lack of results reported in previous works.\n\n> It can be argued that this is incremental work given model simplicity.\n\nThank you for raising this concern. We believe that the simplicity, intuitiveness, and effectiveness of the final graph fusion step is a strong advantage of ERR relative to more complicated fusion methods (e.g., GNN). Indeed, EMNLP follows the [ACL policy](https://2023.aclweb.org/blog/review-acl23/) in recognizing that \"the goal is to solve the problem, not to solve it in a complex way\" and that \"simpler solutions are in fact preferable, as they are less brittle and easier to deploy in real-world settings.\" \n\nOur findings indicate that the most effective method for improving the temporal reasoning capabilities of transformers via temporal graphs is to incorporate them directly into the model's input. This can be achieved without the need for complex graph modeling techniques, such as those in previous state-of-the-art methods (e.g., DocTime). The effectiveness and simplicity of our approach provide valuable insights to the research community and can inform future work on graph fusion methods.\n\n> Could you give some more qualitative examples like in Table 3?\n\nYes, thank you for this suggestion. Due to page constraints, we only displayed 5 out of our 50 manually annotated errors in Table 3 to illustrate each type of mistake. However, we will include more qualitative examples in the final version of our paper, and have provided some of the new examples in the table below. Model predictions and correct answers are in bold. \n\n| Type                                                       | Question                                                     | Relevant Context                                             | Prediction                                | Answer                                                |\n| ---------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------------------- | ----------------------------------------------------- |\n| False Negative                                             | Where was Richard G. Hovannisian educated from 1954 to 1958? | Hovannisian received his B.A . in history (1954) from the **University of California, Berkeley**, and his M.A . in history (1958) and his Ph.D . (1966) from **University of California, Los Angeles (UCLA).** | University of California, Berkeley        | University of California , Los Angeles (UCLA)         |\n| False Negative                                             | Which team did the player Simone Verdi belong to from 2014 to 2015? | On 18 June 2014, the co-ownership between Torino and **Milan** was renewed for a third year, with Verdi loaned to **Empoli** again. | Empoli                                    | Milan                                                 |\n| False Negative                                             | Segenet Kelemu was an employee for whom from Nov 2013 to Nov 2014? | In 2013, Kelemu joined the **Alliance for a Green Revolution in Africa** (AGRA) as Vice President for Programs for about a year. In November 2013, Kelemu became the Director General of the **International Centre of Insect Physiology and Ecology** (icipe) ... | Alliance for a Green Revolution in Africa | International Centre of Insect Physiology and Ecology |\n| Semantic Understanding                                     | Donald Thobega played for which team in 2006?                | ... Between 1999 and 2006 he won a total of five caps for the **Botswana national football team**. | N/A                                       | Botswana national football team                       |\n| Semantic Understanding                                     | Which school did Alexander Dvorkin go to from 1975 to 1978?  | ... in 1972, he became a student in the Faculty of Russian Language and Literature of **Moscow Pedagogical Institute** ... consequently, in the autumn of 1975, Dvorkin was expelled from the third year of the institute ... was expelled from the institute for poor academic performance and non-attendance in early 1975. | Moscow Pedagogical Institute              | N/A                                                   |\n| Semantic Understanding & Insufficient Temporal Information | What was the capital of Kolomna from 1929 to Apr 2017?       | Within the framework of administrative divisions, Kolomna serves as the administrative center of **Kolomensky District** ... | N/A                                       | Kolomensky District                                   |\n| Semantic Understanding & Insufficient Temporal Information | Where did Ryan Gander work from 2007 to 2015?                | ... In 2004, he was made Cocheme Fellow at Byam Shaw School of Art, **London** ... | N/A                                       | London                                                |\n\n> Missing References: Aditya Sharma, Apoorv Saxena, Chitrank Gupta, Mehran Kazemi, Partha Talukdar, and Soumen Chakrabarti. \"TwiRGCN: Temporally Weighted Graph Convolution for Question Answering over Temporal Knowledge Graphs.\" In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2041-2052. 2023.\n\nWe appreciate this reference to interesting prior work, which we will cite in the final version of our paper. While Sharma's paper focuses on a different task (question answering over structured knowledge graphs vs. our focus on time-sensitive question answering over unstructured text), we will add the below discussion of how our work relates to it in Section 2.\n\n\"Sharma et al. (2023) use pre-trained temporal knowledge graph embeddings combined with graph neural networks to reason over structured temporal knowledge graphs, with the goal of addressing temporal questions. In contrast, we concentrate on temporal reasoning in the reading comprehension setting, using unstructured text to deduce answers. As our method does not utilize a predefined structured knowledge base, our answers are derived from segments within unstructured texts rather than entities or relations in knowledge graphs.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-organised and easy to follow.\n- Evaluation is interesting and beats state-of-the-art. In particular, Table 3 helps us understand the model behaviour qualitatively, though I wish there were more examples. \n- The ChatGPT analysis is very intriguing.", "reasons_to_reject": "- There are a lot of missing fields in Table 2, and most baselines have not been directly compared across datasets, so it’s difficult to generalise improvements.\n- It can be argued that this is incremental work given model simplicity.", "questions_to_authors": "- Could you give some more qualitative examples like in Table 3?", "paper_topic_and_main_contributions": "This work focuses on the problem of time-sensitive question answering over the events and times in a question and an accompanying long context document. The authors use existing temporal information extraction systems to construct temporal graphs and investigate different fusion methods to inject them into transformer-based language models. They compare their input fusion method with prior approaches that use graph convolutions. Models are evaluated on SituatedQA and TimeQA datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Aditya Sharma, Apoorv Saxena, Chitrank Gupta, Mehran Kazemi, Partha Talukdar, and Soumen Chakrabarti. \"TwiRGCN: Temporally Weighted Graph Convolution for Question Answering over Temporal Knowledge Graphs.\" In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2041-2052. 2023."}}, {"review_id": "8HVZntcO8M", "reviewer": "Reviewer_ay7N", "summary": "", "questions": "Q1: Are the reported results obtained from a singular program run, or have multiple runs been conducted? In a different run, you may get a lower or higher result compared to other baselines. \n\nQ2: If we have the labels such as <before> or <after>, why must we pass this to a language model for the final reasoning? Cannot we do logical computations over these to find the answer? For example, if we know that event A is before event B and event C is after event B then we can get that event C is after event A too. \n\nQ3: This concept merits exploration beyond its current application in temporal reasoning, showcasing potential generality and utility in broader reasoning contexts. Given its effectiveness in temporal reasoning, one may speculate about the feasibility of applying the same approach in other logical reasoning domains, such as spatial reasoning.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ay7N,\n\nThank you for your comprehensive review, which we found insightful in both its depth and constructive details. We are grateful for your recognition of our work's contributions, especially the ERR method and its state-of-the-art performance. Your remarks on its simplicity, integration with ChatGPT, and the clarity of our writing are particularly appreciated. We address your constructive comments and feedback raised in the review below.\n\n> W1: The paper presents two distinct methodologies, ERR and GNN, to integrate graph knowledge into language models. However, an explicit comparative analysis between these approaches needs to be included. The rationale behind the superior performance of ERR vs. GNN remains unexplored, even though ERR employs a more straightforward data infusion technique than GNN. Investigating the reasons behind this performance disparity could shed valuable light on these approaches' strengths and weaknesses, contributing to a more comprehensive understanding of their effectiveness.\n\nWe appreciate this suggestion and agree that a qualitative comparison of the GNN approach vs. the ERR approach would be a valuable contribution to our paper. We will add the following analysis to the final version of our paper to address this question.\n\nTo better understand the differences in performance between the GNN and ERR methods, we analyzed 20 randomly sampled samples from TimeQA Hard where the ERR method was correct but the GNN method was incorrect. From our manual analysis of these 20 samples, we found that all 20 samples shared the same pattern: the GNN method failed to capture explicit temporal expressions in the context and relate them to the question's timeline, which is crucial for deducing the right answer. This suggests that directly embedding pre-computed relationships between time nodes into the input is more efficient than implicitly doing so through the GNN, allowing the model to utilize them more easily. \n\nTo illustrate, we provide three of our analyzed samples below with an annotated explanation for the error:\n\nExample 1:\n\n- Question: What position did Roland Burris take in May 1989?\n- ERR prediction (correct answer): Illinois Comptroller\n- GNN prediction (wrong answer): Illinois Attorney General\n- Context: ... In 1978, Burris was the first African-American elected to statewide office in Illinois, when he was elected **Illinois Comptroller**. He served in that office until his election as **Illinois Attorney General** in 1990 ...\n- Annotated explanation: The GNN prediction of \"Illinois Attorney General\" is based on the fact that Burris was elected to this position in 1990. However, it failed to correctly link the timeline in the question \"May 1989\" to the context.\n\nExample 2:\n\n- Question: Where was Joe Burrow educated in Feb 2015?\n- ERR prediction (correct answer): Ohio State\n- GNN prediction (wrong answer): Athens High School\n- Context: ... Burrow attended **Athens High School** (2011–14) in The Plains, Ohio ... After redshirting his first year at **Ohio State** in 2015, Burrow spent the next two years as a backup to J. T. Barrett ...\n- Annotated explanation: The GNN's guess of \"Athens High School\" is based on Burrow's education at Athens High School in The Plains, Ohio. But it failed to discern how \"Feb 2015\" in the question aligns with the timeline in the context. As evident, Burrow was at Ohio State in 2015.\n\nExample 3:\n\n- Question: Which school did Kyriakos Mitsotakis attend in Feb 1985?\n- ERR prediction (correct answer): Athens College\n- GNN prediction (wrong answer): Harvard University\n- Context: ... In 1986, he graduated from **Athens College**. From 1986 to 1990, he attended **Harvard University** and earned a bachelor's degree in social studies, receiving the Hoopes Prize ...\n- Annotated explanation: GNN's prediction \"Harvard University\" is based on Mitsotakis attending Harvard from 1986 to 1990. Yet, the timeline \"Feb 1985\" in the question doesn't match the context. From the context, it's clear Mitsotakis graduated from Athens College in 1986, implying he was studying there in 1985.\n\n> W2: Despite the paper's strength lying in its simplicity and ease of application to other benchmarks, a more comprehensive analysis could be achieved by comparing this approach with others that incorporate textual data into large language models. For instance, employing code execution, as explored in the paper \"Unlocking Temporal Question Answering for Large Language Models Using Code Execution, 2023,\" might offer deeper insights and further enrich the investigation.\n\nThank you for acknowledging the simplicity and ease of application of our work, and for providing this relevant reference. Unfortunately, we found that the code and data associated with this paper are unavailable, which makes a direct comparison to it in our experimental setting impossible. We also noticed that it was first uploaded to arXiv on 24 May 2023, which classifies it as contemporaneous to our submission per ACL policy. Nevertheless, we found this paper very interesting and will include the following discussion in Section 2 of our final manuscript. \n\n“Li et al., (2023) primarily focus on exploring large language models for information extraction in structured temporal contexts. They represent the extracted time-sensitive information in code, and then execute Python scripts to derive the answers. In contrast, we adopt the original TimeQA setup, where the context document is long and unstructured. This poses more challenges in information extraction and involves more complex reasoning, which motivates our integration of existing temporal information extraction systems with transformer-based language models.”\n\nXingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng, Shafiq Joty, Lidong Bing, Unlocking Temporal Question Answering for Large Language Models Using Code Execution, 2023\n\n> W3: Acknowledging that ChatGPT exhibits certain limitations in reasoning, some of which have been addressed in models like InbdtructGPT, Flan-T5, PaLM2, or GPT4(ref: Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models, 2023), choosing ChatGPT as the representative language model for this study may not be the optimal choice. It is plausible to hypothesize that more advanced language models, such as GPT4, might already possess the capacity for temporal reasoning without the explicit infusion of graph data. Nevertheless, the proposed approach could still offer valuable enhancements, even for powerful language models like GPT4.\n\nWe appreciate this suggestion and the relevant reference. During the rebuttal period, we repeated our previous ChatGPT experiments using GPT-4. The results are presented in the tables below. We observed that our ERR method achieved similar performance improvements with GPT-4 as it did with ChatGPT. Specifically, it improved the model's performance on both TimeQA Easy and particularly on TimeQA Hard.\n\n|                            | Metric | TimeQA Easy | TimeQA Hard |\n|:--------------------------:|:------:|:-----------:|:-----------:|\n| **GPT-4**                |   EM   |     52.0    |    48.0    |\n|                                  |   F1   |     62.1   |   56.7     |\n| **w/ ERR + DT2QT Subgraph**|   EM   |   **53.0**     |    **55.0**   |\n|                                                    |   F1   |     **64.0**   |   **63.5**    |\n\n\nFor this experiment, we used an identical setting as the ChatGPT experiment described in our paper. However, we randomly sampled 100 examples each from TimeQA Easy and TimeQA Hard (resulting in a total of 200 samples for the experiment) due to the cost of the commercial GPT-4 API. We will update the final version of our paper to include these new results.\n\n> Q1: Are the reported results obtained from a singular program run, or have multiple runs been conducted? In a different run, you may get a lower or higher result compared to other baselines.\n\nThank you for this question. In Table 2, we report the performance of our single best-performing model after hyperparameter tuning experiments. However, we agree that it is important to evaluate the variability in the performance of individual models. To accurately assess this aspect of performance, we followed the method proposed by Dror et al. (2018), employing bootstrap resampling on the test set to obtain confidence intervals of our model's performance. This technique is statistically more appropriate for comparing performance on a test set than running multiple times with different random seeds. We provide the 95% confidence intervals on our models’ performance in Table 7 of Appendix A.3. \n\nNonetheless, we are grateful for the reviewers' suggestions. We will include the average performance over multiple runs as an additional reference in the final version's appendix.\n\n[Dror et al., 2018] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing.\n\n> Q2: If we have the labels such as <before> or <after>, why must we pass this to a language model for the final reasoning? Cannot we do logical computations over these to find the answer? For example, if we know that event A is before event B and event C is after event B then we can get that event C is after event A too.\n\nThank you for this insightful question. The reasoning required by long-document time-sensitive question answering is often complex, which makes it challenging to obtain an answer solely through straightforward temporal logical reasoning. A language model is still essential to understand the text and integrate the information about the question distributed throughout the long document. Our work aims to assist the model in one specific facet of this reasoning: its temporal reasoning. \n\nTo illustrate, consider the example shown in Figure 1 of our paper: \"What was George Washington’s position between 1776 - 1780?\". The reasoning needed to derive the correct answer includes:\n\n1. Identifying and understanding the distributed information about George Washington’s career progression from the text (e.g., “Congress created the Continental Army on June 14, 1775, and Samuel and John Adams nominated Washington to become its Commander-in-Chief.\")\n2. Recognizing events or facts mentioned in the text related to George Washington during the period 1776-1780. For instance, the document mentions Washington bidding farewell to his officers at Fraunces Tavern in December 1783 and resigning his commission days later.\n3. Inferring Washington's activities and roles during this period, and concluding that he primarily served as the Commander-in-Chief of the Continental Army.\n4. Establishing a timeline of these events to ensure that the answer aligns with the queried timeframe. While Washington was elected as the Chancellor of the College of William & Mary in 1788 and began his tenure as the U.S. president the same year, these roles came after the period in question.\n5. Synthesizing information from various sources to produce an accurate answer.\n\nFrom inspecting the temporal graph corresponding to this question (Figure 3), it is evident that relying solely on the temporal graph would be insufficient to address all of the types of reasoning detailed above. Moreover, the text may contain irrelevant or potentially confusing information, like George Washington's roles or activities during other periods. Understanding negations within the context at times can further complicate the reasoning process. Thus, a language model needs not only to find the right information in the text but also to distinguish and filter out irrelevant or potentially misleading content.\n\nWe appreciate you raising this question and will update the final version of our manuscript to include this discussion.\n\n> Q3: This concept merits exploration beyond its current application in temporal reasoning, showcasing potential generality and utility in broader reasoning contexts. Given its effectiveness in temporal reasoning, one may speculate about the feasibility of applying the same approach in other logical reasoning domains, such as spatial reasoning.\n\nThank you for recognizing the generalizability of our method and for your insightful suggestions. While our current work focuses on temporal reasoning, we also see great potential in applying our proposed method to spatial reasoning. For instance, integrating it with geospatial information extraction systems to construct graph structures and fusing them into the input. We will expand our conclusion in the final version of our manuscript to include a discussion of such opportunities for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The proposed approach achieves state-of-the-art performance on TimeQA and SituatedQA datasets.\n\nS2: The proposed approach offers advantages over alternative models in simplicity. It requires only minor adjustments to the Transformer model's input. It outperforms prior work using a model with only 250 million parameters, while other models require additional data, complex parsers, or large pre-trained models. \n\nS3:  The paper also shows the advantages of the ERR approach in the ChatGPT model. \n\nS4: The writing was clear and easy to follow.", "reasons_to_reject": "W1: The paper presents two distinct methodologies, ERR and GNN, to integrate graph knowledge into language models. However, an explicit comparative analysis between these approaches needs to be included. The rationale behind the superior performance of ERR vs. GNN remains unexplored, even though ERR employs a more straightforward data infusion technique than GNN. Investigating the reasons behind this performance disparity could shed valuable light on these approaches' strengths and weaknesses, contributing to a more comprehensive understanding of their effectiveness.\n\nW2: Despite the paper's strength lying in its simplicity and ease of application to other benchmarks, a more comprehensive analysis could be achieved by comparing this approach with others that incorporate textual data into large language models. For instance, employing code execution might offer deeper insights and further enrich the investigation.\n\nW3: Acknowledging that ChatGPT exhibits certain limitations in reasoning, some of which have been addressed in models like InbdtructGPT, Flan-T5, PaLM2, or GPT4(ref: Towards Benchmarking and Improving the Temporal Reasoning Capability\nof Large Language Models, 2023), choosing ChatGPT as the representative language model for this study may not be the optimal choice. It is plausible to hypothesize that more advanced language models, such as GPT4, might already possess the capacity for temporal reasoning without the explicit infusion of graph data. Nevertheless, the proposed approach could still offer valuable enhancements, even for powerful language models like GPT4.", "questions_to_authors": "Q1: Are the reported results obtained from a singular program run, or have multiple runs been conducted? In a different run, you may get a lower or higher result compared to other baselines. \n\nQ2: If we have the labels such as <before> or <after>, why must we pass this to a language model for the final reasoning? Cannot we do logical computations over these to find the answer? For example, if we know that event A is before event B and event C is after event B then we can get that event C is after event A too. \n\nQ3: This concept merits exploration beyond its current application in temporal reasoning, showcasing potential generality and utility in broader reasoning contexts. Given its effectiveness in temporal reasoning, one may speculate about the feasibility of applying the same approach in other logical reasoning domains, such as spatial reasoning.", "paper_topic_and_main_contributions": "The paper explores temporal reasoning in answering time-sensitive questions from long documents. The authors investigate whether large language models can perform this reasoning solely using text or if they can benefit from additional temporal information. They apply existing temporal information extraction systems to construct temporal graphs and propose different approaches for fusing them into Transformer models. Experimental results show that their approach enhances the temporal reasoning capabilities of Transformer models and outperforms other graph convolution-based approaches, achieving state-of-the-art performance on SituatedQA and TimeQA datasets.\n\nThe main contributions of the paper are as follows: \n\n1. Applying existing temporal information extraction systems to construct temporal graphs of events, times, and temporal relations in questions and documents. \n\n2. Investigating different approaches for fusing these temporal graphs into Transformer models. \n\n3. Proposing an approach called Explicit Edge Representation Fusion (ERR) to directly fuse the temporal graphs into the input text of the Transformer model. \n\n4. Another approach is Graph neural network fusion: This approach utilizes a Graph Neural Network (GNN) to fuse the temporal graphs into the token embeddings or the last hidden layer representations of the Transformer model. \n\n5. Shows that the proposed ERR approach enhances the temporal reasoning capabilities of Transformer models and outperforms various graph convolution-based approaches, achieving state-of-the-art performance on SituatedQA and three splits of TimeQA datasets.\n\n\nThe experimental results and the proposed approach to achieving state-of-the-art performance are summarized as follows: \n\n1. Baseline Models: The LongT5 model without temporal graph fusion performs better than or equivalent to all other baseline models on TimeQA and SituatedQA datasets, except for the Easy split. The best-performing model on TimeQA Easy is DocTime. \n\n2. Graph Fusion Methods:  On TimeQA, the LongT5GNN model does not outperform LongT5 without fusion. However, the LongT5ERR model improves over LongT5 on every split and dataset, particularly on the Hard splits. On SituatedQA, both LongT5ERR and LongT5GNN models improve over the no-fusion LongT5 baseline, with ERR providing the best performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a method for addressing time-sensitive questions. The methodology comprises two steps: 1) a temporal extraction system of time and temporal relationships to construct a graph, and 2) the fusion of these temporal graphs with transformer models. Two fusion techniques are explored for this integration: directly as text, and using graph convolutions. The method is evaluated in SituatedQA and TimeQA datasets. The reviewers highlighted the strong empirical results, that the manuscript is well-organised and easy to follow. They also raised issues the work being incremental in relation to similar ideas and that some baseline evaluations were missing. During the rebuttal period those and other concerns were partially addressed by the authors who re-implemented an FiD model.
Decision: Accept-Findings