MTGER: Multi-view Temporal Graph Enhanced Temporal Reasoning over Time-Involved Document
Authors: Zheng Chu
Zekun Wang
Jiafeng Liang
Ming Liu
Bing Qin
Venue: EMNLP 2023 Findings (2023)
Keywords: Question Answering
Temporal Reasoning
Natural Language Processing
TL;DR: Inspired by the human reasoning process, we propose to model the temporal relationship between facts and time via temporal graphs, outperforming SOTA on two document-level temporal reasoning datasets, TimeQA and SituatedQA.

Abstract: The facts and time in the document are intricately intertwined, making temporal reasoning over documents challenging. Previous work models time implicitly, making it difficult to handle such complex relationships. To address this issue, we propose MTGER, a novel Multi-view Temporal Graph Enhanced Reasoning framework for temporal reasoning over time-involved documents. Concretely, MTGER explicitly models the temporal relationships among facts by multi-view temporal graphs. On the one hand, the heterogeneous temporal graphs explicitly model the temporal and discourse relationships among facts; on the other hand, the multi-view mechanism captures both time-focused and fact-focused information, allowing the two views to complement each other through adaptive fusion. To further improve the implicit reasoning capability of the model, we design a self-supervised time-comparing objective. Extensive experimental results demonstrate the effectiveness of our method on the TimeQA and SituatedQA datasets. Furthermore, MTGER gives more consistent answers under question perturbations.

MTGER: Multi-view Temporal Graph Enhanced Temporal Reasoning over Document
Zheng Chu', Zekun Wang', Jiafeng Liang', Ming Liu Bing Qin
Harbin Institute of Technology, Harbin, China Peng Cheng Laboratory, Shenzhen, China {zchu, zk zkwang, jfliang, mliu," qinb) @ir.hit.cdu.cn
Abstract
(a) Time-sensitvve Question Ansser & Constraint
Which school dld Tiger Woods
19657
Stainford (n(mppic Relation Constraint oduccted Time Constraint 1995
The facts and time in the document are intri- cately intertwined, making temporal reason- ing over documents challenging. Previous work models time implicitly, making it dif- ficult to handle such complex relationships. To address this issue, we propose MTGER, novel Multi-view Temporal Graph Enhanced Reasoning framework for temporal reasoning over time-involved documents. Concretely, MTGER explicitly models the temporal rela- tionships among facts by multi-view temporal graphs. On the one hand, the heterogeneous temporal graphs explicitly model the tempo- ral and discourse relationships among facts; on the other hand, the multi-viww view mechanism captures both time-focused and fact-focused information, allowing the two views to comple- ment each other through adaptive fusion. To further improve the implicit reasoning capabil- ity of the model, we design a self-supervised time-compeen comparing objective. Extensive experi- mental results demonstrate the effectiveness of our method on the TimeQA and SituatedQA datasets. Furthermore, MTGER gives more con- sistent answers under question perturbations.
(bi Wikipedia Document: Tiger Woods
Immil In 1904 the age of eigh, son 9-10 boys' event, the yourgest age group assilablee the Junior Worls Chempiooshipps as amaleur player.
When Woods national junior tounament. was years
years old, he played in the 1989 Big I, which was his frstimajor
student at Westem High School Anahalm when yourgest U.S.] Ameteur chempion:
scholarshi
his fistt
legiate the 40th onder profossional plyer goffidusttyy He college
Tucker Invitational, that Separberr.
Event Timeline
Amertur postten))
Prolectioal Plyer
Hish backel
stacked
Mare
Retre
To aae bole
In Edscalat
(ei Reasoning from the Human Perspective
1. Eviidence from Dec.bl [TTie Pairs)
2. Implicit Rewnoning &
3. Deduce Ansseer from Evidence under Relation and Time Constraint
New Evidencs
1984, Amalleu P layer
1991. Wessem High
10: Wheh School in 19957
1989, years
Schanl coms -20500
Ammmttuur Playe Stanrord
15 years old. Weeseem Sigh School 1994, Stanford
<g. Time Cocanatonn
1984-1996, Ameteur Player
1996 CllEgR
1991-1994, Westem High School
Brofesskon. crma +31
1994-1996, S ttanford t
After two years, Coleeo Turn Profession
1996-, Professional
An example of a time-sensitive question in- volving implicit reasoning, best viewed in color. (a) de- scribes the question, answer and constraints (b) shows a time-involved document (c) depicts the timeline of events in the document (d) illustrates the reasoning pro- cess in the human perspective, including three steps.
Figure I:
1 Introduction
the existing facts according to the constraints of the question, including time constraints and relation constraints (right of Figure 1(d)).
In the real world, many facts change over time, and these changes are archived in the document such as Wikipedia. Facts and time are intertwined in docu- ments with complex relationships. Thus temporal reasoning is required to find facts that occurred specific time. To investigate this problem, Chen et al. (2021) propose the TimeQA dataset and Zhang and Choi (2021) propose the SituatedQA dataset. For example, Figure illustrates a ques- tion involving implicit temporal reasoning. From the human perspective, to answer this question, we first need to find relevant facts in the document and obtain new facts based existing facts (left of Figure (d)). We need to deduce the answer from
Recent work (Chen et al., 2021; Zhang and Choi,
2021; Izacard and Grave, 2021b; Zaheer et al., 2020; Beltagy et al., 2020a; Guo al., 2022) di- rectly uses pre-trained models or large-scale lan- guage models answer time-sensitive questions, neglecting to explicitly model the time like human, resulting in lack of understanding of time. Even the state-of-the-art QA models have a large gap compared with human performance (50.61 EM vs. 87.7 EM), indicating that there is still a long way to go in this research area.
Inspired by the human reasoning process de- picted in Figure (d)) we intend to explicitly model the temporal relationships among facts. To this
Corresponding Author.

---

end, we propose Multi-view Temporal Graph En- hanced Reasoning framework (MTGER) for tem- poral reasoning over time-involved documents. We construct a multi-view temporal graph to establish correspondence between facts and time and to ex- plicitly model the temporal relationships between facts. The explicit and implicit temporal relations between facts in the heterogeneous graph enhance the temporal reasoning capability, and the cross- paragraph interactions between facts alleviate the inadequate interaction.
2
MTGER Framework
2.1 Task Definition
Document-level textual temporal reasoning tasks take long document (e.g., typically thousands of characters) with time-sensitive question input and output the answer based the docu- ment. Formally, given a time-sensitive question Q, document D. the goal to obtain the answer A which satisfies the time and relation constraints in question Q. The document = P, P2...., Pk} contains k paragraphs, which are either from a Wikipedia page or retrieved from Wikipedia dumps. The answer can be either an extracted span gen- erated text, and take the generation approach in this paper. Please refer Appendix A. for the definition of time-sensitive questions.
Specifically, each hetcrogencous temporal graph (HTG) contains factual and temporal layers. Nodes in the factual layer are events, and nodes in the temporal layer are the timestamps (or time inter- vals) corresponding to the events. Different nodes are connected according the discourse relations and relative temporal relations. In addition, construct a time-focused HTG and fact-focused HTG to capture information with different focuses, forming a multi-view temporal graph. We com- plement the two views through adaptive fusion to obtain more adequate information. At the decoder side, we use a guestion-guided fusion mechanism to dynamically select the temporal graph informa- tion that is more relevant to the question. Finally, we feed the time-enhanced representation into the decoder to get the answer. Furthermore, we intro- duce a self-supervised time-comp objective to enhance the temporal reasoning capability.
A arg max P(A ([AIQ,D) D;0)
(1)
2.2 Overview
As depicted in Figure 2, MTGER first encodes paragraphs and constructs multi-view tempo- ral graph, then applies temporal graph reasoning the multi-view temporal graph with the time- comparing objective and adaptive fusion, and fi- nally feeds the time-enhanced features into the de- coder to get the answer.
2.3 Text Encoding and Graph Construction Textual Encoder We use the pre-trained FiD (Izacard and Grave, 2021b) model to encode long text. FiD consists of a bi-directional encoder and a decoder. It encodes paragraphs individually at the encoder side and concatenates the paragraph representations as the decoder input.
Extensive experimental results demonstrate the effectiveness of our proposed method on the TimeQA and SituatedQA datasets, with a perfor- mance boost of up to 9% compared to the state- of-the-art QA model and giving more consistent answers when encountering input perturbations. The main contributions of our work can be sum- marized as follows:
Given document D (Pi, Pa,..., Pk) con- taining K paragraphs, each paragraph contains M tokens: P; {λ1, .2. pm)) h represents the hidden dimension. Following the previous method (Izacard and Grave, 2021a), the question and paragraph are concat in a "question: title: paragraph:" fashion. The textual representation H Htant &xxxh is obtained by encoding all para- graphs individually using the FiD Encoder.
We propose to enhance temporal reasoning by modeling time explicitly. As far we know, the first attempt to model time explicitly in temporal reasoning over documents.
We devise a document-level temporal reason- ing framework, MTGER, which models the temporal relationships between facts through heterogencous temporal graphs with com- plementary multi-view fusion mechanism.
Hirnt = Enc(P, P2, Pp.....)))) Ps))
(2)
Graph Construction We construct multi-view heterogeneous temporal graph based on the rela- tionship between facts and time in the document, as illustrated in Figure 2(a). The multi-view tempo- ral graph consists of two views, Gfact = (V, EV,E,uu {
Extensive experimental results demonstrate the effectiveness and robustness of our method on the TimeQA and SituatedQA datasets.

---

(4) Textual Encoder Graph Construction
(b) Temporal Graph Reasoning & Multi-viww fusion
Time emmeeenhnneed Decoder
Time-lacsed
Tengeralone Gruph Beasoring
Queeilin Teat Test Rusion
Maltrsiew fasion
Textual Encoder
Time Comparinen
Trccel stace
Time onhareed Reyessenation Hoyr
Vist
Van
Decoder
After Orariop
question
parigraphs
Figure Overview of MTGER. Best viewed in color. (a) Encoding the context and constructing multi-view temporal graph (b) Temporal garph reasoning over multi-view temporal graph with time-comparing objective and adaptive multi-view fusion (c) Question-guiit text-graph fusion and answer decoding.
and Otime = (V, Esse), which has the same nodes and different edges.
two fact nodes in each of the two paragraphs to con- struet inter paragraph edges. Temporal and factual layers are bridged by time-to-fact edges, which are uni-directed, from times to facts. The global node is connected with all fact nodes, from facts to the global node. The Appendix A.8 and A.9 provide example of temporal graph and graph construction process.
Procedure Constructing a multi-view temporal graph consists of document segmentation, node ex- traction and edge construction. Firstly, we segment the document into chunks (or paragraphs) based on headings, with similar content within each chunk. Afterward, we extract time and fact nodes repre- senting time intervals and events, using regular ex- pressions. Finally, we construct edges based on relationship between nodes and chunks. Besides, to mitigate graph sparsity problem, we introduce a global node to aggregate information among fact nodes. We will introduce nodes, edges, and views in the following.
detailed
Views We construct two views, the fact-focused view and the time-focused view. Multiple views make it possible to model both absolute relation- ships between time expression (e.g. 1995 is before 2000 because 1995 < 2000) and relative relation- ships between events (e.g. Mcssi joins Inter Miami CF after his World Cup championship). If only one view exists, difficult model both rela- tionships simultaneously. In the time-focused view, time comparisons occur between time nodes, and fact nodes interact indirectly through time nodes as bridges; in the fact-focused view, the relative temporal relationships between facts are directly modeled. The model can sufficiently capture the temporal relationships among facts by complement- ing each other between the views. To obtain the fact-focused view, we replace the discourse re- lation edges between fact nodes in the foee-fd view with the temporal relation edges between the corresponding time nodes and replace the edges of time nodes with discourse relations of fact nodes in similar way. The comparison of the two views is shown at the top of Figure 2(b).
Nodes According the roles represented in tem- poral reasoning, we define six node types: global node is responsible for aggregating the overall in- formation; fact nodes represent the events at the sentence-level granularity (e.g. He enrolled at Stan- ford); time nodes are divided into four categories according to their temporal states, before, after, be- tween, and in, which represent the time interval of the events (e.g. Between 1980 and March 1988 [1980, 1998.25]). In the temporal graph, the global node and fact nodes are located at the factual layer and time nodes at the temporal layer.
Edges In the temporal layer, we build edges ac- cording to the temporal relationship between time nodes, including before, after and overlap. In the factual layer, build edges according to the dis- course relationship between fact nodes. For facts in the same paragraph, they usually share com- mon topic. Accordingly, we construct densely- connected intra- paraa-par edges among these fact nodes. For facts in different paragraphs, we pick
2.4 Multi-view Temporal Graph Reasoning
Temporal Graph Reasoning First, we initialize the nodes representations using the text represen- tation and then perform a linear transformation to

---

the nodes according their types.
Adaptive Multi-view Graph Fusion Through the graph reasoning module introduced above, we can obtain the fact-focused and time-focused graph representation, V E Rax Vi E Rnxh, respec- tively. These two graph representations have their own focus, and to consider both perspectives con- currently, we adopt adaptive fusion manner (Li et al., 2022; Zhang al., 2023) to model the inter- action between them, as illustrated in Figure 2(b).
8/1 Pooling h1, h2: hi)
(3)
(0) W' o
(4)
V vij 22
(0) (0) 1
where hi,... h are textual representations corre- sponding to the node v. W E Raxa is the linear transformation corresponding to node type I, n is number of nodes, (()) E R serves as the first layer input to the graph neural network.
À Sigmoid (W, Wyy + W.V)
(13)
fuse (1 A) V + V,
(14)
We refer to the heterogeneous graph neural net- work (Schlichtkrull al., 2018; Busbridge al., 2019) to deal with different relations between nodes. In this paper, we use heterogencous graph attention network.
where W E hx and W. Rixh are learnable
parameters.
Self-supervised Time-comparing Objective To further improve the implicit reasoning capability of models over absolute time, we design self- supervised time-comparing objective. First, we transform the extracted TimeX into time interval represented by floating-point number. After that, the fact nodes involving TimeX in the graph are combined two by two. Three labels are generated according to the relationship between the two time intervals: before, after, and overlap. Assuming that (6) there are N fact nodes in the graph, N(N - 1)/2 self-supervised examples can be constructed, and cross cross-entropy optimize them.
We define the notation as follows: W C Rhxh W E Rlxh W E ghxh represents the node transformation, query transformation and key transformation under relation r. respectively. They are all learnable parameters.
First, we perform a linear transformation on the nodes according to the relations they are located.
O" VW" E R
Q' O' W c Ruch
(7)
K" O' W E Ruxh
(8) hig MLP hyl)
(15)
Afterwards, calculate the attention scores' in order to aggregate the information.
N Lie Σ
Σ yı log( (og((())))))))) (16) -=++1
a q. kj
where hi, hy are the nodes representations and yk is the pseudo-label. [:] denotes vector concatenate.
exp (af
0/.3 ExEN. .
(10)
2.5 Time-enhanced Decoder
Question-guided Text-graph Fusion Now we have the text representation H and the multi- view graph representation Vjuse- Next, dynam- ically fuse the text representation and the graph representation guided by the question to get the time-enhanced representation Huse which will be fed into the decoder to generate the final answer. as illustrated Figure 2(c).
Finally, we obtain the updated node representa- tion by aggregating based on the attention score. Here we use the multi-head attention mecha- nism (Vaswani et al., 2017), where K is the number of attention heads.
K
((
Σ
nh Ln of
(11)
1=
(EN-((
Vluse MHCA (EEEA(Qqun (quuery, Viuse: Viuse) (17)
H'S Text MHCA MHCA(Hectt Htxt, Vinses Viuse)
(18)
We obtain the final graph representation through layer heterogeneous graph neural network.
H Juse AdapGate Htert; Hient)
(19)
where qqeery query embedding, MHCA multi- head cross-attention and AdapGate is mentioned in the graph fusion section.
((+1) HeteroGNN(V
(12)
Formula of multi-head attention is omitted for readability.

---

Time-enhanced Decoding Finally, the time- enhanced representation Juse fed into the de- coder predict the answer, and we use the typical teacher forcing loss to optimize the model.
and QA tasks. In order explore their capabil- ities on textual temporal reasoning, we also in- clude the state-of-the-art LLM, GPT3 and Chat- GPT (Ouyang et al., 2022; OpenAI, 2022) for comparison.
For the large-scale language model baseline, sample 10 answers per question, @mean and @ max represent the average and best results, re- spectively. We use gpt-3,5-turbo max subse- quent experiments unless otherwise stated.
N
Lif
yi log P(y/
H Juse. yes))
(20)
2.6 Training Objective
The training process has two optimization ob- jectives: teacher-foreing loss for generating an- swers by maximum likelihood estimation and time- comparing loss for reinforcing time-comparing ability in the graph module. We use a multi-task learning approach to optimize them.
Implementation Details We use Pytorch frame- work, Huggingface Transformers for pre-trained models and PyG for graph neural networks. We use the base size model with hidden dimension 768 for all experiments. For all linear transformations graph neural networks, the dimension is 768x768 The learning rate schedule strategy is warm-up for the first 20% of steps, followed by cosine decay. We use Adam W as the optimizer (Loshchilov and conducted on single Tesla A100 GPU. The training cost of TimeQA and SituatedQA approximately 4 hours and 1.5 hours, respectively. Please refer to Ap- pendix A.3 for detailed hyperparameters.
Lioial = Las + A Lie
(21)
where X is hyper-parameter.
Hutter, 2019). All experiments
3 Experiments
3.1 Experiments Setup
Dataset We conduct experiments on the TimeQA and SituatedQA (Chen et al., 2021; Zhang and Choi, 2021) datasets. TimeQA is document- level temporal reasoning dataset to facilitate tempo- ral reasoning over time-involved documents. It is crowd-sourced based on Wikidata and Wikipedia. SituatedQA is an open-domain QA dataset where each question has a specified temporal context, which is based on the NQ-Open dataset.
3.2 Main Result
Results on TimeQA Table shows the overall experiment results of baselines and our methods on the TimeQA dataset. The first part shows the results of the LLM baselines. Even though the LLMs demonstrate their amazing reasoning ability, it performs poorly in temporal reasoning. The sec- ond part shows the results of supervised baselines, where BigBird performs on par with gpt-3.5-turbo, and FiD performs better than BigBird.
Following the original experimental setup, we use EM and Fl the TimeQA and EM on the Sit- uatedQA dataset as evaluation metrics. In addition, we calculate metrics for each question type. For more information about the datasets, please refer to Appendix A.I.
Our method MTGER outperforms all baselines, achieving 60.40 EM / 69. .444 F1 on the easy split and 53.19 EM / 61.42 Fl on the hard split. MT- GER obtains 2.43 EM / 1.92 Fl and 3.39 EM 2.95 F1 performance boost compared to the state- of-the-art QA model, FiD. We think this significant performance boost comes from explicit modelling of time and interaction across paragraphs. In addi- tion, MTGER++ further improves the performance through long-context adaptation pre-training for the reader model, achieving 60.95 EM / 69.89 F1 on the easy split and 54.11 EM / 62.40 F1 on the hard split. Please refer to Appendix A.4 for more details about MIGER++
Baseline We choose four typical long-context QA models, FiD, BigBird, LED and LongT5 (Izac- ard and Grave, 2021b; Zaheer et al., 2020; Belt- agy et al., 2020a; Guo et al., 2022), an open- domain QA system DPR (Karpukhin et al., 2020) and close-book QA model BART (Lewis et al., 2020a) baselines. FiD, BigBird and LongT5 are inited with pre-trained checkpoint on NaturalQues- tions (K (Kwiatkowski al., 2019), TriviaQA (Joshi et al., 2017) and NewsQA (Trischler et al., 2017). Besides, large-scalc language models (LLMs) have demonstrated excellent performance on reasoning
ChatGPT May 24 Version

---

Easy
Hard
Method
Dev
Test
Dev
EM
F1
EM
F1
EM
F1
EM
F1
LLM Baseline
Zero-shot
gpt-3,5- 3.5-turroo@men
30.71
42.67
32.79
46.04
28.91
36,94
28.68
35.14
apt-3.5-turoommax
43.14
53.92
43,46
56.18
37.14
45.08
37.64
43.32
Few-shot(n=5)
gpt-3,5-turbo@menn
42.14
51.59
34,80
47.12
35.37
43.52
34,03
42.36
3pt-3.5-trho@@max
54.67
62.92
44.98
58.66
45.69
54.41
41.63
51.71
Supervised Baseline
LED
46.97
56.41
49.41
58.35
39.78
48.45
40.50
48.10
LongT5
48.17
59.59
48. 14
58.94
41.43
52.58
42.94
53.50
BigBird..a
52.00
61.13
50.65
60.21
43.69
51.71
44.73
51.59
BigBird
51,89
62.43
50.61
60.88
44.04
53.83
43.56
53.60
FiDago
55.21
65.82
56.32
64.99
56.74
47.92
56.78
FiDDa
57.25
66.56
58.06
67.52
49.21
58.00
49.80
58.74
Ours
MTGER
59.65
68.05
60.49
69.44
52.31
60.66
53.19
61.42
MTGER++
59.20
67.70
60.95
69.89
52.02
61.06
54.15
62.40
Table Main results on the TimeQA Dataset. All supervised models are base size and we report the average results of three runs. Best and second results are highlighted by bold and underline.
Easy
Hard
Perf. Gap
Temp
Method
Method
EM
F1
EM
F1
EM
Fl
Static
Samp. Start
Avg.
LLM
LLM
Zeroshot
43.45
56.94
36.28
43.06
-1.73%
+0.5%
Zeroshot
11.6
8.5
11.9
9.9
Fewshot(n=5)
52.74
64.75
50.63
57.80
+19.4%
+11.1%
Fewshot(n=5)
9.3
13.4
15.5
13.4
Supervised
Supervised Bascline
BigBird
46.81
55.67
41.75
51.36
-5.8%
-6.4%
LED
18.1
16.2
20.4
17.8
FiD.,
55.91
63.86
47.52
54.82
-4.3%
-6.4%
LongT5
22.0
14.4
18.3
16.8
Ours
BART
26.0
16.2
18.3
18.3
MTGER
58.74
66.
12
50.55
57.00
-3.9%
-5.9%
DPR
39.8
17.2
24.9
23.0
MTGER++
59.35
66.51
51.06
57.91
4.1%
-6.0%
BigBird,,,
34.2
15.7
21.3
20.2
FiDM
28.7
22.2
31.8
26.4
Table 2: Results on TimeQA Human- Paraphhrae dataset. Gap represents the performance gap compared with the main dataset.
Ours
MTGER
45.0
24.8
28.2
28.8
Table 3: Results SituatedQA Temp test set. Eval- uation metric is EM from the correct temporal context.
Results on TimeQA Human-Paraphrased The questions in TimeQA human-pant paraphrased are rewritten by human workers in more natural lan- guage manner, making the questions more diverse and fluent. As shown in Table 2, the performance of the LLMs increases rather than decreases, which we think may be related to the question becoming more natural and fluent. All supervised baselines show some performance degradation. Our method still outperforms all baselines and has less perfor- mance degradation compared with supervised base- lines. The experimental results illustrate that our method is able adapt to more diverse questions.
conduct experiments on an open-domain temporal reasoning dataset, SituatedQA. Since the model ini- tiated with NaturalQuestion checkpoint performs better than TriviaQA, we do not present the results for models initiated with TriviaQA.
As shown in Table 3, MTGER achieves 28.8 EM, exceeding all baseline methods, improving by 9% compared to the best baseline. MTGER does not have reranking module while it outperforms DPR by 5.8 EM, which has reranking module. Our method performs well on all three types of questions and significantly better than others on Static and Samp. types. The experimental results demonstrate that our method can be generalized to
Results on SituatedQA To investigate the gen- eralization of MTGER for temporal reasoning, we

---

Hard-Implicit
Easy-Explicit between
Method
in
between
before
after
avg.
in
avg.
L.M Baseline
Zeroshot
38.79 37.69
34,53 45.36
41,46 56,09
27.50
36.69
33.33
44.39
43,46
Fewshot(n=5)
25.50
41.41
45.27
40.47
44.86
Supervised Baseline
BigBird, 0g
46.31
45,04
40.22
42.36
44.85
28.97
52.34
50.61
FiDag
49.61
48.83
50.18
53.81
49.73
56.10
58.22
58.06
Ours
MTGER
52.99 52,99
52.80 53.57
53.87 56.83
56.10 10
53.28
57.92 57.47
60.69
60.49
MTGER++
57.25
53.99
61.23
60.95
Table 4: Results for different question types the TimeQA test set. Evaluation metric is E.M
different temporal reasoning datasets and achieves excellent performance.
Dev
Test
Method
Easy
Hard
Easy
Hard
MTGER
59.65
52.31
60.49
53.19
4 Analysis
Text-graph Fusion (a)
59.11
51.88
59.52
52.39
TTime-comparing (b)
59.65
51.53
60,66
52.65
4.1 Probe Different Question Types
Multi-view Graph (c.1)
59.17
52.04
60.12
52.42
Heteroorraph (c.2)
58.67
50.52
59.89
50.78
We calculate the metrics for different categories of implicit questions in the hard split and explicit questions in the easy split. Results are shown in Table Our method outperforms baselines on all question types, achieving 53.99 Implicit-FM and 60.95 Explicit-EM, compared to the best base- line of 49.73 Implicit-EM and 58.06 Explicit-ENN reaching relative improvement of 8.6% for im- plicit questions and 5% for explicit questions. According to the results, our method particu- larly good at handling hard questions that require implicit reasoning. We believe the explicit tem- poral modelling provided by the heterogeneous temporal graphs improves the implicit temporal reasoning capability of models. In the next sec- tion, will perform an ablation study on different modules of MIGER to verify our conjecture.
Homoorrphh (c.3) 57.25
49.21
58.06 49.80
Table
Results of Ablation Study on the TimeQA dataset. Evaluation metric is EMM
and test set, respectively. This suggests that the question-guided text-fusion module can improve overall performance by dynamically selecting more useful information.
Effect of Time-comparing Objective As shown in Table 5(b), removing the time-comparing objec- tive has almost no effect on the easy split, and the performance degradation in the dev and test set on the hard split is 0.78 EM / 0.54 EM, indicating that the time-comparing objective mainly improves the implicit temporal reasoning capability of models.
4.2 Ablation Study
Effect of Multi-view Temporal Graph quentially remove the multi-viet -iel temporal graph (c.1 keep only heterogeneous temporal graph); replace the heterogeneous temporal graph with a homogeneous temporal graph (c.2 do not distin- guish the relations between nodes); and remove the graph structure (c.3) to explore the effect of graph structure in detail.
We se-
We investigate the effects of text-graph fusion, time-comparing objective and multi-view temporal graph. For multi-view temporal graphs, we conduct more detailed analysis by sequentially remov- ing the multi-view graph, heterogeneous temporal graph and homogeneous temporal graph. Experi- mental results are shown in Table
Removing the multi-view temporal graph (c. brings overall performance degradation of 0.48 EM/ 0.78 EM and 0.37 EM/ 0.77 EM in the dev set and test set, respectively, implying that the comple- mentary nature of multi-view mechanism helps capture sufficient temporal relationships between facts, especially implicit relationships.
Effect of Text-graph Fusion Question-guides text-graph fusion module dynamically selects the graph information that is more relevant to the ques- tion. As shown Table 5(a), removing this module results in the performance degradation of 0.54 EM 0..3 EM and 0.87 EM/ / 0.80 EM in the dev set

---

Perturbated Question
Context
MIGER
Beatrix often spelled Beatrice uttended Radeliffe College Massachusetts and re- ceived her bachelors degree 1954. 1956 she cameed her masters degree from Brown University working with Carl Pfaffiman. She completed her PhD in zoology at Oxford Uni- versity in 1959 where she studied under the mentorship Niko Tinbergen... ...
Which school did Beatrix Tu- gendhut Gardner go Dec 1955? Q': Which school did Beatrix Tu- gendhut Gardner and 1955?
Answer': Radeliffe College X
Answer'
before 1954
Brown University
Answer: Brown University
... Rockefeller moved to Emmons, West Vir- ginia serve VISTA worker 1964 and first elected public office ber of the West Virginia House of Delegates ( 1966-1968 ). Rockefeller was later elected West Virginia Secretary of State ( (1988-197) (988-973) and was president of West Virginia Wesleyan College 1973-1975 ). He became states senior U.S Senator
Q: Which school did Jay Rockefeller to between Oct 1966 and Dec
Answer': West Virginia
19677
Answer':
Q': Which school did Jay Rocke- feller go to before 1973?
Wesleyan College X
Ununswerable
Answer: Unanswerable
the long- serving Sen-
Robert Byrd died in June 2010
Table Case study from the consistency analysis. Q stands for the original question, Q' stands for the perturbated question and Answer' stands for the answer after question perturbation.
We replace the heterogeneous temporal graph with a homogeneous temporal graph (c.2), which results in the GNN losing the ability to explicitly model the temporal relationships between facts, leaving only the ability to interact across para- graphs. The performance degradation is slight in the easy split, while it causes significant perfor- mance degradation of 1.52 EM/ 1.64 EM com- pared with (c.1) in the hard split, which indicates that explicit modelling of temporal relationships between facts can significantly improve the implicit reasoning capability.
Method
Consistency
BigBird
0.63
FiD,g
0.66
text-davinci-0033 mcan
0.54
gpt-3.5-turbo mean
0.46
MTGER
0.76
MIGER++
0.78
Table 7: Results of consistency analysis the TimeQA
hard test set. The higher the consistency, the better.
4.4
Case Study
We show two examples from the consistency analy- sis to illustrate the consistency of our model in the face of question perturbations, as shown in Table 6. Both examples are from the TimeQA hard dev set.
Removing the graph structure also means remov- ing both text-graph fusion and time-comparing ob- jective, which degrades the model to FiD model (c.3). At this point, the model loses the cross- paragraph interaction ability, and there is an overall degradation in performance, which suggests that the cross- poss-paragraph interaction improve the overall performance by establishing connections between facts and times.
The first example shows the importance of im- plicit temporal reasoning. From the context, we know that Beatrix received her bachelor's degree in 1954 and her master's degree in 1956. The master's came after the bachelor's, s, so we can infer that she was enrolled in master's 's degree between 1954 and 1956, and 1954-1955 lies within this time in- terval, so she was enrolled in Brown University dur- ing this period. Since FiD lacks explicit temporal modelling, its implicit temporal reasoning ability is weak and fails to predict the correct answer.
4.3 Consistency Analysis
To investigate whether the model can consistently give the correct answer when the time specifier of questions is perturbed, we conduct consistency analysis. The experimental results in Table 7 show that our method exceeds the baseline by up to 18%, which indicates that our method is more consistent and robust compared to baselines. Please refer to Appendix A.7 for details of consistency analysis.
The second example shows the importance of question understanding. The question is about which school he attends, which is not mentioned in the context. FiD incorrectly interprets the question as to where he is working for, fails to understand the question and gives the wrong answer. Our method,

---

which uses question-guided fusion mechanism, allows for better question understanding and con- sistently gives the correct answer.
relationships between facts by graph reasoning over multi-view temporal graph.
5.2 Question Answering for Long Context
5 Related Work
Since the computation and memory overhead of Transformer-base models grows quadratically with the input length, additional means are quired reduce the overhead when dealing with long-context input. ORQA (Lee et al., 2019) and RAG (Lewis et al., 2020b) select a small number of relevant contexts to feed into the reader through the retriever, but much useful information may be lost in this way. FiD (Izacard and Grave, 2021b) and M3 (Wen et al., 2022) reduce the overhead at the encoder side by splitting the context into para- graphs and encoding them independently. However, there may be problem of insufficient interaction at the encoder side, and FiE (Kedia et al., 2022) introduces a global attention mechanism to allevi- ate this problem. Longformer, LED, LongT5, and BigBird (Beltagy et al., 2020b; Guo et al., 2022: Zaheer et al., 2020) reduce the overhead by sliding window and sparse attention mechanism.
5.1 Temporal Reasoning in NLP
Knowledge Base Temporal Reasoning Knowl- edge base QA retrieves facts from a knowledge base using natural language queries (Berant et al., 2013; Bao et al., 2016; Lan and Jiang, 2020; He al., 2021; Srivastava et al., 2021). In re- cent years, some benchmarks specifically focus on temporal intents, including TempQuestions (Jia et al., 2018 and TimeQuestions (Jia et al., 2021). TEQUILA (Jia et al., 201 2018b) decomposes complex questions into simple ones by heuristic rules and then solves simple questions via general KBQA systems. EXAQT (Jia et al., 2021) uses Group Steiner Trees to find subgraph and reasons over subgraph by RGCN. SF-TQA (Ding al., 2022) generates query graphs by exploring the relevant facts of entities to retrieve answers.
Event Temporal Reasoning Event Temporal Reasoning focuses the relative temporal re- lationship between events, including event tem- poral QA (Ning et al., 2020; Lu al., 2022; Han et al., 2021), temporal commonsense reason- ing (Qin et al., ; Zhou et al., 2019, 2020), event timeline extraction (Faghihi et al., 2022), and tem- poral dependency parsing (Mathur et al., 2022). TranCLR (Lu et al., 2022) injects event semantic knowledge into QA pipelines through contrastive learning. ECONET (Han al., 2021) equips PLMs with event temporal relations knowledge by con- tinuing pre-training. TACOLM (Zhou et al., 2020) exploits explicit and implicit mentions of temporal commonsense by sequence modelling.
Conclusion
In this paper, devise MTGER, novel temporal reasoning framework over documents. MTGER explicitly models temporal relationships through multi-view temporal graphs. The heterogeneous temporal graphs model the temporal and discourse relationships among facts, and the multi-view mechanism performs information integration from the time-focused and fact-focused perspectives. Furthermore, we design self-supervised objec- tive to enhance implicit reasoning and dynami- cally aggregate text and graph information through the question-guided fusion mechanism. Exten- sive experimental results demonstrate that MTGER achieves better performance than state-of-the-ar methods and gives more consistent answers in the face of question perturbations on two document- level temporal reasoning benchmarks.
Textual Temporal Reasoning Textual temporal reasoning pays more attention the temporal un- derstanding of the real-world text, including both absolute timestamps (c.g. before 2019, the late 1990s) and relative temporal relationships (e.g. A occurs before B). To address this challenge, Chen et al. (2021) proposes the TimeQA dataset, and Zhang and Choi (2021) propose the SituatedQA dataset. Previous work directly adopts long-contex, QA models (Izacard and Grave, 2021b; Zaheer et al., 2020) and lacks explicit temporal modelling. In this paper, we focus on temporal reasoning over documents and explicitly model the temporal
Limitations
Although our proposed method exhibits excellent performance in document-level temporal reason- ing, the research in this field still has long way go. We will discuss the limitations well possi- ble directions for future work. First, the automat- ically constructed sentence-level temporal graphs are slightly coarse in granularity; a fine-grained

---

temporal graph can be constructed by combining an event extraction system in future work to capture fine-grained event-level temporal clues accurately, Second, our method does not give a temporal rea- soning process, and in future work, one can con- sider adding a neural symbolic reasoning module to provide better interpretabillty.
EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages 1867-1877. Association for Computational Linguistics.
Hossein Rajaby Faghihi, Bashar Alhafni, Ke Zhang. Shihao Ran, Joel R. Tetreault, and Alejandro Jaimes. 2022. Crisisltlsum: A benchmark for local crisis event timeline extraction and summarization. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages for Computational Linguisties.
A Acknowledgements
Association
The research in this article is supported by the National Key Research and Development Project (2022YFF0903301,, the National Science Founda- tion of China (U22B2059, 61976073, 62276083), and Shenzhen Foundational Research Funding (JCYJ202001091 100419914444) Major Key Project of PCL (PCL2021A06).
Mandy Guo, Joshua Ainslie, David C. Uthus, Santi- ago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yin- fei Yang. 2022. Longt5: Efficient rext-to-text trans- former for long sequences. In Findings of the Associ- ation for Computational Linguistics: NAACI 2022, Seattle, WA, United States, July 10-15, 2022, pages 724-736. Association for Computational Linguistics.
Rujun Han, Xiang Ren, and Nanyun Peng. 2021. ECONET: effective continual pretraining of language models for event temporal reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5367-5380. Association for Computational Linguistics.
References
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question an- swering with knowledge graph. In COLING 2016, 26th International Conference Computational Lin- guistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages ACL.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In WSDM '21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021, pages 553 561. ACM.
1z Beltagy, Matthew F. Peters, and Arman Cohan. 2020a. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020b. Longformer: The long-document trans- former. CORR, abs/2004.05150.
Gautier Izacard and Edouard Grave. 202 la. Distilling knowledge from reader to retriever for question an- swering. In 9th International Conference Learn- ing Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.ne.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from guestion-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1533-1544, ACL.
Gautier Izacard and Edouard Grave 2021b. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguisties: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 874- 880. Association for Computational Linguisties.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y. Hammerla. 2019. Relational graph attention networks. CoRR, Abs/1104.00881.
Zhen Jia, Ahdalghani Abujahal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018a. Tem- pquestions: A benchmark for temporal question an- swering. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon, France, April 23-27, 2018, pages 1067-1062. ACM.
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive ques- tions. In Proceedings of the Neural Information Pro- cessing Systems Track on Datasets and Benchmarks 1, NeurIPS Dulasets and Benchmarks 2021, Decem- ber 2021, virtual.
Wentao Ding, Hao Chen, Huayu Li, and Yuzhong Qu. 2022. Semantic framework based query generation for temporal question answering over knowledge graphs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b. TEQUILA: temporal question answering over knowl- edge bases. Proceedings of the 27th ACM Inter- national Conference on Information and Knowledge

---

Management, CIKM 2018, Torino, Italy, October 22- 26, 2018, pages 1807-1810. ACM.
Kenton Lee, Ming-Wor Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Pupers, pages 6086 6096. Association for Computational Linguisties.
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal ques- tion answering on knowledge graphs. CIKM '21: The 30th ACM International Conference on Infor- mation and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pages 792-802. ACM.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, ACL 2020, Online, July 5-10, 2020, pages 1871-7880. Association for Computational Linguisties.
pre-training
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611. 1. Association for Computational Linguistics.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval -aagmeeeee generation for wwwwledeeintensiing NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 62, 2020, virtual.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4769-6781. Associa- tion for Computational Linguisties.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and Jingbo Zhu. 2022. On vi- sion features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 6327-6337. Association for Computa- tional Linguistics.
Akhil Kedia, Mohd Abbas Zaidi, and Haejun Lee. 2022. Fie: Building global probability space by leverag- ing early fusion in encoder for open-domain question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 4246-4260. Association for Computational Linguistics.
Jimmy Lin, Xueguang Ma. Sheng-Chiel Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356-2362.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format bound- aries with single QA system. In Findings of the Association for Computational Linguisties: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNIP 2020 of Findings of ACI., pages 1907. Association for Computational Linguisties.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-War Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Junru Lu, Xingwei Tan, Gabriele Pergola, Lin Gui, and Yulan He. 2022. srent-centric question answering via contrastive learning and invertible event transfor- mation. In Findings of the A Association for Computa- tional Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2377- 2389. Association for Computational I. Linguistics.
Yunshi Lan and Jing Jiang, 2020. Query graph genera- tion for answering multi-hop complex questions from knowledge bases. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020. Online, July 5-10. 2020, pages 969-974. Association for Computational Linguistics.
Puneet Mathur, Vlad I. Morariu. Verena Caynig-Fittkan Jiuxiang Gu, Franck Dernoncourt, Quan Hung Tran, Ani Nenkova, Dinesh Manocha, and Rajiv Jain. 2022. Doctime: A document- temporal dependency graph parser. In Proceedings of the 2022 Conference

---

of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI. 2022, Seatrle, WA, United States, July 10-15, 2022, pages 993-1009. Association for Computational Linguistics.
dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, RepANLP@ACL 2017, Vancouver, Canada, August 3, 2017, pages 191-200. Association for Computational Linguistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng. Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering ques- tions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 1178-1172. Association for Computational Linguis- tics.
Liang Wen, Houfeng Wang, Yingwei Luo, and Xiaolin Wang. 2022. M3: A multi-view fusion and multi- decoding network for multi-document reading com- prehension. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1450-1461, Association for Computational Linguistics.
OpenAI. 2022. Introducing chatgpt.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Manzil Zaheer, Guru Gurugamesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He. Yejin Choi. and Manaal Faruqui. 2021. TIME- DIAL: temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Lan- guage Processing, ACI/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7066-7076. Association for Computational Linguis- tics.
Michael J. Q Zhang and Eunsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into QA. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event Punta Cana, Dominicun Repub- lic, 7-11 November, 2021, pages 7371-7387, Associ- ation for Computational Linguistics.
Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn, Res., 11:140:1-140:5
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao. George Karypis, and Alex Smola. 2023. Multi- modal chain-of-thought reasoning in language mod- els. CoRR, abs/2302.00923.
Michael Sejr Schlichtkrull, Thomas N. Kipf. Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web- 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer Science, pages 593-607.660 Springer.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "going for walk": A study of temporal commonsense under- standing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing, Hong Kong, China, November 3-7, 2019, pages 3361- 3367. Association for Computational Linguistics.
2019,
Saurabh Srivastava, Mayur Patidar, Sudip Chowdhury, Puneet Agarwal, Indrajit Bhattacharya, and Gautam Shroff. 2021. Complex question answering on knowl- edge graphs using machine translation and multi-task learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, FACL 2021, Online, April 19 - 23, 2021, pages 3428-3439. Asso- ciation for Computational Linguistics.
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with min- imal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACI 2020, Online, July 5-10, 2020, pages 7579-7589, Association for Computational Linguis- tics.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension

---

Explicit and Implicit Questions Explicit Ques- tions: The timestamps in the questions appear in the document. Implicit Questions: (a) the times- A Appendix
Dataset
#Train
#Dev
#Test
TimeQA Easy
14308
3021
2997
A.1 Datasets Details
TimeQA Hard
14681
3087
3078
TimeQA TimeQA contains main dataset and a puman-paraphrased dataset, where the questions in the main dataset are synthesized by the templates the questions in the human-paraphrased dataset are rewritten manually by humans. The ques- tions in the dataset include both explicit and im- plicit forms. Explicit question types contain in- explicit and between-explicit, and implicit ques- tion types contain in-implicit, between- -impiiii,,,, before-implicit, and after-implicit. The easy split contains only explicit questions, and the hard split contains most implicit questions and few explicit questions. The statistics of the dataset are shown in Table 8 and Table 9.
TimeQA Human- Paraphrased Easy
1171
989
TimeQA Human-Paraphrann Hard
1171
989
SituatedQA Temp
6009
3423
2795
Table 8: Statistic TimeQA and SituatedQA.
Implicit
Explicit bet.
Split
bet.
bef.
aft.
in
Easy
92.43%
7.56%
Hard 37.75% 38.79% 8.32% 7.75% 7.37%
Table The proportion the #question of different categories in the TimeQA training set.
windows of length 100. We take the Pyserini (Lin et al., 2021) library, using DPR as the dense re- triever". We select the top 20 most similar seg- ments for each question as the context. The English Wikipedia dumps we used is of Feb 20, 2021.
SituatedQA The same question may have dif- ferent answers depending on the context. Situat- edQA is an open-domain QA dataset that requires a specific context to produce the correct answer. The dataset contains two types of context, tempo- ral and geographic, and we use the questions with the temporal context in this paper. The evaluation metrics in the original paper include EM-One (the answer exactly matches the correct context) and EM-Any (the answer matches any of the annotated contexts), and we use EM-One as the evaluation metric. The original task form of SituatedQA is open-domain QA, and we transform it into long- context QA by retrieving relevant fragments using dense retriever. Please refer to Appendix A.2 for the retrieval details.
A.3 Hyperparameters
The hyperparameters we used during our experi- ments shown in Table 10. We search through the listed hyperparameters and end up using the bolded ones each row.
Hyperparameters
Values
Learning Rate
20-5 5e-5
Batch Size
Warmup
20%
Num GNN Layers
3 7
Num Graph Relations
4 8
GNN Dropout
0.1 0.5 0.9
Beam Size
Training Epochs
X
0.01 0.001
Table 10: Hyperparameters
tamps in the questions do not appear in the docu- ment (b) vague timestamps, such as the 1990s and st century (c) the timestamps involving common- sense knowledge, such as World War II lasted from 1939 to 1945.
A.4 Stronger Reader with Long-context Adaptation
We perform long-context adaptation pre-training for models that do not support long-context in- put. We replace the reader backbone with Uni- fiedQA (Khashabi et al., 2020), a strong QA model. However, UnifiedQA is built on vanilla T5 (Raf- fel ct al., 2020) and does not support long con- text input, which needs additional adaptations. We use similar approach to FiD (Izacard and Grave,
Time-sensitive Questions We refer to the def- inition in (Chen et al., 2021): (a) each question contains a time identifier (b) changing the time identifier causes the answer to change (c) it requires temporal reasoning to answer the question.
A.2 Details of SituatedQA Retrieval
Following the DPR (Karpukhin et al., 2020), we divide Wikipedia into segments by no overlapping
'tttps: pottps://huntt http///gggggga......g co/ wwaaaeaoo//pp-cccccccccc encoder mul www.uiistttiii base

---

Easy
Hard
Method
Dev
Test
EM
F1
EM
F1
EM
F1
EM
F1
Zero-shot
gpt-3.5-tubbo
30.71
42.67
32,79
46.04
28,91
36,94
28,68
35.14
gpt-3.5-turbo@man max
43.14
53.92
43.46
56.18
37.14
45.08
37.64
43.32
text-davinci-003e mean
21.27
34.59
24.34
33.47
22.21
27.27
23.80
30.52
text-davinci-0036 @ max
32.40
49.05
38.02
51.09
34.26
43.20
38.44
46.83
Few-shot(n=5)
upt-3.5-turbo@mmaa
42.14
51.59
34.80
47.12
35.37
43.52
34.03
42.36
gpt-3.5-trroo@mmen max
54.67
62.92
44.98
58.66
45.69
54,41
41.63
51.71
ext-davvicci-003e @ mean
32.00
43.58
28.87
38.28
30,96
36.82
29.05
33.78
text-davinci-003:
42.54
53.91
40.64
51.70
39.87
48.43
42.23
48.58
Ours
MIGER
59.65
68.05
60.49
69. 04.44
52.31
60.66
53,19
61.42
MTGER++
59.20
67.70
60.95
69.89
52.02
61.06
54.15
62.40
Table 11: Entire results of LLM baseline the TimeQA dataset.
2021b), forcing the model to encode paragraphs separately at the encoder side, delaying the interac- tion across paragraphs to the decoder side, and use a FiD-style input format for training 3000 steps on long-context QA task for adaption. Finally, obtain UnifiedQA model adapted to long- context input. We replace the reader of MTGER with the long-context adapted UnifiedQA to obtain MIGER++.
A.7 Consistency Analysis
This section describes how construct perturbated questions. A question and time interval [s, e] can determine the answer. We randomly sample time interval [s e'] C [s, e]. which ensures that the an- swer to the question does not change. After that, we randomly select 100 questions that the model an- swers correctly (EM=1.0), perturb the timestamps of the questions, and caleulate the results after the perturbation. If the model still gives the correct answer after the perturbation, it is consistent; oth- erwise, it inconsistent.
A.5 Prompts
The prompts use are shown in Figure 4, and Figure We try three different prompts and choose the one that performs best on the TimeQA dev set. We maintain a sho- examples pool. For the k-shot prompt, examples are drawn from it at a time, and we ensure that these k examples can cover all question types in the current dataset split. Due the maximum length limitation of the LLMs, the few-shot examples cannot take the full context. Therefore, we filter some context irrel- evant to the answer based on the original annotation data to ensure that the context does not exceed the maximum length limit.
#cons.
consistency
(22)
#cons. + #incons.
A.8 An Example of Temporal Graph
Figure 3 illustrrtees heterogeneous temporal graph under the time-focused view, with the legend on the right. There are three kinds of edges in the factual layer: Intra-para Fact Edge connects fact nodes within the same paragraph, Inter-para Fact Edge connects fact nodes across paragraphs and Fact Aggregation Edge connects fact nodes and the global node. There are three types of edges in the temporal layer, Before, After and Overlap. de- pending the relative temporal relations between time nodes. Among them, Before and After are inverse edges of each other. Time-to-Fact Edge is the cross-layer edge for bridging fact and time nodes.
A.6 Entire Results of L.MMs
We conduct experiments using ept-3.5-turbof (Ope-
nAI, 2022) and text-davinci-003 (Ouyang et al., 2022) on the TimeQA dataset. The entire experi- ment results are shown in Table 11. Gpt-3.5-turbo has significantly outperformed text-davinci-000 in most cases, but they still have large gap compared to method.
A.9 Graph Construction Details
Document Segmentation We divide the docu- ment into chunks (or paragraphs) based on chapter
WW use gpt-3.5-turbo (May 24 Version). https:// platform. beatform.openai... com/

---

Factual Layer
Types
Maslow
Seppeeede inl Bil.
Fart Nadl
rettee...
Time Node
Strafjed
Global Mode
Edge Types
Wecsem L Smael..
Perkebeeol
Vedaii swien
Aneer
Temporal Layer
Tempural Relation Fdge
ERP3
yell 1984
Patofiso4
2001
Time- to--ttt--cttttt Edge
Sodiee
ARNYY
Intra-para Ldge
Inter-para Part Edgge
1900 Ileee
Fact Aggregation Edge
Ia 2556
2019
Figure 3: An example heterogeneous temporal graph under time-focused view. Best viewed in color.
headings, with similar content within each para- graph. (e.g. Early life, College career, Profession career, Retirement, etc.)
to all fact nodes (Fact Aggr. Edge). Unidiree- tional connections are taken from fact to time nodes (Time-to-Fae Edge).
Node Extraction We design regular expressions extract Time Expression from documents. We treat the extracted TimeX as time nodes containing 4 categories (In, Between, Before, After). The sentence which time nodes are located are treated as fact nodes, corresponding to the time nodes. Examples: time nodes (In 1999, Before March 1988, etc.), fact nodes (He went to UCLA, etc.)
Edge Construction Take a time-focused graph as an example. The graph contains temporal layer and factual layer.
Factual Layer: We construct fact edges accord- ing the document paragraphs divided in the first step. Dense connections (Inter-para Fact Edge) are taken for fact nodes within the same paragraph (two nodes in the same paragraph), and sparse connec- tions (Intra-para Fact Edge) are taken between fact nodes across paragraphs (two nodes are located in two adjacent paragraphs).
Temporal Layer: Each time node can be repre- sented by time interval (e.g. between 1923 and 1924 can be represented [1923, 1924]; before March 1988 can be represented as [-o0, (5)). There are three kinds of interval relations between two time nodes: before, after and overlap. We use this to connect time nodes (Temporal Relation Edge). It is worth noting that the connections be- tween time nodes are also dense within paragraphs and sparse across paragraphs.
Other connections: To mitigate the graph spar- sity problem, we introduce a global node to aggre- gate information, and global nodes are connected

---

Zero-shot Prompt
[User]
will give you question with context.
You need to answer my question based on the context.
you infer the answer from the context, then output your amswer; otherwise, if there is answer, oulput [unanswerable].
Do output anything else.
question:
context:
[Assistant]
answer:
Model generated...
Figure 4: An example of zero-shot prompt.
Few-shot Prompt
[User]
I will give you question with context.
You need to answer my question based on the context.
you can infer the answer from the context, then output your amswer; otherwise, if there is answer, output [unanswerable].
Do output anything else.
question: Where was Clarice Phelps educated from 2016 to 2019?
context: Clarice Phelps Clarice Evone Phelps American nuclear chemist rescarching the processing of radioactive transuranic elements the U.S.
answer: [unanswerable]
Several Examples
question:
Which team did the player Michael Rankine belong to before Feb 2002?
Michael Rankine Michael Lee Rankine boorn January 1985 English former professional footballer who played striker Early career
context:
answer: Doncaster Rovers
question:
context:
[Assistant]
answer: Model generated...
Figure S: An example of few-shot prompt.


Reviews:
[{"review_id": "kM5Gz25GCA", "reviewer": "Reviewer_dd6k", "summary": "", "questions": "1. It is not clear how the time-focued graph is constructed. What are the time nodes? According the description on page 3, I guess the edges in the time-focused graph contain 3 different relations? Is it right? If it is right, the graph is very sparse,, which will bring new issue. I suggest the authors give more details and it is neceesary to give one examples to answer the above questions. \n2. In Table 1, the authors use gpt-3.5. In the implementation details. the authors use one-single A100 GPU. Can GPT-3.5 run on the single A100?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful and valuable review, which plays an important role in improving our paper! We address and clarify the questions and concerns below.\n\n**Re: How the time-focused graph is constructed?**\n\nConstructing the temporal graph is divided into three steps: Document Segmentation, Node Extractions and Edge Construction.\n\n  1. Document Segmentation: We divide the document into chunks(paragraphs) based on chapter headings, with similar content within each paragraph. (e.g. Early life, College career, Profession career)\n\n  2. Node Extraction:\n     1. We design regular expressions to extract Time Expression from documents.\n     2. We treat the extracted TimeX as time nodes, which contain 4 categories  (In, Between, Before, After). The sentence which time nodes are located are treated as fact nodes, corresponding to the time nodes. \n     3. Example: Time node (In 1999, Before March 1988, etc.), fact nodes (He went to UCLA, etc.)\n     \n  3. Edge Construction\n     1. Take time-focused graph as example, the graph contains a temporal layer and a factual layer. Figure 3 in the Appendix shows a temporal graph.\n     \n     2. Factual Layer: We construct fact edges according to the Document paragraphs divided in the first step. Dense connections (Inter-para Fact Edge) are taken for fact nodes within the same paragraph (two nodes in the same paragraph), and sparse connections (Intra-para Fact Edge) are taken between fact nodes across paragraph (two nodes are located in two adjacent paragraphs).\n     \n     3. Temporal Layer: Each time node can be represented by a time interval (e.g. between 1923 and 1924 can be represented as [1923, 1924]; before 1999 can be represented as [~, 1999]). There are three kinds of interval relations between two time nodes, before, after and overlap. We use this to connect time nodes (Temporal Relation Edge). It is worth noting that the connections between time nodes are also dense within paragraphs and sparse across paragraphs.\n     \n     4. Other connections: To mitigate graph sparsity problem, we introduce a global node to aggregate information, and global nodes are connected to all fact nodes (Fact Aggr. Edge). Unidirectional connections are taken from fact nodes to time nodes (Time-to-Fact Edge).\n     \n\n**Re: What are the time nodes**\n\n1. The time nodes are time expressions extracted from the document. There are four types of time nodes accordingg to the type of time expression.\n\n2. Each time node represents a time interval, for example(time, time interval): (In 1999, [1999, 1999]; Before March 1988, [~, 1988.25], etc.)\n\n**Re: Is time-focused graph contain 3 different relations**\n\nThe time-focused graph contains a total of 7 types of edge.  Three types in temporal layer (Before, After, Overlap); three types in factual layer (Intra-para, Inter-para, Fact Aggregation); and one type connects the factual and temporal layers (Time-to-Fact).\n\n**Re: Graph sparse problem**\n\nWe take two approaches to mitigate the sparsity problem.\n\n  1. We chunk the document by chapter into paragraphs and connect nodes densely within paragraphs and sparsely across paragraphs. This solution alleviates the problem of sparsity while preventing connections from becoming too dense.\n  2. We use global node to aggregate information from fact nodes to alleviate the problem of difficult information interaction between two nodes that are too far apart.\n\n**Re: Question about implementation details**\n\n1. All supervised experiments (FiD, BigBird, LED, LongT5, MTGER, etc.) were run on a single A100 GPU.\n2. All LLM experiments (text-davinci-003, gpt3.5) were done by calling OPENAI-API.\n\n**General Response (Complementary experiments)**\n\nWe conduct complementary experiments on two reader models which are capable of handling long text input, LED[1] and LongT5[2].\nThe results of the experiment are shown in the table below. Our method still significantly outperforms all baselines.\n\n  [1] Longformer: The Long-Document Transformer, 2020\n\n  [2] LongT5: Efficient Text-To-Text Transformer for Long Sequences, 2022\n\nTable1: Experimental results on the TimeQA dataset (* represent complementary experiments). \n\n| Method         | Dev-Easy-EM | Dev-Easy-F1 | Test-Easy-EM | Test-Easy-F1 | Dev-Hard-EM | Dev-Hard-F1 | Test-Hard-EM | Test-Hard-F1 |\n| -------------- | :---------: | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ | ------------ |\n| LED*           |    46.97    | 56.41       | 49.41        | 58.346       | 39.78       | 48.45       | 40.50        | 48.1         |\n| LongT5_newsqa* |    48.17    | 59.59       | 48.14        | 58.94        | 41.43       | 52.58       | 42.94        | 53.50        |\n| LongT5_squad*  |    48.67    | 59.54       | 50.85        | 61.00        | 41.64       | 52.63       | 41.93        | 52.07        |\n| BigBird_nq     |    51.89    | 62.43       | 50.61        | 60.88        | 44.04       | 53.83       | 43.56        | 53.60        |\n| FiD_nq         |    57.25    | 66.56       | 58.06        | 67.52        | 49.21       | 58.00       | 49.80        | 58.74        |\n| **MTGER**      |  **59.65**  | **68.05**   | 60.49        | 69.44        | **52.31**   | 60.66       | 53.19        | 61.42        |\n| **MTGER++**    |    59.20    | 67.70       | **60.95**    | **69.89**    | 52.02       | **61.06**   | **54.15**    | **62.40**    |\n\nTable2: Experimental results on the SituatedQA dataset (* represent complementary experiments). .\n\n\n| **Method**    | **Static** | **Samp.** | **Start** | **Avg.** |\n| ------------- | ---------- | --------- | --------- | -------- |\n| LED*           | 18.1       | 16.2      | 20.4      | 17.8     |\n| LongT5_newsqa* | 22.0       | 14.4      | 18.3      | 16.8     |\n| LongT5_squad*  | 14.5       | 13.4      | 22.0      | 16.2     |\n| BigBird_nq    | 34.2       | 15.7      | 21.3      | 20.2     |\n| FiD_nq        | 28.7       | 22.2      | **31.8**  | 26.4     |\n| **MTGER**   | **45.0**   | **24.8**  | 28.2      | **28.8** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research topic is interesting and the solution is clear. \n2. The results show that the proposed method outperforms than other baselines.", "reasons_to_reject": "1. It is not clear how the time-focued graph is constructed. What are the time nodes? According the description on page 3, I guess the edges in the time-focused graph contain 3 different relations? Is it right? If it is right, the graph is very sparse,, which will bring new issue. I suggest the authors give more details and it is neceesary to give one examples to answer the above questions. \n2. In Table 1, the authors use gpt-3.5. In the implementation details. the authors use one-single A100 GPU. Can GPT-3.5 run on the single A100?", "questions_to_authors": "1. It is not clear how the time-focued graph is constructed. What are the time nodes? According the description on page 3, I guess the edges in the time-focused graph contain 3 different relations? Is it right? If it is right, the graph is very sparse,, which will bring new issue. I suggest the authors give more details and it is neceesary to give one examples to answer the above questions. \n2. In Table 1, the authors use gpt-3.5. In the implementation details. the authors use one-single A100 GPU. Can GPT-3.5 run on the single A100?", "paper_topic_and_main_contributions": "This paper presents a multi-view temporal graph enhanced temporal reasoning over time-involved document. The authors also conduct extensive experiments on two time-invoved QA datasets. The research topic is interesting and the solution is clear. The results show that the proposed method outperforms than other baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cFUEBPtc7T", "reviewer": "Reviewer_ZXhY", "summary": "", "questions": "A) How to obtain the time nodes? Are the time expressions annotated in the datasets, or extracted by another tool?\n\nB) How to judge the relationships between time nodes and fact nodes? Are they annotated in the datasets, or classified by another tool? \n\nC) How to classify the type of time nodes? For example, in Figure 3, what type does the node ``15 years old'' belong to?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful and valuable review, which plays an important role in improving our paper! We address and clarify the questions and concerns below.\n\n**Re: How to obtain the time nodes? Are the time expressions annotated in the datasets, or extracted by another tool?**\n\n1. We design regular expressions to extract time expressions from documents.\n2. We treat the extracted TimeX as time nodes, which contain 4 categories  (In, Between, Before, After).  Each time node indicates the time interval in which the event occurred.\n3. Example(time, time interval): (In 1999, [1999, 1999]; Before March 1988, [~, 1988.25], etc.)\n\n**Re: How to classify the type of time nodes? For example, in Figure 3, what type does the node '15 years old' belong to?**\n\n1. We categorize time nodes based on the textual features of the temporal expression (e.g. before 2000 is categorized as Before).\n    1. before, until are categorized as Before type.\n    2. between ... and and from ... to are categorized as Between type.\n    3. In, on and at are categorized as In type.\n    4. after and since are categoried as After type.\n2. For some other types of abstract time expressions, we categorize as In type. (e.g. 1940s, xx years old, xx century, etc.). \n\n**Re: How to judge the relationships between time nodes and fact nodes? Are they annotated in the datasets, or classified by another tool?**\n\nThe relations between nodes are rule-based rather than from annotations. \n\nBefore introducing how to construct edges, it is necessary to introduce document segmentation.\n\n1. Document Segmentation: We divide the document into chunks (paragraphs) based on chapter headings, with similar content within each paragraph. (e.g. Early life, College career, Profession career)\n2. Edge between fact nodes: We construct fact edges according to the Document segmentation. Dense connections (Inter-para Fact Edge) are constructed for fact nodes within the same paragraph (two nodes in the same paragraph), and sparse connections (Intra-para Fact Edge) are constructed between fact nodes across paragraph (two nodes are located in two adjacent paragraphs).\n3. Edge between time nodes: Each time node can be represented by a time interval (e.g. between 1923 and 1924 can be represented as [1923, 1924]; before 1999 can be represented as [~, 1999]). There are three kinds of interval relations between two time nodes, before, after and overlap. We use this to connect time nodes (Temporal Relation Edge). It is worth noting that the connections between time nodes are also dense within paragraphs and sparse across paragraphs.\n   1. Example (time1, time2, time relation): (In 1999 [1999, 1999], Before 1988 [~, 1988], Overlap), (After 2005 [2005, ~], Between March 2001 and Sep 2002 [2001.25, 2002.75], After)\n4. Other connections: To prevent graph sparsity, we introduce a global node to aggregate information, and global nodes are connected to all fact nodes (Fact Aggregation Edge). Unidirectional connections are taken from fact nodes to time nodes (Time-to-Fact Edge).\n\n**Re: The supervised baselines are from 2020. The comparison of more recent baselines, if available, may better demonstrate the effectiveness of the proposed method.**\n\n1. We conduct complementary experiments on two reader models which are capable of handling long text input, LED[1] and LongT5[2].\n\n   The results of the experiment are shown in the table below. Our method still significantly outperforms all baselines.\n\n   [1] Longformer: The Long-Document Transformer, 2020\n\n   [2] LongT5: Efficient Text-To-Text Transformer for Long Sequences, 2022\n\nTable1: Experimental results on the TimeQA dataset (* represent complementary experiments). \n\n| Method         | Dev-Easy-EM | Dev-Easy-F1 | Test-Easy-EM | Test-Easy-F1 | Dev-Hard-EM | Dev-Hard-F1 | Test-Hard-EM | Test-Hard-F1 |\n| -------------- | :---------: | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ | ------------ |\n| LED*           |    46.97    | 56.41       | 49.41        | 58.35       | 39.78       | 48.45       | 40.50        | 48.1         |\n| LongT5_newsqa* |    48.17    | 59.59       | 48.14        | 58.94        | 41.43       | 52.58       | 42.94        | 53.50        |\n| LongT5_squad*  |    48.67    | 59.54       | 50.85        | 61.00        | 41.64       | 52.63       | 41.93        | 52.07        |\n| BigBird_nq     |    51.89    | 62.43       | 50.61        | 60.88        | 44.04       | 53.83       | 43.56        | 53.60        |\n| FiD_nq         |    57.25    | 66.56       | 58.06        | 67.52        | 49.21       | 58.00       | 49.80        | 58.74        |\n| **MTGER**      |  **59.65**  | **68.05**   | 60.49        | 69.44        | **52.31**   | 60.66       | 53.19        | 61.42        |\n| **MTGER++**    |    59.20    | 67.70       | **60.95**    | **69.89**    | 52.02       | **61.06**   | **54.15**    | **62.40**    |\n\nTable2: Experimental results on the Situated dataset (* represent complementary experiments). .\n\n\n| **Method**     | **Static** | **Samp.** | **Start** | **Avg.** |\n| -------------- | ---------- | --------- | --------- | -------- |\n| LED*           | 18.1       | 16.2      | 20.4      | 17.8     |\n| LongT5_newsqa* | 22.0       | 14.4      | 18.3      | 16.8     |\n| LongT5_squad*  | 14.5       | 13.4      | 22.0      | 16.2     |\n| BigBird_nq     | 34.2       | 15.7      | 21.3      | 20.2     |\n| FiD_nq         | 28.7       | 22.2      | **31.8**  | 26.4     |\n| MTGER(ours)    | **45.0**   | **24.8**  | 28.2      | **28.8** |\n\n2. Explain baseline selection:\n\n   1. Our task is performed on long text, which typically accepts inputs larger than 4000 tokens. This leads to that most QA models would be inapplicable due to the input length limitation. Therefore our selected baseline methods must be able to handle long text inputs, such as BigBird, LED, LongT5 (by sparse attention mechanism) and FiD (by segmentating document and delaying interactions). Among them, FiD and BigBird are generative and extractive SOTA long text readers, respectively.\n\n\n   2. Why don't we select some ODQA methods as a generic baseline, e.g. DPR[3], KG-FiD[4]:\n\n      Open-domain Q&A methods usually use the long-context model as a base reader, and improve the performance of the whole system by retrieval and reranking module.  However, our task is performed over given long documents that do not need retrieving and reranking, so these ODQA methods are equivalent to the long-context reader model when the retrieval and reranking modules are removed. Therefore we do not select methods specifically designed for the ODQA task as general baselines (e.g. KG-FiD[3], which is equivalent to FiD when removing the reranking module and DPR is equivalent to BERT when removing the retrieval module).\n\n       [3] Dense Passage Retrieval for Open-Domain Question Answering, 2020\n\n       [4] KG-FiD: Infusing knowledge graph in fusion-in-decoder for opendomain question answering, 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of constructing the temporal graph sounds reasonable and interesting.\n2. This paper proposes a novel framework to fuse temporal and factual information during reasoning.\n3. This paper conducts extensive experiments to show the effectiveness of the proposed method.", "reasons_to_reject": "1. Some details in the graph construction part are not clearly described.\n2. The supervised baselines are from 2020. The comparison of more recent baselines, if available, may better demonstrate the effectiveness of the proposed method.\n", "questions_to_authors": "A) How to obtain the time nodes? Are the time expressions annotated in the datasets, or extracted by another tool?\n\nB) How to judge the relationships between time nodes and fact nodes? Are they annotated in the datasets, or classified by another tool? \n\nC) How to classify the type of time nodes? For example, in Figure 3, what type does the node ``15 years old'' belong to?\n", "paper_topic_and_main_contributions": "This paper focuses on temporal reasoning over documents, where QA models are required to answer time-related questions based on documents. This paper is inspired by the human reasoning process. Specifically, this paper extracts the temporal relations and discourse relations explicitly and constructs a multi-view heterogeneous temporal graph. Then, it models the relations via a GNN-based graph reasoning component. Finally, it generates the answer via a time-enhanced decoder. The experimental results show the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0MCV60HmtT", "reviewer": "Reviewer_DJLB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful and valuable review, which plays an important role in improving our paper! We address and clarify the questions and concerns below.\n\n**Re: Why to segment heterogeneous temporal graph into two views and then fuse them?**\n\n1. Motivation: Multiple views make it possible to model both absolute relationships between time expression (e.g. 1995 is before 2000 because 1995 < 2000) and relative relationships between events (e.g. Messi joins Inter Miami CF after his World Cup championship). If there is only one view, it is difficult to model both relationships simultaneously.\n2. Explanation:\n   1. Modeling relationships between time nodes (time expressions) in the time-focused view focuses more on numerical comparisons, i.e., absolute magnitude relationships of the time in time expression.\n   2. Modeling relationships between fact nodes in the fact-focused view focuses more on the sequence of events, i.e., the relative temporal relationship of events.\n   3. We model these two temporal relationships simultaneously through a multi-view mechanism and make absolute and relative temporal relations complementary through a fusion mechanism, which makes time modeling more adequate.\n\n3. Example: \n   1. Suppose there are two event-time pairs, (go to high school, In 1988) and (go to university, In 1992). \n   2. In the time-focused view, the relation is (In 1998, After, In 1992), which models the absolute relation of time expression (1998 > 1992, so the relation is After).\n   3. In the fact-focused view, the relation is (go to high school, before, go to university), which models the relative temporal relation of events.\n\n**Re: Why to use different baselines in different QA experiments.**\n\n1. We include BART and DPR in the table simply follows the results of the original paper and has no other implications. If this creates ambiguity and misunderstanding, we can remove these two results from the table.\n\n2. Our task form is QA over long context, and we need to pick methods that can handle long text input as a baseline. Thus we select FiD and BigBird as the generalized baseline for both datasets. FiD and BigBird are SOTA generative and extractive long text reader model , respectively.\n\n3. BART and DPR are the baselines for closed-book setting and open-ended setting in the original SituatedQA paper, respectively. Our task form is neither closed-book nor open-ended but QA over long context, therefore FiD and BigBird are the baselines that should be compared.\n\n\n\n**Re: More state-of-the-art QA baselines are needed**\n\n1. We conduct complementary experiments on two reader models which are capable of handling long text input, LED[1] and LongT5[2].\n\n    The results of the experiment are shown in the table below. Our method still significantly outperforms all baselines.\n\n   [1] Longformer: The Long-Document Transformer\n\n   [2] LongT5: Efficient Text-To-Text Transformer for Long Sequences\n\n  Table1: Experimental results on the TimeQA dataset (* represent complementary experiments). \n\n| Method         | Dev-Easy-EM | Dev-Easy-F1 | Test-Easy-EM | Test-Easy-F1 | Dev-Hard-EM | Dev-Hard-F1 | Test-Hard-EM | Test-Hard-F1 |\n| -------------- | :---------: | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ | ------------ |\n| LED*           |    46.97    | 56.41       | 49.41        | 58.35        | 39.78       | 48.45       | 40.50        | 48.1         |\n| LongT5_newsqa* |    48.17    | 59.59       | 48.14        | 58.94        | 41.43       | 52.58       | 42.94        | 53.50        |\n| LongT5_squad*  |    48.67    | 59.54       | 50.85        | 61.00        | 41.64       | 52.63       | 41.93        | 52.07        |\n| BigBird_nq     |    51.89    | 62.43       | 50.61        | 60.88        | 44.04       | 53.83       | 43.56        | 53.60        |\n| FiD_nq         |    57.25    | 66.56       | 58.06        | 67.52        | 49.21       | 58.00       | 49.80        | 58.74        |\n| **MTGER**      |  **59.65**  | **68.05**   | 60.49        | 69.44        | **52.31**   | 60.66       | 53.19        | 61.42        |\n| **MTGER++**    |    59.20    | 67.70       | **60.95**    | **69.89**    | 52.02       | **61.06**   | **54.15**    | **62.40**    |\n\nTable2: Experimental results on the Situated dataset (* represent complementary experiments). .\n\n\n| **Method**     | **Static** | **Samp.** | **Start** | **Avg.** |\n| -------------- | ---------- | --------- | --------- | -------- |\n| LED*           | 18.1       | 16.2      | 20.4      | 17.8     |\n| LongT5_newsqa* | 22.0       | 14.4      | 18.3      | 16.8     |\n| LongT5_squad*  | 14.5       | 13.4      | 22.0      | 16.2     |\n| BigBird_nq     | 34.2       | 15.7      | 21.3      | 20.2     |\n| FiD_nq         | 28.7       | 22.2      | **31.8**  | 26.4     |\n| **MTGER**    | **45.0**   | **24.8**  | 28.2      | **28.8** |\n\n2. Explain baseline selection:\n\n   1. Our task is performed on long text, which typically accepts inputs larger than 4000 tokens. This leads to that most QA models would be inapplicable due to the input length limitation. Therefore our chosen baseline methods must be able to handle long text inputs, such as BigBird, LED, LongT5 (by sparse attention mechanism) and FiD (by segmentating document and delaying interactions). Among them FiD and BigBird are generative and extractive SOTA long text readers, respectively.\n\n\n   2. Why don't we select some ODQA methods as a generic baseline, e.g. DPR[3], KG-FiD[4]:\n\n        Open-domain Q&A methods usually use the long-context model as a base reader, and improve the performance of the whole system by retrieval and reranking module.  However, our task is performed over given long documents that do not need retrieving and reranking, so these ODQA methods are equivalent to the long-context reader model when the retrieval and reranking modules are removed. Therefore we do not select methods specifically designed for the ODQA task as general baselines (e.g. KG-FiD[3], which is equivalent to FiD when removing the reranking module and DPR is equivalent to BERT when removing the retrieval module).\n   [3] Dense Passage Retrieval for Open-Domain Question Answering, 2020\n\n   [4] KG-FiD: Infusing knowledge graph in fusion-in-decoder for opendomain question answering, 2022\n\n**Re: The experimental analyses are not sufficient. (Table 5, Results of SituatedQA)**\n1. I think the reason why our method performs slightly worse on the Start type may be that the model relies too much on the temporal graph to solve the problem.\n2. Static type questions involve a large number of timestamps, Sampled type questions involve a large number of time intervals. While Start type problems only contain timestamps for the start time of the answer.  In other words, questions of type Start contain fewer time expressions. Since rich temporal graph can be constructed when facing the first two types of question, the model relies on temporal graphs to answer complex questions. However, when faced with Start type questions, the content of temporal graphs is relatively insufficient, resulting in a slight performance degradation.\n3. Despite the slight performance degradation on Start type, our method significantly outperforms baselines in the remaining types as well as in the overall performance, demonstrating the effectiveness of our method.\n\n**Re: The experimental analyses are not sufficient. (Table 3, Ablation Study)**\n\nWe will explain this from both experimental and module design perspectives.\n\n\n|  Method   | Dev-Easy  | Dev-Hard   | Test-Easy  |Test-Hard\n|  ----  | ----  | ----  | ----  | ----  |\n|  MTGER  | **59.65**  | **52.31**  | 60.49  | **53.19**  |\n|  -Time Cmp (original paper) | **59.65**  | 51.53  | **60.66**  | 52.65  |\n|  performance gap | -0.00  | -0.78  | +0.17  | -0.54  |\n|  -Time Cmp (rerun)  | 59.58  | 51.44  | 60.42 | 52.69  |\n|  performance  gap  | -0.07  | -0.87 | -0.07  | -0.5  |\n|  average  gap  | -0.035  | -0.825 | +0.05  | -0.52  |\n1. Experimental perspective: We rerun the experiment three times and obtained average results as shown above.\n\n   1. As shown above, the effect of removing this module on the performance in the Easy-split is less than 0.1% ($\\frac{0.05}{60.49}$), which is negligible.\n   2. Based on the repeated experimental results, we believe that the performance fluctuation comes from random factors, and removing the module does not isignificantly improve or reduce the performance in the Easy-split.\n\n\n2. Module design perspective\n\n   1. The time comparison module is designed to strengthen the model's implicit reasoning ability, while all of the questions in the easy-split do not need implicit reasoning.\n\n   2. Therefore removing this module should have little or no effect on the performance of the Easy-split. And the experimental results are also in line with our expectations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The studied task, document-level textual temporal reasoning, is important. Extensive experiments demonstrate the effectiveness of the proposed method to a certain degree.", "reasons_to_reject": "There are some main weaknesses:\n1. The design of the proposed method is not well-motivated. Why to segment heterogeneous temporal graph into two views and then fuse them? Keeping all nodes and edges in a single graph is better for information integrity consideration.\n2. State-of-the-art QA baselines after 2022 are needed. Besides, why to use different baselines in different QA experiments.\n3. The experimental analyses are not sufficient. For example, in Table 3, the proposed method is poorer than FiD_{nq} on Start type and in Table 5, removing time comparing is better than applying it on the easy split of the test set. However, no further explanation is provided.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a multi-view temporal graph enhanced reasoning framework for temporal reasoning over time-involved documents, which explicitly models the temporal relationships among facts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper Topic And Main Contributions:
* This paper proposes a multi-view temporal graph enhanced reasoning framework for temporal reasoning over time-involved documents, which explicitly models the temporal relationships among facts. The authors conduct extensive experiments on two time-invoved QA datasets.

Reasons to accept:
* The topic is interesting and the solution is clear. The results show that the proposed method outperforms baselines.

Reasons to reject:
* Some details in the graph construction part are not clearly described. The authors explain more details in the rebuttal.
* The supervised baselines are not up-to-date. The authors add results in the rebuttal that show the superiority over additional, newer methods for long-input decoding: LED and LongT5.
Decision: Accept-Findings