Once Upon a ${\it Time}$ in ${\it Graph}$: Relative-Time Pretraining for Complex Temporal Reasoning
Authors: Sen Yang
Xin Li
Lidong Bing
Wai Lam
Venue: EMNLP 2023 Main (2023)
Keywords: Temporal Question Answering
Time-aware Pre-training
TL;DR: We devise a graph view on temporally-scoped events and thus propose a time-aware pre-training method for complex temporal reasoning.

Abstract: Our physical world is constantly evolving over time, rendering challenges for pre-trained language models to understand and reason over the temporal contexts of texts. Existing work focuses on strengthening the direct association between a piece of text and its time-stamp. However, the knowledge-time association is usually insufficient for the downstream tasks that require reasoning over temporal dependencies between knowledge. In this work, we make use of the underlying nature of time, all temporally-scoped sentences are strung together through a one-dimensional time axis, and suggest creating a graph structure based on the relative placements of events along the time axis. Inspired by the graph view, we propose \textsc{RemeMo} ($\underline{Re}lative Ti\underline{me} \underline{Mo}deling$), which explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. Experimental results show that \textsc{RemeMo} outperforms the baseline T5 on multiple temporal question answering datasets under various settings. Further analysis suggests that \textsc{RemeMo} is especially good at modeling long-range complex temporal dependencies.

Once Upon a Time in Graph:
Relative-Time Pretraining for Complex Temporal Reasoning
Sen Yang 123
Xin Li 231 Lidong Bing 2
Wai Lam
The Chinese University of Hong Kong
DAMO Academy, Alibaba Group
3 Hupan Lab, 310023, Hangzhou, China
senyang stu@gmail.com com (xinting. 1x, 1. (alng))@libaba-innccoom
wlam@se. cuhk. ek
Abstract
Context: LeBron Raymone James Sr. (born December 30, 1984) is an American professional baskethall player ...... James was drafted first overall by the Cleveland Cavaliers in 2003. ..... In 2010, James famously announced his decision to join the Miami Heat, ..... Jumes returned to the Cleveland Cavaliers in 2014 and led the team to their first ever NBA championship in 2016, ending the city s 52-year championship drought. ......
Our physical world is constantly evolving over time, rendering challenges for pre-trained lan- guage models understand and reason over the temporal contexts of texts. Existing work focuses on strengthening the direct association between piece of text and its time-stamp. However, the knowledge-time association is downstream tasks
Conventional Question Answering Question (i): What rank was LeBron James drufted?
Answer (i): First overall.
usually insufficient that require reasoning over temporal dependen- cies between knowledge. In this work, we make use of the underlying nature of time, all temporally-scoped sentences are strung to- gether through ame-dimensiona, time axis, and suggest creating graph structure based the relative placements of events along the time axis. Inspired by the graph view, we propose REMEMO (Relative Time Modeling). which explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. Experimental results show that REMEMO outperforms the baseline T5 on mul- tiple temporal question answering datasets un- der various settings. Further analysis sug- gests that REMEMO especially good at modeling long-range complex temporal de- pendencies. We release our code and pre- trained checkpoints at https: thub. com/ MANO-NLP-SG/RemeMM
Temporal Question Answering
Question (ii): What team LeBron James play for 2003?
Answer (i)): Cleveland Cavaliers.
Question (ii):: What team did LeBron James play for in 2012?
Answer (iii): Miami Heat.
Question (iv): What team did LeBron James play for after he left the Miami Heat?
Answer (iv): Cleveland Cavaliers.
Question (v): What team did LeBron James play for when got into his first NBA Finals?
Answer (V): Cleveland Cavaliers.
Table Examples of conventional QA and temporal QA. In conventional QA, the answer to the question does not change over time. In temporal QA, the answer depends both the question and the temporal context.
context of the question changes. It is thus neces- sary for PLMs to understand the temporal context of user queries and reason over temporally-scoped facts.
1 Introduction
Knowledge plays a crucial role in downstream NLP applications. Pre-trained language models (PLMs) generate accurate responses various user queries by reasoning over their internal knowledge and the knowledge contained in texts.
To enhance the temporal understanding capa- bility PLMM, recent studies have focused on strengthening the one-to-one associations between an event and its corresponding time tokens or time stamps (Dhingra al., 2022; Rosin and Radinsky, 2022; Rosin al., 2022). By modeling such super- ficial temporal dependencies, these methods gain improvements on various sime-sensitive tasks.
Real-world knowledge, however, frequently ex- pires and updates. Taking example in Table 1, the correct answer the question "Whal leam did LeBron James play for?" varies when the temporal
These methods, however, assume all temporally- scoped facts as independent of cach other, ignor- ing the complex temporal dependencies among
This work was supported by Alibaba Group through the Alihaba Innovative Research (AIR) Program (TA2217728). XL is the corresponding author.

---

lemporally-scoped facts. This may pose limita- tions because time-related queries often require reasoning over multiple temporally-scoped facts. For example, in Table 1, Question (iv) Whha team did LeBron James play for after he left the Miami Heat?" requires reasoning over two facts: (I) James "join the Miami Heat in 2010" and (2) James "returned to the Cleveland Cavaliers in 2014". By performing the temporal reasoning over path: {"2014" is later than "2010": "returned to the Cleveland Cavaliers" is associated with "in 2014"; "join the Miami Heaf" is associated with in 2010") } - ("returned to the Cleveland Cava- liers" is later thin""oon the Miami Hear"), Ques- tion (iv) can be answered by combining the two facts together. Modeling such complex temporal dependencies benefits downstream time-sensine sensitive tasks, but requires the PLM to understand the con- cept of time beyond the superficial association be- tween one single fact and its time stamp.
tive, the LM objective can be seen as node feature learning, while the TRC objective operates edge classification.
REMEMO is pre-trained on regular corpora and temporally-scoped sentences extracted from Wikipedia articles. It is then evaluated seven downstream temporal open open-book question answer- ing datasets. REMEMO consistently gives better results compared with the baseline T5 model under multiple settings.
Overall, our contributions are summarized as follows:
We devise a graph view represent temporally-scoped events. Grounded on that, we propose TRC, a new pre-training objective that exploits relative time to systematically model complex temporal dependencies.
We pre-train REMEMO jointly with the LM objective and the proposed TRC objective. Extensive experiments show that REMEMO gains significant improvements over T5 base- lines under various settings.
The most well-known concept of time is the linear concept, which views time as a series of moments that unfold sequentially from the past, through the present, and into the future. As shown in Figure la, cemporally-scoper events are strung together through the one-dimensional time axis. By inspecting the relative positions between any two events, we can build fully-connected digraph, where events act as graph nodes and temporal re- lations among them serve as directed graph edges, as shown in Figure 1b. Complex temporal depen- dencies can thus be systematically modeled from the perspective of graph learning. For example, as shown in Figure 1b, Fact-1 1 (as the start node) is associated with Fact-2 (as the destination node) ei- ther directly through the earlier edge or indirectly by passing through two edges via Fact-3.
We present processing pipeline to obtain normalized time tags for raw texts.
Related Work
Temporal Question Answering Temporal ques- tion answering has attracted much research atten- tion in recent years. An early line of work fo- cused on temporal question answering over knowl- edge graphs, for which several datasets were built, such as TempQuestions (Jia et al., 2018b), TEQUILA (Jia et al., 2018a), TimeQuestions (Jia et al., 2021) and CRONKGQA (Saxena et al., 2021). Under this temporal KGQA setting, the model is required to do complex temporal reasoning over temporal knowledge graphs and to select an entity node from a given node list as the answer.
Inspired by this graph view, we present Relative- Time Modeling (REMEMO): a novel time-aware pre-training framework that exploits relative posi- tions on the time-axis to model complex temporal dependencies. Specifically, we extract time-span tags from sentences and then pre-train the PLM with two joint objectives: (i) the Language Mod- eling (L.M) objective; (ii) novel Time Relation Classification (TRC) objective, which classifies the relation between any two sentences, i.e., Sentence- A is earlier, later or contemporary 1 compared with Sentence-B. From the graph learning perspec-
More recently, another line of work has fo- cused on temporal question answering either un- der the closed booed-book setting to evaluate models' internal memorization of temporal facts or under the open-book setting to evaluate models' tempo- ral understanding and reasoning capability over unstructured texts. Some example datasets in- clude TempLAMA (Dhingra al., 2022), Situat- edQA (Zhang and Choi, 2021), TimeQA (Chen et al., 2021), ArchivalQA (Wang et al., 2022).
WWe omit the mull label for simplicity, which corresponds to either Sentence-A A or Sentence-B B does not contain a valid time-span tag.

---

the lears thear line awer NSS champiorafio 2015.
Faxt
James returned the Cleveland Cavallers 2014 the their finst ever NBA championship in 2018.
Golden State Warriors the Cleveland Cavalees in six games in the 2015 NBA Finals.
2010, James famously announced to join the Marni Hest.
Fact
Golden State Wirmore delected the Coveland Canaler garric
fumousiy arnouncod it's docsion lo the
Fact
Fact
Fact
Lakerr
Earter
2011
2013
2016
2017
Contemporary
(a) Temporally-sceee scoped facts strung together through time-axis.
(b) A fully-comeccted digraph consisted of facts and time relations.
Figure 1: A fully-conmen cully-connected digraph (Figure 1b) can be constructed based on Figure the graph, each piece of fact serves as the head and tail nodes for multiple directed edges specificalll. temporal relations.
StreamingQA (Liska et al., 2022) and TempRea- son (Tan et al., 2023). Among them, SituatedQA, TimeQA and TempReason are designed to evaluate the temporal understanding and reasoning abilities of the model, so we adopt them to evaluate RE- MEMO.
the temporal relations between event pairs from text. The main concept shared by our work and pre- vious ETRC works is the temporal relation between two events. However, some task differences exist. Our work focuses temporal question answering (TQA), in which most clues in existing datasets are explicit and coarse-grained time expressions, such as "June 2012", "18th century" and "1980s". In contrast, the line of ETRE papers focused on time information that more implicit, fine- fineggraindd, and nuanced. Here is an example from Cassidy et al. (2014):
Time-aware Modeling Lots of work has at- tempted to incorporate time into the modeling process (Wijaya and Yeniterzi, 2011; Hamilton et al., 2016; Bamler and Mandt, 2017; Frermann and Lapata, 2016; Dubossarsky et al., 2019; Giu- lianelli et al., 2020; Röttger and Pierrchumbert, 2021; Lazaridou et al., 2021; Qin et al., 2021; Luu et al., 2022: Su et al., 2022; Zhao et al., 2022; Cao and Wang. 2022). Among them, some studies that are relevant to this paper explore time-aware pre-training by modeling the one-to-one associa- tion between a piece of text and its corresponding time-stamp. For example, Dhingra et al. (2022) pre-pended document-creatioo time (e.g.. "year: 2018")) the beginning of each document so that the PLM better memorizes a fact along with its published time; Rosin et al. (2022) masked out time-informing tokens in a sentence for masked lan- guage modeling (MLM) training; Rosin and Radin- sky (2022) added a sentence-level time-slot embed- ding into the self-attention module of Transformer- based models. Different from these methods, our REMEMO is built upon the graph view of time so to represent complex many-to mony-to-mmny temporal dependencies.
Example Sentence:
Police confirmed Friday that the body found along a highway in San Juan belonged to Jorge Hernandez.
TimeBank- Dense Labels:
belonged before confirmed; belonged before found; found before confirmed; belonged before Friday; confirmed is-included-in Friday, found is- included-in Friday.
REMEMO
Motivated by the graph view shown in Figure I,, we present REMEMO to model the temporal de- pendencies among temporally-scope sentences Specifically, given a document containing multiple sentences, first apply time-identification & Lime-normalizationn pipeline to assign a time-span tag for each sentence ($3.1). Time-relation labels are then created over any pair of temporally-scopee sentences uccording to the chronological order the events/facts in the sentences and will be utilized for later time-aware pre-training. Pre-trained on
Event Temporal Relation Extraction A line of work (Cassidy et al., 2014; Ning al., 2018; Mathur al., 2021; Zhang al., 2022; Huang et al., 2023) has been focusing on event temporal relation extraction (ETRE), which aims to extract
Tempporaly sentences are defined as sentences that contuin explicit time-informing texts.

---

both the Language Modeling (LM) objective and the Time Relation Classification (TRC) objective (83.2), REMEMO is capable capturing complex temporal dependencies beyond simple time-token co-occurrences. Finally, we discuss variants of R MEMO by controlling the graph density of the TRC objective ($3.3).
Time Expression
Normalized Value
21 July 1924
1924-07-21
November 7, 2006
2006-11-44
June 2012
2012-06
18th century
17XX
fourteen fifty
1454
Table 2: Examples of the extracted time expressions and their normalized values.
3.1
Pre-processing Pipeline
3.1.1
Time Identification & Normalization
tence as example, the time expression "2005" would be ignored if all three time-expressions were merged using our strategy. Generally speaking. if one sentence contains z time-xxpressions (where x > 2), then at least x time-expressions would be ignored by our merging strategy. As for our pre- training corpora, roughly 75% of all temporally- scoped sentences have < 2 time-expressions per sentence, so the other 25% would see losses of granularity.
To allow time-aware pre-training, we develop pipeline to explicitly assign the time tag for each training sentence. The pipeline is comprised of time token identification model and rule-based time normalization model. The time-identification model is trained on TimeBank 1.2 (Pustejovsky et al., 2006) and the time-normalization script is adopted from Filannino (2020). Given a document D. we first sentence- tokenize D into N sentences: D S2, ..... Sr). Each sentence is fed into the time-identiffcation model extract time pressions, which works similarly named-entity recognition, but only for time expressions. Those time expressions are then processed by the time- normalization script to obtain the TimeML (Saurí et al., 2006) type (e.g., date. duration) and value attributes. We further filter out the invalid outputs so that time tag is assigned for each valid time expression, with few examples shown in Table 2.
We include more details and manually verify the reliability of our pre- prepprcesssing pipeline in Ap- pendix $ A.1.
3.2 Pre-training Framework
As shown in Figure la, temporally-scoped sen- tences are strung together through the one- dimensional time-axis, on which each sentence covers a specific time-span. A fully-connected di- graph can thus be constructed to connect all these sentences by inspecting the relative positions of any two time-spans, as shown in Figure 1b. In- spired by this graph view, we propose the Time Relation Classification (TRC) objective to simulate the graph structure with textual inputs.
3.1.2 Creating Time-Span Tags
Let day be the unit of the time-axis, we propose to map time expression to the corresponding inter- val of the time-axis and regard such interval as the time-span tag. For instance, the time-span for "June 2012" can be expressed as [2012-06-01, 2012-07- 01), where the starting date is inclusive and the ending date is exclusive. For those sentences con- merge
Creating TRC Input Instances To simulate graph consisting of multiple nodes, put mul- tiple temporally-scoped sentences into one con- text, where sentences act as nodes and the whole context acts as the graph. Because nodes in the graph are clearly separated, we mark the bound- aries of these sentences with special tokens so that the model can correctly identify each sen- tence (node). Specifically, we insert two tempo- ral special tokens, [TIME] and |/TIMEJ, at the be- ginning and the end of each sentence, respec- tively. Overall, given M temporally-scoped sen- tences (S]' ..., S.. M one training instance looks like: 
 [TIME] ST [/TIME] [TIME] S [/TIME]... [TIME] S [/TIME] 
.
taining more than one time-expression, all valid time-spans. Note that we merge these time- spans even if they are not strictly consecutive. For example, for the sentence "In 2003, Tim Duncun won his first NBA Finals MVP award, followed by two more in 2005 and 2007. "" the time-span tag is 22003-01-1, 2008-01-01), which is the merger of [2003-01-00 -01, 2004-01-01), [2005-01-01, 2006- 01-01) and [2007-01-01, 2008-01-01). We adopt this loose strategy because these time-span tags used to model time relations, for which the leftmost and rightmost time boundaries mostly matter.
One notable defect of this strategy the loss of granularity. Take the above "Tim Duncan" sen-

---

TRC Objective The aforementioned fully- connected digraph is constructed by inspecting the relative positions of any two lemporally-scoped sentences in the same context. These relative positions act as edges in the graph shown in Figure lb. The directed edge from Sentence-i to Sentence-j may lie in one of the three cate- gories: (i) Earlier: The right time-span bound- ary of Sentence-é is smaller than the left time- span boundary of Sentence-j; (ii) Later: The left time-spa boundary of Sentence-é is larger than the right time-span boundary of Sentence-j; (iii) Contemporary: The time-spans of Sentence- and Sentence-j overlap. These directed edges are adopted training labels for the TRC objec- tive. Formally, given a training instance that sim- ulates the graph structure, 
 [TIME] ST [/TIME] (IME) S [/TIME]
Fact 1
Fact
Fact
Fact
N/A
Earlier
Earlier
Fact 2
Later
N/A
Contemp
Fact 3
Loter
Contemp
N/A
Figure 2: TRC labels (i.e., adjacency matrix) for the example graph in Figure Contemp refers to Contem- porary.
where
LLM
Eoog p(2;
where is the i-th token of the input instance and h; is the corresponding token representation.
? [/TIME]
[TIME] S/ [/TIME] 
, the TRC objective aims classify the directed edge rij connecting ST to sJ. where rij E (Earlier, Later, Contemporary); Vi.j E [I, and + j. Taking the adja- cency matrix shown Figure/ 2 an example, '12 Earlier and r32 Contemporary. In prac- tice, we adopt the representation of the [TIME] spe- cial token positioning at the start of S as the temporal representation for STT and denote it by [IME
3.3 Graph Density
The density of graph is defined to be the ratio of the number of edges EE with respect to the maximum possible edges. In the case of digraph, the density is
E
D
(4)
V
We concatenate the two representations, TIME and h [TTMME)
where [V] is the number of vertices (nodes).
As shown in Figures 1b and 2, given n graph nodes (i.e., temporally-scoped sentences) in the digraph, there would be n(n 1) = O( di- rected edges (e.g., time relations of Sentence-i Sentence-j). ) In other words, the vanilla TRC ob- jective adopts a graph density D = 1.0. It thus raises an interesting question of how graph den- sity influences model performance. In & 4.5, we empirically investigate variants of REMEMO by controlling the graph density to be D D respectively.
represent the directed edge
pointing from Node S.' Node ST" The concate- nated representation is then fed into a two-layer MLP to predict riy. We build REMEMO based on T5 checkpoints (Raffel et al., 2020) and apply the TRC objective to T5 encoder hidden states. The TRC loss function is written
M
M
LTRC
log p
Tis
h
IINE
h
TIME
log and n
1=1
(1)
where M is the total number of temporally-scoped sentences in the context. [o:o]] is the vector con- catenation operator.
Experiments
We present the empirical study in this section. Ex- perimental settings are in & 4.1 and & 4.2. Main results are in $ 4.3. We further investigate the chal- lenges of modeling complex temporal dependen- cies in $ 4.4.
Pre-training Two objectives are adopted for pre- training REMEMO, including the Language Mod- eling (L.) objective and the Time Relation Clas- sification (TRC) objective. Since adopt T5 ar- chitecture, the LM objective is the same that of T5, i.c., the denoising objective. The LM objective applies to all instances, while the TRC objective applies to temporal instances that we formulate earlier. Overall, the loss function is:
4.1 Pre-training Setup
4.1.1 Pre-processing
We adopt the pre-trained RoBERTa-base (Liu et al., 2019) checkpoint to initialize our time- identification model and further fine-tune it on TimeBank 1.2 (Pustejovsky et al., 2006). As for
L LIM + LTRO
(2)

---

TImeQA
TempReason
SituatedQA
Avg.
Hard
ReasonQA L,2
OBQA
ReasonQA
OBQA L33
EM
EM
F1
EM
F1
F1
EM
EM
EM
F1
EM
Thess
68.23
59.99
64.06
55.56
90.62
87.56
44.99
25.96
59.56
48.92
41,76
23.79
52.55
45.72
60.25
49.64
These 1.M
69.81
61.06
65.71
56,99
94.64
92.34
49.03
31.02
62.71
53.91
43.02
26.11
54.07
47.19
62.71
$2.66
TempT5
95.05
92.67
49.60
31.77
63.92
54.73
43.00
26.06
REMEMO
70.35
61.36
67.28
5K.19
97.67
96.22
51.58
33.62
91.17
89.29
44.91
28.48
55.06
48.12
68.29
59.33
TSung
71.60
63.10
68.11
59.49
96.68
94.80
50,91
32.72
60.24
50.21
46.78
28. 84
55.56
48.77
64.27
53.99
TSuag LL
71.72
62.76
70.59
61.89
97.78
96.17
53.73
35.98
63.50
55.41 41
48.26
31.93
54.71
47.76
65.76
55.99
REMEMO
72.25
63.73
69.31
60:49
98.15
96.74
54,94
37.40
94.02
92.69
49,31
33.37
56.06
49.12
70,58
61.93
GPT--.5-turbo"
53.14
44.00
26.32
19.00
32.83
28.00
9.52
6.00
44.30
36.00
22.01
7:00
17.19
7.00
29:33
21,00
GPT-4"
56.19
48.00
35,90
29.00
90.15
84.00
36.84
19:00
82.15
76.00
46.35
29
60.79
54.00
58.34
48.43
Table 3: Main results under the single-context setting. The datasets include TimeQA (two splits: (Easy, Hard)), ) TempReason (four splits: {Reason Level-2. OBQA Level-2, Reason Level-3, OBQA Level-3)) and SituatedQA. We report both Fl and exact match (EM) scores. LLM results (with *) are evaluated on 100 randomly-sampled test-set instances and are thus not directly comparable to the full-size test-set results.
time normalization, we only keep those predicted values that are valid time spans. We manually ver- ify the high accuracy of our pre-processing pipeline, with details shown in Appendix & A.I.
TimeQA
Easy
Hard
EM
EM
F1
EM
Thease
63.65
54.19
50.21
41.16
56.93
47.675
Theas + L.M
65.61
55.92
57.46
48.05
61.54
51.99
4.1.2 Pre-training Implementation
REMEMOpss
66.14
56.56
$9.84
50.71
62.99
$3.64
We adopt the pre-trained T5 v1.1 (Raffel et al., 2020) checkpoints to initialize our model and perform continual pre-training on Wikipedia and Bookcorpus. The peak learning rates are set to 5e- 5 and 3e-4 for base-sized and large-sized models, respectively. We follow the pre-training of T5 to take Adafactor (Shazeer and Stern, 2018) as the optimizer with a batch size of 2048 and parameter- scaling enabled across all the experiments. We pre- train all models for 8,000 steps, take a warm-up ratio of 10% to linearly increase the learning rate, and adopt the cosine learning rate scheduler after warm-up. We implement our code with Hugging- Face Transformers library (Wolf et al., 2020). It takes about 60 hours train the base-size model on 4 Nvidia V100-32GB GPUs and about 75 hours to train the large-size model on 2 Nvidia A A100-80GB GPUs with BF16.
(Hyper-parameter Search on TimeQA-Hard)
TStarg
52.15
42.48
62.15
53.09
57.15
47.79
TS + LM
68.90
59.46
62.54
53.48
65.72
56.47
REMEMOLagg
69.53
60.26
63.67
54.84
66.60
57.55
Table 4: Fusion-in -eeeoderrr (FiD) results TimeQA (two splits: {Easy, Hard)). ). Note that the large-scale hyper-parameter search performed on Taae and then directly applied to TimeQA-Eas due to the high computational cost of FiD.
TimeQA (Chen et al., 2021) is designed to evaluate the model's reasoning capability of time- evolving facts. This dataset contains two difficulty levels: The easy-level split tends to be answered based on surface form rather than temporal rea- soning while the hard-level split is more likely necessitate reasoning over the implicit time infor- mation.
TimeQA
4.2 Evaluation Setup
SituatedQA SituatedQA (Zhang and Choi, 2021) contains questions where extra-linguistic contexts need to be considered, e.g., depending on the tem- poral or geographical context. We adopt their temporal-deeendent dataset for our experiments.
We evaluate REMEMO on seven downstream tem- poral QA datasets and under three fine-tuning set- tings. We report both Fl and exact match (EM) for all experiments, using the evaluation code of SQuAD (Rajpurkar et al., 2016) and SQuAD-v2 (Rajpurkar et al., 2018).
TempReason TempReason (Tan et al., 2023) is designed to evaluate models' temporal reason- ing ability. consists of two levels (i.e., L-2: "time-event" time relation; L-3:tvvent-event" time- relation) and two settings (i.e., OBQA: conven-
4.2.1 Datasets
We give a brief description of our adopted datasets in this section.

---

TimeQA
TempReason
Easy
Hard
ReasonQA L2
OBQA L2
F1
EM
F1
EM
Fl
EM
FI
EM
Toase
8,02
2.95
8,95=1.00
2.78
14.60
0,00
8.17=a.0
0.09
Thsse I.M
13.30
9.88
15.51
12.64
14.39
0,31
16.69
3.07
REMF:Mobaze
14.0511.99
11.21
14.17
13.8512.48
23.12
9.96_s331
14.06
1.98
These
12.37
2.92.2211
11.79
3.31
24.87:2.01
2.02
21.20
2.83
Thass LM
21.61
14.02
24.1143.00
16.04
39.21
24. 24.99
23. 23.7220.57
5.44
REMEMOpase
26.54
18.20
22.11+3.50
14.57+261
55.47+11.87
42.17+13.9
26.43+2.75
7.12+2.09
TempReason
SituatedQA
Avg.
(Continued)
ReasonQA L33
OBQA L,3
F1
EM
FM
F1
FM
TS
15.48
0.57
13.02-4.54
0.15g031
3.16-1002
0.16g0
10.20.2..7
0.82
104 MASii
TSease
I.M
23.8913.07
9.42
12.10
1.19
3.30
0.09
14.31
5.23
REMEMOpase
24.69
10.83
11.18_4.02
1.11
3.73
0.28
15.09
7.17
T5
18.60
0.00
20.80
3.02g1.65
6.12_3338
0.37
16.54_3.44
2.18
TSease + L.M
13.10
19.40
2..5
4.37
26.6410.91
20.64.0.18
28.991..28
16.41 12.96
REMEMOcase
44.79
31.38
25.50
5.99
26.4627.35
20.38
32.47
19,97=5.68
Table 5: Few-shot results under the setting of single-context fine-tuning. We report average Fl and exact match (EM) scores along with their standard deviations across five rums with five different random seeds.
tional open-book QA; ReasonQA: the simplified version of OBQA by concatenating all relevant facts into a paragraph), rendering four splits in total. Note that our reported baseline results different from those reported in Tan et al. (2023) due to few differences in implementations. More details are shown in Appendix $ A.5.
least once in the truncated context.
Since Fusion-in Dession-n-Decoder (FiD) (Izacard and Grave, 2021) was adopted as standard baseline in the original paper of TimeQA, we also conduct FiD experiments for comprehensive comparison.
Fusion-in-Decoder
Example instances of the above datasets can be found in Appendix $ A.3
Few-shot We conduct few-shot experiments investigate the ability of REMEMO to efficiently adapt time-related QA tasks. For k-shot setting, we randomly sample training and development instances, respectively, while the test set remains the same. We run experiments for 16-shot and 64-shot.
4.2.2 Fine-tuning Settings
We adopt three settings to evaluate REMEMO: (1) single-context fine-tuning, (2) few-shot fine-tuning, and (3) Fusion-in- -Deeddeer (FiD) fine-tuning. All 7 datasets are evaluated under the single-context and the few-shot settings, while two of these datasets are included in the FiD setting. All models are opti- mized with AdamW (Loshchilov and Hutter, 2019). We perform hyper-parameter search in all exper- iments except FiD of large-sized models. More details are shown in Appendix & A.2.
4.2.3 Baselines
T5 stands for the vanilla T5 checkpoints, which are directly applied for supervised fine-tuning.
T5 + LM is pre-trained using the LM objective. The only difference between REMEMO and T5 LM is that REMEMO adopts the TRC objective but T5 LM does not.
Single-Context Under the cingle-context setting, the context and the question are concatenated to- gether as the input to the model and the model is required to generate the answer by comprehending the context. The maximum input length is set to 512 after tokenization. To keep the input length lower than 512, we truncate the context to 1500 chars while ensuring the correct answer occurs at
TempTS (Tan al., 2023) is reinforcement learning (RL) method that further improves super- vised fine-tuned (SFT) T5 models. It requires the dataset to have negative answer candidates, so it can only be applied to TempReason (Tan et al., 2023). It is added upon a fine-tuned checkpoint, so

---

AF1-Score
F1-Scoora
F3
Bemello
-10
RemeMNo
Aug.
500 1000
1000 11500
Avg
Avg
Change Context Lenqth
Change of Context Length
Change of Context Length
TimeQA-Hard,
(b) TempRcason ReasonQA-1.3.
(c) Average all 7 datasets.
Figure 3: Changes in Fl-scores response to the changes context lengths (i.e., 500 = 1000, 1000 = 1500, and the average the two). Blue bars with slash "/" LLM while orange bars with cross "+" are REMEMO Positive -slorce indicate that the model obtains improvements as contexts become longer, while negative AFl-scores indicate declines with longer contexts. Standard deviations on 7 datasets are also reported for Figure 3c.
TimeQA
TempReason
SituatedQA
Avg.
Easy
Hard
ReasonQA
OBQA L2
ReasonQA
OBQA L3
F1
F1
EM
F1
EM
17 1.0
70.35
61.36
67.28
58.19
97.67
96.22
51.58
33.62
91.17
89.29
44.91
28.48
55.06
48.12
68.29
59.33
D (loge)/n
67.25
97.63
95.84
51.69
26.83
55.36
D 1/n
69.48
60.56
67.92
58.93
97.03
95.37
50.71
33.33
89.54
87.33
44.49
28.32
54.49
47.55
67.67
58.77
TS + LM
69.81
61.06
65.71
56.99
94.64
92.34
49.03
31.02
62.71
53.91
43.02
26.11
$4.07
47.19
62.71
$2.66
TSes
68.23
59.99
64.06
55.56
90.62
87.56
44.99
25.96
59.56
48.92
41.76
23.79
52.55
45.72
60.25
49.64
Table 6: Ablation results graph density of the TRC objective. We modify the graph density to be D) 1.0.
log and = n respectively, where n the number of temporally-seee scoped sentences a training instance.
we can directly apply it on all other baselines (TS and T5 LM) and REMEMO.
Fusion-in in-Decoder Table 4 shows the FiD re- sults TimeQA-Easy & -Hard. REMEMO out- performs the baseline T5 on both datasets. The average improvements in terms of EM are +1.65 for the base scale and +1 for the large scale.
GPT-3.5-turbo GPT-4 are two powerful large language models (LLMs). We evaluate their zero- shot performances on 100 randomly-samples test- ing instances under the single-context setting due budget limit. The prompt templates and the model versions that adopt shown in Ap- pendix A.4.
Few-shot Table 5 shows the 16- and 64-shot results under the single-context setting with the base-scale models. REMEMObase outperforms the strongest baseline on most datasets, with aver- age improvements +1.9 EM under 16-shot and +3.6 EM under 64-shot. Notably, REMEMOpase achieves performance gains more rapidly com- pared with T5 (3247 z 215.2% versus 28.99 14.31 ~ 202.6% in terms of relative F1), when the number of available instances increases from 16 to 64. These results suggest that REMEMO is more effective to reach peak performance with fewer in- stances.
4.3 Main Results
Single-Context Table shows the single-context results on all seven datasets. REMEMO outper- forms T5 on most datasets, with an average im- provement of +5.58 F1-scores over T5base LM and +4.82 over 151arge + LM. TempT5 improves the performance of T5 + LM on most splits but does not surpass REMEMO. Notably, REMEMO outperforms T5 for large margins with both base and large scales on ReasonQA Level-3, which the model is required reason over "event-event" temporal relations. This result suggests that the TRC training objective is effective for modeling complex temporal dependencies.
15.09
4.4 Better Modeling of Complex Temporal Dependencies
One crucial challenge faced by open-book QA sys- tems is how to connect the question with relevant sentences in the context while not being misguided

---

by irrelevant ones. This problem becomes more se- vere when the context becomes longer because the number of possible dependencies among context sentences could increase enormously, rendering more complexity when dealing with those depen- dencies.
MEMO jointly with the LM objective and the TRC objective and evaluated it on seven downstream temporal QA datasets. Experimental results show that REMEMO outperforms the baseline T5 under various settings. The performance improvements are remarkably large on datasets that require rea- soning over complex "event-event" temporal rela- tions. Further analysis suggests that (i) REMEMO is especially good at modeling long-range complex temporal dependencies; (ii) The key to the success of REMEMO the graph-inspired modeling rather than the larger density of the graph.
As for temporal question answering, the most practical dependencies are mostly temporal depen- dencies. To investigate how our model handles much more complex temporal dependencies, we modify the context lengths to be 1,000 and 500 chars, respectively, in addition to the regular setup of 1,500 chars under the single-context setting, Fig- ures 3a, 3b and 3c show the changes in Fl-scores of base +LM and REMEMObase TimeQA A-Hadd, TempReason ReasonQA-L2 and on the average of all datasets when context lengths increase. On average, the Fl-score decrease of REMEMObase is much lower than that of +aa On Tem- pReason ReasonQA-L3, longer contexts hurt the performance of +LM, but surprisingly boost the performance of REMEMMbase- The relatively smaller Fl-score performance drop (or even im- provement) of REMEMO suggests that REMEMO can better models long-range and complex tempo- ral dependencies.
Limitations
We observe two limitations regarding our work:
Because the TRC objective requires time-span tags of textual sentences, implement pre- processing pipeline (53.1), which contains rule-based method normalize time expres- sions. This may limit our method from gener- alizing to other languages since each language requires its corresponding time-normalizaton method.
The TRC objective is designed to simulate the graph-like structure, described in $3.2. However, the order of the graph (i.c., the num- ber of nodes in the graph) limited by the length of the context, restricting us from in- creasing the order of the graph to arbitrarily large.
4.5 Ablation Study: Graph Density
Table shows the ablation results on graph den- sity of the TRC objective (5 3.3). In addition to the regular setting (i.e., D = 1.0), we pre-train another two REMEMMObase variants with differ- ent graph densities by controlling the number of edges in the digraph (i.e., D log and D ) As the TRC training labels become more sparse
Ethics Statement
In this paper, we adopt Wikipedia and BookCor- pus for pre-training the language model. These two corpora are publicly available and widely used for research purposes in the NLP area. We adopt TimeQA, SituatedQA and TempReason for evalua- tion. These datasets are all publicly available and are for research purposes only. However, these cor- pora/datasets may still contain improper or harmful content. None of such content reflects the opinions of the authors of this paper.
(1.0 12 2), downstream performances on temporal QA show mild decreases on most datasets. All three variants, however, still outperform T5base baselines significantly. These results suggest that the key to the superior performances of REMEMO is the graph-like modeling of temporally-scoped events rather than the density of graph edges.
5 Conclusions
In this paper, we devised a graph view of temporally events based on the fact that these events are strung together through the one- dimensional time axis. Inspired by this graph view, we proposed the TRC pre-training objective that exploits relative time as directed graph edges for complex temporal reasoning. We pre-trained RE-
Acknowledgements
We thank Qingyu Tan for sharing TempReason Dataset. We thank the anonymous reviewers for their thorough and constructive suggestions that help make this work better.

---

References
Quzhe Huang, Yutong Hu, Shengqi Zhu, Yansong Feng. Chang Liu, and Dongyan Zhao. 2023. More than classification: A unified framework for event tem- poral relation extraction. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9631- 9646, Toronto, Canada. Association for Computa- tional Linguistics.
Robert Bamler and Stephan Mandt. 2017. Dynamic word embeddings. In Proceedings of the 34th Inter- national Conference
Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Re- search, pages 380-389. PMLR.
Shuyang Cao and Lu Wang. 2022. Time-aware prompt- ing for text generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7231-7246, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Gautier Izacard and Edouard Grave 2021. Leveraging passage retrieval with generative models for open do- main question answering- In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume. pages 974-880. Online. Association for Computa- tional Linguistics.
Taylor Cassidy, Bill McDowell, Nathanael Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. Proceedings of the 52nd Annual Meeting of the Association f Computational Linguistics (Volume 2: Short Papers), pages 501-506, Baltimore, Maryland, Association for Computational Linguistics.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy. Jannik Strötgen, and Gerhard Weikum. 2018a. TEQUILA: temporal question answering over knowl- edge bases. In Proceedings of the 27th ACM Inter- national Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22- 26, 2018, pages 1807-1810. ACM.
Wenhu Chen, Xinyi Wang, and William Yang Wang- 2021. A dataset for answering time-sensitive ques- tions. In Thirty-fifth Conference Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018b. Tem- pquestions: A benchmark for temporal question an- swering. In Companion Proceedings of the The Web Conference 2018 WWW '18, page 1057-1062, Re- public and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguisties, 10:257- 273.
Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM International Conference Informa- tion & Knowledge Munagement, CIKM '21, page 792-802, New York, NY, USA. Association for Com- puting Machinery.
Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-out: Tem- poral referencing for robust modeling of lexical se- mantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 457-470, Florence, Italy, Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1769-6781 Online. Association for Computational Linguistics.
Michele Filannino. 2020. timex- https:
/ //gi tigb. ctw/ fi 1annimmmtmttt tomex-normallllieoeeoeo
Lea Frermann and Mirella Lapata. 2016. A Bayesian
model of diachronic meaning change. Transactions of the Association for Computational Linguistics, 4:31-45.
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomás Kociský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Sysiems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages
Mario Giulianelli, Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualised word representations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3960- 3973, Online. Association for Computational Lin- guistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1501, Berlin, Germany. Association for Com- putational Linguistics.
Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng. Shafiq Joty, and Lidong Bing. 2023. Unlocking tem- poral question answering for large language models using code execution.

---

Adam Liska, Tomás Kociský, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener. Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-MeMahon Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. Streamingga: A benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learn- ing Research, pages 166222 PMLR.
Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-tex, trans- former. J. Mach. Learn. Res, 21: 10:11440..7
Pranav Rajpurkar, Robin Jia, and Perey Liang. 2018 Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. ArXiv preprint, abs/ 1et//997.11669 1692.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22. page 833-841, New York, NY, USA. Association for Com- puting Machinery.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time
waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944-5958, Seattle, United States. Association for Computational Lin- guistics.
Guy D. Rosin and Kira Radinsky. 2022. Temporal at- tention for language models. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 1498-1508, Seattle, United States. Association for Computational Linguistics.
Paul Röttger and Janet Pierrchumbert. 2021. Temporal adaptation of BERT and performance downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021. pages 2400-2412 Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Puneet Mathur, Rajiv Jain, Franck Dernoncourt, Vlad Morariu. Quan Hung Tran, and Dinesh Manocha. 2021. TIMERS: Document-level temporal relation extraction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 524-533, Online. Association for Computa- tional Linguistics.
Roser Saurf, Jessica Moszkowicz, Bob Knippen, Rob Gaizauskas, Andrea Setzer, and James Pustejovsky. 2006. Timeml annotation guidelines version 1.2.1.
Qiang Ning, Hao Wu, and Dan Roth. 2018 A multi- axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume
Apoorv Saxena, Soumen Chakrabarti, and Partha Taluk- dar. 2021. Question answering over temporal knowl- edge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 6663-6676, Online. Association for Computational Linguistics.
1: Long Papers), pages 1318-1328 Melbourne, Aus- tralia. Association for Computational Linguistics.
James Pustejovsky et al. 2006. Timebank 1.2 Ide2006108. Web Download.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He. Yejin Choi, and Manaal Faruqui. 2021. TIME- DIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 7066-7076, Online. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stem. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmäs- san, Stockhoim, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4603-4611. 1. PMLR.
Zhaochen Su, Zecheng Tang, Xinyan Guan, Lijun Wu, Min Zhang, and Juntao Li. 2022. Improving tem- poral generalization pre-trained language mod- els with lexical semantic change. In Proceedings of
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou,

---

the 2022 Conference on Empirical Methods Nat- ural Language Processing, pages 1380-6393, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
A
Appendix
A.1 Details and Reliability of Pre- Pre-processing Pipeline
As for the iime-ieentification model, we train it with a learning-rate to be 2e-5 and a batch-size to be 32.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of ACL.
We generally adopt aggressive filtering during pre-processing. Specifically, we focus on two parts:
Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022. Archivalqa: A large-scale benchmark dataset for open-domain question answering over historical news collections. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 3025-3035, New York, NY, USA. Association for Computing Machinery.
Filtering out invalid time-identification tags produced by the fine-tuned RoBERTa model. Specifically, following the NER-like tagging manner, our time-identification model pre- dicts tag among (B-TIME, I-TIME, O for each token. We exclude all sequences of tokens that start with I-TIME (a valid NE should start with B-TIME), even though these sequences starting with I-TIME still might be valid time expressions in some cases.
Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Under- standing semantic change of words over centuries. In Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural DiversiTy on the Social Web, DETECT '11, page 35 40, New York, NY, USA. Association for Computing Machinery.
Filtering out time-normalization results that are somehow unlikely to be valid dates. Below we show a few examples of what we included and excluded. Each example is tuple contain- ing four elements: (temporal expression on, TYPE (DATE, TIME, SET, DURATION] according to TimeML, normalized VALUE according to TimeML, specific format). We only focus on tuples with TYPE=DATE. Our filtering strategy works by checking the spe- cific format.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Included:
('2015', 'DATE', '2015', 'Year')
Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating extra- ling-liin contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 1 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
("the early 90s", 'DATE', '199X',
'90s nineties')
('may 1913', 'DATE', '1913-05', 'Month Year')
(november 1 2015', 'DATE', '2015-11-01', 'Month. Date, MM DD YYYY")
Shuaicheng Zhang, Qiang Ning, and Lifu Huang. 2022. Extracting temporal event relation with syntax- guided graph transformer. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 379-390, Seattle, United States. Association for Computational Linguistics.
(30 november 1945', 'DATE', "1945-11-30', 'Day Month Year mic')
(11tth century'. 'DATE', '18XX', '20th century')
Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang. and Hai Jin. 2022. Can language models serve as temporal knowledge bases? In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pages 2024-2037, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
('01/01//022",, 'DATE', '2022-01- 01', 'DCT3")
('seventeen hundred and fifty two', "DATE', '1752', 'Year number 90s&before')

---

((2043/11/05', "DATE", '2043-11- 05', "'icl")
results of the models obtaining the best devel- opment set Fl-scores. For TimeQA, we adopt the value of key "contex'" in the released JSON files as context. For SituatedQA, since their officially released dataset files do not contain contexts, we follow the authors to adopt DPR (Karpukhin et al., 2020) to retrieve contexts for each question from the Wikipedia Dump of 2021-Feb-20. We concate- nate the top-2 retrieved snippets to formulate context.
Excluded:
('03130001000) 'DATE', '0313- 00-01', "DCTImic')
Unlikely to be time expressions.
('2019/11/25', 'DATE', 225-11-19', 'DCT4")
Ambiguous: may be 2019/11/25 or 2029/1
* ('2004 third quarter', 'DATE', -2004- Q3', 'year quarter")
Fusion-in-Decoder For the base-size models, we perform hyper- paper-parameter search: batch-size = {32, 128); learning-rate = (5c-5, le-4, 1e-3). For the large-size models, due to the heavy computational cost of FiD, we only perform hyper-parameter search for TimeQA-Hard to obtain the best setting (Ir=1c-4, bsz=32) and directly apply it to the other two datasets. We train all models for epochs, lin- early warm up the model for the initial 10% steps. and then linearly decrease the learning rate. The maximum number contexts is set to 100 and each context contains 250 tokens at most. We report the test set results of the models that obtain the best development set results. For TimeQA, we adopt the value of the key "parugraphs" in the released JSON files as FiD contexts.
By manually checking, we found that the results of year quarter for- mat often contain errors.
The above strategies mainly supported by man- ual qualitative evaluations conducted by the authors of this paper.
Averaged Procision
Averaged Recall
Averaged Fl
Averaged EM
0.89
0.84
0.76
Table 7: Reliability evaluation of our pre- peeprccessingg pipeline on 100 randomly-sample instances.
Besides, to measure the accuracy of our pre- processing pipeline, we randomly sample 100 sen- tences that are predicted to contain valid time-span tags, manually annotate the time-span tags, and compare those tags with the automatically pre- dicted ones. We adopt the overlap between the annotated and the predicted time-spans in terms of day to calculate "Fl-score" 3, As can be seen from Table 7, the average precision is 0.93, which indicates the effectiveness of our strategy.
Few-shot We perform hyper-parametee search (batch-size (8, 64); learning-rate (1e-4, 5e-4)) and run each setting five times with five different random seeds to minimize the effects of random- ness. We use Adam to train all models for 10 epochs with the linear learning rate scheduler.
A.3 Example Data Points
Example data instances are shown in Tables 8, 9 and 10. We underline those context sentence(s) that contain the necessary information to answer the question.
A.2 Evaluation Settings
Single-Context The maximum input length is set to 512 after tokenization. We adopt hyper- parameter search to find out the best setting for each dataset: batch-size (16, 32); learning-rate (3c-5, Ic-4. 3c-4). We train all models for 10 epochs, use a warm-up ratio of 10% for the llarg-- size experiments and no warm-up for the -base size, and adopt the linear learning rate scheduler. During fine-tuning, the model is evaluated on the development set every half epoch. We report test
A.4 LLM Settings
Our LLM experiments were conducted in June 2023. We adopt GPT-4-32K. We filter out re- sponses like "The context does not provide infor- mation on which employer Corine Mauch worked for in Jun 2008. " and replace them with empty strings by manually writing few simple rules.
We write our prompts based on the prompts of Li et al. (2023). For datasets that do not contain null answers (i.e., all four splits of TempReason and SituatedQA), we use the following prompt:
'For example, suppose the annotated time-span is (2023-
01-01, 2023-01-24] and the predicted time-span is [2023-
01-01, 2023-02-011, then precision 23 = 74.2%, recall
31
100% and F1-score 85.2%.

---

PROMPT_TEMPLAN
fine-tuning settings, which might be the main cause for the higher performance of base- lines compared with the results in Tan et al. (2023).
Answer the question based on the context.
Only answer the name.
Dataset Pre-processing To fit the T5 fine-tuning process into one single GPU, we truncate the lengths of the contexts to 1,500 chars for all down- stream datasets under single-context and few-shot settings. This may boost performance since slightly shorter contexts make the modeling of dependen- cies little easier.
If there are more than one answer, only give me the most suitable answer.
\nContext: O
AnQuestion: {
AnAnswer:
12
For TimeQA-Hard and TimeQA-Easy, which include null answers, the following prompt: PROMPT_TENPLATE_V2 Н*"
The original TempReason pa- per simply chose a set of hyper-parametery for all experiments. In comparison, we adopted hyper- parameter search in all downstream experiments to avoid biases resulting from dyper-paramend choices. This may boost the performance of our implementation.
preee-pprrmeter
Answer the question based on the context. Only answer the name.
If there are more than one answer, only give me the most suitable answer
If the question cannot be answered based on the context, answer \NNo Answer\".
AnContext: O
\nQuestion: ()
AnAnswer:
A.5 Differences Baseline Implementations
The different baseline performances between this work and Tan et al. (2023) result from few differ- ences in implementations.
T5 Versions As mentioned in ș 4.1, we adopted T5-v1.1 for all experiments, while the original TempReason papers adopted T5-v1.0. We did not adopt T5-v1.0 because its pre-training corpora included downstream datasets, such as SQuAD, which makes the evaluation of few-shot QA unfair. In comparison, 1 is pre-trained on raw texts in a fully unsupervised manner, so the comparison for few-shot QA will be fair. Empirically speaking, in preliminary experiments, we found that
Vanilla T5-v1.0 always outperformed the con- tinually pre-trained T5-v1.0 significantly der the few-shot setting.
Vanilla T5-v1.1 outperformed vanilla 1,-v1.0 on most adopted datasets under the full-set

---

TAmaQA-Easy
Context Dorothy Hansine Andersen (May 15. 1901- March 1963) was an American pathologist and pediatrician who was the first person to identify cystic fibrosis and the first American physician to describe the disease. In 2001 she was inducted into the National Women's Hall of Fame. Farly life. Dorothy Hansine Andersen was born Asheville, North Carolina on May 15, 1901. 1914 her father, Hans Peter Andersen, died and she took the full responsibility for caring for her invalid mother. Andersen's 's mother died in 1920 and after they had moved to St. Johnsbury, Vermont. In 1922 Andersen gruduated with a Bachelor of Arts in zoology and chemistry from Mount Holyoke College. Later, she went on to allend Johns Hopkins School of Medicine which is where she first began to perform research under Florence Rena Sabin. Andersen's first two research papers were on the lymphatic and blood vessels in the reproductive organs of female pigs. Both of these papers were published in Contributions to Embryology. Once she graduated from Johns Hopkins, Andersen served as teaching assistunt in anatomy at the Rochester School of Medicine. A year later she became an intern for surgery at the Strong Memorial Hospital in Rochester, New York. After completing her internship year, Andersen was denied a residency in general surgery at the hospital because she was a woman. This drove Andersen to focus on her research instead and in 1929, she began working at Columbia University
Question: Dorothy Hansine Andersen went to which school from 1921 to 1922?
Answer
Mount Holyoke College
TimeQA-Hard
w............................... Context ; Mei Li Vos Mei Li Vos (born 31 March 1970) is a Dutch Dutchpoliiiiian, former trade unionist and editorialist. A member of the Lubour Party ( PvdA ), she was member of the House of Representutives from 1 March 2007 to 17 June 2010 and again from 20 September 2012 until 23 March 2017, She hus been member of the Senate since 11 June 2019, Early life. Mei Li Vos was born on 31 March 1970 in Eindhoven in the Netherlands. Vos comes from family with five bruthers. Her mother was Chinese Indonesian. established in the Dutch East Indies. During her youth her family lived in Christian commune in Veldhoven. Between 1982 and 1988 she attended the
the Christian Lyceum in Arnhem, Academic career. Between 1988 and 1989 she studied General Social Science the University of Utrecht. Vos studied political science at the University of Amsterdam. Between 1994 and 2002 she worked as teacher and researcher at the political science department of the University of Amsterdam, In 1994-1995 she also worked mart-time as manager of the Moarten Altena Ensemble. From 1995 and 2000 she worked on her thesis, which concerned the relationship between the Netherlands and Indonesia. She specifically researched what the effect was of Indonesia refusing development cooperation of the
Question What was the position of Mei LJ Vos between Feb 2016 and Dec 2016?
Answer member of the House of Representatives
Table 8: Examples from TimeQA- A-maQd Hard and -Easy. We underline those context sentence(s) that contain the necessary
information to answer the question.

---

TempReason -eassonnRaasnnOA-12
Context Paul Dubreil studied at Lycée Saint-Louis from Jan 1921 to Jon 1923. Paul Dubreil worked for Université de
Nancy from Jan 1933 to Jan 1946, Paul Dubreil studied at Lcole normale supérieure (Paris) from Jan 1923 to Jan 1926. Paul Dubreil worked for University of Paris from Jan 1946 to Jan 1975. Poul Dubreil worked for University of Lille from Jan 1931 to Jan 1933. Paul Dubreil studied at University of Hamburr from Jan 1929 to Jan 1930.
Question: Where was Paul Dubreil educated in May 1921?
Answer
Lycée Saint-Louis
TempReason-OBQA--1
Benoit DoraisBenoit Dorais a city councillor from Montreal, Quebec, Canada. He has served as the borough moyor of Le Sud- Ouest since 2009. From his first election to 2013. Dorais was a member of Vision Montreal, before joining Coalition Montréal in 2013 and Projet Montréal prior to the 2017 municipal election. Dora's was born and raised in the Saint- -enri neighbourhood. Prior to his election as city councillor, Dorais served as a political staff member to former Bloc Québécois MP Thierry St-Cyr. He has also served as commissioner with the Commission scolaire de Montréal since 2007. He holds a university degree in philosophy and social ethics. Following Marcel Côté's 's death on May 26, 2014, Dorais became leader of Coalition Montreal. In 2017 he resigned that role to sit as an independent, joining Projet Montréal soon after. prior to the 2017 elections, Dorais chaired the City of Montreal committee on social development and Montreal diversity. Following the 2017 election, Mayor Valérie Plante named him chair of the Montreal Executive Committee, with responsibility for finances, human resources, and legal affairs.
Context
Question: Which political party did Benoit Dorais belong to in Aug 2016?
Answer Coalition Montréal
TempReason- TenpRenson-Reesonnn... ReasonQA-L3
Clifford Truesdell studied at California Institde of Technology from Jan 1938 to Jan 1942. Clifford Truesdell worked for Naval Ordnance Laborutory from Jun 1946 to Jan 1948. Clifford Truesdell worked for Johns Hopkins University from Jan 1961 to Jan 1989, Clifford Truesdell worked for Massachusetts Institute of Technology from Jun 1944 to Jan 1 1946. Clifford Truesdell worked for the United States Naval Research Laboratory from Jan 1948 to Jan 1950. Clifford Truesdell worked for Brown University from Jan 1942 Jan 1943. Clifford Truesdell worked for Indiana University Bloomington from Jan 1950 to Jan 1961. Clifford Truesdell worked for University of Michigan from Jon 1943 to Jan 1944.
Context
Question Which employer did Clifford Truesdell work for after Indiana University Bloomington?
Answer
Johns Hopkins University
TenpReason-OBQQA-..
Piotr WilczekPiotr Antoni Wilczek (born April 26, 1962 Churzów) Polish intellectual historian. specialist in comparative literature and literary translator, who serves the Ambassador Poland to the United Stotes, Wilezek gruduated from the University of Silesia in Katowice, Poland (1986) where he received his Ph.D. (1992) and Habilitation (2001) degrees. In 2006 he was Rominated professor the humanities by the President of the Republic of Poland. Doctor of Humane Letters (honoris causa) of Cleveland State University. He was assistant and associate professor at the University of Silesia (1986-2008). where he also served as Dean of the Faculty Languages 2002)) Since 2008 he has been temured full professor at the Faculry of ..rtes Liberales". "."". University of Warsaw and until 2016 served there as Head of the Collegium Artes Liberales (College of Liberal Arts and Sciences) and Head of the Centre for the Study of the Reformation and Intellectual Culture Early Modern Eumoe..H He did his postgraduate work intellectual history and Neo-Latin Studies at the Universities of Oxford (St Anne's 's College, 1988) and Lódš, Poland (1989). He was a visiting translator at The British Centre for Literary Translation, University o
Context
Question:
Which employer did Piotr Wilczek work for before University of Warsaw?
Answer
University of Silesia
Table 9: Examples from TempReason, including ReasonQA Level-2 & Level-3 and Open-domain QA (OBQA) Level-2 & Level-3. We underline those context sentence(s) that contain the necessary information to answer the question.

---

SituatedQAA
Context USA on her first try, then went on to participate in Miss USA 1997 at Shreveport, Louisiana on February 5, 1997, where she was crowned the winner by outgoing titleholder Ali Landry of Louisiana. Lee represented the United States in the Miss Universe 1997 pageant in Miami Beach, Florida. On May 16, 1997, she won the crown at 26 years and 128 days, became the oldest Miss Universe to win. Lee and Al Masini along with funding from the state were forces behind the Miss Universe 1998 pageant being held in her home state of Hawaii, in Honoldu, for the first time. Inmediately prior to winning the Miss USA and Miss Universe crowns Lee Miss Universe, after winning Miss USA. Six titleholders represented the United States Hawaii at major international pageants, with Miss Hawaii USA 1997, Brook Lee, winning the title of Miss USA 1997 and capturing the crown of Miss Universe 1997, one of only 8 Miss USA winners to become Miss Universe in the history of the pugeant. Hawaii holds record of 24 placements at Miss USA.
Question When was the last time USA won miss universe as of July 22, 1998?
Answer May 16, 1997
Table 10: Examples from SituatedQA. We underline those context sentence(s) that contain the necessary information to answer the question.


Reviews:
[{"review_id": "FsUM3zGixi", "reviewer": "Reviewer_bSAa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer bSAa for the insightful comments and suggestions that have helped identify our work's strengths and potential improvements.\n\n## 1. Comparison with Existing Methods\n\n### TL;DR:\n\n1. There are only a small number of existing methods focusing on temporal question answering. Among them, TempT5 is the only proper one for comparison. It was proposed in the TempReason dataset paper.\n2. TempT5 improves the performance of T5+LM for +$0.54\\%$ F1-scores, but does not improve over our best model (REMEMO+SFT) and even slightly hurts its performance.\n3. Our best model remains the best of all.\n\n### Detailed Author Response:\n\nTemporal question answering (TQA), as a specific task, has garnered limited attention in earlier years, resulting in few existing methods for comparison. As far as we know, all existing non-GPT methods that conducted experiments on the three datasets we adopted come from their corresponding original dataset papers.\n\n- **TimeQA**: Their original dataset paper adopts Fusion-in-Decoder (FiD), for which we have implemented the experiments and reported the results in Table 4 of our paper.\n- **TempReason**: Their dataset paper proposed TempT5, a reinforcement learning (RL) method that further improves supervised fine-tuned (SFT) T5 models. It is a meaningful comparison for our experiments. We thus conducted additional experiments using this RL method and include the results in Table 1 below.\n- **SituatedQA**: Their dataset paper adopts Dense Passage Retrieval (DPR) [1] as the baseline. Apart from the retrieval component, DPR is no more than a simple BERT-based extractive QA baseline. We list the DPR results reported by the SituatedQA paper in Table 2 below.\n\n**Table 1**: F1-scores on TempReason. All experiments are conducted using T5-v1.1-base. \"SFT+RL\" refers to TempT5.\n\n| Pre-training Method | Fine-tuning Method | Temp-L2 | OBQA-L2 | Temp-L3 | OBQA-L3 | Average |\n| --- | --- | --- | --- | --- | --- | --- |\n| T5 | SFT | 90.62 | 44.99 | 59.56 | 41.76 | 59.23 |\n| T5 | SFT+RL | 90.61 | 44.64 | 59.88 | 41.42 | 59.14 |\n| T5+LM | SFT | 94.64 | 49.03 | 62.71 | 43.02 | 62.35 |\n| T5+LM | SFT+RL | 95.05 | 49.60 | 63.92 | 43.00 | 62.89 |\n| REMEMO | SFT | 97.67 | 51.58 | 91.17 | 44.91 | 71.33 |\n| REMEMO | SFT+RL | 97.61 | 51.14 | 91.28 | 44.70 | 71.18 |\n\nAs can be seen from Table 1 above, SFT+RL (i.e., TempT5) only improves the averaged performance of T5+LM but does not work for all other models. Specifically, REMEMO+SFT (our best model) already obtains the best results among all of them; and further applying RL to our REMEMO+SFT slightly hurts its performance. This may be caused by the fact that all our T5+SFT checkpoints (including baselines and REMEMO) are obtained through extensive hyperparameter search so little room is left for further improvement through further dataset-specific RL training.\n\n**Table 2**: Results on SituatedQA. DPR and Human* results are from the original SituatedQA dataset paper. Human* results were from human evaluation conducted on 50 randomly-sampled instances.\n\n| Method | F1 | EM |\n| --- | --- | --- |\n| DPR | N/A | 23.00 |\n| REMEMO-base | 55.06 | 48.12 |\n| REMEMO-large | 56.06 | 49.12 |\n| Human* | N/A | 57.00 |\n\n> Notes:\n> \n> 1. Because TempT5 requires the dataset to have negative answers in addition to the correct answer, we can only conduct TempT5 experiments on TempReason, which is the only dataset that contains negative answer candidates.\n> 2. The original TempReason paper adopted T5-v1.0 for experiments, while we adopted T5-v1.1 in all our experiments. We did not adopt T5-v1.0 because its pre-training corpora included downstream datasets, such as SQuAD, which will make the evaluation of few-shot QA unfair. In comparison, T5-v1.1 is pre-trained on raw texts in a fully-unsupervised manner.\n> 3. The original SituatedQA paper only reported exact match (EM) scores.\n> 4. For the results in Table 1 above, the default setting in TempReason's released code is 'num_layers_unfrozen'=6 and 'max_len'= 768. We additionally tried 'num_layers_unfrozen'$\\in$ {-1, 6, 2, 1} and 'max_len' $\\in$ {768, 512}. We then take the models that obtain the best validation F1-scores for testing.\n\n## 2. Noise and Ambiguity of Temporal Contexts in Text\n\n### TL;DR:\n\nWe adopted an aggressive filtering strategy for data pre-processing to avoid noises as much as we could.\n\n### Detailed Author Response:\n\nWe agree with the reviewer that temporal contexts in text can sometimes be ambiguous or noisy. Below we discuss how we handle such noise:\n\n- Generally speaking, we maintained strict filtering during pre-training data pre-processing, i.e., high precision rather than high recall. We adopted such a strategy because even little bits of noise might be highly harmful and the amount of raw pre-training articles is quite sufficient.\n- Specifically:\n    - Unclear temporal information: We regard such sentences as \"not containing a valid time-span tag\" and do not include them when constructing time-relation classification (TRC) labels.\n    - Conflicting temporal information: As discussed in Section 3.1 of our paper, we merge multiple time spans, supported by the intuition that time expressions that appeared in the same sentence tend to describe events that happened during a similar period of time. Therefore, when multiple mentions are found, they are regarded as a whole. For example,\n        - Victor Hugo's most famous works are the novels *The Hunchback of Notre-Dame* (1831) and *Les Misérables* (1862).\n        - García Márquez started as a journalist and wrote many acclaimed non-fiction works and short stories, but is best known for his novels, such as *One Hundred Years of Solitude* (1967), *Chronicle of a Death Foretold* (1981), and *Love in the Time of Cholera* (1985).\n        - Both sentences above contain more than one time-expressions, but the two mergers of time-spans, $[18310101, 18630101)$ and $[19670101, 19860101)$, are meaningful and distinguishable from each other.\n\n## References\n\n- [1] Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, *EMNLP 2020*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Comprehensive Temporal Understanding: REMEMO goes beyond conventional knowledge-time association methods and offers a more detailed understanding of temporal contexts. \n\n2. Long-Range Dependency Handling: The paper's focus on modeling long-range complex temporal dependencies is a significant contribution. This capability is highly relevant in real-world scenarios where understanding distant relationships in time is crucial for accurate comprehension and reasoning.\n\n\nMain benefits for the NLP community:\n1. The paper contributes to advancing the state-of-the-art in temporal reasoning within NLP. By addressing a critical challenge in understanding temporal contexts, it provides valuable tools and techniques for researchers and practitioners to build more accurate and sophisticated models\n\n2.  Incorporating the REMEMO framework could lead to improved performance of NLP models across various tasks that involve temporal information. This could have a cascading effect on downstream applications, leading to better user experiences and more reliable results.", "reasons_to_reject": "1. Comparison with Existing Methods: While REMEMO outperforms the baseline T5, a more comprehensive comparison with other state-of-the-art methods could provide a clearer understanding of its relative performance and strengths. Ensuring that the comparison covers a diverse set of temporal reasoning models could add more depth to the evaluation.\n\n2. Impact of Noise and Ambiguity: Temporal contexts in text can sometimes be ambiguous or noisy. The paper could discuss how REMEMO handles cases where temporal information is unclear or conflicting, and whether it has limitations in such scenarios.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the challenge of enhancing the understanding and reasoning capabilities of pre-trained language models in dealing with temporal contexts in text.The main problem at hand is the difficulty of capturing and utilizing temporal dependencies between pieces of knowledge in various downstream tasks. While existing approaches mainly focus on connecting text snippets directly to timestamps, they often fall short when more complex reasoning over temporal relationships is required.\n\nThe main contributions of the paper can be summarized as follows:\n\n1. REMEMO Framework: The paper introduces the REMEMO framework, which goes beyond simple knowledge-time association and instead focuses on capturing complex temporal dependencies between different pieces of knowledge. This framework utilizes a graph-based approach to represent and model these dependencies, offering a more comprehensive understanding of temporal contexts in text.\n\n2. Improved Performance: Experimental results demonstrate that REMEMO outperforms the baseline model T5 on multiple temporal question answering datasets under various settings. This indicates that REMEMO's approach of explicitly modeling time relations between sentences leads to more accurate and effective reasoning over temporal contexts.\n\n3. Long-Range Temporal Dependencies: The paper provides further analysis indicating that REMEMO excels at modeling long-range and complex temporal dependencies. This capability is crucial for a wide range of applications where understanding distant relationships in time is essential.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9XXVNLFN9j", "reviewer": "Reviewer_ZZGY", "summary": "", "questions": "How about the efficiency of REMEMO, e.g., the training time on each dataset, which can be used to evaluate whether the proposed method can be used for scale up over real-world application?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer ZZGY for the insightful comments and suggestions that have helped identify our work's strengths and potential improvements.\n\n## 1. The baseline model seems to be less.\n\n### TL;DR:\n\n1. There are only a small number of existing methods focusing on temporal question answering. Among them, TempT5 is the only proper one for comparison. It was proposed in the TempReason dataset paper.\n2. TempT5 improves the performance of T5+LM for +$0.54\\%$ F1-scores, but does not improve over our best model (REMEMO+SFT) and even slightly hurts its performance.\n3. Our best model remains the best of all.\n\n### Detailed Author Response:\n\nTemporal question answering (TQA), as a specific task, has garnered limited attention in earlier years, resulting in few existing methods for comparison. As far as we know, all existing non-GPT methods that conducted experiments on the three datasets we adopted come from their corresponding original dataset papers.\n\n- **TimeQA**: Their original dataset paper adopts Fusion-in-Decoder (FiD), for which we have implemented the experiments and reported the results in Table 4 of our paper.\n- **TempReason**: Their dataset paper proposed TempT5, a reinforcement learning (RL) method that further improves supervised fine-tuned (SFT) T5 models. It is a meaningful comparison for our experiments. We thus conducted additional experiments using this RL method and include the results in Table 1 below.\n- **SituatedQA**: Their dataset paper adopts Dense Passage Retrieval (DPR) [1] as the baseline. Apart from the retrieval component, DPR is no more than a simple BERT-based extractive QA baseline. We list the DPR results reported by the SituatedQA paper in Table 2 below.\n\n**Table 1**: F1-scores on TempReason. All experiments are conducted using T5-v1.1-base. \"SFT+RL\" refers to TempT5.\n\n| Pre-training Method | Fine-tuning Method | Temp-L2 | OBQA-L2 | Temp-L3 | OBQA-L3 | Average |\n| --- | --- | --- | --- | --- | --- | --- |\n| T5 | SFT | 90.62 | 44.99 | 59.56 | 41.76 | 59.23 |\n| T5 | SFT+RL | 90.61 | 44.64 | 59.88 | 41.42 | 59.14 |\n| T5+LM | SFT | 94.64 | 49.03 | 62.71 | 43.02 | 62.35 |\n| T5+LM | SFT+RL | 95.05 | 49.60 | 63.92 | 43.00 | 62.89 |\n| REMEMO | SFT | 97.67 | 51.58 | 91.17 | 44.91 | 71.33 |\n| REMEMO | SFT+RL | 97.61 | 51.14 | 91.28 | 44.70 | 71.18 |\n\nAs can be seen from Table 1 above, SFT+RL (i.e., TempT5) only improves the averaged performance of T5+LM but does not work for all other models. Specifically, REMEMO+SFT (our best model) already obtains the best results among all of them; and further applying RL to our REMEMO+SFT slightly hurts its performance. This may be caused by the fact that all our T5+SFT checkpoints (including baselines and REMEMO) are obtained through extensive hyperparameter search so little room is left for further improvement through further dataset-specific RL training.\n\n**Table 2**: Results on SituatedQA. DPR and Human* results are from the original SituatedQA dataset paper. Human* results were from human evaluation conducted on 50 randomly-sampled instances.\n\n| Method | F1 | EM |\n| --- | --- | --- |\n| DPR | N/A | 23.00 |\n| REMEMO-base | 55.06 | 48.12 |\n| REMEMO-large | 56.06 | 49.12 |\n| Human* | N/A | 57.00 |\n\n> Notes:\n> \n> 1. Because TempT5 requires the dataset to have negative answers in addition to the correct answer, we can only conduct TempT5 experiments on TempReason, which is the only dataset that contains negative answer candidates.\n> 2. The original TempReason paper adopted T5-v1.0 for experiments, while we adopted T5-v1.1 in all our experiments. We did not adopt T5-v1.0 because its pre-training corpora included downstream datasets, such as SQuAD, which will make the evaluation of few-shot QA unfair. In comparison, T5-v1.1 is pre-trained on raw texts in a fully-unsupervised manner.\n> 3. The original SituatedQA paper only reported exact match (EM) scores.\n> 4. For the results in Table 1 above, the default setting in TempReason's released code is 'num_layers_unfrozen'=6 and 'max_len'= 768. We additionally tried 'num_layers_unfrozen'$\\in$ {-1, 6, 2, 1} and 'max_len' $\\in$ {768, 512}. We then take the models that obtain the best validation F1-scores for testing.\n\n## 2. The process of extracting relation (i.e., Later, Earlier, Contemp) is not illustrated explicitly.\n\nWe thank the reviewer for pointing out this issue. We will add more detailed explanations and input-output pair examples of our pre-processing pipeline in the next version of our paper. \n\nTo help better illustrate the details of our pre-processing pipeline, we plan to do the following:\n\n- We will add detailed explanations of the time-identification phase and the time-normalization phase to the appendix for the next version of our paper.\n    - The time-identification phase is to fine-tune a RoBERTa model using TimeBank v1.2 Dataset. We will add several cases on (i) how this phrase works; and (ii) what the input and output could be.\n    - The time-normalization phase is to normalize the time-expressions that were extracted from the previous phase using a rule-based script. We have provided the link to the script in Line #207 of our paper and shown several input-output examples in Table 2 of our paper.\n- We will release the code for all experiments, including the pre-processing pipeline and the time-tagging fine-tuned RoBERTa checkpoint after the anonymous period, so that all details will be clear then.\n\n## 3. How about the efficiency of REMEMO, e.g., the training time on each dataset, which can be used to evaluate whether the proposed method can be used for scale up over real-world application?\n\n### TL;DR:\n\nFine-tuning normally takes 1~3 hours using a single GPU.\n\n### Detailed Author Response:\n\nThe efficiency of REMEMO is two-fold:\n\n- **Pre-training Cost**: We reported the pre-training cost in Section 4.1.2 of our paper.\n- **Fine-tuning Cost**: The fine-tuning cost of REMEMO is the same as a usual T5 model. Specifically, we fine-tune each model for 10 epochs on each dataset. The specific training time depends on the size of the corresponding dataset. Below are a few examples:\n    - Fine-tuning a base-size model on TempReason-ReasonQA-L2 takes about 3 hours using a single V100-32G GPU.\n    - Fine-tuning a base-size model on TimeQA-hard takes about 2.5 hours using a single V100-32G GPU.\n    - Fine-tuning a large-size model on TimeQA-hard takes about 2.5 hours using a single A100-80G GPU.\n    - Fine-tuning a large-size model on SituatedQA takes about 1.5 hours using a single A100-80G GPU.\n\nWe will add a discussion paragraph regarding the time cost of REMEMO to Section *Limitations* in the next version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A TRC pre-training objective is designed for complex temporal reasoning.\n2. The performance improvements are remarkable.", "reasons_to_reject": "1. The baseline model seems to be less.\n2. The process of extracting relation (i.e., Later, Earlier, Contemp) is not illustrated explicitly.", "questions_to_authors": "How about the efficiency of REMEMO, e.g., the training time on each dataset, which can be used to evaluate whether the proposed method can be used for scale up over real-world application?", "paper_topic_and_main_contributions": "This paper proposes REMEMO, aiming to make use of time information by modeling time relations among sentences. Experimental results demonstrate the effectiveness of REMEMO.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gozJh5vPSU", "reviewer": "Reviewer_r1Nz", "summary": "", "questions": "1. In few-shot settings, REMEMO performs worse than T5+LM in many tasks, like OBQA L3. This is strange for me, because REMEMO has been pretrained on the domain-specific task and I think it should performe much better than T5 in fewshot settings. \n2. For those sentences that don't contain time token, how to determine the temporal relation between them and other sentences?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer r1Nz for the insightful comments and suggestions that have helped identify our work's strengths and potential improvements.\n\n## 1. Why use `Graph` to explain the proposed method. The motivation could be very easy to understand, that is identifying the temporal relation between two sentences during pretraining.\n\nWe use `graph` instead of temporal relation to explain our method because it is not mere temporal relation itself works. We show that REMEMO remains highly effective as long as “high-order” temporal relations exist among sentences during pre-training. This is rather similar to the scenario where there exist paths (either one-edge or multi-edge) from any node to another in a graph. This suggests that the internal dynamics during REMEMO pre-training may go beyond simple “one-order” time relation classification. Therefore, it might be better to explain REMEMO from a graph view.\n\n### 1.1 Using Only Time Relation Classification\n\nWhen initiating this work, our very first idea was to construct BERT-like pre-training instances, i.e., ```[CLS] Sentence-A [SEP] Sentence-B [SEP]```. The model is required to predict the time relation of Sentence-A to Sentence-B during pre-training. However, preliminary experiments showed that this method did not offer any substantial improvement over the baseline. We thus tried out a few modifications and found out that **including multiple sentences in one training instance** works well.\n\n### 1.2 Intuitive Explanation\n\nOur time-relation classification (TRC) objective aims to classify the time relation between any two sentences, which can be roughly seen as \"first-order\" time relation classification. However, the input of our REMEMO usually contains far more than two sentences (8~9 on average). Such a large number of sentences induces complex sentence-level dependencies and there may exist \"high-order\" sentence-to-sentence temporal connections that will affect the \"first-order\" relation classification (i.e., the TRC objective).\n\nAs an intuitive example, suppose we have 4 sentences, $\\{S_1, S_2, S_3, S_4\\}$, in one instance. The prediction from $S_1$ to $S_2$ may be influenced by the predictions (i) from $S_1$ to $S_3$; (ii) from $S_2$ to $S_3$; (iii) ... et al. If the model tends to predict that (i) $S_1$ is earlier than $S_3$ and (ii) $S_2$ is later than $S_3$, it is thus likely that the model would predict $S_1$ is earlier than $S_2$. Here, $S_3$ serves as a bridge node to construct a \"second-order\" time relation between $S_1$ and $S_2$. Similar ideas apply to higher-order relations. Overall, though the final prediction target is a simple \"first-order\" time relation, all these 8~9 sentences may have complex interactions with each other either directly or indirectly across many Transformer layers.\n\n### 1.3 Empirical Analysis\n\nThe above intuition may be supported by Table 6 of our paper. We briefly list the results below. `# Paths` refers to the number of paths from any node to another. In other words, `# Paths ≥ 1 for most nodes` implies that there exist “high-order” connections among most sentences. \n\n| Method | # Paths | Averaged F1 | Averaged EM |\n| --- | --- | --- | --- |\n| T5+LM | # Paths $=0$ | 62.71 | 52.66 |\n| REMEMO with $n$ TRC Labels | # Paths $\\geq 1$ for most nodes if the edges distribute randomly | 67.67 | 58.77 |\n| REMEMO with $\\frac{n(n-1)}{\\log n}$ TRC Labels | # Paths $\\geq 1$ for almost all nodes if the edges distribute randomly | 68.09 | 58.89 |\n| REMEMO with $n(n-1)$ TRC Labels | # Paths $=13,699$ for all nodes (suppose $n=8$) | 68.29 | 59.33 |\n\nAs shown in the table above, the three respective REMEMO models with $n$, $\\frac{n(n-1)}{\\log n}$ and $n(n-1)$ edges show similar performances and all significantly outperform T5+LM (i.e., with $0$ edges). This indicates that REMEMO remains effective as long as “high-order” temporal connections exist, which is similar to a graph where nodes are connected through multiple edges. We thus explain our method from the `Graph` view.\n\n## 2. The performance of T5 in TempReason seems to be much higher than what is reported in the original paper. The comparison with previous works is not sufficient.\n\n## TL;DR:\n\n1. The higher performances compared with the results reported from the TempReason paper are caused by a few differences between our implementation and the implementation of the authors of TempReason.\n2. We add two tables to report comparisons with other existing methods, including (i) TempT5 on the TempReason dataset; and (ii) DPR and human evaluation on the SituatedQA dataset. Results show that REMEMO remains the best of all.\n\n## Detailed Author Response:\n\n### 2.1 The performance of T5 in TempReason seems to be much higher than what is reported in the original paper [1].\n\nWe appreciate the reviewer for pointing out this issue. The different performances result from a few differences between our implementation and the implementation of the authors of TempReason.\n\n- **T5 Versions**: As mentioned in Section 4.1.2 of our paper, we adopted T5-v1.1 for all experiments, while the original TempReason papers adopted T5-v1.0. We did not adopt T5-v1.0 because its pre-training corpora included downstream datasets, such as SQuAD, which makes the evaluation of few-shot QA unfair. In comparison, T5-v1.1 is pre-trained on raw texts in a fully unsupervised manner, so the comparison for few-shot QA will be fair. Empirically speaking, in preliminary experiments, we found that\n    - Vanilla T5-v1.0 always outperformed the continually pre-trained T5-v1.0 significantly under the few-shot setting.\n    - Vanilla T5-v1.1 outperformed vanilla T5-v1.0 for roughly 5 to 10 points in terms of F1-scores on most adopted datasets under the full-set fine-tuning settings. We assume that this is the main cause for the higher performance of our baselines compared with the results in the original TempReason paper.\n- **Dataset Pre-processing**: To fit the T5 fine-tuning process into one single GPU, we truncate the lengths of the contexts to 1,500 chars for all downstream datasets under single-context and few-shot settings. This may boost performance since slightly shorter contexts make the modeling of dependencies a little easier. We mentioned this in Line 423 of our paper.\n- **Hyper-parameter**: The original TempReason paper simply chose a set of hyper-parameters for all experiments. In comparison, we adopted hyper-parameter search in all downstream experiments to avoid biases resulting from hyper-parameter choices. This may boost the performance of our implementation. We listed detailed hyper-parameters in Appendix A.2 of our paper.\n\nWe will highlight the above implementation differences in the next version of our paper.\n\n### 2.2 The comparison with previous works is not sufficient.\n\n### TL;DR:\n\n1. There are only a small number of existing methods focusing on temporal question answering. Among them, TempT5 is the only proper one for comparison. It was proposed in the TempReason dataset paper.\n2. TempT5 improves the performance of T5+LM for +$0.54\\%$ F1-scores, but does not improve over our best model (REMEMO+SFT) and even slightly hurts its performance.\n3. Our best model remains the best of all.\n\n### Detailed Author Response:\n\nTemporal question answering (TQA), as a specific task, has garnered limited attention in earlier years, resulting in few existing methods for comparison. As far as we know, all existing non-GPT methods that conducted experiments on the three datasets we adopted come from their corresponding original dataset papers.\n\n- **TimeQA**: Their original dataset paper adopts Fusion-in-Decoder (FiD), for which we have implemented the experiments and reported the results in Table 4 of our paper.\n- **TempReason**: Their dataset paper proposed TempT5, a reinforcement learning (RL) method that further improves supervised fine-tuned (SFT) T5 models. It is a meaningful comparison for our experiments. We thus conducted additional experiments using this RL method and include the results in Table 1 below.\n- **SituatedQA**: Their dataset paper adopts Dense Passage Retrieval (DPR) [1] as the baseline. Apart from the retrieval component, DPR is no more than a simple BERT-based extractive QA baseline. We list the DPR results reported by the SituatedQA paper in Table 2 below.\n\n**Table 1**: F1-scores on TempReason. All experiments are conducted using T5-v1.1-base. \"SFT+RL\" refers to TempT5.\n\n| Pre-training Method | Fine-tuning Method | Temp-L2 | OBQA-L2 | Temp-L3 | OBQA-L3 | Average |\n| --- | --- | --- | --- | --- | --- | --- |\n| T5 | SFT | 90.62 | 44.99 | 59.56 | 41.76 | 59.23 |\n| T5 | SFT+RL | 90.61 | 44.64 | 59.88 | 41.42 | 59.14 |\n| T5+LM | SFT | 94.64 | 49.03 | 62.71 | 43.02 | 62.35 |\n| T5+LM | SFT+RL | 95.05 | 49.60 | 63.92 | 43.00 | 62.89 |\n| REMEMO | SFT | 97.67 | 51.58 | 91.17 | 44.91 | 71.33 |\n| REMEMO | SFT+RL | 97.61 | 51.14 | 91.28 | 44.70 | 71.18 |\n\nAs can be seen from Table 1 above, SFT+RL (i.e., TempT5) only improves the averaged performance of T5+LM but does not work for all other models. Specifically, REMEMO+SFT (our best model) already obtains the best results among all of them; and further applying RL to our REMEMO+SFT slightly hurts its performance. This may be caused by the fact that all our T5+SFT checkpoints (including baselines and REMEMO) are obtained through extensive hyperparameter search so little room is left for further improvement through further dataset-specific RL training.\n\n**Table 2**: Results on SituatedQA. DPR and Human* results are from the original SituatedQA dataset paper. Human* results were from human evaluation conducted on 50 randomly-sampled instances.\n\n| Method | F1 | EM |\n| --- | --- | --- |\n| DPR | N/A | 23.00 |\n| REMEMO-base | 55.06 | 48.12 |\n| REMEMO-large | 56.06 | 49.12 |\n| Human* | N/A | 57.00 |\n\n> Notes:\n> \n> 1. Because TempT5 requires the dataset to have negative answers in addition to the correct answer, we can only conduct TempT5 experiments on TempReason, which is the only dataset that contains negative answer candidates.\n> 2. The original TempReason paper adopted T5-v1.0 for experiments, while we adopted T5-v1.1 in all our experiments. We did not adopt T5-v1.0 because its pre-training corpora included downstream datasets, such as SQuAD, which will make the evaluation of few-shot QA unfair. In comparison, T5-v1.1 is pre-trained on raw texts in a fully-unsupervised manner.\n> 3. The original SituatedQA paper only reported exact match (EM) scores.\n> 4. For the results in Table 1 above, the default setting in TempReason's released code is 'num_layers_unfrozen'=6 and 'max_len'= 768. We additionally tried 'num_layers_unfrozen'$\\in$ {-1, 6, 2, 1} and 'max_len' $\\in$ {768, 512}. We then take the models that obtain the best validation F1-scores for testing.\n\n## 3. Related to event temporal relation classification\n\nWe agree with the reviewer our work is closely related to previous work that focused on event temporal relation classification (ETRC). TimeBank, on which we trained the time-identification model, belongs to this line of work. We do appreciate the efforts that were made by this line of work. Below we discuss some similarities and differences between our work and previous ETRC works. We will add these discussions and related references ([1], [2], [3], [4], [5]) in the next version of our paper.\n\n### 3.1 Focus on temporal relations but on different granularities\n\nThe main concept shared by our work and previous ETRC works is the temporal relation between two events. However, some task differences exist. Our work focuses on temporal question answering (TQA), in which most clues in existing datasets are **explicit and coarse-grained time expressions**, such as \"*June 2012*\", \"*18th century*\" and \"*1980s*\". In contrast, the line of ETRC works focused on time information that is **more implicit, fine-grained, and nuanced**. Below we show some contrasting examples.\n\n**Temporal Question Answering (Clues to answer the question are bold.)**\n\n- Example-1 (from TimeQA-hard):\n    \n    Context: Mei Li Vos Mei Li Vos (born 31 March 1970 ) is a Dutch politician, former trade unionist and editorialist. A member of the Labour Party ( PvdA ), **she was a member of the House of Representatives from 1 March 2007 to 17 June 2010 and again from 20 September 2012 until 23 March 2017.** She has been a member of the Senate since 11 June 2019.\n    \n    Question: What was the position of Mei Li Vos between Feb 2016 and Dec 2016?\n    \n    Answer: member of the House of Representatives.\n    \n\n- Example-2 (from SituatedQA):\n    \n    Context: Ray Allen is widely considered to be one of the greatest three-point shooters of all-time, and **he held the record for most three-pointers made in a career from 2011 until 2021**, when he was surpassed by Stephen Curry. ........... During a game against the Los Angeles Lakers **on February 10, 2011, Allen became the all-time NBA leader in total three-pointers made (2,562), surpassing Reggie Miller's record of 2,560**.\n    \n    Question-1: Who made the most three point shots in total in the NBA as of 2012?\n    \n    Answer-1: Ray Allen.\n    \n    Question-2: Who made the most three point shots in total in the NBA as of 2010?\n    \n    Answer-2: Reggie Miller.\n    \n\n**Event Temporal Relation Classification**\n\n- Example-1 (from [1]):\n    \n    Police **confirmed** **Friday** that the body **found** along a highway in San Juan **belonged** to Jorge Hernandez.\n    \n    TimeBank-Dense Labels: *belonged* before *confirmed*; *belonged* before *found*; *found* before *confirmed*; *belonged* before *Friday*; *confirmed* is-included-in *Friday*; *found* is-included-in *Friday*.\n    \n- Example-2 (from [2]):\n    \n    At one point , when it **became** clear controllers could not contact the plane, someone **said** a prayer.\n    \n    MATRES Label: *became* before *said*.\n    \n- Example-3 (from [2]):\n    \n    The US is bolstering its military presence in the gulf, as President Clinton **discussed** the Iraq crisis with the one ally who has **backed** his threat of force, British prime minister Tony Blair.\n    \n    MATRES Label: *discussed* after *backed*.\n    \n\n### 3.2 Modeling\n\nSeveral methods on ETRC ([3], [4]) adopted graph neural networks to model ETRC from a graph view. We also propose a graph-like view over our extracted temporal relations from raw corpora and incorporate this view into our modeling process. However, we make efforts to incorporate our proposed graph view into large-scale pre-training and empirically demonstrate the effectiveness of such pre-training.\n\n### 3.3 Can we first finetune LMs in event temporal relation classification and then apply them to solve temporal QA?\n\nWe did not try the exact method above, but we used to conduct experiments under a similar setting which may provide some information. We constructed time relation labels on the “context” of temporal question-answering (TQA) datasets. We tried to directly fine-tune a vanilla T5 with two joint objectives: the answer generation objective and the time relation classification (TRC) objective. Below are the F1 scores. Note that these results were early trials and **were conducted in a different setting from those of the reported results in our paper, making them not directly comparable**.\n\n| Method | TimeQA-Easy | TimeQA-Hard | ReasonQA-L2 | OBQA-L2 | ReasonQA-L3 | OBQA-L3 |\n| --- | --- | --- | --- | --- | --- | --- |\n| T5-base | 65.77 | 61.69 | 85.10 | 37.04 | 60.89 | 31.89 |\n| T5-base + TRC fine-tuning | 61.13 | 54.37 | 86.56 | 34.82 | 74.08 | 32.53 |\n\nThe table above shows that directly applying TRC in the fine-tuning stage yields mixed results. Though TRC shows a large improvement on ReasonQA-L3, it also gives significant decreases on TimeQA-Easy, TimeQA-Hard, and OBQA-L2. Based on the results above, we believe pre-training is a necessary step for adapting temporal relation classification to solve temporal QA.\n\n## 4. In few-shot settings, REMEMO performs worse than T5+LM in many tasks, like OBQA L3. This is strange because REMEMO has been pre-trained on the domain-specific task and it should perform much better than T5 in few-shot settings.\n\nThis may be caused by\n\n1. Even though REMEMO is pre-trained with one additional objective: time relation classification (TRC), this classification objective is not directly aligned with the answer generation objective of downstream temporal QA tasks.\n2. T5+LM is only pre-trained with seq-to-seq generation and does not inherit \"noise\" from TRC, so it might be more directly applicable to QA, whose format is also seq-to-seq generation.\n3. With slightly more instances (i.e., increasing from 16-shot to 64-shot), REMEMO starts to outperform T5+LM on most datasets. This may suggest that REMEMO requires a reasonable but not large number of instances to rectify its behavior into answer generation.\n\n## 5. For those sentences that don't contain time token, how to determine the temporal relation between them and other sentences?\n\nWe excluded such sentences when constructing time relation classification (TRC) labels.\n\n## References:\n\n- [1] Taylor Cassidy et al. An annotation framework for dense event ordering. ACL 2014.\n- [2] Qiang Ning et al. A multi-axis annotation scheme for event temporal relations. ACL 2018.\n- [3] Puneet Mathur et al. TIMERS: Document-level Temporal Relation Extraction. ACL 2021.\n- [4] Shuaicheng Zhang et al. Extracting Temporal Event Relation with Syntax-guided Graph Transformer. NAACL 2022.\n- [5] Quzhe Huang, et al. More than Classification: A Unified Framework for Event Temporal Relation Extraction. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed pretraining task, REMEMO, is simple yet effective. This method might be useful to alleviate the problem that LLMs cannot handle temporal reasoning well.\n\n2. The ablation study with different context length shows the REMEMO could alleviate the degration brought by long input. This result shows the REMEMO has the ability to capture the most important temporal dependency and ignore the irrelevant information.", "reasons_to_reject": "1. I don't understand why the authors use ``Graph`` to explain the proposed method. I think the motivation could be very easy to understand, that is identifying the temporal relation between two sentences during pretraining. But adding ``Graph`` makes me confused when reading the paper. \n\n2. The comparison with previous works is not sufficient. In the experiment, the authors only report their own baselines but don't include any previous efforts, like the  TempT5 proposed by Tan[1]. Besides, the perfromance of T5 in TempReason seems to be much higher than what is reported in the original paper[1]. Any explanation for it?\n\n3. The authors hope to model the temporal dependencies facts, but they seem to ignore a line of works that directly solve this problem, that is event temporal relation classification[2-6]. Can we first finetune LMs in such task and then apply them to solve temporal QA? At least, I think the authors should discuss them in related work.\n\n[1] Qingyu Tan, et al. Towards benchmarking and improving the temporal reasoning capability of large language models. ACL 2023.\n\n[2] Taylor Cassidy et al. An annotation framework for dense event ordering. ACL 2014.\n\n[3] Qiang Ning et al. A multi-axis annotation scheme for event temporal relations. ACL 2018.\n\n[4] Puneet Mathur, et al. TIMERS: Document-level temporal relation extraction. ACl 2021.\n\n[5] Shuaicheng Zhang, et al. Extracting temporal event relation with syntactic-guided temporal graph transformer. NAACL 2022.\n\n[6] Quzhe Huang, et al. More than Classification: A Unified Framework for Event Temporal Relation Extraction. ACL 2023.", "questions_to_authors": "1. In few-shot settings, REMEMO performs worse than T5+LM in many tasks, like OBQA L3. This is strange for me, because REMEMO has been pretrained on the domain-specific task and I think it should performe much better than T5 in fewshot settings. \n2. For those sentences that don't contain time token, how to determine the temporal relation between them and other sentences?", "paper_topic_and_main_contributions": "To better capture the temporal dependencies, this paper proposes a new pretraining task, REMEMO, which predicts the temporal relation between any two sentences in a document. Specifically, REMEMO first identifies the time token in the documents and builds a normalized time span for every sentences, like [2003-01-01, 2004-01-01). Then, by comparing the time spans of two sentences, the temporal relation between two sentences could be classified as Earlier, Later and Contemporary. During pretraining, REMEMO requires the model to predict the temporal relation based on the sentence representations. This paper apply REMEMO to T5 and shows greaat improvements on several temporal question answering datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for incorporating time-sensitive information into pretrained language models. The authors extract time-stamps, encode them as a graph and propose the Time Relation Classification (TRC) pretraining objective. The paper shows promising results on temporal QA and, interestingly, offers better modeling of complex dependencies. This is an important advancement since temporal reasoning is indeed problematic for SOTA LMs.

One reason the paper got initially moderate scores with the reviewers was due to problematic evaluation: all the three reviewers raise concerns wrt baseline. This, however, seems more like a presentation issue that has been resolved with the authors' thorough rebuttal (cf. discussion). I would strongly suggest, however, that the authors incorporate the relevant discussion into the next version to improve clarity.

Another issue, raised by all the reviewers in some aspect, is the approach for extracting temporal information from text (pipeline). As it seems, the authors only extract explicit timestamps and then merge them rather straightforwardly. They do not assign temporal information to sentences with no explicit timestamps and do not model non-trivial cases of temporal expressions (e.g., "later") -- for example, "X submitted a paper to EMNLP-2023. They resubmitted it again to ACL" would be problematic as it seems? However, even with a simplistic approach to extracting temporal expressions, the model achieves interesting results.
Decision: Accept-Main