SlowGuess/NAACL_2022_OCR huggingface dataset:
Jingyi You, Dongyuan Li, Hidetaka Kamigaito, Kotaro Funakoshi and Manabu Okumura
{youjy, lidy, kamigaito, funakoshi, oku}@lr.pi.titech.ac.jp
Previous studies on the timeline summarization (TLS) task ignored the information interaction between sentences and dates, and adopted pre-defined unlearnable representations for them. They also considered date selection and event detection as two independent tasks, which makes it impossible to integrate their advantages and obtain a globally optimal summary. In this paper, we present a joint learning-based heterogeneous graph attention network for TLS (HeterTLS), in which date selection and event detection are combined into a unified framework to improve the extraction accuracy and remove redundant sentences simultaneously. Our heterogeneous graph involves multiple types of nodes, the representations of which are iteratively learned across the heterogeneous graph attention layer. We evaluated our model on four datasets, and found that it significantly outperformed the current state-of-the-art baselines with regard to ROUGE scores and date selection metrics.
Timeline summarization (TLS) is designed to extract sentences that describe an evolutionary story from a massive amount of web articles with respect to a specific topic in chronological order. TLS has drawn much attention in recent years (Chen et al., 2019; Martschat and Markert, 2018; You et al., 2021b; Ghalandari and Ifrim, 2020; Yu et al., 2021) since it releases people from burdensome manual creation of summaries and gives readers a faster but comprehensive access to track events from many aspects, such as start and end, causality, and the main protagonists involved.
Most studies on TLS seek ways to combine two individual subtasks: date selection and event detection. Depending on different strategies for them, current methods are generally divided into three categories (Ghalandari and Ifrim, 2020): 1) direct summarization approaches (Chieu and Lee, 2004;
Tran et al., 2013; Martschat and Markert, 2018; Duan et al., 2020) directly identify topic-related sentences from a collection of news articles to form a timeline; 2) date-wise summarization methods (Wang et al., 2016; Ghalandari and Ifrim, 2020; Li et al., 2021; Quatra et al., 2021) first select salient dates then construct a timeline for each date individually with sentences of the highest score; and 3) event detection algorithms (Steen and Markert, 2019; Duan et al., 2020; Yu et al., 2021) detect events by clustering sentences from multi-timeline news articles then identify several of the most important events and summarize them separately.
Although great successes have been achieved in conducting TLS, several issues remain unsolved. First, current TLS methods mainly adopt statistical hand-designed features to represent dates, e.g., the number of published articles and topic-related sentences in a specific time duration (Yu et al., 2021; Ghalandari and Ifrim, 2020), and employ sentence-BERT (Reimers and Gurevych, 2019) and other predefined unchangeable representations for sentences. The low-level or unlearnable representations tend to ignore the semantic and temporal information interaction between sentences and dates, which significantly degrades the performance of downstream tasks. Secondly, traditional approaches focus on either date selection or event detection. Although excellent date selection algorithms can pinpoint accurate timeline dates, they usually extract topic-irrelevant sentences. While event detection algorithms are capable of avoiding redundant summaries by various clustering strategies, they sometimes capture wrong timeline dates. To the best of our knowledge, there is no framework that jointly learns the advantages of the above two subtasks to accurately capture salient dates and eliminate topic-irrelevant sentences in a timeline.
To circumvent the above dilemma, we propose to jointly learn date/sentence representations and event detection-based sentence clustering in a het
Human Language Technologies, pages 4091 - 4104
erogeneous graph attention network (HAN) for TLS. Specifically, we construct a heterogeneous graph with dates, words, and sentences as semantic units to solve the first problem. In this graph, words act as a bridge between dates and sentences, enabling date nodes to learn different granularities (word- and sentence-level) of semantic information and sentence representations to be complemented with a date-related intra- and cross-sentence message. As for the second issue, semi-supervised date prediction and event detection-based clustering are integrated into an overall objective, where labeled dates guide and facilitate sentence clustering, and sentence-level clustering information indicating main events improves the accuracy of unlabeled date prediction. Note that we create a new way beyond the above-mentioned three categories as a joint end-to-end approach since we no longer have to handle each subtask step by step.
- This study is the first to construct a model for automatic TLS as a HAN that propagates heterogeneous information with different granularities, of date-word-sentence, to effectively learn flexible and accurate representations for both date and sentence nodes.
- Date selection and event detection subtasks are incorporated into an overall objective so that they can be jointly optimized to obtain a globally optimal solution.
- We have empirically shown that HeterTLS outperformed all existing competitors on four benchmark datasets. Its effectiveness and robustness were further confirmed via ablation studies and parameter analysis.
2.1 Timeline summarization
Unlike multi-document summarization (MDS), TLS executes both date selection and summary extraction (Zhou et al., 2021). In accordance with different strategies for defining the two subtasks, available approaches are categorized into three classes, whose major methods are reviewed as follows.
Direct summarization approaches (Allan et al., 2001; Yan et al., 2011a; Li and Li, 2013; Zhao et al., 2013; Suzuki and Kobayashi, 2014) treat the task as MDS with time-stamped textual summaries. Chieu and Lee (2004) directly rank and extract sentences
relevant to a query from a collection of documents and place them along a timeline. As the current state-of-the-art method for direct summarization, revised submodular-function optimization, which is commonly used for MDS, is applied to search for a combination of sentences from an entire document collection (Martschat and Markert, 2018).
Date-wise summarization methods (Li et al., 2021) first select dates then extract sentences corresponding to the dates. Tran et al. (2013, 2015b) propose a supervised graphical model for selecting salient dates and tracking events on each date. In another study, text and image embeddings are jointly learned using a scalable low-rank approximation approach to generate a more readable timeline summary (Wang et al., 2016).
Event detection algorithms (Tran et al., 2015c; Pasquali et al., 2019; Duan et al., 2020) usually cluster documents by affinity propagation to detect events and summarize them individually along a timeline (Steen and Markert, 2019) or implement multi-timeline summarization (Yu et al., 2021).
2.2 Heterogeneous graph for summarization
A heterogeneous graph contains different types of nodes and multiple relationships between nodes (Xu et al., 2021; Hu et al., 2021). Wang et al. (2020) present a HAN for single or multiple document extractive summarization to enrich cross-sentence relations through additional semantic units. Jia et al. (2020) leverage a sentence-level redundancy layer into a HAN to remove excessive phrases. Although much research has gone into constructing source documents as heterogeneous graphs and using graph attention network-based first-order neighbors during information dissemination, longer-distance heterogeneous paths have not been considered. Inspired by Wang et al. (2019), we extended a HAN to TLS and developed HeterTLS to learn better node representations for downstream tasks.
3.1 Problem definition and preliminaries
Given a collection of news documents  $\mathcal{D}$  within  $\mathcal{T}$  dates, TLS involves 1) predicting a sequence of date labels  $\{y_1,\dots ,y_{\mathcal{T}}|y_i\in \{0,1\} \}$ , where  $y_{t} = 1$  represents the  $t$ -th date included in the timeline; and 2) ranking and extracting sentences from candidates for each selected date. The number of dates as well as the length of the daily summaries are typically controlled by the user.
Figure 1: Model overview. HeterTLS consists of three chief components: (a) graph constructor and initializer, (b) heterogeneous graph encoder with sentence clustering constraint, and (c) timeline summary extractor. We first construct heterogeneous network for date, sentence, and word nodes with two initialization strategies. We then extract meta-paths and iteratively update node representations via HAN under nuclear norm constraint on sentence nodes. Finally, we predict unlabeled date nodes and extract sentences from candidate clusters.
Given a graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  with  $\mathcal{V} = V_d\cup$ $V_{w}\cup V_{s}$  and  $\mathcal{E} = E_{w - d}\cup E_{w - s}$ , where  $V_{d}$ ,  $V_{w}$  and  $V_{s}$  respectively denote a node set for dates, words, and sentences and  $E_{w - d}$  and  $E_{w - s}$  are a set of undirected edges between word-date and word-sentence. Specifically,  $V_{d} = \{d_{1},\dots ,d_{\mathcal{T}}\}$ $V_{w} = \{w_{1},\dots ,w_{m}\}$ , and  $V_{s} = \{s_{1},\dots ,s_{n}\}$  correspond to  $\mathcal{T}$  dates,  $m$  unique words, and  $n$  sentences within  $\mathcal{D}$ .  $e_{ij}\neq 0$  ( $i\in \{1,\dots ,m\},j\in$ $\{1,\dots ,n\}$ ) of  $E_{w - s}$  indicates that the  $i$  th word appears in the  $j$  -th sentence.  $e_{ij}\neq 0$  ( $i\in$ $\{1,\dots ,m\},j\in \{1,\dots ,\mathcal{T}\}$ ) of  $E_{w - d}$  signifies the  $i$  -th word appears in the articles published on the  $j$  -th date. No edge exists between nodes of the same type, e.g., word pairs. We then define a metapath and meta-path-based neighbors for disseminating information among heterogeneous nodes.
Definition 1 Meta-path  $\Phi$  is defined as a path in the form of  $v_{1} \xrightarrow{e_{1}} \cdots \xrightarrow{e_{q}} v_{q+1}$ , which describes a composite edge relation  $e = e_{1} \circ \cdots \circ e_{q}$  between nodes  $v_{1}$  and  $v_{q+1}$ , where  $\circ$  denotes the composition of relations.
Definition 2 Meta-path-based neighbors  $\mathcal{N}_i^\Phi$  of the  $i$ -th node are defined as all nodes in a single meta-path  $\Phi$ .
Figure 1 exhibits an overview of HeterTLS, which consists of three main components: (a)
graph constructor and initializer, (b) heterogeneous graph encoder with sentence clustering constraint, and (c) timeline summary extractor. Each component is introduced subsequently in detail in the following subsections.
3.2 Graph constructor and initializer
Let  $\mathbf{X}_d\in \mathbb{R}^{\mathcal{T}\times r_d}$ ,  $\mathbf{X}_w\in \mathbb{R}^{m\times r_w}$ , and  $\mathbf{X}_s\in \mathbb{R}^{n\times r_s}$  respectively denote input feature matrices for date, word, and sentence nodes, where  $r_d,r_w$ , and  $r_s$  are dimensions of date representations, word embeddings, and sentence representations. We initialize the  $j$ -th sentence node in Figure 1 (a) by concatenating its local n-gram feature  $p_j$  and sentence-level global feature  $q_{j}$  as  $X_{s_j} = [p_j;q_j]$ .  $p_j$  is captured by a convolutional neural network (CNN) (LeCun et al., 1998) with different kernel sizes, and  $q_{j}$  is gripped by a bidirectional long short-term memory (Bi-LSTM) (Hochreiter and Schmidhuber, 1997). Considering the success of transformer-based pretrained models, we also provide another initialization strategy: using BERT (Devlin et al., 2019) and sentence-BERT (Reimers and Gurevych, 2019) as word and sentence encoders. Date nodes take the average-pooling of their connected sentences as initialization for both aforementioned strategies.
To leverage the saliency of each word in differ
ent sentences or dates, we propose term frequency-inverse sentence frequency (TF-ISF) and term frequency-inverse date frequency (TF-IDATEF) weights to initialize edges in  $E_{w - s}$  and  $E_{d - w}$ . Specifically, TF is the number of occurrences of  $w_{i}$  in  $s_j$  or  $d_t$ , and ISF/IDATEF is determined by dividing the total number of sentences or dates in  $\mathcal{D}$  by the number of sentences or dates containing  $w_{i}$  (refer to Appendix D for more details).
3.3 Heterogeneous graph encoder with sentence clustering constraint
As Figure 1 (b) illustrates, we first iteratively update node representations via meta-paths in heterogeneous graph attention layers. We then introduce how we constrain sentence representations to reserve a low-rank-based clustering structure, which helps sentence nodes learn better event-related information. Finally, the semi-supervised date classification and sentence clustering structure are jointly learned in an overall objective.
3.3.1 Heterogeneous graph attention layer
Node representations are updated by hierarchical heterogeneous graph attention layers, where the node-level attention layer ensures information propagation and aggregation in a single meta-path, while the semantic-level one is committed to merging messages from multiple meta-paths. Specifically, referring to  $\mathbf{h}_i$  as the hidden state of the  $i$ -th node, the node-level attention layer is calculated as
$$
e _ {i j} ^ {\Phi_ {p}} = \text {L e a k y R e L U} \left(\mathbf {W} _ {a} \left[ \mathbf {W} _ {\phi_ {i}} \mathbf {h} _ {i}; \mathbf {W} _ {\phi_ {j}} \mathbf {h} _ {j} \right]\right), \tag {1}
$$
$$
\alpha_ {i j} ^ {\Phi_ {p}} = \frac {\exp \left(e _ {i j} ^ {\Phi_ {p}}\right)}{\sum_ {l \in \mathcal {N} _ {i} ^ {\Phi_ {p}}} \exp \left(e _ {i l} ^ {\Phi_ {p}}\right)}, \tag {2}
$$
$$
\mathbf {z} _ {i} ^ {\Phi_ {p}} = \underset {k = 1} {\overset {K} {\parallel}} \sigma \left(\sum_ {j \in \mathcal {N} _ {i} ^ {\Phi_ {p}}} \alpha_ {i j} ^ {\Phi_ {p}} \mathbf {W} _ {\phi_ {j}} \mathbf {h} _ {j}\right), \tag {3}
$$
where  $\mathbf{W}_a, \mathbf{W}_{\phi_i}$ , and  $\mathbf{W}_{\phi_j}$  are trainable parameters,  $\mathbf{z}_i^\Phi$  is the representation of the  $i$ -th node learned from the node-level attention layer by  $\Phi$ ,  $\alpha_{ij}^{\Phi}$  measures the importance of the  $j$ -th node to the  $i$ -th node via  $\Phi$ ,  $\mathcal{N}_i^\Phi$  contains all nodes in single metapath  $\Phi$ , and  $K$  is the number of multi-heads.
Afterwards, the semantic-level attention layer fuses all the meta-path information for the  $i$ -th node. We extract meta-paths  $\hat{\Phi}_{d_1\sim 3} = \{\text{date-word, date-word-date, date-word-sent}\}$  for date nodes,  $\hat{\Phi}_{w_{1\sim 2}} = \{\text{word-sent, word-date}\}$  for word nodes,
and  $\hat{\Phi}_{s_1\sim 3} = \{$  sent-word, sent-word-sent, sent-word-date\} for sentence nodes (Figure 1 (b)), while long-distance meta-paths are discarded due to their limited impact. With the assumption that the  $i$  -th node has  $P$  meta-paths as  $\{\Phi_1,\dots ,\Phi_P\}$  , the representation of the  $i$  -th node is updated as
$$
w _ {\Phi_ {p}} = \frac {1}{| \mathcal {V} |} \sum_ {i \in \mathcal {V}} \mathbf {q} ^ {T} \tanh  \left(\mathbf {W} \mathbf {z} _ {i} ^ {\Phi_ {p}} + \mathbf {b}\right), \tag {4}
$$
$$
\beta_ {\Phi_ {p}} = \frac {\exp \left(w _ {\Phi_ {p}}\right)}{\sum_ {l = 1} ^ {P} \exp \left(w _ {\Phi_ {l}}\right)}, \tag {5}
$$
$$
\mathbf {z} _ {i} = \sum_ {p = 1} ^ {P} \beta_ {\Phi_ {p}} \mathbf {z} _ {i} ^ {\Phi_ {p}}, \tag {6}
$$
where  $\mathbf{q}$ ,  $\mathbf{W}$ , and  $\mathbf{b}$  are learnable parameters and  $\beta_{\Phi_p}$  represents the importance of the  $p$ -th metapath for the final embedding of the  $i$ -th node.
In the same manner described by Wang et al. (2020), to avoid gradient vanishing after certain iterations, a residual connection and position-wise feed-forward network (FFN) layer with two linear transformations (Vaswani et al., 2017) are added after the semantic-level attention layer.
Iterative update: We alternately update each type of node to realize information propagation and aggregation. The updating process for the  $t$ -th iteration is measured as
$$
Z _ {w _ {1} \sim 2} ^ {t + 1} = N L e v e l \left(H _ {d} ^ {t}, H _ {s} ^ {t}, H _ {w} ^ {t}\right), \tag {7}
$$
$$
H _ {w} ^ {t + 1} = F F N \left(S L e v e l \left(Z _ {w _ {1} \sim 2} ^ {t + 1}\right) + H _ {w} ^ {t}\right), \tag {8}
$$
$$
Z _ {d _ {1} \sim 3} ^ {t + 1} = N L e v e l \left(H _ {d} ^ {t}, H _ {s} ^ {t}, H _ {w} ^ {t + 1}\right), \tag {9}
$$
$$
H _ {d} ^ {t + 1} = F F N \left(S L e v e l \left(Z _ {d _ {1} \sim 3} ^ {t + 1}\right) + H _ {d} ^ {t}\right), \tag {10}
$$
$$
Z _ {s _ {1} \sim 3} ^ {t + 1} = N L e v e l \left(H _ {d} ^ {t + 1}, H _ {s} ^ {t}, H _ {w} ^ {t + 1}\right), \tag {11}
$$
$$
H _ {s} ^ {t + 1} = F F N \left(S L e v e l \left(Z _ {s _ {1 \sim 3}} ^ {t + 1}\right) + H _ {s} ^ {t}\right), \tag {12}
$$
where  $NLevel$  and  $SLevel$  respectively indicate node-level and semantic-level attention layers, and  $H^t$  is the stacked hidden state of a certain type of node at the  $t$ -th timestep. Eqs. 8, 10, and 12 represent the residual connection and FFN layer.
3.3.2 Sentence clustering constraint
Detecting main events from  $\mathcal{D}$  can effectively reduce the redundancy when generating summaries. Current TLS methods (Yu et al., 2021) identify major events by applying K-means directly to sentence representation matrix  $H_{s}$ , which has two limitations. First, K-means is sensitive to initialization and outliers, resulting in unstable outputs (Ding
and Li, 2007). Furthermore, the clustering performance is undesirable due to the independence of sentence representation learning and sentence clustering. Even though structure learning has the potential to address the above issues by co-clustering on a newly created bipartite graph to extract the clustering structure (You et al., 2021a; Nie et al., 2017), it is not suitable for our framework to make  $H_{s}$  a block-diagonal matrix with  $k$  components. Theorem 1 paves the way to detect the clustering structure of  $H_{s}$  by adding a low-rank constraint.
Theorem 1 (Chung and Graham, 1997) The multiplicity of eigenvalue 0 of the normalized Laplacian matrix of  $H_{s}$  is equal to the number of clusters in  $H_{s}$ .
Theorem 1 indicates that the block-diagonal clustering structure relies on newly constructing an adjacency network of sentence nodes, which increases the complexity of the model. Haeffele and Vidal (2020); Piao et al. (2019) propose the nuclear norm and prove that the constraint on the Laplace matrix of  $H_{s}$  is mathematically equal to the constraint on sentence representation matrix  $H_{s}$  as
$$
\mathcal {L} _ {\text {c l u s t e r}} = \| H _ {s} \| _ {*}, \tag {13}
$$
where  $\| H_{s}\|_{*}$  is defined as the sum of the  $k$  smallest eigenvalues, i.e.,  $\sum_{i = n - k}^{n}\lambda_{i}$  with  $\lambda_{i}$  as the  $i$ -th smallest eigenvalue (Piao et al., 2019). When  $\| H_{s}\|_{*}$  is set to 0, we obtain  $k$  clusters in  $H_{s}$  by reorganizing its columns or rows and converting it into a block-diagonal form with  $k$  blocks, as shown in Figure 1 (c). We also determine parameter  $k$  by the elbow method (Bholowalia and Kumar, 2014).
3.3.3 Joint learning framework
Past work considered date selection and sentence clustering-based event detection as independent tasks. In HeterTLS, they are jointly trained to combine their advantages into an overall objective:
$$
\mathcal {L} = \mathcal {L} _ {\text {c l a s s i f y}} + \lambda \mathcal {L} _ {\text {c l u s t e r}}, \tag {14}
$$
where  $\mathcal{L}_{\text {classify }}$  minimizes the cross-entropy over all labeled date nodes between the ground-truth during training, and  $\lambda$  serves as a weighted coefficient to balance  $\mathcal{L}_{\text {classify }}$  with  $\mathcal{L}_{\text {cluster }}$ . Eq. 14 can be optimized via stochastic gradient descent (SGD) (Zinkevich et al., 2010) in an end-to-end manner. Readers can also refer to (Piao et al., 2019; Liu and Vandenberghe, 2009) for the detailed nuclear norm optimization strategy of  $\mathcal{L}_{\text {cluster }}$ .
Our date classification is trained in a transductive learning-based semi-supervised manner. We iterate
T17	Crisis	Ent.	Covid.
Topics	9	4	47	1
Timelines	19	22	47	1
Avg.Documents	508	2,310	959	26,376
Avg.Sentences	20,409	82,761	31,545	791,280
Avg.Dates	124	307	600	218
Avg.Duration	212	343	4,437	266
Table 1: Basic dataset statistics. Avg.X demonstrates average X for each topic, and Timelines refers to number of ground-truths in each dataset.
all node representations in the heterogeneous graph simultaneously with  $50\%$  labeled date nodes (40% for training and  $10\%$  for verification) and  $50\%$  unlabeled date nodes as the test set. The joint learning model is able to effectively find event-based candidate clusters (see Figure 1 (c)), thereby save much running time and improve the accuracy of TLS.
3.4 Timeline summary extractor
With  $l$  selected dates and their corresponding representations  $\{\mathbf{h}_{d_1},\dots ,\mathbf{h}_{d_l}\}$ , we represent the  $k$  clusters as  $\{\mathbf{h}_{c_1},\dots ,\mathbf{h}_{c_k}\}$  by averaging sentence representations inside that cluster. As shown in Figure 1 (c), candidate clusters for the  $t$ -th selected date are determined by calculating the cosine similarity between the date representation with all cluster representations as  $\cos (\mathbf{h}_{d_t},\mathbf{h}_{c_j})(j\in \{1,\dots ,k\})$ . If the cosine similarity is larger than the pre-defined threshold  $\delta$  (see Section 5.4), the corresponding cluster is considered a candidate for the date. Finally, we apply CENTROID-OPT (Ghalandari, 2017) as a sentence ranking algorithm within a cluster and summarize each date individually by selecting one sentence per cluster with the highest ranking score.
We carried out our experiments on the four most widely used benchmark datasets, i.e., 17 Timelines (T17) (Tran et al., 2013), Crisis (Tran et al., 2015a), Entities (Ghalandari and Ifrim, 2020), and CovidTLS (Quatra et al., 2021). All contain human-written timelines concerning certain topics, the source news articles of which are retrieved from the web at a given point in time.
Using these datasets makes it possible to comprehensively verify the effectiveness and generalization of HeterTLS because both the number of
Datasets Metrics	T17	Crisis	Entities
CR1-F	CR2-F	AR1-F	AR2-F	CR1-F	CR2-F	AR1-F	AR2-F	CR1-F	CR2-F	AR1-F	AR2-F
Full Oracle	0.500	0.180	0.312	0.128	0.490	0.160	0.360	0.150	0.348	0.079	0.232	0.075
CHIEU (2004)	0.290	0.072	0.067	0.019	0.374	0.070	0.052	0.012	0.275	0.053	0.036	0.011
TRAN (2013)	0.336	0.065	0.094	0.022	0.271	0.034	0.054	0.012	0.275	0.052	0.042	0.012
MARTSCHAT (2018)	0.383	0.092	0.105	0.030	0.333	0.072	0.075	0.016	0.275	0.052	0.042	0.011
DATEWISE (2020)	0.385	0.097	0.121	0.035	0.347	0.075	0.089	0.026	0.271	0.051	0.057	0.017
DASG (2021)	0.333	0.064	0.118	0.029	0.323	0.068	0.077	0.018	0.282	0.052	0.045	0.010
SDF (2021)	0.401	0.101	0.106	0.033	0.360	0.073	0.064	0.014	0.275	0.052	0.041	0.011
HeterTLS-HAN	0.398	0.101	0.141	0.052	0.372	0.070	0.092	0.026	0.272	0.052	0.054	0.015
HeterTLS-Joint	0.392	0.101	0.132	0.042	0.323	0.068	0.079	0.015	0.271	0.048	0.049	0.012
HeterTLS+Pre-trained	0.401	0.103	0.142	0.053	\( \mathbf{0.379}^{\dagger} \)	\( \mathbf{0.078}^{\dagger} \)	\( \mathbf{0.107}^{\dagger} \)	\( \mathbf{0.028}^{\dagger} \)	0.282	0.054	0.057	0.019
HeterTLS	\( \mathbf{0.408}^{\dagger} \)	\( \mathbf{0.108}^{\dagger} \)	\( \mathbf{0.145}^{\dagger} \)	\( \mathbf{0.058}^{\dagger} \)	0.374	0.075	0.105	0.028	\( \mathbf{0.288}^{\dagger} \)	\( \mathbf{0.058}^{\dagger} \)	\( \mathbf{0.059}^{\dagger} \)	\( \mathbf{0.019}^{\dagger} \)
Table 2: Concatenation- and alignment-based ROUGE-1/2 F1-scores for T17, Crisis, and Entities datasets. Best results among model-generated timelines are marked in bold. Symbol  $\dagger$  indicates that our results significantly surpass all baselines using bootstrap test (Dror et al., 2018) with  $p < 0.005$ .