SlowGuess/ACL_2020_OCR huggingface dataset:
Demian Gholipour Ghalandari $^{1,2}$  and Georgiana Ifrim $^{1}$
<sup>1</sup>Insight Centre for Data Analytics, University College Dublin, Ireland
$^{2}$ Aylien Ltd., Dublin, Ireland
{demian.gholipour|georgiana.ifrim}@insight-centre.org
Previous work on automatic news timeline summarization (TLS) leaves an unclear picture about how this task can generally be approached and how well it is currently solved. This is mostly due to the focus on individual subtasks, such as date selection and date summarization, and to the previous lack of appropriate evaluation metrics for the full TLS task. In this paper, we compare different TLS strategies using appropriate evaluation frameworks, and propose a simple and effective combination of methods that improves over the state-of-the-art on all tested benchmarks. For a more robust evaluation, we also present a new TLS dataset, which is larger and spans longer time periods than previous datasets. The dataset will be made available at https://github.com/complementizer/news-tls.
Timelines of news events can be useful to condense long-ranging news topics and can help us understand how current major events follow from prior events. Timeline summarization (TLS) aims to automatically create such timelines, i.e., temporally ordered time-stamped textual summaries of events focused on a particular topic. While TLS has been studied before, most works treat it as a combination of two individual subtasks, 1) date selection and 2) date summarization, and only focus on one of these at a time (Tran et al., 2013a,b, 2015b). However, these subtasks are almost never evaluated in combination, which leaves an unclear picture of how well TLS is being solved in general. Furthermore, previously used evaluation metrics for the date selection and timeline summarization tasks are not appropriate since they do not consider the temporal alignment in the evaluation. Just until recently, there were no established experimental settings and appropriate metrics for the full TLS task (Martschat and Markert, 2017, 2018).
Date	Summary
2001-11-29	Enron could cost Dutch group $195m
2001-11-30	1,100 UK jobs go in Enron collapse
2001-12-02	Barclays: Enron bankruptcy will not affect business
2002-01-15	As Enron scandal spreads, US starts to question cash for influence culture
[...]
2004-07-08	Jury indicts Lay for inflating Enron earnings
2006-05-25	Former Enron bosses found guilty
2006-07-05	Enron founder Lay dies
2008-02-22	US prison beckons British bankers who got cosy with Enron
Table 1: Excerpt of an automatically constructed timeline about the company Enron, using article headlines as summaries. The shaded parts indicate that the date or summary matches entries in a corresponding human-written ground-truth timeline.
In this paper, we examine existing strategies for the full TLS task and how well they actually work. We identify three high-level approaches: 1) Direct summarization treats TLS like text summarization, e.g., by selecting a small subset of sentences from a massive collection of news articles; 2) The date-wise approach first selects salient dates and then summarizes each date; 3) Event detection first detects events, e.g., via clustering, selects salient events and summarizes these individually. The current state-of-the-art method is based on direct summarization (Martschat and Markert, 2018). We therefore focus on testing the two remaining strategies, which have not been appropriately evaluated yet and allow for better scalability.
We propose a simple method to improve date summarization for the date-wise approach. The method uses temporal expressions (textual references to dates) to derive date vectors, which in turn help to filter candidate sentences to summarize particular dates. With this modification, the date-wise approach obtains improved state-of-the
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1322-1334 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
art results on all tested datasets. We also propose an event-based approach via clustering, which outperforms (Martschat and Markert, 2018) on one of three tested datasets. We use purpose-build evaluation metrics for evaluating timelines introduced by Martschat and Markert (2017). For a more robust evaluation, we also present a new dataset for TLS, which is significantly larger than previous datasets in terms of the number of individual topics and time span.
1. We compare different TLS strategies side-by-side using suitable evaluation metrics to provide a better picture for how well the full TLS task for news is solved so far.
2. We propose a simple addition to existing methods to significantly improve date-wise TLS, achieving new state-of-the-art results.
3. We present a new TLS dataset that is larger than previous datasets and spans longer time ranges (decades of news timelines).
Timeline summarization for news articles has received some attention in the last two decades (Swan and Allan, 2000; Allan et al., 2001; Chieu and Lee, 2004; Yan et al., 2011a,b; Kessler et al., 2012; Tran et al., 2013a,b; Li and Li, 2013; Tran et al., 2015a,b; Wang et al., 2015, 2016; Martschat and Markert, 2017, 2018; Steen and Markert, 2019). The task is commonly split into date selection and date summarization subtasks.
2.1 Date Selection
Supervised machine learning has been proposed to predict whether dates appear in ground-truth timelines (Kessler et al., 2012; Tran et al., 2013a). Tran et al. (2015b) use graph-based ranking of dates, which is reported to outperform supervised methods<sup>1</sup>.
2.2 Date Summarization
Several approaches construct date summaries by picking sentences from ranked lists. The ranking is based on regression or learning-to-rank to predict ROUGE scores between the sentence and a ground-truth summary (Tran et al., 2013a,b). Tran
et al. (2015a) observe that users prefer summaries consisting of headlines to summaries consisting of sentences from article bodies. Steen and Markert (2019) propose abstractive date summarization based on graph-based sentence merging and compression. Other works propose the use of additional data, such as comments on social media (Wang et al., 2015), or images (Wang et al., 2016).
2.3 Full Timeline Summarization
Chieu and Lee (2004) produce timelines by ranking sentences from an entire document collection. The ranking is based on summed up similarities to other sentences in an  $n$ -day window. Nguyen et al. (2014) propose a pipeline to generate timelines consisting of date selection, sentence clustering, and ranking. Martschat and Markert (2018) adapt submodular function optimization, commonly used for multi-document summarization, for the TLS task. The approach searches for a combination of sentences from a whole document collection to construct a timeline and is the current state-of-the-art for full TLS. Steen and Markert (2019) use a two-stage approach consisting of date selection and date summarization to build timelines. Other examples of automatic timeline generation can be found in the social media-related literature, where microblogs are often clustered before being summarized (Wang et al., 2014; Li and Cardie, 2014). We explore a similar framework for evaluating clustering-based TLS.
3 Strategies for Timeline Summarization
We define the TLS setup and task as follows. Given is a set of news articles  $A$ , a set of query keyphrases  $Q$ , and a ground-truth (reference) timeline  $r$ , with  $l$  dates that are associated with  $k$  sentences on average, i.e.,  $m = k * l$  sentences in total. The task is to construct a (system) timeline  $s$  that contains  $m$  sentences, assigned to an arbitrary number of dates. A simpler and stricter setting can also be used, in which  $s$  must contain exactly  $l$  dates with  $k$  sentences each.
Approach Types
A number of different high-level approaches can be used to tackle this task:
1. Direct Summarization:  $A$  is treated as one set of sentences, from which a timeline is directly extracted, e.g., by optimizing a sentence
<sup>1</sup>Despite our best efforts, we could neither obtain code for this method from the authors nor reproduce its reported performance, and therefore did not include it in our experiments.
Figure 1: Counts of published articles and textual mentions across dates in an article collection about Enron.
combination (Martschat and Markert, 2018), or by sentence ranking (Chieu and Lee, 2004). Among these, Matschat and Markert (2018)'s solution for the full TLS task has state-of-the-art accuracy but does not scale well.
2. Date-wise Approach: This approach selects  $l$  dates and then constructs a text summary of  $k$  sentences on average for each date.
3. Event Detection: This approach first detects events in  $A$ , e.g., by clustering similar articles, and then identifies the  $l$  most important events and summarizes these separately.
Since no prior work has analyzed the latter two categories for the full TLS task, we discuss and develop such approaches next.
3.1 Date-wise Approach
The approach described here mostly consists of existing building blocks, with a few but important modifications proposed from our side.
Defining the Set of Dates
First, we identify the set of possible dates to include in a timeline. We obtain these from (i) the publication dates of all articles in  $A$  and (ii) textual references of dates in sentences in  $A$ , such as 'last Monday', or '12 April'. We use the tool HeidelTime $^{2}$  (Strömgen and Gertz, 2013) to detect and resolve textual mentions of dates.
Date Selection
Next, we select the  $l$  most important dates. We compare the following date selection methods introduced by Tran et al. (2013a):
- PUBCOUNT: Ranking dates by the number of articles published on a date.
- MENTIONCOUNT: Ranking dates by the number of sentences that mention the date.
- SUPERVISED: Extracting date features and using classification or regression to predict whether a date appears in a ground-truth timeline. These features mostly include the publication count and different variants of counting date mentions.
Our experiments show that SUPERVISED works best, closely followed by MENTIONCOUNT (Appendix A.1). Figure 1 shows an example of publication and date mention counts and ground-truth dates over time. Two challenges are evident that date selection methods face: 1) These count signals usually do not perfectly correlate with ground-truth dates, and 2) high values often cluster around important dates, i.e., a "correct" date is often surrounded by other, "incorrect" dates with similarly strong signals.
Candidate Sentences for Dates
To summarize a particular date  $d$ , we first need to decide which articles or sentences we use as a source to create a summary from. Previous research has not explored this aspect much due to the separated treatment of subtasks. We propose a simple but effective heuristic to do this. We consider the following two sets to be the primary source of suitable candidate sentences:
-  $P_{d}$ : Sentences published on or closely after  $d$ . These often contain initial reports of events occurring on  $d$ .
-  $M_d$ : Sentences that mention  $d$ . These sentences are from articles published at any point in time, and may retrospectively refer to  $d$ , or announce events on  $d$  beforehand<sup>3</sup>.
We evaluate these two options in our experiments, and propose an heuristic that combines these, which we call PM-MEAN. We aim to
In practice, we include the first 5 sentences in the body of each article published on  $d$  and up to 2 days after  $d$  into  $P_{d}$ . We include all sentences found in  $A$  that mention  $d$  into  $M_{d}$ .
$^{2}$ https://github.com/HeidelTime/heideltime
find a subset of sentences in  $P_d \cup M_d$  that are likely to mention important events happening on  $d$ . We convert all the sentences in the collection  $A$  to sparse bag-of-words (unigram) vectors with sentence-level TF-IDF weighting. We represent the sets of sentences  $P_d$  and  $M_d$  using the mean of their respective sentence vectors,  $\overline{x}_{P_d}$  and  $\overline{x}_{M_d}$ . The core assumption of the method is that the content shared between  $P_d$  and  $M_d$  is a good source for summarizing events on  $d$ . To capture this content, we build a date vector  $x_d$ , so that we can compare sentence vectors against it to rank sentences. We set the value of  $x_d$  for each dimension  $i$  in the feature space as follows:
$$
x _ {d} ^ {i} = \left\{ \begin{array}{l l} \frac {1}{| P _ {d} |} \bar {x} _ {P _ {d}} ^ {i} + \frac {1}{| M _ {d} |} \bar {x} _ {M _ {d}} ^ {i} & \text {i f} \bar {x} _ {P _ {d}} ^ {i} > 0 \text {a n d} \bar {x} _ {M _ {d}} ^ {i} > 0 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {1}
$$
Thus the date vector  $x_{d}$  is an average of  $\overline{x}_{P_d}$  and  $\overline{x}_{M_d}$  weighted by the sizes of  $P_{d}$  and  $M_{d}$ , with any features zeroed out if they are missing in either  $P_{d}$  or  $M_{d}$ . To rank sentences, we compute the cosine similarity between the vector  $x_{s}$  of each candidate sentence  $s\in (P_d\cup M_d)$  to  $x_{d}$ . We select the best-scoring candidate sentences by defining a threshold on this similarity. To avoid tuning this threshold, we use a simple knee point detection method (Satopaa et al., 2011) to dynamically identify a threshold that represents the "knee" (or elbow) in the similarity distribution. This set of best-scoring sentences is then used as the input for the final date summarization step.
Date Summaries
To construct the final timeline, we separately construct a summary for the  $l$  highest ranked dates. Prior to our main experiments, we test several multi-document summarization algorithms:
- TEXTRANK: Runs PageRank on a graph of pairwise sentences similarities to rank sentences (Mihalcea and Tarau, 2004).
- CENTROID-RANK: Ranks sentences by their similarity to the centroid of all sentences (Radev et al., 2004).
- CENTROID-OPT: Greedily optimises a summary to be similar to the centroid of all sentences (Ghalandari, 2017).
- SUBMODULAR: Greedily optimizes a summary using submodular objective functions that represent coverage and diversity (Lin and Bilmes, 2011).
The only modification to these algorithms in our TLS pipeline is that we prevent sentences not containing any topic keyphrases from query  $Q$  to be included in the summary. CENTROID-OPT has the best results (Appendix A.1) and is used in the main experiments.
Timeline Construction
The date-wise approach constructs a timeline as follows: first, rank all potential dates using one of the date selection approaches described, then pick the  $l$  highest ranked ones, pick candidate sentences for each date, and summarize each date individually from the according candidate set, using  $k$  sentences. We might not be able to summarize a particular date due to the keyword constraint in the summarization step. Whenever this is the case, we skip to the next date in the ranked list, until  $l$  is reached.
3.2 Event Detection Approach
When humans are tasked with constructing a timeline, we expect that they reason over important events rather than dates. Conceptually, detecting and selecting events might also be more appropriate than selecting dates because multiple events can happen on the same day, and an event can potentially span multiple days.
To explore this, we test a TLS approach based on event detection by means of article clustering. The general approach can be summarized as follows: (1) Group articles into clusters; (2) Rank and select the  $l$  most important clusters; (3) Construct a summary for each cluster. Similarly to the date-wise approach, this mostly consists of existing building blocks that we adapt for TLS.
For each input collection  $A$ , we compute sparse TF-IDF unigram bag-of-words vectors for all articles in  $A$ . We apply clustering algorithms to these vectors. To cluster articles, we use Markov Clustering (MCL) with a temporal constraint. MCL (Van Dongen, 2000) is a clustering algorithm for graphs, i.e., a community detection algorithm. It is based on simulating random walks along nodes in a graph. Ribeiro et al. (2017) use this approach for clustering news articles.
We convert  $A$  into a graph where nodes correspond to articles so that we can cluster the articles using MCL, with the following temporal constraint: Articles  $a_1$ ,  $a_2$  are assigned an edge if their publication dates are at most 1 day apart from each other.
The edge weight is set to the cosine similarity between the TF-IDF bag-of-words vectors of  $a_1$  and  $a_2$ . The constraint on the publication dates ensures that clusters do not have temporal gaps. Furthermore, it reduces the number of similarity computations between pairs of articles considerably. We run MCL on this graph and obtain clusters by identifying the connected components in the resulting connectivity matrix<sup>4</sup>.
Assigning Dates to Clusters
We define the cluster date as the date that is most frequently mentioned within articles of the cluster. We identify date mentions using the HeidelTime tool.
Cluster Ranking
To construct a timeline, we only need the  $l$  most important clusters. We obtain these by ranking and retaining the top- $l$  clusters of the ranked list. We test the following scores to rank clusters by:
SIZE: Rank by the numbers of articles in a cluster.
- DATEMENTIONCOUNT: Rank by how often the cluster date is mentioned throughout the input collection.
- REGRESSION: Rank using a score by a regression model trained to predict importance scores of clusters.
For the regression-based ranking method, we represent clusters using the following features: number of articles in a cluster; number of days between the publication dates of the first and last article in the cluster; maximum count of publication dates of articles within a cluster; maximum mention count of dates mentioned in articles in a cluster; sum of mention counts of dates mentioned in articles in a cluster. We test two approaches to label clusters with target scores to predict.
- Date-Accuracy: This is 1 if the cluster date appears in the ground-truth, else 0.
- ROUGE: The ROUGE-1 F1-score<sup>5</sup> between the summary of the cluster and the ground-truth summary of the cluster date. If the cluster date does not appear in the ground-truth, the score is set to 0.
We evaluate these different options (Appendix A.2) and observe that ranking by DATEMENTION-COUNT works better than the supervised methods, showing that predicting the suitability of clusters for timelines is difficult.
Cluster Summarization
We use the same multi-document summarization method that works best for the date-wise approach (CENTROID-OPT).
In summary, the clustering approach builds a timeline as follows: 1) cluster all articles, 2) rank clusters, 3) build a summary with  $k$  sentences for the top- $l$  clusters, skipping clusters if a summary cannot be constructed due to missing keywords. Furthermore, we skip clusters if the date assigned to the cluster is already "used" by a previously picked cluster. Conceptually, this implies that we can only recognize one event per day. In initial experiments, this leads to better results than alternatives, e.g., allowing multiple summaries of length  $k$  per day.
Tran et al. introduced the 17 Timelines (T17) (Tran et al., 2013a) and the CRISIS (Tran et al., 2015a) datasets for timeline summarization from news articles. However, we see the need for better benchmarks due to 1) a small number of topics in the T17 and CRISIS datasets (9 and 4 topics respectively), and 2) relatively short time span, ranging from a few months to 2 years.
Therefore, we build a new TLS dataset, called ENTITIES, that contains more topics (47) and longer time-ranges per topic, e.g., decades of news articles. In the following, we describe how we obtain ground-truth timelines and input article collections for this dataset.
Ground-Truth Timelines: We obtain ground-truth timelines from CNN Fast Facts<sup>6</sup>, which has a collection of several hundred timelines grouped in categories, e.g., 'people' or 'disasters'. We pick all timelines of the 'people' category and a small number from other categories.
Queries: For each ground-truth timeline, we define a set of query keyphrases  $Q$ . By default, we use the original title of the timeline as the keyphrase. For people entities, we use the last token of the title to capture surnames only, which increases the
<sup>4</sup>We use the implementation and default parameters from https://github.com/GuyAllard/markov_clustering
$^{5}$ ROUGE-1 obtained a better overall performance than ROUGE-2 for this purpose.
http://edition.cnn.com/specials/world/fast-facts
coverage. We manually inspect the resulting sets of keyphrases and correct these if necessary.
Input Articles: For each entity from the ground-truth timelines, we search for news articles using TheGuardian API<sup>7</sup>. We use this source because it provides access to all published articles starting from 1999. We search for articles that have exact matches of the queries in the article body. The timespan for the article search is set so that it extends the ground-truth timeline by  $10\%$  of its days before its first and after its last date.
Adjustments and Filtering: The ground-truth timelines are modified to be usable for TLS and to ensure they do not contain data not present in the document collection:
- We remove entries in the ground-truth timelines if they do not specify year, month, and day of an event.
- Ground-truth timelines are truncated to the first and last date of the input articles.
- Entries in the ground-truth timeline are removed if there is no input article published within  $\pm 2$  days.
Afterwards, we remove all topics from the dataset that do not fulfill the following criteria:
- The timeline must have at least 5 entries.
- For at least  $50\%$  of the dates present in the ground-truth timeline, textual references have to be found in the article collection (e.g., 'on Wednesday' or 'on 1 August'). This is done to ensure that the content of the timelines is reflected to some degree in the article collection.
- There are at least 100 and less than 3000 articles containing the timeline-entity in the input articles. This is done to reduce the running time of experiments.
Dataset Characteristics: Tables 2 and 3 give an overview of properties of the two existing datasets and our new dataset, and mostly show averaged values over tasks in a dataset. An individual task corresponds to one ground-truth timeline that a TLS algorithm aims to simulate. #PubDates refers to the number of days in an article collection  $A$  on which any articles are published. The compression ratio w.r.t. sentences ("comp. ratio (sents)") is  $m$  divided by the total number of sentences in  $A$ ,
and the compression ratio w.r.t dates is  $l$  divided by #PubDates. "Avg. date cov" refers to the average coverage of dates in the ground-truth timeline  $r$  by the articles in  $A$ . This can be counted by using publication dates in  $A$ , ("published"), or by textual date references to dates within articles in  $A$  ("mentioned"). The fact that there are generally more ground-truth dates covered in textual date references compared to publication dates suggests making use of these date mentions.
T17 has longer  $(l)$ , and more detailed  $(k)$  timelines than the other datasets, CRISIS has more articles per task, and ENTITIES has more topics, publication dates and longer time periods per task.
In our experiments, we measure the quality of generated timelines with the following two evaluation metrics, which are also used by Martschat and Markert (2018):
- Alignment-based ROUGE F1-score: This metric compares the textual overlap between a system and a ground-truth timeline, while also considering the assignments of dates to texts.
- Date F1-score: This metric compares only the dates of a system and a ground-truth timeline.
We denote the alignment-based ROUGE-1 F1-score as AR1-F and Date F1-score as Date-F1.
5.2 Experimental Settings
Concerning the datasets and task, we follow the experimental settings of Martschat and Markert (2018):