SlowGuess/ACL_2020_OCR huggingface dataset:
- Each dataset is divided into multiple topics, each having at least one ground-truth timeline. If a topic has multiple ground-truth timelines, we split the topic into multiple tasks. The final results in the evaluation are based on averages over tasks/ground-truth timelines, not over topics.
Each task includes a set of news articles  $A$ , a set of keyphrases  $Q$ , a ground-truth timeline  $r$ , with number of dates (length)  $l$ , average number of summary sentences per date  $k$ , and total number of summary sentences  $m = l * k$ .
<sup>7</sup>http://open-platform.theguardian.com/
Table 2: Dataset Statistics for the TLS task (i)
Dataset	#Topics	#TLs	Avg. #Docs	Avg. #Sents	Avg. #PubDates	Avg. Duration (in days)
T17	9	19	508	20409	124	212
CRISIS	4	22	2310	82761	307	343
ENTITIES	47	47	959	31545	600	4437 (≈ 12 years)
Table 3: Dataset Statistics for the TLS task (ii)
Dataset	Avg. l	Avg. k	Avg. m	Avg. comp. ratio (sents)	Avg. comp. ratio (dates)	Avg. date cov. (published)	Avg. date cov. (mentioned)
T17	36	2.9	108	0.0117	0.43	81%	93%
CRISIS	29	1.3	38	0.0005	0.11	90%	96%
ENTITIES	23	1.2	26	0.0017	0.06	51%	65%
- In each task, we remove all articles from  $A$  whose publication dates are outside of the range of dates of the ground-truth timeline  $r$  of the task. Article headlines are not used.
- We run leave-one-out cross-validation over all tasks of a dataset.
- We test for significant differences using an approximate randomization test (Marcus et al., 1993) with a p-value of 0.05.
We use the following configurations for our methods:
- A stricter and simpler version of the output size constraint: We produce timelines with the number of dates  $l$  and  $k$  sentences per date.
- In the summarization step of our methods, we only allow a sentence to be part of a summary if it contains any keyphrase in  $Q$ . As opposed to Martschat and Markert (2018), we still keep sentences not matching  $Q$ , e.g., for TF-IDF computation, clustering, and computing date vectors.
5.3 Methods Evaluated
We compare the following types of methods to address the full news TLS task.
Direct summarization approaches:
- CHIEU2004: Chieu and Lee (2004) An unsupervised baseline based on direct summarization. We use the reimplementation from Martschat and Markert (2018).
- MARTSCHAT2018: Martschat and Markert (2018) State-of-the-art method on the CRISIS and T17 datasets. It greedily selects a combination of sentences from the entire collection
A maximizing submodular functions for content coverage, textual and temporal diversity, and a high count of date references<sup>8</sup>.
Date-wise approaches:
- TRAN 2013 (Tran et al., 2013a): The original date-wise approach, using regression for both date selection and summarization, and using all sentences of a date as candidate sentences.
- PUBCOUNT: A simple date-wise baseline that uses the publication count to rank dates, and all sentences published on a date for candidate selection. We use CENTROID-OPT for summarization.
- DATEWISE: Our date-wise approach after testing different building blocks (see Appendix A.1). It uses supervised date selection, PM-MEAN for candidate selection and CENTROID-OPT for summarization.
Event detection approach based on clustering:
- CLUST: We use DATEMENTIONCOUNT to rank clusters, and CENTROID-OPT for summarization, which are the best options according to our tests (see Appendix A.2).
Note that all methods apart from DATEWISE and CLUST have been proposed previously.
Oracles:
To interpret the alignment-based ROUGE scores better and to approximate their upper bounds, we measure the performance of three different oracle methods:
Multiple variants of this approach were introduced in the paper. We picked the variant called "AsMDS  $+f_{\mathrm{TempDiv}} + f_{\mathrm{DateRef}}$  due to its good results.
- DATE ORACLE: Selects the correct (ground-truth) dates and uses CENTROID-OPT for date summarization.
- TEXT ORACLE: Uses regression to select dates, and then constructs a summary for each date by optimizing the ROUGE to the ground-truth summaries.
- FULL ORACLE: Selects the correct dates and constructs a summary for each date by optimizing the ROUGE to the ground-truth summaries.
We give more detail about these in Appendix A.3.
Table 4 shows the final evaluation results. We reproduced the results of CHIEU2004 and MARTSCHAT2018 reported by Martschat and Markert (2018) using their provided code<sup>9</sup>. The other results are based on our implementations. Table 10 in Appendix A.6 shows several output examples across different methods.
6 Analysis and Discussion
6.1 Performance of TLS Strategies
Among the methods evaluated, DATEWISE consistently outperforms all other methods on all tested datasets in the alignment-based ROUGE metrics. The Date-F1 metric for this method is close to other methods, and not always better, which shows that the advantage of DATEWISE is due to the sentence selection (based on our heuristic date vectors) and summarization. Note that the date selection method is identical to TRAN2013. We conclude from these results that the expensive combinatorial optimization used in MARTSCHAT2018 is not necessary to achieve high accuracy for news TLS.
CLUST performs worse than DATEWISE and MARTSCHAT2018, except on ENTITIES, where it outperforms MARTSCHAT2018. We find that for the other two datasets, CLUST often merges articles from close dates together that would belong to separate events on ground-truth timelines, which may suggest that a different granularity of clusters is required depending on the task.
DATE ORACLE and FULL ORACLE should theoretically have a  $100\%$  Date-F1. In practice, their Date-F1 scores turn out lower because, for some dates, no candidate sentences that match query  $Q$
can be found, which causes the dates to be omitted from the oracle timelines.
Based on the performance of different systems, the hardest dataset is ENTITIES, followed by CRISIS.
6.2 What makes TLS difficult?
While the ranking of methods is fairly stable, the performance of all methods varies a lot across the datasets and across individual tasks within datasets. To find out what makes individual tasks difficult, we measure the Spearman correlation between AR1-F and several dataset statistics. The details are included in Appendix A.5. The correlations show that a high number of articles and publication dates and a low compression ratio w.r.t to dates generally decreases performance. This implies that highly popular topics are harder to summarize. The duration of a topic also corresponds to lower performance, but in a less consistent pattern.
The generally low performance across tasks and methods is likely influenced by the following factors:
- The decision for human editors to include particular events in a timeline and to summarise these in a particular way can be highly subjective. Due to the two-stage nature of TLS, this problem is amplified in comparison to regular text summarization.
- Article collections can be insufficient to cover every important event of a topic, e.g., due to the specific set of news sources or the search technique used.
6.3 Running Time
DATEWISE and CLUST are up to an order of magnitude faster to run than MARTSCHAT2018 (Appendix A.4) since their date summarization steps only involve a small subset of sentences in an article collection.
6.4 Adjacent Dates and Redundancy
Automatically constructed timelines often contain a high amount of multiple adjacent dates, while this is not the case in ground-truth timelines. Summaries of such adjacent dates often tend to refer to the same event and introduce redundancy into a timeline. To quantify this, we count the proportion of those "date bigrams" in a chronologically ordered timeline, which are only 1 day apart. The results (see Table 5) show that this is an issue
<sup>9</sup>With the exception of CRISIS due to memory issues.
T17 Dataset	CRISIS Dataset	ENTITIES Dataset
AR1-F	AR2-F	Date-F1	AR1-F	AR2-F	Date-F1	AR1-F	AR2-F	Date-F1
Text Oracle	0.198	0.073	0.541	0.136	0.052	0.297	0.069	0.023	0.20
Date Oracle	0.179	0.057	0.926	0.202	0.063	0.974	0.17	0.047	0.757
Full Oracle	0.312	0.128	0.926	0.367	0.15	0.974	0.232	0.075	0.757
CHIEU2004	0.066	0.019	0.251	0.052	0.012	0.142	0.036	0.01	0.102
MARTSCHAT2018	0.105	0.03	0.544 •	0.075 •	0.016	0.281	0.042	0.009	0.167
TRAN2013	0.094	0.022	0.517 •	0.054	0.011	0.289	0.042	0.012	0.184 †
PUBCOUNT	0.105	0.027	0.481	0.067	0.012	0.233	0.033	0.009	0.107
DATEWISE	0.12 • †	0.035 •	0.544 •	0.089 •	0.026 •	0.295 •	0.057 • †	0.017 • †	0.205 • †
CLUST	0.082	0.020	0.407	0.061	0.013	0.226	0.051 †	0.015 †	0.174
DATEWISE (titles)	-	-	-	0.072	0.016	0.287	0.057	0.017	0.194
Table 4: Results on the full TLS task.  $\star$  indicates a significant improvement over Tran 2013,  $\bullet$  over CLUST, and † over MARTSCHAT2018. DATEWISE (titles) is not included in the significance testing.
Ground-truth	MARTSCHAT2018	DATEWISE	CLUST
T17	0.45	0.63	0.62	0.25
CRISIS	0.18	-	0.52	0.06
ENTITIES	0.03	0.18	0.3	0.05
Table 5: Proportion of adjacent dates of timelines produced by different methods, and the ground-truth timelines.
for MARTSCHAT2018 and DATEWISE, but less so for CLUST, which is designed to avoid this behavior. Note that MARTSCHAT2018 includes an objective function to reward diversity within a timeline, while DATEWISE has no explicit mechanism against redundancy among separate dates. Interestingly, when forcing DATEWISE to avoid selecting adjacent dates (by skipping such dates in the ranked list), the performance in all metrics decreases. In this case, high redundancy is a safer strategy for optimizing TLS metrics compared to enforcing a more balanced spread over time. Because of such effects, we advise to use automated evaluation metrics for TLS with care and to conduct qualitative analysis and user studies where possible.
6.5 Use of Titles
While using article titles can make timelines more readable and understandable (Tran et al., 2015a), we do not involve titles in our main experiments, in order to directly compare to MARTSCHAT2018, and due to the lack of titles in T17. The last row in Table 4 shows the results of a separate experiment with DATEWISE in which we build date summaries using titles only. Using only titles generally increases AR Precision at the cost of Recall. AR-F is negatively affected in CRISIS but does not change in ENTITIES. Figure 1 shows parts of a title-based timeline produced by DATEWISE.
In this study, we have compared and proposed different strategies to construct timeline summaries of long-ranging news topics: the previous state-of-the-art method based on direct summarization, a date-wise approach, and a clustering-based approach. By exploiting temporal expressions, we have improved the date-wise approach and yielded new state-of-the-art results on all tested datasets. Hence, we showed that an expensive combinatorial search over all sentences in a document collection is not necessary to achieve good results for news TLS. For a more robust and diverse evaluation, we have constructed a new TLS dataset with a much larger number of topics and with longer time-spans than in previous datasets. Most of the generated timelines are still far from oracle timeline extractors and leave large gaps for improvements. Potential future directions include a more principled use of our proposed heuristic for detecting content relevant to specific dates, the use of abstractive techniques, a more effective treatment of the redundancy challenge, and extending the new dataset with multiple sources.
James Allan, Rahul Gupta, and Vikas Khandelwal. 2001. Temporal summaries of new topics. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in in
formation retrieval. ACM, 10-18.
Hai Leong Chieu and Yoong Keok Lee. 2004. Query based event extraction along a timeline. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 425-432.
Demian Gholipour Ghalandari. 2017. Revisiting the Centroid-based Method: A Strong Baseline for Multi-Document Summarization. In Proceedings of the Workshop on New Frontiers in Summarization. 85–90.
Remy Kessler, Xavier Tannier, Caroline Hagege, Véronique Moriceau, and André Bittar. 2012. Finding salient dates for building thematic timelines. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers Volume 1. Association for Computational Linguistics, 730-739.
Jiwei Li and Claire Cardie. 2014. Timeline generation: Tracking individuals on twitter. In Proceedings of the 23rd international conference on World wide web. ACM, 643-652.
Jiwei Li and Sujian Li. 2013. Evolutionary hierarchical dirichlet process for timeline summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 556-560.
Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 510-520.
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics 19, 2 (1993), 313-330.
Sebastian Martschat and Katja Markert. 2017. Improving rouge for timeline summarization. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Vol. 2. 285-290.
Sebastian Martschat and Katja Markert. 2018. A Temporally Sensitive Submodularity Framework for Timeline Summarization. In Proceedings of the 22nd Conference on Computational Natural Language Learning. 230-240.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. 404-411.
Kiem-Hieu Nguyen, Xavier Tannier, and Véronique Moriceau. 2014. Ranking multidocument event descriptions for building thematic timelines. In Proceedings of COLING 2014, the 25th International
Conference on Computational Linguistics: Technical Papers. 1208-1217.
Dragomir R Radev, Hongyan Jing, Małgorzata Stys, and Daniel Tam. 2004. Centroid-based summarization of multiple documents. Information Processing & Management 40, 6 (2004), 919-938.
Swen Ribeiro, Olivier Ferret, and Xavier Tannier. 2017. Unsupervised Event Clustering and Aggregation from Newswire and Web Articles. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism. 62-67.
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a" needle" in a haystack: Detecting knee points in system behavior. In 2011 31st International Conference on Distributed Computing Systems Workshops. IEEE, 166-171.
Julius Steen and Katja Markert. 2019. Abstractive Timeline Summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. 21-31.
Jannik Strötgen and Michael Gertz. 2013. Multilingual and Cross-domain Temporal Tagging. Language Resources and Evaluation 47, 2 (2013), 269-298. https://doi.org/10.1007/s10579-012-9179-y
Russell Swan and James Allan. 2000. Automatic generation of overview timelines. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 49-56.
Giang Tran, Mohammad Alrifai, and Eelco Herder. 2015a. Timeline summarization from relevant headlines. In European Conference on Information Retrieval. Springer, 245-256.
Giang Tran, Eelco Herder, and Katja Markert. 2015b. Joint graphical models for date selection in timeline summarization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Vol. 1. 1598-1607.
Giang Bihn Tran, Mohammad Alrifai, and Dat Quoc Nguyen. 2013a. Predicting relevant news events for timeline summaries. In Proceedings of the 22nd International Conference on World Wide Web. ACM, 91-92.
Giang Binh Tran, Tuan A Tran, Nam-Khanh Tran, Mohammad Alrifai, and Nattiya Kanhabua. 2013b. Leveraging learning to rank in an optimization framework for timeline summarization. In SIGIR 2013 Workshop on Time-aware Information Access (TAIA).
Stijn Marinus Van Dongen. 2000. Graph clustering by flow simulation. Ph.D. Dissertation.
Lu Wang, Claire Cardie, and Galen Marchetti. 2015. Socially-Informed Timeline Generation for Complex Events. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
William Yang Wang, Yashar Mehdad, Dragomir R Radev, and Amanda Stent. 2016. A Low-Rank Approximation Approach to Learning Joint Embeddings of News Stories and Images for Timeline Summarization.. In HLT-NAACL. 58-68.
Zhenhua Wang, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2014. On summarization and timeline generation for evolutionary tweet streams. IEEE Transactions on Knowledge and Data Engineering 27, 5 (2014), 1301-1315.
Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a. Timeline generation through evolutionary trans-temporal summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 433-443.
Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011b. Evolutionary timeline summarization: a balanced optimization framework via iterative substitution. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. ACM, 745-754.
A.1 Testing Variants of DATEWISE
Table 6 shows results for different variants of the date-wise approach.
Date Selection: While testing different date selection methods, we use PM-MEAN for candidate selection and CENTROID-OPT for summarization. The supervised date selection methods work best, closely followed by MENTIONCOUNT.
Candidate Sentence Selection: We compare different strategies of defining the set of sentences associated with a date prior to summarization. The results show that the PM-method can improve the performance, especially for the Crisis dataset.
Date Summarization: Finally, we test different unsupervised text summarization algorithms to summarize each selected date. CENTROID-OPT works best and is used in our main experiments.
A.2 Testing Variants of CLUST
For the clustering-based TLS approach, we only test different options for ranking clusters. For the summarization step, we use CENTROID-OPT, which works best for the date-wise approach. Table
Date Selector	T17	Crisis	Entities
PubCount	0.49	0.243	0.135
MentCount	0.528	0.295	0.159
Tran 2013 (Reg)	0.535	0.297	0.191
Tran 2013 (Clf)	0.541	0.295	0.172
Candidate Selector	T17	Crisis	Entities
Sents mentioning d	0.11	0.077	0.041
Sents published on d to d + 2 (first 5)	0.112	0.078	0.045
Sents published on d to d + 2 (all)	0.113	0.079	0.041
PM-MEAN	0.118	0.089	0.047
Summarizer	T17	Crisis	Entities
TEXTRANK	0.113	0.086	0.046
CENTROID-RANK	0.112	0.085	0.046
CENTROID-OPT	0.118	0.089	0.047
SUBMODULAR	0.116	0.088	0.047
7 shows somewhat inconsistent results, but overall DATEMENTIONCOUNT obtains the best performance in terms of alignment-based ROUGE.
Table 6: Variants of Date-wise TLS and their alignment-based ROUGE-1 score.
Cluster Ranking (AR1-F)	T17	Crisis	Entities
SIZE	0.08	0.06	0.048
DATEMENTIONCOUNT	0.081	0.061	0.051
REGRESSION (DATES)	0.08	0.055	0.045
REGRESSION (ROUGE)	0.082	0.055	0.048
Cluster Ranking (Date-F1)	T17	Crisis	Entities
SIZE	0.41	0.22	0.16
DATEMENTIONCOUNT	0.41	0.23	0.15
REGRESSION (DATES)	0.46	0.23	0.16
REGRESSION (ROUGE)	0.44	0.24	0.16
Table 7: Variants of Date Clustering-based TLS.
A.3 Oracles
For the text and full oracles, we use Algorithm 1 for constructing a summary for a date, using ROUGE-1 F1-score as the objective. We include all sentences that mention  $d$ , as well as the first 5 sentences of all articles published between  $d$  and  $d + 5$  days, as candidate sentences for the oracles to summarize dates.
Algorithm 1: Greedy summarization oracle.
1 Input: Candidate sentences  $C$ , reference summary  $R$ , summary length  $k$
2 Output: Summary sentences  $S$
3  $S\gets \{\}$
4 while  $|S| < k$  and  $|C| > 0$  do
5  $s^{*}\gets \arg \max_{s\in C}ROUGE(S,R)$
6  $S\gets S\cup \{s^{*}\}$
7  $\begin{array}{rl}\big{\downarrow} & C\leftarrow C\setminus \{s^{*}\} \end{array}$
8 Return  $S$
Dataset	Method	1	k	#articles	#dates	Comp. ratio (sents)	Comp. ratio (dates)	duration
T17	MARTSCHAT 2018	-0.116	0.381	-0.421	-0.586 *	0.298	0.57 *	-0.376
T17	DATEWISE	-0.196	0.616 *	-0.354	-0.714 *	0.319	0.638 *	-0.577 *
T17	CLUST	0.283	0.429	-0.247	-0.411	0.504 *	0.576 *	-0.197
Crisis	DATEWISE	0.19	-0.3	0.393	0.096	-0.271	0.147	0.028
Crisis	CLUST	-0.087	0.038	-0.184	0.029	0.061	-0.037	0.013
Entities	MARTSCHAT 2018	-0.05	-0.012	-0.657 *	-0.682 *	0.644 *	0.649 *	-0.338 *
Entities	DATEWISE	-0.038	-0.056	-0.394 *	-0.406 *	0.348 *	0.39 *	-0.103
Entities	CLUST	0.028	-0.044	-0.461 *	-0.501 *	0.422 *	0.515 *	-0.358 *
A.4 Running Time
In Table 9 we compare the running time of DATE-WISE and MARTSCHAT2018 on the T17 and ENTITIES datasets<sup>10</sup>. The implementations of both our methods and of MARTSCHAT2018 make use of parallel computation to obtain pairwise similarities between sentences or documents where required. We do not parallelize our methods in any other way. We could not run MARTSCHAT2018 on the CRISIS dataset since it requires too much memory, which demonstrates the need for more scalable state-of-the-art methods. DATEWISE and CLUST are considerably faster on both datasets, due to their "divide-and-conquer" nature: The summarization step is applied to only  $l$  smaller portions of articles and sentences, instead of the entire set. Note that part of the time is required to run the evaluation tool to compute alignment-based ROUGE.
Table 8: Correlations between Task Properties and Method Performance.
Dataset	Method	Avg. seconds per topic
T17	MARTSCHAT2018	176
T17	DATEWISE	16
T17	CLUST	15.9
ENTITIES	MARTSCHAT2018	106.3
ENTITIES	DATEWISE	29.5
ENTITIES	CLUST	34.7
Table 9: Running time comparison between current state-of-the-art method MARTSCHAT2018 and the methods we implemented.
A.5 Correlations between Performance and Dataset Characteristics
Detailed results of correlations between different methods and different dataset characteristics are shown in Table 8.
A.6 Output Examples
Table 10 shows parts of timelines produced by different methods for a selection of dates that all methods have selected. The topics are taken from the ENTITIES dataset. The examples demonstrate different levels of detail in describing particular events.
$^{10}$ On a machine with 16 3.70GHz Intel CPUs and 32GB memory.