SlowGuess/NAACL_2022_OCR huggingface dataset:
topics and their time spans are completely different. Specifically, the Entities dataset contains dozens of topics and spans decades per topic, while the others involve only a few topics within two years. The basic statistics are summarized in Table 1.
In our experiments, the evaluation of model-generated timelines depended on the ROUGE metric and its variants as follows (Yu et al., 2021):
Concatenation-based ROUGE F1 Similar to conventional ROUGE, it compares a concatenated system summary with its corresponding ground-truth by referring only to the textual overlap while ignoring all time stamps of the timeline (Yan et al., 2011b; Nguyen et al., 2014; Wang et al., 2016).
Alignment-based ROUGE F1 On the basis of the above concatenation metric, it linearly penalizes the ROUGE score by the distance of date alignments (Martschat and Markert, 2017).
Date selection F1 It only measures how well the model selects dates contained in the ground-truth (Martschat and Markert, 2018).
Since each topic has at least one ground-truth timeline, we considered each timeline independently if multiple ground-truths exist, and the final evaluation results were obtained by averaging scores over all timelines. We split training/verification/test sets in accordance with the ratio of  $40\% / 10\% / 50\%$  mentioned to Sec. 5.5. All experiments for a dataset were subject to leave-one-out cross-validation, and significant differences were determined by bootstrap test (Dror et al., 2018) with p-value of 0.005.
For our heterogeneous network, the vocabulary size was limited to 50,000 and tokens were initialized with 400-dimensional GloVe embeddings (Pennington et al., 2014). We truncated an input document to a maximum length of 40 sentences and removed  $10\%$  of vocabulary with the lowest TF-IDATEF values to eliminate noise. Date/sentence nodes and edge features individually included  $r_d = r_s = 128$  and 40-dimensional vectors for initialization. We set the learning rate and regularization hyper parameter  $\lambda$  to  $5e - 4$  and 1.5, respectively. Each HAN layer had 8 heads and 64-dimensional hidden size. The inner hidden size of the FFN layer was set to 512. An early stop was carried out when the validation loss did not descend for three continuous epochs. We trained all baselines as well as HeterTLS on a single Titan RTX GPU.
The following excellent baselines were used for comparison and to demonstrate the effectiveness of HeterTLS: direct summarization including CHIEU (Chieu and Lee, 2004) and MARTSCHAT (Martschat and Markert, 2018); date-wise summarization such as TRAN (Tran et al., 2013), DATEWISE (Ghalandari and Ifrim, 2020), and SDF (Quatra et al., 2021); and event detection method DASG (Liu et al., 2021). We additionally follow Ghalandari and Ifrim (2020) to obtain full oracle.
5.1 Performance of HeterTLS
According to Tables 2 and 3, HeterTLS outperformed all baselines in terms of all metrics. Con
Dataset	T17	Crisis	Entities
Metric	Date-F1	Date-F1	Date-F1
Full Oracle	0.926	0.974	0.757
CHIEU (2004)	0.252	0.142	0.102
TRAN (2013)	0.517	0.289	0.185
MARTSCHAT (2018)	0.544	0.281	0.167
DATEWISE (2020)	0.544	0.295	0.205
SDF (2021)	0.553	0.302	0.397
HeterTLS-HAN	0.668	0.455	0.432
HeterTLS-Joint	0.620	0.418	0.395
HeterTLS+Pre-trained	0.688	0.494†	0.478
HeterTLS	0.703†	0.492	0.488†
Table 3: Date F1-scores on T17, Crisis, and Entities datasets. Bold-faced characters and  $\dagger$  indicate best results and significant improvements over all baselines.
sidering that DASG ignores date information, we excluded it from the Date F1 experiment. We noticed that HeterTLS with pre-trained initial node representations surpassed HeterTLS only on Crisis and CovidTLS (refer to Appendix A) datasets. This indicates that pre-trained models require larger downstream datasets (Crisis or CovidTLS datasets) to escape from the local optimum, while CNN- and Bi-LSTM-based initialization can better capture the characteristics of small-scale datasets and reach the globally optimal solution in a few epochs.
We consider three possible reasons for the excellent performance of HeterTLS. First, the HAN is configured to learn multi-level semantic features for date representations. Compared with hand-designed statistical low-level features, these features are much more distinguishable, so they improve the accuracy of date selection. Second, regarding the improvement of ROUGE scores, the introduction of low-rank-based regularization helps sentence representations learn a diagonal clustering structure, which enables HeterTLS to effectively capture the topic-related events and informative sentences. Third, date selection and sentence clustering-based event detection are jointly learned and optimized to obtain a globally optimal solution.
We investigated the contribution of each module to HeterTLS via ablation studies using each dataset.
HeterTLS-HAN To verify the interaction within heterogeneous connections, we show the ablation performance in Table 2 by removing the HAN and simply using unlearnable semantic features with nuclear norm constraint. We suspect that the HAN
Figure 2: Comparison of running time of current state-of-the-art models and HeterTLS
layer plays a critical role in facilitating date selection with semantic messages and providing sentence nodes with temporal clustering information, which cannot be replaced with fixed features. Metapaths also provide abundant iterative patterns to pass semantic and temporal information.
However, sentence nodes initialized by CNN and Bi-LSTM layers help capture local and global sentence relationships, which has been proved predominant with regard to the extractive summarization task (Wang et al., 2020). Furthermore, the nuclear norm constraint can effectively reduce the redundancy between selected summary sentences. The above two components ensure the promising performance of the ablation model.
HeterTLS-joint learning Based on the assumption that the remarkable improvement of HeterTLS compared with baselines is due to jointly training node representations and clustering regularization, we show the performance in a separate learning pattern. Date representations are first learned using a HAN to predict which date should be selected to form a timeline. We then cluster sentence nodes in the graph to produce center cluster representations.
From the last block in Tables 2 and 3, implementing the subtasks individually degrades the performance to a great extent. We consider that in the joint learning framework of HeterTLS, vertices learn more discriminative features under the guidance and constraint of sentence clustering and in turn improve clustering accuracy, which cannot be imitated by separate learning. This result further indicates the superiority of HeterTLS, implying that
Figure 3: Impact of parameters on Entities dataset
the combination of node representations and clustering structure is promising for identifying salient dates and sentence candidates simultaneously.
5.3 Running time
We conducted an investigation of running time with all models being trained with the same device and show the results in Figure 2. HeterTLS ran up to an order of magnitude faster than most baselines, while it achieved comparable running efficiency to the current fastest baseline DATEWISE (2020).
The following two reasons may explain the efficiency of HeterTLS. 1) Accurate node initialization enables the model to converge to a globally optimal solution in less than eight epochs. Since transductive semi-supervised learning requires fewer labeled data nodes, it can simplify the scale of the training model and reduce the training time caused by parameter updates. 2) Previous methods rank all candidates by measuring informativeness, redundancy, coherence, and diversity (Yan et al., 2011b). In contrast, our strategy reduces the time complexity by measuring the similarity between date and cluster representations to select candidate clusters that exceed a pre-defined threshold. It can thus extract the most informative sentence in each candidate cluster as a summary without consuming time on the multi-index optimization problem.
5.4 Impact of parameters
There are two essential hyper parameters in our experiments:  $\lambda$  is adopted to balance the importance between  $\mathcal{L}_{\text {classify }}$  and  $\mathcal{L}_{\text {cluster }}$  in Eq. 14, and  $\delta$  acts as a threshold to decide the most related clusters for selected dates (Figure 1(c)). Several sets of  $\lambda$  and  $\delta$  were tested in terms of AR1-F and AR2-F. We can clearly see the best performance from Figure 3 when  $\lambda \in [2.0, 2.5]$  and  $\delta = 0.55$  on Entities dataset. It is explicit that a larger  $\delta$  works better. A plausible reason is that a relatively high threshold can effectively filter irrelevant clusters and reduce the redundancy of generated timelines. However, the gradual changes in histograms indicate that our
Figure 4: Ratio of labeled date nodes on training set vs. corresponding accuracy on test set
method rarely fails to converge as parameters vary because it is robust and insensitive to parameters. The parameter impact on other datasets is discussed in Appendix B due to limited space.
5.5 Ratio of labeled dates
Figure 4 shows that our model achieved promising Macro-F1 scores for date classification on test sets when the ratio of labeled dates was set to 40 or  $50\%$  in the training phase. Therefore, we reasonably believe that our HAN-based transductive learning earns high-quality date classification even with small-scale labeled data, so it can be effectively applied to real TLS tasks. Specifically, HeterTLS learns high-order semantic features implied in a small amount of labeled dates, which can help predict critical time stamps that should be preserved.
5.6 Consecutive dates and redundancy
The proportions of consecutive dates in chronologically ordered model-generated timelines and ground-truth timelines were experimentally measured according to Ghalandari and Ifrim (2020). News articles and sentences published on adjacent dates tend to refer to the same story, especially in a long-time-span dataset such as Entities.
T17	Crisis	Entities	CovidTLS
Ground-truth	0.45	0.18	0.03	0.48
MARTSCHAT	0.63	-	0.18	0.68
DATEWISE	0.62	0.52	0.30	0.66
HeterTLS	0.48	0.23	0.10	0.56
Table 4: Proportions of consecutive dates of timelines produced with different methods and ground-truths
Combining with Table 1, Table 4 reveals that because the time duration of Entities dataset is the longest, up to 12 years, the proportion of adjacent dates is the lowest among all datasets. Therefore, we reasonably believe that the trend of adjacent
Topic: Steve_Jobs Ground-truth timeline
2003-04-28Apple launches the iTunes store, a download music service.2004-07-31Undergoes surgery to remove a tumor related to the cancer.2006-04-01Apple celebrates its 30th birthday.2007-01-09Jobs unveils the iPhone at the Macworld conference.2008-06-27A class action suit is filed against Jobs and several members of the Apple's board of directors, claiming that they had participated in the backdating of stock option grants. In 2006, Apple was forced to restate its financial results after acknowledging that an internal investigation had revealed irregularities in its stock option grants between 1997 and 2001.2008-2009......
2009-06-29Apple spokesman Steve Dowling announces that Jobs has returned to work 2010-01-27Jobs introduces the iPad. The half-inch-thick, 1.5pound 9.7inch iPad allows users to read books, play games or watch video.2011-03-02Jobs receives a standing ovation when he takes the stage to unveil the iPad 2.2011-06-06At the Worldwide Developers Conference (WWDC) Jobs introduces iCloud the new online media storage system. Other Apple officials demo the new operating systems OS-X Lion and iOS-5.2011-08-24Resigns as CEO of Apple, but announces he will stay on as chairman. Tim Cook is promoted to CEO.
Topic: Steve_Jobs HeterTLS-generated timeline
2003-04-28Apple launches the Powerbook laptop.
2004-07-31When Jobs was recovering from surgery to remove the original cancer.
2006-04-01Appropriately enough, April 1 is the date Apple plans to celebrate its 30th birthday.
2008-06-27The computer maker said it has brought in independent counsel to review the handing out of options between 1997 and 2001, including a batch for chief executive Steve Jobs, after an internal inquiry found potential irregularities, Steve Jobs cooperated with Apple 's independent investigation and with the government 's investigation of stock option grants at Apple, directors said.
2008-2009
2009-06-29Apple spokesman Steve Dowling said the decision to pull Jobs out of the show indicated the company 's intention to stop exhibiting at Macworld. Big Brother permitting the choice that Jobs has made this time round.
2010-01-27Apple says that it "lets users browse the web, read and send email, enjoy and share photos, watch videos, listen to music, play games, read eBooks and much more on the Mac, the iPad, the iPod, the iPhone".
2011-03-02We should know in a couple of hours, when Jobs takes the stage to keynote (sic) Apple 's Worldwide Developer Conference 2005, which opens today in San Francisco. A rapturous standing ovation follows.
2011-06-06Next week is the Apple Worldwide Developers' Conference, where Steve Jobs will address the adoring masses. Details of the new operating systems have been dribbling out for months, with the official unveiling in October last year.
2011-08-24Tim Cook is widely tipped as a possible replacement as CEO. I hereby resign as CEO of Apple.
Figure 5: Partial timelines on topic of Steve Jobs from Entities dataset produced with ground-truth and HeterTLS
date proportion is the same as that of redundancy. The results in Table 4 indicate that HeterTLS is the closest to the ground-truth, thereby proving its ability to predict salient dates.
5.7 Case study
We now show the quality of timelines generated by HeterTLS through a cases study. The topic Steve Jobs is taken from Entities dataset with the time duration from 2003-04-28 to 2011-08-24. In Figure 5, parts of the ground-truth timeline of certain dates are shown on the left, while the right side lists the HeterTLS-generated timeline with similar period coverage as the ground-truth. We manually colored some keywords to illustrate consistent contents in both timeline summaries. The examples demonstrate different levels of detail in describing particular events. Three advantages of HeterTLS are explicit by comparing it with the ground-truth:
- The semi-supervised date prediction component of HeterTLS can accurately position salient dates as the ground-truth, which is the very principle for extracting TLS sentences.
- Our model can capture the major object of each event or topic well (marked in orange) in a daily summary. For example, the subject of the ground-truth on 2003-04-28 is Apple launches, and HeterTLS also generates the same phrase as the subject. On 2009-06-29, Steve Dowling announces and Steve Dowling said serve as subjects in the ground-truth and
model-generated summary, respectively.
- Although HeterTLS generates timelines in an extractive manner, the generated summaries are short and accurate. Current extractive methods always adopt greedy or beam search to extract an uncertain number of sentences as timelines, which greatly increases redundancy. We use clustering-based constraints and intraclass extraction to ensure that HeterTLS generates short but accurate sentences.
We addressed several fundamental problems concerning TLS and proposed a joint learning model called HeterTLS, which trains a HAN by utilizing clustering structure learning-based event detection. The proposed model facilitates node representations with information of different semantic units. Meanwhile, the sentence representations with clustering structure are rich in date- and semantic-level features, which significantly reduce redundancy and improve clustering accuracy. Experimental results, including those of the ablation studies of each part of the overall architecture, demonstrated the effectiveness of HeterTLS.
We would like to gratefully thank the anonymous reviewers for their helpful comments and feedback. Jingyi You and Dongyuan Li acknowledge the support from China Scholarship Council (CSC).
James Allan, Rahul Gupta, and Vikas Khandelwal. 2001. Temporal summaries of news topics. In SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, September 9-13, 2001, New Orleans, Louisiana, USA, pages 10-18. ACM.
Purnima Bholowalia and Arvind Kumar. 2014. Ebk-means: A clustering technique based on elbow method and k-means in wsn. International Journal of Computer Applications, 105(9):17-24.
Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019. Learning towards abstractive timeline summarization. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJ-CAI 2019, Macao, China, August 10-16, 2019, pages 4939-4945. ijcai.org.
Hai Leong Chieu and Yoong Keok Lee. 2004. Query based event extraction along a timeline. In SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25-29, 2004, pages 425-432. ACM.
Fan RK Chung and Fan Chung Graham. 1997. Spectral graph theory. Number 92 in CBMS Regional Conference Series in Mathematics. American Mathematical Soc., Fresno State University.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Chris H. Q. Ding and Tao Li. 2007. Adaptive dimension reduction using discriminant analysis and  $K$ -means clustering. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, volume 227 of ACM International Conference Proceeding Series, pages 521-528. ACM.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392.
Yijun Duan, Adam Jatowt, and Masatoshi Yoshikawa. 2020. Comparative timeline summarization via dynamic affinity-preserving random walk. In ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de
Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 1778-1785. IOS Press.
Demian Gholipour Ghalandari. 2017. Revisiting the centroid-based method: A strong baseline for multi-document summarization. In Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 85-90. Association for Computational Linguistics.
Demian Gholipour Ghalandari and Georgiana Ifrim. 2020. Examining the state-of-the-art in news timeline summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1322-1334. Association for Computational Linguistics.
Benjamin D. Haeffele and René Vidal. 2020. Structured low-rank matrix factorization: Global optimality, algorithms, and applications. IEEE Trans. Pattern Anal. Mach. Intell., 42(6):1468-1482.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Yun Hu, Yeshuang Zhu, Jinchao Zhang, Changwen Zheng, and Jie Zhou. 2021. Toward fully exploiting heterogeneous corpus: A decoupled named entity recognition model with two-stage training. In *Findings of the Association for Computational Linguistics: ACL/IJCNLP* 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of *Findings of ACL*, pages 1641-1652. Association for Computational Linguistics.
Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural extractive summarization with hierarchical attentive heterogeneous graph network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3622-3631. Association for Computational Linguistics.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2324.
Jiwei Li and Sujian Li. 2013. Evolutionary hierarchical dirichlet process for timeline summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 2: Short Papers, pages 556-560. The Association for Computer Linguistics.
Xi Li, Qianren Mao, Hao Peng, Hongdong Zhu, Jianxin Li, and Zheng Wang. 2021. Automated timeline
length selection for flexible timeline summarization. CoRR, abs/2105.14201.
Jingzhou Liu, Dominic J. D. Hughes, and Yiming Yang. 2021. Unsupervised extractive text summarization with distance-augmented sentence graphs. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2313-2317. ACM.
Zhang Liu and Lieven Vandenberghe. 2009. Interior-point method for nuclear norm approximation with application to system identification. SIAM J. Matrix Anal. Appl., 31(3):1235-1256.
Sebastian Martschat and Katja Markert. 2017. Improving ROUGE for timeline summarization. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 285-290. Association for Computational Linguistics.
Sebastian Martschat and Katja Markert. 2018. A temporally sensitive submodularity framework for timeline summarization. In Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1, 2018, pages 230-240. Association for Computational Linguistics.
Kiem-Hieu Nguyen, Xavier Tannier, and Véronique Moriceau. 2014. Ranking multidocument event descriptions for building thematic timelines. In COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, pages 1208-1217. ACL.
Feiping Nie, Xiaoqian Wang, Cheng Deng, and Heng Huang. 2017. Learning A structured optimal bipartite graph for co-clustering. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4129-4138.
Arian Pasquali, Vitor Mangaravite, Ricardo Campos, Alipio Mário Jorge, and Adam Jatowt. 2019. Interactive system for automatically generating temporal narratives. In Advances in Information Retrieval - 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part II, volume 11438 of Lecture Notes in Computer Science, pages 251-255. Springer.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.