Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries
Authors: Nishanth Sridhar Nakshatri
Siyi Liu
Sihao Chen
Dan Roth
Dan Goldwasser
Daniel Hopkins
Venue: EMNLP 2023 Findings (2023)
Keywords: news stream clustering
event discovery
political discourse characterization
LLM
TL;DR: We propose a versatile framework that leverages LLM to enhance current document clustering techniques for identifying key events within a vast collection of news articles.

Abstract: Understanding and characterizing the discus- sions around key events in news streams is important for analyzing political discourse. In this work, we study the problem of identification of such key events and the news articles associated with those events from news streams. We propose a generic framework for news stream clustering that analyzes the temporal trend of news articles to automatically extract the underlying key news events that draw significant media attention. We characterize such key events by generating event summaries, based on which we form document clusters in an unsupervised fashion. We evaluate our simple yet effective framework, and show that it produces more coherent event-focused clusters. To demonstrate the utility of our approach, and facilitate future research along the line, we use our framework to construct KeyEvents, a dataset of 40k articles with 611 key events from 11 topics.

Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries
Nishanth Nakshatri* Siyi Liu Sihao Chen
Daniel J. Hopkins Dan Roth
Dan Goldwasser*
Purdue University Uniiversity of Pennsylvania (nnakshat, dgoldwas J@purdue edu (siyiliu, sihaoc, danhop, danroth]@upenn.ce edu
Abstract
news events usually not known in advance, past works have typically formulated the problem as a form of non non-parnan non-parametric clustering of news arti- cles, using Hierarchical Dirichlet Processes (Zhou et al., 2015; Beykikhoshk et al.. 2018) or Stream Clustering (Laban and Hearst, 2017; Miranda et al., 2018; Staykovski et al., 2019; Saravanakumar al.. 2021). Rather than relying the output of such clustering algorithms directly, we view the dis- covered clusters as event candidates, and lever- age recent advances in Large Language Modeling (LLM) (Brown et al., 2020) to characterize these candidates and reason about their validity. From a bird's eye view, the process is related to past work on interactive clustering (Hu et al., 2014 10;4; Pacheco et al., 2022, 2023), but instead of using human feedback to shape the emergent clusters, we rely on LLM inference.
Understanding and characterizing the discus- sions around key events in news streams is im- portant for analyzing political discourse. In this work, study the problem of identification of such key events and the news articles associated with those events from news streams. We pro- pose generic framework for stream clus- tering that analyzes the temporal trend of news articles to automatically extract the underlying key news events that draw significant media attention. We characterize such key events by generating event summaries, based on which we form document clusters in an unsupervised fashion. We evaluate our simple yet effective framework, and show that it produces more co- herent event-focused clusters. To demonstrate the utility of our approach, and facilitate future research along the line, we to construct KEyEVSNTS', dataset of 40k articles with key events from topies.
framework
We proposc framework for clustering archive of news articles into temporally motivated news events. A high-level overview of our ap- proach shown in Figure 1. We first retrieve rele- vant issue-specific articles (dctails about the docu- ment retrieval module are in App A) and perform temporal analysis to identify "peaks", in which the number of articles is significantly higher. We then use HDBSCAN (Campello et al.. 2013) a non- parametric clustering algorithm generate didate event clusters. We then characterize the candidate clusters by performing few-shot multi- document summarization of the top-K articles as- signed to each cluster, identify inconsistent clusters by assessing the (dis)agreement between the sum- mary and each article individually, and redundant clusters by assessing the similarity between cluster pairs' summaries (details in Sec. 2.1). These low- quality candidates are removed, resulting in higher quality event clusters. We demonstrate this prop- erty over the NELA dataset (Horne et al., 2022) and show the improvement both in terms event coherence and document mapping quality.
1 Introduction
Analyzing the dynamics of discussions within the stream of news coverage has been an important tool for researchers to visualize and characterize media discourse around a topic (Field et al., 2018; Liu et al., 2019; Li and Goldwasser, 2019; Roy and Goldwasser, 2020; Luo et al., 2020; Liu et al., 2021; Lei et al., 2022; Dutta et al., 2022). News media discourse is typically centered around real-world events that catch media attention and gives rise news reports streams. With the vast, ever-growing amount of news information available, we need automatic ways for identifying such key events.
In this paper, we study the problem of identi- fying and characterizing key events from large collection of news articles. Since the number of hwwp./////aaasatt////eeeeeees

---

Discovery
Inference
GPT-3.5
Morge Remove
Characterization
Events
Membership
Issues
Keywords
GPT-3.5
Neywords
Peak Specific
Peak Speciffic
Events
Clustering
User
haaue hoywordds
KEYEVENTS
Issue Specific Articles
NÉLA Retriever
Detection
Document Rebrivval Module
Temporal Filtering Module
Figure I: High-level overview of our framework KEYEVENTS identification.
2 Event Discovery and Article Inference
Incoherent Cluster (Top-3 documents shown)
Event Title: Climate Justice and African Activists Event Deseription: This is about the challenges faced by African climate activists in bringing attention the climate crisis and the need for climate justice.
2.1 Event Discovery
Temporal Filtering. The first step towards gener- ating event candidates is to identify temporal land- marks or peaks, where the media coverage surges with respect to one or more real-world events. We represent the news articles as time-series data, where T (t1,t2,- .2,...,,,,,) } denote time, and
Doc. 1: There Will Never Be Climate Justice If African Activists Keep Being Ignored
We go Kampala, U Uganda, to speak climate activist Vanessa Nakate on the occasion of her first book heing published, A Bigger Picture.
C {e,, ; denote the number of ar- ticles published each time step. The task to identify set of peaks, P = P2, PPm) at different points in time. With this formulation, we hypothesize that the resulting clusters from our framework would be able to segregate discussions at various time steps and form coherent events com- pared to other approaches. We use an existing outlier detection algorithm (Palshikar et al., 2009) towards this task. More details in Appendix B.
The Looking Glass World Of "Climate Injustice' In our wacky world where almost nothing makes sense anymore, there is no shortage of examples of politicians, let alone self-important academics, journalists, and wealthy clites, looking foolish with self- coff-contradcctryy policy demands...
Doc. 2:
Doc. 3: New Mixs Universe Urges Action on Climate Change: Choice to Kill or Save Nature A new Miss Universe has been crowned and she a climate alarmist.
Table I: Incoherent cluster removal. The cluster sum- mary aligns with the 1st and the 2nd article, while the 3rd article is off-topic compared the other two.
Peak-Specific Clustering. Within each peak, the increased media coverage can be attributed to mul- tiple relevant events. We categorize the docu- ments each peak pi into a set of events, & (e,,,,,...,, and form an overall event set, E (,, Em pertaining to the issue. We embed the title and first lines of a news article instance using dense retriever (Ni et al., 2021) model. The embedded documents are clustered us- ing HDBSCAN to identify key news events. Prior to clustering, reduce the dimensions of doc- ument embedding using UMAP (Mclnnes al., 2018). Details are in Appendix C.
multi-document summary using GPT-3.5. The prompts are engineered to generate short event- specific summaries in a two-shot setting. The two closest documents to each centroid are used in the prompt to generate event summaries.
Post summary generation, we perform cluster inconsistency check. A cluster is deemed to be incoherent if the top-K closest documents to the centroid do not align with the summary embedding. We embed the event summaries using the same dense retriever model, and compute the cosine sim- ilarity score between the summary embedding and the top-K documents for the cluster (k 5). Based on threshold value, treat the incoherent clus- ters as noise and discard them. Note that we only discard clusters but not documents associated with them. They are still used for cluster membership as- signment the next stage of our framework. Tab.
Event Characterization. The event set obtained each peak (E4), is still prone to noise and is not easily interpretable without significant effort. Char- acterizing the news events makes the clusters in- terpretable and helps remove inconsistencies. The candidate events are characterized by generating

---

Summary of Article 1
Summary of Article 2
Event Title: President Biden's Climate Plan
Event Title: Biden's Climate Change Actions Event Description: This is about President Joe Biden's exce- utive actions to combat climate change by prioritizing science and evidence-based policy across federral agencies, pausing oil drilling on public lands, and aiming to cut oil. gas, and coal emissions.
Event Description: This is about President Joe Biden's 's ex- ecutive orders aimed tackling climate change reducing the U.S. carbon footprint and emissions, stopping oil and gas leases on public lands, and prioritizing climate change as national security concern.
Event Title:
Texas Abortion Ban
Event Title: Texas Abortion Law
Event Description: This is about new Texas law that bans abortions after 6 weeks and empowers regular citizens to bring civil lawsuits against anyone who aids woman looking to terminate pregnancy.
Event Description: This is about the controversial Texas abortion law that bans abortions after six weeks and has been condemned by President Joe Biden as unprecedented as- sault on women's 's rights.
Table 2: Illustrates cases of cluster merge from issue Climate Change, and Abortion respectively.
shows example of the discarded cluster.
pendix A). Using our document retrieval module, collect total of 335k: relevant news articles on 11 contemporary issues?. The application of tem- poral filters reduces the article count to 90k, which is the basis for our analysis. The retrieved articles are mapped to a four-way (left, right, center, and political rating. Details about the dataset, document retrieval module, and four-way political rating can found in Appendix A.
We do an additional cleaning step by merging the clusters that share a similar event summarry We devise a simple greedy algorithm which uti- lizes GPT-3.5 for inference. In the first iteration of the algorithm, we start by constructing set, S = (2), s2)....,((.n)1))) (5m-1, s, that contains ev- ery pairwise combination of event summaries. For each element in S, we prompt LLM to infer if the pair of summaries are discussing about the same event. the event summarics, say 82), are equivalent, then we merge these summaries, and update the set S by removing every element in the set that contains or 82. In the second iteration, we construct new set, S', that holds every com- bination of updated event summaries, and repeat the previous step. We run the algorithm for two iterations or halt if there are no merges after the first iteration. Tab. shows an example where the event summaries clearly indicate that the clusters need to merged. Details about the hyperparameter selections, and prompts are in Appendix C, B.
Evaluation Metries. We evaluate our frame- work's s ability to create coherent event clusters at the desired granularity with three automatic metrics inspired by Mimno et al. (2011). Given an event e; and the top- 10 relevant entities V = {xt] hee[..1 to e, by TF-IDF, entity purity measures the percent- age of the documents that mention at least one of the top- 10 entities; coverage counts the percentage of documents accounted for in the cluster assign- ments. In addition, entity coherence considers co- occurrences of central entity pairs in the clustered documents to measure coherency for an event.
M 1 Flust eit + Clei, Vol = Σ
Σ log
2.2 Inference: Map Articles to Events
F( (1)
In this stage of our framework, we decide the clus- ter membership using a similarity module. We em- bed the updated event summaries using the same encoder, and compute the cosine similarity score between the summary and the document of interest, By thresholding, we determine if the article can be mapped to event. For cluster membership, we extend the temporal window by d days before and after the peak (d 1), and consider all the documents published in that timeframe.
where F(r rf(t) indicates the co-occurrence fre- quency of two entities in documents. An entity coherence value closer to zero indicates a highly coherent news event cluster. We offer more de- tailed explanation of the metries in Appendix D.
Baselines. We compare our method's perfor- mance against various competitive topic model as baselines. We consider LDA (Blei et al., 2003; Hoffman et al., 2010) in two different settings LDA, and LDA (Temporal). The topics are esti- mated individually at each temporal peak for LDA (Temporal), whereas the topics are estimated across
3 Experiments and Results
We conduct experiments on the dELA-dataset, which is a large collection of news articles (see Ap-
chttps://wwer allsides. con/ henn/opiccs issues

---

Model
Coverage |
Entity Purity
Entity Coherence
Event Count
LDA (baseline)
99.69
31.52
-1008.42
60.0
Temporal fittrrng
28.15
-1061 160
18.7
LDA (Temporal)
89,02
38.62
-1005.3 37
63.7
HDBSCAN
81.78
62.55
-776.80
58.4
BERTopic
84.04
66.00
-726.11
62.3
Our Method
44,29
82.69
-477.89
55.5
Our Method (iter 2)
56.83
77.49
-579.48
55.5
Table 3: Evaluation results averaged for all issues. Last column shows the average of the total event count from each peak and for each issue. For LDA(Temporal), we assigned the document to its most probable topic if the probability was 0.5.
all peaks at once for LDA. We include three addi- tional baselines Temporal Filtering, HDBSCAN, and BERTopic (Grootendorst, 2022). Note that BERTopic is an off-the-shelf neural baseline for clustering documents. For methods other than ours, we do not incorporate cluster membership module we directly estimate the topics for all the docu- ments in an extended temporal window of c days before and after the peak (d 1). Preprocessing and hyperparameter details are in Appendix
method in terms of purity, and this can be primarily attributed to our inference mechanism that based on generated event summaries.
To address low coverage issue from our method, we propose to run our framework for the second iteration by updating event summary embedding with the mean value of top- 10 most representative document embeddings in the cluster (from the first iteration). In doing so, average coverage increased by + 12.5 points across all issues, with minimal decrease < 5 points in purity. Tab. 6 shows the results for each issue after the second iteration.
Results. Tab. 3 shows the aggregated results ob- tained for various methods across all the issues. For LDA (baseline), the events are estimated over a union of all the documents from every peak for an issue. We study the impact of event estimation with the temporal component by comparing LDA (base- line) and Temporal Filtering methods. We observe only slight drop in average purity (-3 points) for the Temporal Filtering method. Further, Tab. 8 shows that in case of Free Speech, Abortion, Im- migration issues, the purity scores are higher than LDA (baseline), which validates our hypothesis that adding temporal dimension to event identifi- cation can help form coherent events.
4.2 Impact of Merge/Remove Operations
We investigate the impact of removing cluster in- consistencies over the generated candidate events. For this analysis, we compare HDBSCAN with the same hyperparameters and input data as our method. We observe that average of the inter-event cosine similarity score between event-pairs pains, and across all issues is lesser by 0.14 for method. This indicates that our method achieves improved cluster separability after climinating inconsisten- cies. Tab. 5 shows the report for each issue. Over- all, the score is reduced, with one exception for the issue of Corruption. Manual inspection suggest that the increase can be due to removal of "good" clusters. An example is shown in Fig. 7.
4 A Anayssi and Discussion
4.1 Coverage vs Purity Trade off
We evaluate the trade-off between coverage and en- tity purity among the methods that take event tem- porality into account. We observe that LDA (Tem- poral) has a very high coverage with the least purity, which can be attributed to noise associated with the topic distributions. BERTopic improves over this method in both coverage, and purity measures across 11 issues. It even outperforms HDBSCAN in both the metrics. However, while BERTopic has increased coverage, it still fails to outperform
4.3 KEYEVENTS > More Event Coherence
To better understand the advantages and disadvan- tages of our method, the authors manually annotate small set of data samples for Climate Change. We test for event coherence, and mapping quality over this dataset. We define event to be coherent if the top-K most representative documents of that event are in agreement with each other (k: =3). We also annotate to verify the validity of document- to-event assignments (mapping quality), where we check for agreement between the document and its
Phttps://maarten gighaarttuggg..hhub.... github. Boruub.oo/BERR io/BERTopic

---

Mapping Quality (Precision)
Entity: Joe
Model
Event Coherence 1
Right
HDBSCAN
84,90
62.27
BERTopic
85 48
69.87
Our Method
91.07
72.19
Trequency
Table Human evaluation results of our method.
respective event summary. The details about the experimental setup can be found in Appendix E.
The test is conducted across all events for our method, HDBSCAN, and BERTopic. To mea- sure coherence, we first identify the top-K doc- uments for an event based on their cosine similarity scores with the event centroid. In addition, we esti- mate mapping quality by judging if document pairs should be clustered together or not.
Figure 2: Frequency of the entity Joe Manchin (y-axis: #entity mentions per article within each event) in Cli- mate Change events (X-axis: event indices across time).
Tsvetkov, 2019; Roy al., 2021) and analyze men- tions of Joe Manchin, a democratic senator and the chair of Senate Energy Committee, in Climate Change articles. Fig. 2 shows that left-leaning arti- cles mention him significantly more than the other two ideologies in some of the events (e.g. the 5th, gth and 14th). Analyzing these events' articles show that left leaning articles criticize his ties to the coal industry and opposition to climate change legislation, while fewer (or no) mentions in articles with other idcology leanings under the same events.
Results. The results of the human evaluation are shown in Tab. 4. Our method failed to generate coherent events for 5 out of the 56 cases for Cli- mate Change, while BERTopic failed in 9 out of 62 cases (ignoring 3 cases where the annotator pro- vided label of 1).. HDBSCAN failed in 8 out of 53 cases. Overall, the event coherence scores from BERTopic and HDBSCAN closely trail our method by a margin of approximately -6 6 points, implying that the generated events from these meth- ods are coherent. However, considering the event purity scores, we conclude that these two methods more noisy. In terms of mapping quality, our method outperforms HDBSCAN by a large mar- gin. The precision score from BERTopic is better than HDBSCAN, indicating the effectiveness of BERTopic in grouping 'good' item pairs together over a small sample of randomly selected data- points for the issue Climate Change. More details in Appendix E.
Different ideologies also persist different senti- ments when mentioning the same entity. In Biden's 's Executive Actions on Climate Change (16th event in Fig. 2), articles from different idcologies have comparable mention frequencies of Joe Manchin. We prompt GPT-3.5 to classify the sentiment ex- pressed towards him (positive, neutral, negative). Interestingly, none of the articles from any ideology expresses a positive sentiment; 86% of the articles from the left endure negative attitude towards him, whereas only 38% and 0% of the articles from the center and the right have negative sentiments. This distinction shows that even the same entities could be portrayed diffcrently within each event to strengthen the beliefs along their political lines.
4.4
LLM Usage and Efficiency
As temporal filtering results in an average of 55 event clusters per issue, we observe that using LLM for event summarization and cluster-merging incurs reasonable cost, as we discuss in Limitations.
Conclusion
5 Broader Impact
We present a framework for key events identifica- tion and showed that events generated from our approach were coherent through quantitative mea- sures, and human evaluation. We also presented a simple qualitative study showcase the potential of KEY EVENTS, for investigating various political perspectives under nuanced settings.
Our method and the resulting KEYEVENTS dataset could be useful for analyzing political discourse across different ideologies. As simple case study, we illustrate how the portrayal of events varies for different political idcologies. We take an entity- based approach (Rashkin et al., 2016; Field and

---

Limitations
References
As the temporal filtering step of our framework relies on the publicaiton date of documents as in- put, we work with the assumption that the docu- ments have a timestamp attached to them. How- ever, the main idea of event characterization using LLM, and associating the documents to their clos- est event summary is applicable to other cases with changes.
Adham Beykikhoshk, Ognjen Arandjelović, Dinh Phung, and Svetha Venkatesh. 2018. Discovering topic structures of temporally evolving document corpus. Knowledge and Information Systems, 55:599- 632.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Ja))993-1102..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
Our approach relies on GPT-3.5 for generat- ing a multi-document event summary and cluster- merging. We choose to use GPT-3.5 instead of the open-source counterparts mostly due to computa- tional resource constraints. Since all GPT calls are made on the cluster-level, we are able to maintain the total experimental cost of the paper under $5 with respect the OpenAl API. To minimize the reliance and cost associated with LLM usage, we are using only pairs of documents with most similar vector representation to generate event summary. We opt for more an efficient approach here, and leave the exploration of efficiency vs. performance trade-off for future work.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based hierarchical density estimates. Advances in Knowl- edge Discovery and Data Mining: 17th Pacific-Asia Conference, PAKDD 2013, Gold Coast, Australia, April 14-17, 2013, Proceedings, Purt II 17, pages 160-172. Springer.
Sihao Chen, William Bruno, and Dan Roth. 2023. To- wards corpus-scale discovery of selection biases in news coverage: Comparing what sources say about entities as start. arXiv preprint orXiv:2304.034/4
Acknowledgements
Sujan Dutta, Beibei Li, Daniel S Nagin, and Ashiqur R KhudaBukhsh. 2022. murder and protests, the capitol riot, and the chauvin trial: Estimating dis- parate news media stance. In Proceedings of the Thirty-First International Joint Conference on Artifi- cial Intelligence, pages 5059-5065.
We thank the anonymous reviewers of this paper for all of their vital feedback. The project par- tially funded by NSF award 11S-2135573, and in part by the Office of the Director of National Intel- ligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-22072200003. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies, ei- ther expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for governmental purposes notwithstanding any copy- right annotation therein.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Fram- ing and agenda-setting in russian news: a computa- tional analysis of intricate political strategies. In Pro- ceedings of the 2018 Conference Empirical Meth- ods Natural Language Processing, pages 3570- 3580.
Anjalie Field and Yulia Tsvetkov. 2019. Entity- contextual affective analysis. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2550-2560, Florence, Italy. Association for Computational Linguistics.
Ethics Statement
Maarten Grootendorst. 2022. Bertopie: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint 2203.0009
To the best of our knowledge, we did not violate any ethical code while conducting the research work described in this paper. We report the technical details needed for reproducing the results and will release the code and data collected. We make it clear that the KEY EVENTS dataset is the result of an automated algorithm not human annotation (though human evaluation was used in assessing its performance over a subset of the data).
Matthew Hoffman, Francis Bach, and David Blei. 2010. Online learning for latent dirichlet allocation. ad- vances in neural information processing systems, 23.
Benjamin Horne, Mauricio Gruppi, and Sibel Adali. 2022. NELA-GT-2021.
Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong. Denis Peskov, Jordan Boyd-Graber, and Philip

---

Resnik. 2021. Is automated topic model evaluation broken? the incoherence of coherence. Advances in Neural Information Processing Systems, 34:2018 88- 2033.
Davoud Moulavi, Pablo A Jaskowiak, Ricardo JGB Campello, Arthur Zimek, and Jörg Sander. 2014. Density-based clustering validation. In Proceedings of the 2014 SIAM international conference on data mining, pages 839-847. SIAM.
Yuening Hu, Jordan Graber, Brianna Satinoff, and Alison Smith. 2014. Interactive topic modeling. Ma- chine learning. 034223-49..
Federico Nanni, Simone Paolo Ponzelto, and Laura Di- etz. 2017. Building entity-centric event collections. In 2017 ACM/IEEE Joint Conference on Digital Li- braries (JCDL), pages 1-10. IEEE.
Philippe Laban and Marti A Hearst. 2017. newslens: building and visualizing long-ranging news stories. In Proceedings of the Events and Stories in the News Workshop, pages 1-9.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández. Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Woi Wei Chang, et al. 2021. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv: (rXiv22122.78899
Yuanyuan Lei, Ruihong Huang, Lu Wang, and Nick Beauchamp. 2022. Sentence-level media bias analy- sis informed by discourse structures. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10040-10050, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Maria Leonor Pacheco, Tunazzina Islam, Monal Maha- jan, Andrey Shor, Ming Yin, Lyle Ungar, and Dan Goldwasser. 2022. A holistic framework for analyz- ing the COVID-19 19 vaccine debate. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5821-5839, Seattle, United States. Association for Computational Linguistics.
Chang Li and Dan Goldwasser. 2019. Encoding so- cial information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2594- 2604.
Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Goldwasser. 2023. Interactive concept learning for uncovering latent themes in large text collections. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5059- 5080, Toronto, Canada. Association for Computa- tional Linguistics.
Siyi Liu, Sihao Chen, Xander Uyttendacle. and Dan Roth. 2021. MultiOpEd: A corpus of multi- perspective news editorials. In Proceedings of the 2027 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4345-4361 On- line. Association for Computational Linguisties.
Girish Palshikar et al. 2009. Simple algorithms for peak detection in time-series, In Proc. 1st Int. Conf. Advanced Data Analysis, Business Analytics and In- telligence, volume 122.
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news fram- ing trends surrounding us gun violence. In Proceed- ings of the 23rd conference on computational natural language learning (CoNLL), pages 504-514.
Hannah Rashkin, Sameer Singh, and Yejin Choi. 2016. Connotation frames: A data-driven investigation. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ), pages 31 311-321, Berlin, Germany. As- sociation for Computational Linguisties.
Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. De- tecting stance in media on global warming. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 3296-3315.
Lev Ratinov, Dan Roth, Doug Downey, and Mike An- derson. 2011. Local and global algorithms for disam- biguation wikipedia. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 1375-1384.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and pro- jection for dimension reduction. arXiv preprint 1rX.v:1802.03426.
David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In Proceed- ings of the 2011 conference on empirical methods in natural language processing, pages 262 222.
Radim Rehurek and Petr Sojka. 2011. Gensim-python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2):2.
Sebastião Miranda, Arturs Znotins, Shay B Cohen, and Guntis Barzdins. 2018. Multilingual clustering of streaming news. In Proceedings of the 2018 Con- ference on Empiricul Methods in Natural Language Processing, pages 4544.
Shamik Roy and Dan Goldwasser. 2020. Weakly su- pervised learning of nuanced frames for analyzing polarization in news media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7698-7716.

---

Shamik Roy, María Leonor Pacheco, and Dan Gold- wasser. 2021. Identifying morality frames in political tweets using relational learning. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 9939-9958.
to four-way (left, right, center, conspiracy- pseudoscience). The ratings are decided based on MBFC 4, Using the scores provided by MBFC, we categorize left-center and right-center political ratings to one of the [center, left, right) ratings.
Kailash Karthik Saravanakumar, Miguel Ballesteros, Muthu Kumar Chandrasekaran, and Kathleen Mck- cown. 2021. Event-driven news stream clustering using entity- aware contextual embeddings. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2330-2340.
B Event Candidate Generation
Temporal Filtering We implement an outlier de- tection algorithm (Palshikar al., 2009) which considers a temporal window of 2k points around each data point, x. These are points before x, and points after x. Using these 2k data points, we compute the mean and standard deviation. The data point is considered as local peak if it is at least standard deviation away from the mean value. Among the detected local peaks, we further apply filter to retrieve global peaks. We do this by com- puting the mean and standard deviation values for the detected local peaks. the value at the local peak is above the mean value, we mark that as a global peak. In the case of multiple peaks within a temporal window of days, we merge them to form single peak. We set the value of l = 3 for experiments. Figure shows the result of this algorithm for the issue Abortion.
Andreas Spitz and Michael Gertz. 2018. Exploring entity-centric networks entangled news streams. In Companion Proceedings of the The Web Conference 2018, pages 555-563.
Todor Staykovski. Alberto Barrón- Cedonn, Giovanni Da San Martino, and Preslav Nakov. 2019. Dense vs. sparse representations for news stream clustering.
Deyu Zhou, Haiyang Xu, and Yulan He. 2015. An unsu- pervised bayesian modelling approach for storyline detection on news articles. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1943-1948.
A Document Retrieval Module
This module retrieves news articles relevant to an issue of interest. User is expected to provide an issue name or set of issue names around which the documents are to be retrieved. Using this input, we generate set of relevant keywords associated with each issue by prompting GPT-3.5. We craft the prompt such a way that GPT-3.5 generates list of keywords that appear in the context of the issue specified by the user. We then use BM25 algorithm on the indexed NELA data to retrieve documents associated with each keyword for the issue. We use BM25 with the default settings for b. and only vary the term frequency saturation kl = 1.3 as we are dealing with longer news documents.
Mo
Figure 3: Dynamic Analysis of documents from Jan to Dec 31, 2021, for the issue Abortion. X-axis represents time (one day interval). Red dots indicate detected peaks.
C Models and Hyperparameters
NELA Dataset It collection of 1.8. news documents from 367 news outlets between January 1st 2021, and December 31st, 2021. NELA is successful in organizing the news articles based on their ideological bias. However, this structure is not well-suited to characterize the differences in discourse between the political ideologies in online news media.
To obtain topics from LDA with Variational Bayes sampling (under both settings), use Gensim (Re- hurek and Sojka, 2011) 1) implementation. We follow the preprocessing steps shown in (Hoyle et al., 2021), and estimate the number of topics driven manner by maximizing We do a grid- search over a set of (2.3, 4, 5} for LDA (Temporal) method. The set of topics for LDA (baseline) is {10, 20, -0,--..,00)
data-
In this work, we primarily focus on 207 news sources that are based out of USA. The political rating corresponding to these sources are mapped
"htps//mmcciaaassticccccc..cmmm

---

In the case of HDBSCAN, when used for our method, and as standalone clustering model, we use a data-driven approach to estimate the best number of topics by maximizing the DBCV score (Moulavi et al., 2014). We retain the default settings for cluster_selecte seleetion__eehhod,..ooo and metric parameters, while we change the min_cluster___a to get more sensible topics. This number is selected based grid search whose values are sensitive the number of input data points. Suppose [X] denote the number of data points, then the grid parameters for HDBSCAN used in our method in- clude {0.05 x |,066 is updated to consider only the last three elements for HDBSCAN (standalone). If not, we see un- usually high number of topics per peak. We set the neingghbors parameter in UMAP embedding model min_ ciir__cuuter_rii
event. Note that in order remove commonly repeated entities in news such as Biden, Trump etc.. we consider top-K TF-IDF based entities for an
event as central entities. A purity score of 100% for an event indicates that every document the cluster has atleast mention of one of the top-K central entities, suggesting that each document is potentially discussing about that event.
We also define entity coherence metric as additional measure validate the cluster quality. We adapt the topic coherence metric from (Mimno et al., 2011) to define entity coherence C, for an event, e; as
0.06 x [X], ... ..10.1 x [X]}. This
M m-1 Σ
Flush elt) +E F(oft)
Clei, Vol =
log Σ
where, 6 denotes an event, V denotes the top-10 TF- IDF based entities for e;. F( oft) indicates the co-document frequency (counts the joint document frequency for entity types Um, vt). F F(er) indicates document frequency for entity type tn and is a smoothing factor. Informally, it considers co-occurrences of central entity pairs (as opposed to topic words) in the clustered documents to measure coherency for an event. Note that a higher value indicates a highly coherent news event. By virtue of using log in formula, a value closer to zero is more desirable than largely negative value. We further observe that this measure is positively correlated with entity purity, indicating that purity can be a good measure to represent cluster coherence.
[uft "
For cluster incoherency check, we choose a threshold of 0.6. If the cosine similarity score between the event summary embedding and the document embedding is lower than this threshold, we discard those documents noise.
For our method's 's similarity module, we choose a threshold of 0.69 based on evaluating the trade-off between purity, coherence and coverage values.
Prior to computing the TF-IDF scores to retrieve the top-K entities, we simple yet effective method for entity linking (Ratinov al., 2011) I) that is based on Wikipedia mentions.
D Evaluation Metrics
In this section, describe the evaluation metrics proposed in our work.
In addition to these, we have an additional metric coverage, which essentially counts the number of documents accounted for in the clustering process. Ideally, we want any clustering algorithm to reject noise and cluster every document in the corpus. We do not want to exclude any document. Post noise removal, a good clustering algorithm is expected to have a coverage of 100% in an ideal scenario.
Several studies (Nanni et al., 2017; Spitz and Gertz, 2018; Chen et al., 2023) in the past have shown that entities and the context associated with them can potentially represent topic or event. With this as the premise, we have devised entity- based evaluation metrics that helps quantify the quality of the resulting clusters. We further vali- date our results through simple human evaluation process on partially annotated data for the issue Climate Change.
E Human Evaluation
For the event coherence case, the annotators are asked to verify if the top-3 documents for the event are in agreement with each other. They are asked to provide score of 1 if the documents are in agreement, a score 0 if they are not, or score of 1 if they are not sure about the label. We show only the title and first four lines of the news article. We did not receive any - for this case.
We define entity purity for an event to be the proportion of the documents that are mapped to that event, where the document has at least one entity that overlaps with the top-K TF-IDF based entities for that event (k 10). The idea is that central entities associated with a news event must be reflected in the documents clustered for that

---

To evaluate the mapping quality of our model, we randomly sample a set of peaks, and within each peak, we randomly sample 50 documents to form an overall set of 430 documents mapped to various events for the issue Climate Change. We show the title and first four lines of news article, and the event summary to the annotators. Similar to coherence case, ask the annotators to provide score of 1 if the document aligns with the summary, 0 for no alignment, or 1 they are not sure. There is no clear definition of alignment, and we let the annotators make this judgement. We received a total of 6 not sure labels. On eliminating unsure instances, our method got 352 out of 424 instances correct, which translates to precision value of R 0.83.
hoer
Moli
Coleee Stedlaity
Eiran
Opersikes
Operation
Caplid Insurection
SURSCAN
055487765
Minhod
HDBSCAN
122
Cheak Clenge
IDDSCAN
1LE052266
34
Free Sperch
Nakol
95
De Nathl
31
20
benigratio
Mickal
81900189
Gan Central
HDBSCANN
Nakel
Crtedial Jajastec Lave Inferecnntt
TIOS AX
0
73
Radal Kopait
SOROCAN
020611541
Minhol
(476641133
SDESCAN
4,39452209
Du Mickal
11
Dur Nokal
33
Table 5: Shows the impact of cluster merge and remove operations for each issue. Note that input data and hyper- parameters used by HDBSCAN in this setting are the same as our method. Lower the similarity score the better.
However, in order to compare the performance of our method with other model, we devise strat- egy to derive 'good' and 'bad' example-pairs by treating human-labeled data as the gold standard. We assume that if the two documents receive a score of 1 within the same event, then they must be 'qquivalent'.
With this assumption, for a given tempo- ral peak and within every event, construct "good/positivee example by considering every possible document-pairs from valid cluster assign- ments. To construct 'bad/negative' example set, we consider a union the following (a) Document- pairs from valid cluster assignments between dif- ferent events; (b) Document-pain pairs from an invalid, and valid cluster assignment within each event.
Bone
Aug. Finiity Perity
Rantyy Cubvrunce
Cipitd Itsumection
65.164
72.253
619 226
Concenvouus
$4.561
56.159
-762.147
Cliaa Charge
56.003
$4.509
119.816
Free Speech
78.124
Inmigratiin
67
76.275
618.273
Car Cornol
46 M4
18
422 161
Crininal Injusiue Law Enfoseenccl
38,701
$7.209
.61.00 951
Raacal Equil
41 985
K2.548
-3491000
Deferse and National Secorny
55 254
54
451.647
Cueraption:
79.943
The task is to evaluate how well each method per- forms in retaining the good example-pairs within the same cluster. We ensure to remove all the doc- uments that are not mapped to any event by each method. Owing to the nature of data collection, we report only the precision values for all the three methods under consideration.
Table Statistics for our proposed method after in- creased coverage with acceptable reduction in entity purity (=z -5 points on an average for all issues).
Re Pasidin Dru Chuasia menbies
F Results
Neu rowe then zomboe Chens srate
rnd fon fet (roe of Whim rusting Ao foll Erriiee fir
G Prompt Templates
clice fowd de E3s Somuamaeeeee
and
dioa of
and bria Kreed d Adtee bntive de Crupris and Auiriaa peral be in whire Aaumonos les losaoookk
This section shows the prompt templates used to generate
summary (Fig. 9), and to verify if a pair of cluster characterization is equiva- lent (Fig. 10).
Table 7: Illustrates an example where the cluster was removed due to the document being present in the top-5 list for this event. We see that the document talking about the same issue from a different frame and merely, using similarity module to identify cluster incoherency is not sufficient in this case.

---

Istne
Medel
Coverage
Aeg. Entty Purity
Ang Bnttey Coherence
Agg Exent Count
LDA (raadine))
99.781
36,055
-1027214
60
Tempoed Faering
27.867
1092
Capitd Insurvcttion
(lunpend)
37.120
-10158877
HDBSCAN (itnaklure)
77.964
54.155
-368,38
Mehhod
47.349
76821
.547.66
LBA (baseline)
17.855
1003.54
Tengoru Fatciig
8.9
1184.476
21
Cunueriias
LDA (Temponnl)
62.754
14.47
1130.409
83
HDSSCAN (sanchlonns)
55.555
34.45
-1004 468
64
51731
15.155
-901 467
Michod
41
$6.299
349,448
LDV (baseline)
42.439
583.566
Tempoei litreng
Climote Change
LDA (Tenpord'
90.59
39,906
957,887
HDBSCAN (stzddioone)
$4.01
763.008
BEXTipic
$3.995
67.635
689.429
65
Our Mehhod
45015
$1.838
4533923
56
LIIA (headinc)
19
21.285
-1000 102
60
Temporal Fitering
30039
1305.5
20
Free Speech
EJbK (lemponi)
93135
41441
-0022 918
HDBSCAN (rznkkuu))
53.175
65.337
572,847
BTXTopic
$3,649
704.514
Methoo
35.46
$7.964
439.136
LDA (baselise)
99.078
33,739
917.643
Tangond Fitering
16.01
-10458897
LDA (Temponil)
93
46161
914.519
48
HI HASCAN purndbiiss)
7608
16161
-995.99
57
BEKhgp
$5.655
71.85
733.281
42
Method
77.198
70.822
-94.95
LDA (rasciuc))
99.746
242233
-003.22
Temporal Fitering
24
1060.21
Immigration
(langond)
$7.848
34.72
300.808
HDBSCAN (etnnbiiine)
944
61818
-726400
86.59 519
67.64
413.325
Our Method
53.944
80.07
<535.755
48
LDA (hsoeline)
606
26.003
-1009.5
60
Teuporul Faaring
35.
903.333
Gun Contrd
Clamponll
42.544
-055088
HDBSCAN bitasbluss)
91,494
$7.047
.50.78
IVTCCopic
Mithod
16124
-323
LDA (baseline)
99.35
40.422
996.408
Tomporal Fitrring
153
1005 909
Criminal Enjosties
Law Enforcement
LDA (Tempand)
96.648
45.
1027.712
66
HURSCAN (unddinns))
$7.968
67.118
-06.3117
68
BEKTapic
58.725
67.105
-756.799
76
Mirhod
94.944
4603682
LIAA (haudinc)
99.79
31.229
-1073
Temponal Piteeing
1109.5
Rocial Equity
(lomponi)
HDBSCAN (stadbbo)))
63.346
$11.005
BEXTipic
$5.394
56.644
-747669
75
Our Meslaod
33.206
89082
309.184
68
1J1 (houtino)
99951
18
.400366
60
langond Fitering
25.312
-1006 041
24
Defense & National Security
LBN (Tempon))
91.09
40003
1000 138
HSSCAN punniiins)
919
BEXToplc
74.519
617.425
Mesleod
-38.2291
LDA (baxdius)
99.572
34,557
10233875
Torpond Filzring
10965
-00.232 222
Corraptton
LDA (lanpond)
93.38
40.95
490.941
34
HUSSCAN (unddiim)))
263
262
663.4
36
BEKhogic
$2.
73.365
615.773
50
Our Mehood
45.233
$7.577
447.75
Table 8: Compares the results obtained for each method and issue. Last column shows summation of all event counts (from each detected temporal peak). For LDA (emmoopo,,,), we assigned the document to its most probable topic if the probability was 0.5.

---

You need to provide title and sentence long deseription for the news event based on news article snippets shown below. The title and description should not be too specific the articles shown below but rather, they need to focus on the main event.
News Article1: **Titl*** *Description**
News Article2: **iitl*** *Description**
News Event Title: Response*** News Event Description: ** **Reppons****
News Articlel: **Title** *Description**
News Article2: **Title** *Description*** **
Table Prompt template for multi-document summary generation (shown one-shot).
You need tell if the following two news event descriptions belong to the same news event. You need say yes or no and nothing more.
News Event Titlel: **Titl***
*Description***
News Event Title2: **Title** *Description**
Answer:
Table Prompt template check for entailment (shown as zero-shot).


Reviews:
[{"review_id": "BfdAg4FhzW", "reviewer": "Reviewer_RPBd", "summary": "", "questions": "(a)  Why only HDBSCAN is used for clustering?\n(b)  Why other methods of dimensionality reduction except UMAP not used?\n(c)   Why only NELA dataset is used?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n1. This paper could be improved by exploring recent state of the art topic modelling models such as TopicBert can be explored.\n\nThanks for suggesting that we compare BERTopic (SBERT + UMAP + HDBSCAN) as an added baseline. We have compared our method against it, and will be reporting full results in the final version of the paper. In fact, keeping the temporal clustering aside, our current baseline - HDBSCAN (standalone), can be considered as a reimplementation of BERTopic. This is because we first obtain a distributed representation of documents, then reduce the dimensionality using UMAP, and finally, apply HDBSCAN algorithm to retrieve news events.\n\nWe found that off-the-shelf BERTopic outperformed HDBSCAN (standalone) baseline in terms of coverage (+3 points), and entity purity metrics (+3.5 points). The reason for this variation can be primarily attributed to the change in hyperparameters (in UMAP and HDBSCAN algorithm) underlying the readily available implementation of BERTopic. This change in parameters led the stochastic clustering process to create more events (+4 events on average across all issues), and help reduce the noise in event clusters. \n\nOur proposed method outperforms both BERTopic and HDBSCAN baselines by a sizable margin. The most notable improvements from our methods are – (1) our method produces a lesser number of events overall due to the merge/remove operations on event candidates. The increase in the purity metric (+11 points vs. BERTopic) can be directly attributed to our additional inference mechanism that is solely based on generated event summaries. This inference mechanism is absent in clustering approaches used in baselines.\n\nWe have also conducted human evaluation on results from BERTopic baseline, and will be added as part of Table 4 in the paper. Event Coherence and Mapping Quality scores from BERTopic increased by +0.5 points and +7.5 points respectively, as compared to baseline - HDBSCAN (standalone). The scores from BERTopic are still lesser than our method by -5.5 points and -2.3 points for event coherence and mapping quality, respectively. After combining these observations along with the reduced overall purity scores from BERTopic, we can conclude that events produced by BERTopic are still prone to noise compared to our method. \n\n\n\n2. Why only HDBSCAN is used for clustering?\n\nThe idea is to use a density-based estimate such as HDBSCAN to retrieve an event representation by estimating areas of highly similar documents in the semantic space. Further, given that this is a short paper, it was out-of-scope for us to compare various clustering approaches. \n\n\n\n3. Why other methods of dimensionality reduction except UMAP not used?\n\nWe have tried using other popular dimensionality reduction techniques such as t-SNE, and PCA. However, we chose to go with UMAP as it preserves both local and global structure in the data, which makes it very useful for us to compare inter-cluster distances. Furthermore, it is known that UMAP scales well for large datasets in comparison to t-SNE, which incentivized us to use UMAP for our use case.\n\n\n\n4. Why only NELA dataset is used?\n\nWe have used NELA dataset-2021 as it is an accumulation of a large collection of news articles (~1.8 Million) from 367 different news sources for the year 2021.  We have leveraged it to illustrate our conceptual idea of using LLMs as an inference engine to retrieve better clustering results. However, in practice we can leverage this approach to identify themes in any large corpus of news articles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work seems to be novel and the first of its kind that generates fine grained and consistent clusters of news articles. The objective is well motivated and experiments seem to justify the author's claims. The authors have used different metrics like purity, coverage and also human evaluation to benchmark their experiments. Different baselines have been compared with and the proposed method seems to outperform all baselines. The dataset 'KEYEVENTS' proposed by the authors would help the research community to steer research in social media news analysis.", "reasons_to_reject": "This paper could be improved by exploring recent state of the art topic modelling models such as TopicBert can be explored.", "questions_to_authors": "(a)  Why only HDBSCAN is used for clustering?\n(b)  Why other methods of dimensionality reduction except UMAP not used?\n(c)   Why only NELA dataset is used?", "paper_topic_and_main_contributions": "This paper is about generating key event aggregation from news articles, specifically NELA. The authors try to cluster different news articles based on specific topics for better analysis of social media news understanding and propagation. The authors have used GPT 3.5 with prompt assistance to further overcome the challenge of under-specificity of social media news articles. They use different merge and remove operations to generate consistent clusters, ultimately mapping articles to events.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IgyvZDjpfu", "reviewer": "Reviewer_DScZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n1. The motivation in this article is not elaborated, the most interesting is the peak, but with a relevant simple operation.\n\nOur goal is to retrieve major news events from a large corpus of news articles with minimum human intervention. The event identification information derived from temporal clustering alone is very ambiguous. This is because of the presence of multiple news events that are often conflated around every temporal peak. \n\nOur main challenge evolves to come up with a principled approach to identify news events at every peak. To tackle this problem, we propose a framework to effectively cluster the news articles into various events at each peak by using LLM for inference purposes. Specifically, we propose to use LLM to characterize and explain the identified event candidates, and then clean up the candidate events through merge/remove operations. The generated event explanations of the final set of clusters are utilized to decide what documents are mapped to each event cluster.\n\n\n2. It is not clear how event clustering is implemented. What is the form of events contained in a peak, what is the connection with the passage, and how to obtain the summary, it is recommended to reorganize. \n\n- Event Clustering (Main idea): Post application of temporal clustering, we use HDBSCAN algorithm over a distributed representation of documents to identify event candidates. We prompt LLM to generate an explanation for each event candidate. This explanation is utilized to purify candidate events through remove/merge operations. The resulting output after purification of event candidates denotes the final set of event clusters. LLM-generated explanations associated with the final set of event clusters are used to retrieve news articles related to each news event.\n\n- Form of events: Intuitively, this can be thought of as a theme extraction process. We have a corpus of news articles at different points in time (peak). At each peak, we are to identify various points of discussion that constitute a major chunk of news articles, which we coin as “news events”. Each news event has a title and a brief description that tells what the event is about, and each event is associated with a collection of news articles.\n\n- Connection with passage and Obtaining Summary: To obtain an event summary, the identified event candidates from HDBSCAN algorithm are characterized using LLM. Specifically, we retrieve the top-3 representative news articles from each event candidate based on cosine similarity score between event centroid and document embeddings for that event. To obtain an event summary, we prompt the LLM with the news article title and first four lines in a few-shot manner to generate a short explanation about that news event.\n\n\n3. The evaluation indicators can be explained in more detail. This article uses multiple evaluation indicators, how they are defined, and what they represent. I hope they can be supplemented. \n\nWe can certainly add more details about the evaluation metrics in the appendix section of the final draft. This can perhaps help in explaining them better. We provide a brief overview here.\n\n- Broadly, we have two kinds of evaluation - human evaluation, and other measures that are based on traditional clustering evaluation metrics. \n\n- Human Evaluation: We have evaluation indicators - Event Coherence and Mapping Quality that are based on human judgements, and can be found in Table 4 in the paper. Event Coherence for an event is measured by checking if the top-3 documents from each event cluster are in agreement with each other, whereas Mapping Quality models if the contents of the news article align with the event summary. Note that there is no clear definition of alignment and we let the annotators make this judgement.\n\n- Other measures: The other indicators are based on traditional clustering-related evaluation metrics that considers coherence for an event (Table 3) and separability across different clusters (Table 6). For instance, to quantify the coherence of the identified events, we propose entity purity. The motivation underlying this metric is that central entities associated with a news event must be repeated in the documents associated with that event. A purity score of 100% for an event indicates that every document in the cluster has atleast a mention of one of the top-K central entities, suggesting that each document is potentially discussing that event.\n\n\n4. The image layout in the appendix is confusing.\n\nThank you. We will fix this issue in the final draft of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An efficient method is proposed for a novel task, and experimental results show that it is computationally inexpensive and produces more interpretable coherent clusters of key events.", "reasons_to_reject": "The motivation in this article is not elaborated, the most interesting is the peak, but with a relevant simple operation.\n\nIt is not clear how event clustering is implemented. What is the form of events contained in a peak, what is the connection with the passage, and how to obtain the summary, it is recommended to reorganize. \n\nThe evaluation indicators can be explained in more detail. This article uses multiple evaluation indicators, how they are defined, and what they represent. I hope they can be supplemented. \n\nThe image layout in the appendix is confusing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "this paper studies how to identify key events and news articles related to these events from the news flow. In this paper, a general framework of news stream clustering is proposed, which makes use of the time trend of news. LLM generates event summaries to describe such key events, and on this basis, document clusters are formed in an unsupervised way. The framework is effective after a series of evaluations and shows that it produces a more coherent event cluster.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xkw7AqQdn5", "reviewer": "Reviewer_9es6", "summary": "", "questions": "Less of a question and more of a comment:\nIn the discussion of limitations, you argue for using GPT 3.5 and against open models, which rather feels like a statement made out of convenience. Yes, using a nicely curated OpenAI API is easy, but it has inherent limitations in applicability, suffers from privacy cocerns, and offers no control over the model. Adding support for an open model would not only allow further and more in-depth experiments, but also make the final produc more accessible.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n1. Limited novelty The proposed approach is a effectively a combination of existing tools and well-established ideas. Novelty is only in the engineering / data, not in the methods.\n\nWhile clustering approaches - probabilistic or neural methods is an established idea, the resulting clusters are still far from perfect. Our main goal and novelty in this paper is to come up with a principled way of using LLM’s capability as an inference engine to help improve the overall cluster quality.\n\nPrecisely, post temporal filtering, we identify the event candidates by clustering the documents using HDBSCAN algorithm. Each event candidate is characterized with a short explanation using LLM. This characterization is utilized to remove incoherent events, and also to merge similar events descriptions together. In the inference stage, we use the LLM-generated event characterization to map the documents to the event cluster. \n\nAs far as our knowledge is concerned, we do not know of anybody who has used LLM’s inference capability in this manner to generate coherent news event clusters. Our empirical results show the potential in our approach. \n\n\n2. Limited evaluation The used baselines are quite outdated, to the point where the comparison is unfair (LDA is 20 years old at this point). It would be good to see the relative performance of more recent neural topic models or best-effort adaptations of LLMs to the task.\n\nThanks for suggesting that we compare our proposed approach against recent neural topic models. We have compared our method against BERTopic, and will be reporting full results in the final version of the paper. In fact, keeping the temporal clustering aside, our current baseline - HDBSCAN (standalone), can be considered as a reimplementation of BERTopic. This is because we first obtain a distributed representation of documents, then reduce the dimensionality using UMAP, and finally, apply HDBSCAN algorithm to retrieve news events.\n\nOn using the ‘off-the-shelf’ model - BERTopic for key event identification, we found that it outperformed HDBSCAN (standalone) baseline in terms of coverage, and entity purity metrics. Coverage increased by +3 points, and entity purity increased by +3.5 points. The reason for this variation can be primarily attributed to the change in hyperparameters (in UMAP and HDBSCAN algorithm) underlying the readily available implementation of BERTopic. This change in parameters led the stochastic clustering process to create more events (+4 events on average across all issues), and help reduce the noise in event clusters. \n\nHowever, our method produced a lesser number of events overall due to the merge/remove operations on event candidates, and also, entity purity scores are atleast +11 points higher than BERTopic. The increase in the purity metric can be directly attributed to our additional inference mechanism that is solely based on generated event summaries. This inference mechanism is absent in clustering approaches used in baselines.\n\nWe have also conducted human evaluation on results from BERTopic baseline, and will be added as part of Table 4 in the paper. Event Coherence and Mapping Quality scores from BERTopic increased by +0.5 points and +7.5 points respectively, as compared to baseline - HDBSCAN (standalone). The scores from BERTopic are still lesser than our method by -5.5 points and -2.3 points for event coherence and mapping quality, respectively. After combining these observations along with the reduced overall purity scores from BERTopic, we can conclude that events produced by BERTopic are still prone to noise compared to our method. \n\n\n3. Less of a question and more of a comment: In the discussion of limitations, you argue for using GPT 3.5 and against open models, which rather feels like a statement made out of convenience. Yes, using a nicely curated OpenAI API is easy, but it has inherent limitations in applicability, suffers from privacy cocerns, and offers no control over the model. Adding support for an open model would not only allow further and more in-depth experiments, but also make the final produc more accessible.\n\nWe agree with your point. It is indeed a statement made out of convenience – With the limited computational resource we had for this project, it was more difficult to setup an open-source model (e.g. LLaMA-7B) compared to using GPT3.5 (which didn’t cost much in our experiments) for inference. In the final draft, we will try to add an ablation study for using open-source LLMs, just so we understand how much performance we will lose/gain when switching to open-source counterparts.\n\n\n4. Typos\n\nThanks for pointing this out. We will fix them in the final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Presentation\nThe paper is well written and the proposed approach is presented clearly, with extensive information regarding the implementation being provided in the appendix. \n\n* Usefulness\nWhile the proposed pipeline is of limited novelty and based on the combination of existing solutions and concepts, the final product is of good quality and could be useful to the community. Providing a useable code repository / package would be even more beneficial.\n\n* Data \nThe authors provide a data set alongside their experiments.", "reasons_to_reject": "* Limited novelty\nThe proposed approach is a effectively a combination of existing tools and well-established ideas. Novelty is only in the engineering / data, not in the methods.\n\n* Limited evaluation\nThe used baselines are quite outdated, to the point where the comparison is unfair (LDA is 20 years old at this point). It would be good to see the relative performance of more recent neural topic models or best-effort adaptations of LLMs to the task.", "questions_to_authors": "Less of a question and more of a comment:\nIn the discussion of limitations, you argue for using GPT 3.5 and against open models, which rather feels like a statement made out of convenience. Yes, using a nicely curated OpenAI API is easy, but it has inherent limitations in applicability, suffers from privacy cocerns, and offers no control over the model. Adding support for an open model would not only allow further and more in-depth experiments, but also make the final produc more accessible.", "paper_topic_and_main_contributions": "The paper considers the task of topical event extraction from news, for which the authors propose a a neural pipeline that combines large language models with outlier detection. The proposed approach is evaluated against topic models as a baseline.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is very well written and enjoyable to read with a nice overall presentation. Great job!\n\nTypos:\n* Line 244: \"We observe that average\"\n* Line 255: \"results in on average\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to cluster news articles into key events using a large language model (LLM) and outlier detection. The paper uses the NELA dataset of news articles from various sources and domains. The paper first uses the LLM to generate event summaries from the articles and then remove irrelevant or noisy articles from the clusters. The authors evaluate the proposed method against topic models as baselines and shows that it produces more coherent and informative event clusters.

I thank the reviewers for providing very constructive feedback and the authors for addressing concerns by executing additional experiments. Especially, the inclusion of BERTopic as an additional baseline is commendable. I would encourage the authors to consider other suggestions such as using another LLM such as Llama2 in addition to GPT3.5 model currently used.
Decision: Accept-Findings