Comprehensive Abstractive Comment Summarization with Dynamic Clustering and Chain of Thought (2024)

Abstract: Real-world news comments pose a significant challenge due to their noisy and ambiguous nature, which complicates their modeling for clustering and summarization tasks. Most previous research has predominantly focused on extractive summarization methods within specific constraints. This paper concentrates on Clustering and Abstractive Summarization of online news Comments (CASC). First, we introduce an enhanced fast clustering algorithm that maintains a dynamic similarity threshold to ensure the high density of each comment cluster being built. Moreover, we pioneer the exploration of tuning Large Language Models (LLMs) through a chain-of-thought strategy to generate summaries for each comment cluster. On the other hand, a notable challenge in CASC research is the scarcity of evaluation data. To address this problem, we design an annotation scheme and contribute a manual test suite tailored for CASC. Experimental results on the test suite demonstrate the effectiveness of our improvements to the baseline methods. In addition, the quantitative and qualitative analyses illustrate the adaptability of our approach to real-world news comment scenarios.

heading: 1 Introduction
text: With the continued development of handheld devices and the boom of social media, the proliferation of online news is swift and relentless. This has fueled heightened interest among researchers on the topic of public opinion analysis, including tasks like opinion mining (Barker et al., 2016a; Pecar, 2018; Gao et al., 2019; Yang et al., 2019; Huang et al., 2023), sensitive comments detection (Pavlopoulos et al., 2017; Chowdhury et al., 2020; Moldovan et al., 2022; Sousa and Pardo, 2022), and news comments summarization (Khabiri et al., 2011; Dalal and Zaveri,
∗Contribution during the internship at Institute for Infocomm Research.
2013; Aker et al., 2016; Žagar and Robnik-Šikonja, 2021). This study concentrates on the Clustering and Abstractive Summarization of online news Comments (CASC). While the past decade has witnessed substantial progress in text summarization across diverse frameworks (Rush et al., 2015; Liu et al., 2015; Yasunaga et al., 2017; See et al., 2017; Lewis et al., 2020; Wang et al., 2021, 2022; Ouyang et al., 2022; OpenAI, 2023), research specifically targeting CASC remains scarce.
Comment summarization involves extracting a collection of comments either by selecting sentences directly from the comments, referred to as extractive comment summarization, or by compressing the original comments through generative techniques for abstractive summarization. While previous studies have explored a methodology of initially clustering comments into distinct groups followed by summarization (Khabiri et al., 2011; Ma et al., 2012; Llewellyn et al., 2016; Barker et al., 2016b; Pecar, 2018; Žagar and Robnik-Šikonja, 2021), these endeavors have focused on extractive summarization. Our investigation of Reddit comments reveals a spectrum of characteristics influenced by context, platform, and user demographics. In particular, these comments exhibit diversity in style, length, level of detail, and relevance to the contextual discussion, posing significant challenges for summarization using extractive approaches.
In this status quo, this work focuses on abstractive comment summarization with the goal of providing well-organized and extensive coverage of user comments across diverse topics. We confront two main challenges: Firstly, the CASC task, in its nascent phase, suffers from a lack of a formal task definition, an open-source evaluation scheme, and accessible data. This absence largely hinders the expeditious and streamlined evaluation of diverse methods, models, or algorithms. Secondly, although recent Large Language Models (LLMs) have achieved impressive performance in text sum-
2884
marization (Zhang et al., 2022; Chung et al., 2022; Touvron et al., 2023a; Ouyang et al., 2022; OpenAI, 2023), there are still limitations when applying such LLMs to CASC. On the one hand, online comments on trending topics can be lengthy, surpassing the strict input sequence length constraints of existing foundational models. On the other hand, while existing LLMs excel as text generators, their efficacy in clustering remains underdeveloped.
In this paper, we propose a novel task on abstractive news comment summarization to address the aforementioned challenges. Our contributions encompass three points:
• This paper introduces a straightforward yet effective comment clustering algorithm, coupled with a task-specific ranking mechanism, to dynamically adjust the similarity threshold for comment clustering and rank candidate clusters based on both their density and topical relevance. This method ensures that each comment cluster yields high compactness and discernibleness.
• Based on the clustering algorithm, we propose harnessing moderate-scale LLMs for comment cluster summarization. Particularly, we fine-tune the model through a chain-of-thought manner to first predict aspect terms (ATs) for each comment cluster and then perform AT-based summarization.
• In particular, this work contributes the first annotation scheme and manual test suite tailored for CASC. Notably, different from previous works on topic-based clustering (Khabiri et al., 2011; Ma et al., 2012), we introduce the concept of aspect-based comment clusters to restrict the granularity of each cluster quantitatively.
Quantitative experiments and qualitative analysis of the proposed CASC system demonstrate the effectiveness of our improvements over the baseline systems. Moreover, a real-scenario case study further indicates the practicability of our approach.

heading: 2 Related Work
text: Comment cluster summarization. Previous research on comment cluster summarization can be categorized into three main groups: some solely study comment clustering (Aker et al., 2016; Llewellyn et al., 2016), others focus on summarizing comment clusters in different forms (Dalal and Zaveri, 2013; Barker et al., 2016a; Gao et al.,
2019; Huang et al., 2023), while a third group explores both clustering and summarization of news comments (Khabiri et al., 2011; Hsu et al., 2011; Llewellyn et al., 2014; Barker et al., 2016b; Žagar and Robnik-Šikonja, 2021). Our study falls within the latter category that emphasizes both clustering and summarization of comments.
Previous studies of (Khabiri et al., 2011; Ma et al., 2012) are the first to explore extractive summarization of comment clusters, which first group comments into clusters under different topics and then extract comments from each cluster to form the summary. Hsu et al. (2011) alternatively proposed a hierarchical clustering method to summarize YouTube comments based on comment term normalization and key term extraction. Subsequently, Llewellyn et al. (2014) conducted a comprehensive comparison of various clustering methods, including topic-driven clustering, keywordbased clustering, k-means, and cosine distance, demonstrating the superior performance of topicdriven clustering in producing comment clusters suitable for extractive summarization.
However, the above topic-oriented approaches often assume that the resulting summary, derived from topically clustered and extracted comments, aligns with user preferences and presents the crux of the discussed matters. Such assumptions may not always hold. Recognizing this problem, Barker et al. (2016b) conducted a series of data analyses and argued that a good comment summary should not only convey the central issues discussed but also encapsulate the opinions surrounding those issues, suggesting that a good summary should cover finer-grained information beyond just topics. In order to capture more informative comment representations, Žagar and Robnik-Šikonja (2021) proposed to utilize neural sentence embedding methods for extractive summarization of comment clusters. Despite these efforts have been made in comment cluster summarization, all previous works concentrated on extractive summarization, while the abstractive CASC task is largely unexplored. In this paper, we aim to conduct an in-depth study of CASC to fill the gap in this field.
LLM-based summarization. The recent popular LLMs (Zhang et al., 2022; Chung et al., 2022; Touvron et al., 2023a; Ouyang et al., 2022; OpenAI, 2023) have achieved impressive performance in many NLP tasks including text summarization. Notably, the recent paper (Pu et al., 2023) argues
that “summarization is almost dead”. In this LLM era, our stance is that summarizing large-scale online user comments remains a challenge even for state-of-the-art (SoTA) foundation models for two reasons: (1) The strict input sequence length limitation of current foundation models makes it infeasible to take all comments as inputs for CASC; (2) Current LLMs are usually pre-trained on text generation tasks, their clustering abilities are far from perfect. Moreover, the quality of text clustering is intricately related to user preference, which is hard to describe using a textual prompt. Our work contributes to building a CASC test suite with userpreferred summaries, which aligns with the new horizons discussed in (Pu et al., 2023).

heading: 3 CASC Test Suite
text: This section introduces the proposed annotation scheme coupled with the manually built test suite for CASC evaluation. Diverging from conventional comment clustering works where the comments are clustered based on the coarse-grained topics, this study focuses more on the finer aspect terms within each news comment. In this paper, we introduce a concept of Comment Aspect Term (CAT), which refers to primary objects of the entire discourse, either extracted or abstracted to represent the central focus. The aspect terms serve as the focal points of explicit or implicit opinion expression within the comment. For example, in the following comment, The current Russian government doesn’t represent us, and we definitely don’t want a war with our closest people., the CATs are determined as the current Russian government and the war, and the comment is expressing disapproval towards the CATs.
With the above background, we formulate the CASC annotation process in three stages: comment aspect term annotation, comment cluster construction, and comment cluster summarization, as detailed below. CAT annotation. The process of annotating CAT presents two challenges: (1) Since aspect terms can be either extracted or abstracted from the original comment, it is difficult to control the granularity of each annotated CAT; (2) User comments usually suffer from unclear reference caused by ellipsis or co-reference, which is hard to avoid when reading each comment independently. The two issues may essentially reduce the annotation consistency.
In this work, we allow the case that each comment has multiple CATs, but we impose a quan-
titative limit of fewer than 10 words to control granularity. Moreover, to better cover the meaning of each comment, the annotators are instructed to read comments chronologically and refer to the pre-context when labeling aspect terms. The scope of permissible context for reference is limited to a single comment tree (the comment-reply structure). To ensure the quality of annotations and discern optimal candidates, we establish the following rules: (1) Preference is given to CATs directly extracted from the original comment rather than artificially induced ones; (2) When CATs convey the same meaning, we prefer using the same textual expression; (3) Aspect terms are not labeled for comments without opinion expression or those devoid of practical meaning.
As aspect term annotation inherently involves subjective judgment from annotators, we estimate the consistency of CAT annotation by comparing the annotations of 200 news comments between two annotators. We approximate consistency by calculating the word-level coverage between each pair of summaries, resulting in a score of 43.67%.1
Comment cluster annotation. Manually clustering a large volume of comments from scratch, depending on the limited memory of human annotators, is time-consuming and challenging, fraught with difficulties in ensuring annotation quality. As stated before, we study clustering comments based on aspect terms. Specifically, we consider semantic affinity inferred from the frequency of word co-occurrence between CATs as a reliable indicator of similarity. First, we represent each comment by averaging the GloVe vectors within the manually annotated aspect terms. Then, the comments are clustered based on the cosine distance between aspect term representations through fast clustering with a high similarity threshold of 0.85. Comments lacking manual aspect terms are assigned to a single unique cluster, labeled as Trivial. Lastly, annotators manually check the correctness of these clusters according to the following guidelines: (1) Given a cluster G = {c1, . . . , cN} with N comments, if the aspect terms and content of ck significantly differ from others in the cluster, exclude ck from G and assign it to a suitable comment cluster or the Trivial group. (2) If cluster Gi shares a
1Given two manually annotated aspect terms after removing the stop words and reducing repetitions, if the word coverage between the two CATs higher than 30% of the averaged word number, we take the two annotated aspect terms as successful coverage.
closely related topic with cluster Gj , combine them into a unified cluster. (3) Check comments annotated with aspect terms in the Trivial group to determine if they can be reallocated to an existing non-Trivial cluster. Comment summary annotation. Text summarization annotation has long been contentious, primarily due to the inherent subjectivity of annotators, which can easily influence the standard of summary annotation. Recent studies (Goyal et al., 2022; Zhang et al., 2023; Yang et al., 2023; Pu et al., 2023) have demonstrated that LLM summarizers can perform on par with or better than human summarizers. Among them, the research in (Pu et al., 2023) shows that when quantitatively and qualitatively assessed against human annotators, LLM summaries with increased factuality are notably favored by human evaluators. Inspired by this, we transition from manual summary annotation to LLM-assisted summary annotation. Specifically, based on the manually adjusted comment clusters, we first use GPT3.5 to generate an LLM summary for each comment cluster. Then, we manually postedit the summaries and refine those suffering from hallucinations to produce the final test suite. Annotated data overview. Based on comments of news articles sourced from Reddit and New York Times 2017,2 our annotated dataset contains 11 article clusters (ACs) covering various topics, which involve a total of 20 news articles and 7,958 comments. Guided by our annotation guidelines and with the assistance of four annotators, we constructed 848 comment clusters (CCs) along with their corresponding summaries. Further details are shown in Table 1.

heading: 4 Approach
text: Task definition. Given a group of m news articles under the same topic, A = {a1, . . . , am}, the articles contain a group of n comments in total, C = {c1, . . . , cn}, which comment on the articles from various aspects. Each comment yields a flexible length distribution ranging from a short sentence to a paragraph. The CASC task first groups
2https://www.kaggle.com/datasets/ aashita/nyt-comments
the n news comments into a set of k comment clusters, G1, . . . , Gk, and each cluster represents a hot topic discussed by the netizens. Subsequently, a summary is abstracted for each comment cluster represented by S1, . . . , Sk.

heading: 4.1 Comment Clustering
text: Baseline method. We employ the fast clustering algorithm (Reimers and Gurevych, 2019)3 as our baseline approach for comment clustering, which offers advantages over classical clustering techniques such as K-Means and Affinity Propagation and provides faster processing times while delivering superior performance. Operating on a cosinesimilarity threshold coupled with SBERT sentence representation, it retrieves and ranks local communities with highly similar sentences greedily. Tailored for CASC, this paper seeks improvements to the baseline method from the following two aspects for more robust comment clustering. Dynamic comment clustering. Our investigation reveals that comment clusters with hot topics tend to attract more user comments, yielding high comment densities, while those tackling less popular topics exhibit relatively sparse clusters. However, the baseline algorithm maintains a static similarity threshold throughout clustering, which struggles to adapt to fluctuating distributions of comment cluster densities. This paper argues that an adaptive dynamic threshold according to the scale of each comment cluster is more practical. Algorithm 1 shows our core clustering algorithm in pseudocode. As cluster scale increases (+∆), the similarity threshold increases accordingly, resulting in a higher cluster density. Drawing inspiration from increasing the similarity threshold in the early stage of dynamic clustering to ensure denser comment clusters, this paper proposes a dynamic similarity threshold adjustment process based on a concave function. Formally, the algorithm automatically adjusts the threshold by fitting the following function:
ςt = √ K1×(κt + K2) (1)
where κt denotes the cluster size, ςt denotes the similarity threshold, and K1 and K2 refer to the hyper-parameters to tune on the DEV set. Cluster ranking. The vanilla algorithm maintains a static similarity threshold to ensure that the distance between comments within each cluster is
3https://www.sbert.net/examples/ applications/clustering/README.html
below the threshold. However, this algorithm may still lead to redundancy as some comments may be shared across different clusters. Then, the clusters are ranked with the redundant comments filtered out for final clustering. The baseline method ranks candidate clusters solely based on cluster size, which is not comprehensive enough. This limitation is particularly evident in CASC, where user preference is more focused on both topic relevancy and cluster compactness. Different from the baseline method, we argue that cluster density and relevance between the cluster and article background4 are vital for CASC. With this motivation, we introduce a new ranking mechanism as:
Rt = Σ n i cosine(ci, ct)×max(log(ξt + 1), 0.1)
where ct denotes the centroid comment of the t-th cluster, ξt denotes the stem word intersection between the centroid comment and the background, and Rt refers to the resulting ranking score. Notably, the first term in the formula can be regarded as the product of the cluster size and the cluster density. Therefore, our ranking scheme considers the cluster size, density, and the correlation between the cluster and background concurrently.

heading: 4.2 Summarization of Comment Clusters
text: With the rapid development of foundation models (Zhang et al., 2022; Chung et al., 2022; Touvron et al., 2023a; Ouyang et al., 2022; OpenAI, 2023), researchers increasingly recognize the effectiveness of LLMs across various NLP tasks. With this inspiration, we study adapting the compact yet effective Flan_T5_XL (3B) (abbr. FTX) as a benchmark summarization system to fill the gap of foundation models in CASC. Instruction-tuning. In line with our human annotation efforts, we aim to enhance the foundation model’s capability to summarize each comment cluster by focusing on the aspect terms central to each cluster. With this goal, we design instruction-tuning data in a chain-of-thought style to tune LLMs to extract aspect terms from a comment cluster and then abstract the summary by considering the extracted aspect terms and comment texts. However, manual annotation of such tuning
4Cluster density refers to the average cosine distance from each data point to the centroid comment of the cluster. In the outer loop of Algorithm 1, ct is the centroid comment of round t. The background is established by randomly sampling six titles from the article cluster where the comments are located.
Algorithm 1 Dynamic Fast Clustering Input: N vectorized news comments C Output: A list of comment clusters G Initialization: initial cluster size: κ, initial threshold: ς , ceiling threshold: thrM=0.9 Begin for t-th comment ct in C do
scores← pairwise_cos_sim(ct, C) scores-k← scores.top-k(κ) ς ′ ← ς while scores-k[-1] > ς ′ and κ < N do κ←Min(N , κ+∆) ς ′ ←Min( √ K1×(κ+ K2), thrM )
scores-k← scores.top-k(κ) end while Gt ← [] for si in scores-k do Gt ← Gt ∪ {ci} when si ≥ ς ′ end for G← G ∪ {Gt} Calculate the ranking score Rt
end for Rank the N clusters in G based on R1...N End
data is labor-intensive and time-consuming. Besides, recent research suggests that current SoTA foundation models perform annotation on par with or better than human (Goyal et al., 2022; Zhang et al., 2023; Yang et al., 2023; Pu et al., 2023). Inspired by this, we automatically build comment clusters from Reddit comments (excluding those in the test suite) and utilize the publicly available gpt-3.5-turbo to build the instruction-tuning data. In particular, to strengthen the model’s capability in chained aspect term and summary generation, we transform each tuning instance into three forms with hybrid learning goals: (1) Given clustered news comments, tune the LLM model to predict aspect terms for each comment cluster; (2) Given the comment cluster along with the aspect terms that the cluster is centered on, tune the model to generate a summary for the cluster; (3) Given the comment cluster, tune the model to predict aspect terms first and then generate a summary based on the predicted aspect terms and the comments within the comment cluster. In total, we obtain 26,438 tuning instances, consisting of 25,438 instances for tuning and 1,000 for validation. Hallucination alleviation. Upon manual examination, it was discovered that 28% of the com-
ment cluster summaries generated by our finetuned LLM model suffer from hallucinations. This phenomenon arises primarily due to the model’s inability to grasp the context of the original articles being commented on. To solve this issue, we harness the in-context learning capability of the pretrained LLM and include background information for comment cluster ranking (see Subsection 4.1) as part of the prompt.

heading: 5 Experiments

heading: 5.1 Evaluation Metrics
text: This research evaluates CASC from three aspects: comment clustering, comment cluster summarization, and pipeline evaluation. Clustering. This work considers both the homogeneity and completeness levels between the gold and predicted clusters by reporting the Normalised Mutual Information (NMI) score 2×I(C;L)H(C)+H(L) as performance, where C refers to the class labels, L refers to the cluster labels, H(·) is the entropy function, and I(C;L) denotes the mutual information between the two group of labels C and L. Summarization. For comment cluster summarization evaluation, we take the gold standard comment clusters as inputs to generate summaries. Afterward, we calculate the F1-based Rouge-1, -2, and -L scores (ROUGE-1.5.5) between the generated summaries and the ground truth as performance. Pipeline evaluation. In the pipeline system, the automatic clustering process could result in mismatches between predicted and standard clusters, making the pipeline evaluation hard to realize. This work introduces a scheme for CASC evaluation. Initially, we define a correct hit5 as when a predicted summary closely matches the ground truth with a cosine similarity exceeding 0.9 (see examples in Appendix C). On this basis, we report precision (P), recall (R), and F1 scores of the correct hits as the CASC performance.6

heading: 5.2 Clustering Results
text: For comment clustering, we compare our approach with the following four baseline methods.
5We utilize INSTRUCTOR (Su et al., 2023) for summary representation. It is an embedding model fine-tuned on 330 tasks through instruction-based methods. This model can generate task-aware embeddings by considering both task instructions and input text. To maintain consistency, we follow the original paper to use the instruction “Represent the statement:” to represent each summary for similarity calculation.
6For details regarding system settings, please refer to Appendices A, B, and C.
• K-Means (MacQueen et al., 1967) is a traditional approach that alternatively moves comments to the nearest comment cluster centers and updates these centers accordingly. The value of K, representing the number of clusters, needs to be manually determined.
• Affinity Propagation (Frey and Dueck, 2007) is a clustering algorithm that identifies exemplars or representatives within a set of data points. The algorithm can automatically determine the number of clusters based on the input data.
• Topic-focused clustering (Llewellyn et al., 2014) is one of the most representative works on comment clustering, which comprehensively proves that LDA topic-driven clustering produces better comment clusters for extractive summarization.
• Fast clustering (Reimers and Gurevych, 2019) is the benchmark system that our dynamic comment clustering algorithm sources from. It utilizes a cosine-similarity threshold in conjunction with SBERT sentence representation to avidly retrieve local communities containing sentences with high similarity.
The overall results are presented in Table 2. The baseline fast clustering method yields better performance than K-Means, Affinity Propagation, and the topic-focused clustering method. Moreover, enhancing the fast clustering method with our proposed dynamic clustering algorithm further improves the performance. On top of the dynamic clustering algorithm, we further evaluate our proposed ranking mechanism in two distinct scenarios: one considering only cluster density and the other incorporating both cluster density and background information. The last two rows show our ranking mechanism improves the proposed dynamic clustering algorithm, with the final system settings achieving the best performance.

heading: 5.3 Summarization Results
text: Abstractive comment summarization is challenging as it requires more training data to understand and summarize noisy online comments. Due to this limitation, previous works have focused solely on extractive summarization. This work takes the compact yet robust FTX model (Chung et al., 2022) as the baseline abstractive summarizer, which uses only 3B parameters while achieving impressive performance. In particular, we contribute two main enhancements to the baseline system. As shown in Table 3, TuneCoT means employing our designed instruction-tuning instances to fine-tune the baseline model on CAT and summary generation in a chain-of-thought style. BG means harnessing the background information extracted from source articles for in-context learning to preserve the contextual integrity of the comment cluster as reflected in the summary. The first two rows show that enhancing the base model with the ability of CAT prediction and CAT-focused summarization improves performance significantly. Moreover, the last two rows indicate that normalizing the output of the LLM with background further improves performance, a point we will illustrate in Section 6.

heading: 5.4 Overall CASC Results
text: We build the benchmark CASC system by combining the SBERT-based fast clustering (Reimers and Gurevych, 2019) module and the FTX-based summarizer. To conduct a thorough performance evaluation, we additionally experiment with recent popular LLMs, including Llama-7B (Touvron et al., 2023a), Llama2-7B (Touvron et al., 2023b), and ChatGPT (gpt-3.5-turbo). The overall results are presented in Table 4.
The first two rows show that our instructiontuning method significantly improves the overall performance, and Line 3 indicates the proposed dynamic clustering method can further boost the performance. To emphasize the importance of clustering in the CASC task, we conducted an ablation experiment with topic-focused clustering applied. The results in Line 4 demonstrate that such an action leads to a significant drop in performance. Fur-
thermore, the results in the lower part of Table 4 show: (1) Our instruction-tuning data and methods can be well extended to other LLMs to enhance their CASC performance. (2) Our tuned FTX-3B model performs better than the larger Llama models and is comparable to ChatGPT. This finding is meaningful for real-world CASC applications.

heading: 6 Analysis & Discussion
text: Comment cluster distribution. Figure 1 presents the distribution of 1,467 comment clusters constructed by our CASC system. The column bars labeled with tag ① depict the distribution of comment cluster sizes, the poly-line marked with tag ② denotes the similarity threshold applied to each cluster, and the poly-line labeled with ③ indicates the density value of each cluster. The diagram shows that as the cluster size increases, our dynamic algorithm progressively elevates the similarity threshold, to impose stringent criteria for cluster formation. Since a comment is included in the cluster only when its similarity to the centroid comment surpasses the threshold, the cluster density is distributed above the threshold line, with higher similarity thresholds leading to increased
Auto Aspect Terms: 1. poverty line, 2. generation poverty line, 3. average wage, 4. minimum income standards
Auto Summary: The user comments discuss various approaches to defining and calculating the poverty line. Some users suggest that every generation has its own unique definition of poverty while others argue that the average wage should be used as a starting point. There is also discussion about whether including irrelevant people in the calculation would make it useless. One user mentions the UK definition of poverty which is 60% below median household income after housing costs but is criticized by some groups due to being a relative measure rather than an absolute measure. Additionally, there is a question about how inflation affects those living.
cluster density. Upon reviewing the real data, it becomes evident that popular topics tend to attract more comments within the cluster, while smaller clusters usually revolve around unpopular topics triggered by the leading comment. In this situation, our method raises the threshold to maintain the integrity of larger clusters and assigns lower t r shold values to small r on to encourage diversity.
Factual error analysis. LLMs enriched with dense knowledge exhibit heightened creativity but also ent il risks of factual inaccuracies. Figure 2 presents an example processed by our CASC system, where all the comments are from Reddit under the topic of “swine flu”. We observe that the aspect terms encompass most key points expressed in the cluster and are also embodied in the summaries. The results indicate that by establishing a coherent chain of thought for generating aspect terms and summaries, our method mitigates factual errors to some extent. Nevertheless, the first summary (w/o BG), which revolves around the COVID-19 pandemic, is inconsistent with the topic discussed within the original comments. This is due to an information gap, as the pre-trained LLM model lacks specific knowledge about the themes of the articles that the comments are focusing on. In contrast, after manually incorporating the background knowledge “H1N1 virus, swine flu” into the prompt context, the second summary correctly references the background information, thus avoiding factual errors.
To quantitatively analyze the impact of the background information incorporated, we further con-
duct CASC on Reddit comments of ten randomly selected article clusters in real scenarios. On this basis, we perform a manual factuality evaluation based on the summaries of the first five longer comment clusters among each article cluster (fifty summaries in total). For the evaluation guideline, we adhere to the rule that a comment cluster summary is deemed factually accurate only when all the entities, events, and opinions it encompasses are entirely in line with the content of the original comments. The evaluation results in Table 5 demonstrate that utilizing background in our final system significantly enhances its factuality.

heading: 7 Conclusion
text: This paper presents an initial exploration into Clustering and Abstractive Summarization of online news Comments (CASC). We introduce a manual annotation scheme, create a test suite, and propose an evaluation metric tailored for the CASC task. On this basis, we introduce a dynamic clustering algorithm and a novel ranking mechanism for effective comment clustering. In particular, we perform the first practice of tuning foundation models for comment cluster summarization. To foster further studies, the data and codes will be released to the research community upon application.

heading: 8 Limitations
text: As an early exploration in CASC, our study faces two notable limitations: (1) Our annotation efforts have been confined to English news comments due to constraints in human resources. In future research endeavors, we will aim to broaden our resources to construct a multi-lingual CASC test suite. (2) This paper establishes a baseline CASC system using LLM. Although we utilize chainof-thought fine-tuning along with the in-context learning of background information to alleviate the hallucination issue, the strategy is not thorough enough. We envision undertaking task-specific retraining of LLMs in future work to address this issue further. (3) Currently, the pipeline method outlined in this paper does not fully leverage LLMs. Given the rapid advancement of foundation models, further investigation is needed to explore methods that bridge the gap between clustering and summarization and the capacity to learn clustering and summarization using LLMs simultaneously. (4) When constructing the test suite, we considered the comment tree a context when performing manual annotation, but our system does not consider the comment tree. Inspired by previous work (Tan et al., 2022), exploring discourse theory within comment trees could help with more intelligent news comment analysis in future work.

heading: 9 Acknowledgments
text: We thank Tharini De Silva, Yang Ding, Tarun Kumar Vangani, and other team members for their useful discussions and contributions to the data construction. We express our gratitude to the anonymous reviewers who highly rated this paper and offered insightful comments that significantly contributed to the enhancement of this work.

heading: A System Settings
text: Our system is based on the Flan_T5_XL (Chung et al., 2022) model with 3B parameters. We tuned all the model parameters for 3 epochs using our built 26,438 instruction-tuning instances. We implemented the CASC system using the PyTorch framework. We tuned our model on A40 GPU cards, and a complete model tuning took around 10 GPU hours. All tests were finished in a single run with the random seed 19. We tuned the hyper-parameters like the number of titles used as background (Table 6), the ∆ value (Table 7), the cluster number of the K-Means algorithm (Table 8), the parameters of the curve in Equation 1 (Appendix B), etc., based on the DEV set.

heading: B Tuning Clustering Curve Steepness
text: Our proposed dynamic clustering algorithm maintains two hyper-parameters in Equation 1 to shape
the curve to control the dynamic similarity thresholds. Figure 3 illustrates an example curve. The curve steepness is determined by the two hyperparameters K1 and K2, which are tuned on DEV by sampling data points with the cluster sizes κ1=50 and κ1=200 on the X-axis. The two parameters are determined by the empirical investigation that a comment cluster with 50 comments is considered a common size, so we need to establish a general similarity threshold ς1 to ensure that the comments within the cluster are identifiable; a cluster with 200 comments is deemed long-winded, where we need a higher threshold ς2 to strictly filter out those unrelated comments to make the cluster more readable. To achieve the above goal, we first assume the curve to be a plain horizontal line to test the effects when ς1 takes different values on the Y-axis. Table 9 shows that as the value of ς1 grows, the performance drops sharply. Therefore, we obtain ς1=0.52 with the highest NMI score on DEV. Then we anchor (κ1, ς1) to tune the data point (κ2, ς2), where the higher the value of ς2, the steeper the curve. It shows that it obtains the best performance when the similarity ς2 is 0.8 for the cluster with 200 comments.

heading: C Clustering Speed
text: The section presents the speed of the fast clustering algorithm, K-Means, and Affinity Propagation for reference, as shown in Table 10. Here, the KMeans and Affinity Propagation algorithms we use are implemented by scikit-learn7. Notably, the running speed fluctuates greatly with changes in the total number of comments.
7https://scikit-learn.org/stable/ modules/clustering.html

heading: D Successful Hit Summary Pairs in the Pipeline Evaluation
text: As stated in the main body, this research proposes the pipeline evaluation scheme tailored for the CASC task where we set the cosine similarity threshold to 0.9 to strictly evaluate the quality of the generated summaries. This section presents some summaries that successfully hit the gold standard summaries for reference.
D.1 Example 1
Ground truth. The user comments discuss the impact of high cost of living on the poorest in society and how they cannot switch to cheaper brands or travel less. They also criticize msm for claiming that high cost of living has little effect on the poor. Some users express frustration with an article that talks about the poor only focusing on wealthy individuals. There is a call for increasing GST to help the poor while others suggest taking half of mayors’ salaries to subsidize them directly. Model prediction. The user comments discuss the high cost of living in Singapore and its impact on the poor. Some users criticize the choice of interviewee for the article, questioning whether it accurately represents the struggles faced by those living in poverty. Others point out that this issue
is not unique to Singapore and highlights the need to reduce the gap between rich and poor. There is also discussion about supermarket chains like ntuc fairprice not passing on higher prices to consumers despite being strategically placed. Overall, there is a sense that rising prices are causing hardship for those living in poverty.
D.2 Example 2
Ground truth. The user comments revolve around the topic of defining and calculating the poverty line. Some users suggest using minimum income standards as a starting point while others argue that every generation’s poverty line is different. There is also discussion about household division and how it can be too low below sustainable levels. The UK’s definition of poverty is 60% below median household income after housing costs which some people dislike as it is a relative measure rather than an absolute measure of poverty. Some users suggest using the average wage as a metric but others argue that doing so would include irrelevant people into the calculations, making the metric useless. Model prediction. The user comments discuss different approaches to defining poverty, including the UK’s 60% below median household income after housing costs and the living wage as a personal income measure. They also mention the UK’s state pension system and minimum income standards for defining the poverty line. One comment suggests that Singapore should have an official poverty line defined by the bottom 0.1%, which may result in fewer poor people.
D.3 Example 3
Ground truth. The user comments discuss the high number of people being helped by food banks since COVID. One user questions if this is truly
CMT-1. how do you think you came in contact with flu? CMT-2. so, if i caught it, do you think i would know or would i just think it was the regular flu? CMT-3. i am currently trying to avoid other humans. although, that sounds like a good revenge for some people. i probably picked up the flu in a public place. CMT-4. how would you explain the news describing people with "mild symptoms". do they mean mild compared to regular flu, or mild compared with dying? CMT-5. or maybe from a patient who was there for something else and was unknowingly serving as an incubater for the disease. excuse me if you've already answered this, but were you not given every possible vaccination, considering your profession? and even if none are effective, are health-care workers continuing to take every precaution against infection of this flu, or have things been a bit laxer lately? do you mind giving the region of texas in which you work? Auto aspect terms: flu, contact, symptoms, mild symptoms, vaccination, health-care workers. Summary w/o BG: The user comments revolve around various aspects related to the COVID-19 pandemic. Users express concerns about how they may have contracted the flu and how they would know if they had it if they caught it. They also discuss symptoms and severity levels of the virus as well as health-care worker precautions taken against it. Some users speculate about potential incubators of the disease such as patients who may have served as incubators for others. Overall, there is a focus on understanding how best to prevent the flu. Summary w/ BG (H1N1 virus, swine flu): The user comments revolve around various aspects related to the H1N1 virus and swine flu. Users express concerns about how they may have contracted the flu and how they would know if they had it compared to regular flu symptoms. They also discuss how mild symptoms are interpreted and whether they refer to a comparison with dying or if they indicate an unknowingly serving as an incubater for the disease. Additionally, users question whether health-care workers are taking proper precautions against infection and if things have become laxer.
Auto Aspect Terms: 1. poverty line, 2. generation poverty line, 3. average wage, 4. minimum income standards
Auto Summary: The user comments discuss various approaches to defining and calculating the poverty line. Some users suggest that every generation has its own unique definition of poverty while others argue that the average wage should be used as a starting point. There is also discussion about whether including irrelevant people in the calculation would make it useless. One user mentions the UK definition of poverty which is 60% below median household income after housing costs but is criticized by some groups due to being a relative measure rather than an absolute measure. Additionally, there is a question about how inflation affects those living.
happening while another explains that food banks are used to providing support for those in the poorest 10% of the population but are now supporting more people. It is noted that many families are cl s to eeding their help due to u xpected vents such as illness or job loss. Model prediction. The user comments discuss the claim that Singapor ’s food banks are helping 1 million people since COVID-19. However, one user questions if this is truly the case and suggests that it may be mor accu ate to say that th poorest 10% of the population is now needing help. Another user points out that many families are close to needing assistance due to unexpected events like illness or job loss. Despite this, there are still over 30,000 households reliant on foodbanks pre-pandemic and over 100k registered with foodbank.sg.
E Abstractive vs. Extractive Comment Summarization
In addition to the final dataset presented in the main body of this paper, our initial exploratory research
extends to the extractive summarization of news comment clusters. In extractive summarization annotation, the annotators are guided to manually extract comments from the original comment cluster following two rules: (1) Since each comment is manually annotated with aspect terms, the summary content should cover all the annotated aspect terms; (2) Each aspect term could be shared among multiple comments, the annotators should read and select the most representative comments as the summary to avoid duplication and redundancy. With this guideline, we annotated summaries for 6 article clusters. For reference, we present a comment cluster with extractive and abstraction summaries annotated in Figure 4. When compared, extractive summarization proves superior in accuracy, fluency, and conciseness. This showcases the adaptability of LLM-based summary generation to noisy realworld data.

heading: Future Work
text: In future research endeavors, we will aim to broaden our resources to construct a multi-lingual CASC test suite. (2) This paper establishes a baseline CASC system using LLM. Although we utilize chainof-thought fine-tuning along with the in-context learning of background information to alleviate the hallucination issue, the strategy is not thorough enough. We envision undertaking task-specific retraining of LLMs in future work to address this issue further. (3) Currently, the pipeline method outlined in this paper does not fully leverage LLMs. Given the rapid advancement of foundation models, further investigation is needed to explore methods that bridge the gap between clustering and summarization and the capacity to learn clustering and summarization using LLMs simultaneously. (4) When constructing the test suite, we considered the comment tree a context when performing manual annotation, but our system does not consider the comment tree. Inspired by previous work (Tan et al. , 2022), exploring discourse theory within comment trees could help with more intelligent news comment analysis in future work