Indicative Summarization of Long Discussions
Authors: Shahbaz Syed
Dominik Schwabe
Khalid Al Khatib
Martin Potthast
Venue: EMNLP 2023 Main (2023)
Keywords: Summarization
Computational Argumentation
Large Language Models
Social Media Discussions
TL;DR: We present a fully unsupervised approach to generate indicative summaries of long discussions resembling a Table of Contents.

Abstract: Online forums encourage the exchange and discussion of different stances on many topics. Not only do they provide an opportunity to present one's own arguments, but may also gather a broad cross-section of others' arguments. However, the resulting long discussions are difficult to overview. This paper presents a novel unsupervised approach using large language models (LLMs) to generating indicative summaries for long discussions that basically serve as tables of contents. Our approach first clusters argument sentences, generates cluster labels as abstractive summaries, and classifies the generated cluster labels into argumentation frames resulting in a two-level summary. Based on an extensively optimized prompt engineering approach, we evaluate 19 LLMs for generative cluster labeling and frame classification. To evaluate the usefulness of our indicative summaries, we conduct a purpose-driven user study via a new visual interface called **Discussion Explorer**: It shows that our proposed indicative summaries serve as a convenient navigation tool to explore long discussions.

Indicative Summarization of Long Discussions
Shahbaz Syed t
Dominik Schwabe 1* Khalid Al-Khatib
Martin Potthast
18
LLiipiig University
University of Groningen
ScaDS.AI
syedeuz
Abstract
exchanged, so still necessary read most of them for comprehensive overview. In this paper. we depart from previous approaches to summariz- ing long discussions by using indicative summaries instead of informative summaries. 3 Figure 1 illus- trates our three-step approach: first, the sentences of the arguments clustered according to their latent subtopics. Then, large language model gen- erates a concise abstractive summary for each clus- ter as its label. Finally, the argument frame (Chong and Druckman, 2007; Boydstun et al., 2014) each cluster label is predicted as a generalizable operationalization of perspectives on a discussion's ss topic. From this, hierarchical summary is created in the style of a table of contents, where frames act as headings and cluster labels subheadings. To our knowledge, indicative summaries of this type have not been explored before (see Section 2).
Online forums encourage the exchange and discussion of different stances on many top- ics. Not only do they provide an opportunity present one's own arguments, but may also gather a broad cross-section of others' argu- ments. However, the resulting long discussions are difficult to overview. This paper presents a novel unsupervised approach using large lan- guage models (LLMs) generating indicative summaries for long discussions that basically serve as tables of contents. Our approach first clusters argument sentences, generates cluster labels as abstractive summaries, and classifies the generated cluster labels into argumentation frames resulting in two-level summary. Based on an extensively optimized prompt engineer- ing approach, we evaluate 19 LLMs for genera- tive cluster labeling and frame classification. To evaluate the usefulness our indicative summaries, conduct purpose- driven user study via new visual interface called DISCUS- SION EXPLORER: shows that our proposed indicative summaries serve as convenient nav- igation tool to explore long discussions.'
Our four main contributions are: (1) A fully un- supervised approach to indicative summarization of long discussions (Section 3). We develop robust prompts for generative cluster labeling and frame assignment based on extensive empirical evaluation and best practices (Section 4). (2) A comprehen- sive evaluation of 19 state-of-the-art, prompt-based, large language models (LLMs) for both tasks, sup- ported by quantitative and qualitative assessments (Section 5). (3) A user study of the usefulness of in- dicative summaries for exploring long discussions (Section 5). (4) DISCUSSION EXPLORER, an inter- active visual interface for exploring the indicative summaries generated by our approach and the cor- responding discussions." Our results show that the GPT variants of OpenAl (GPT3.5, ChatGPT, and GPT4) outperform all other open source models at the time of writing. LLaMA and TO perform well, but are not competitive with the GPT models. Regarding the usefulness of the summaries, users preferred our summaries to alternative views to ex- plore long discussions with hundreds of arguments.
1 Introduction
Online discussion forums are popular medium for discussing a wide range of topics. As the size of a community grows, so does the average length of the discussions held there, especially when current controversial topics are discussed. On Change- My View (CMV),2 for example, discussions often go into the hundreds of arguments covering many perspectives the topics in question. Initiating, participating in, or reading discussions generally has two goals: to learn more about others' views on a topic and/or share one's own.
To help their users navigate large volumes of arguments in long discussions, many forums offer basic features to sort them, for example, by time of creation or popularity. However, these alternative views may not capture the full range of perspectives
Unlike an informative summary, an indicative summary does not capture much information as possible from a texi. but only its gist. This makes them particularly suitable for long documents like books in the form of tables of contents. "ntps://discussion cecission-xpppreeee explorer web. web
*Equal contribution.

---

Grouping- approaches group discussion units like posts sentences, either implicitly explic- itly. The groups based queries, aspects, top- ics, dialogue acts, argument facets, key points annotated by experts. Once the units are grouped. individual summaries are generated for cach group by selecting representative members, respectively. Summary generation Unit clustering
Summary presentation
Discussion
Cluster labeling frame assignment
Conceptual
Example (simplified)
Topic
Cluster
Topic
This the best lime humen history (655)
Label
Frames: A,
Frame A
Pollical
(repfies; click label view)
estate policy
Label
prices soaring since the 90s. (85)
issues. (45)
Cluster 2
Label 2
Frames: C, D,
Frame
Economic
Label
estate pollcy
-Label
prices soaring slince the 90s. (88)
Pragmented economic power. (20)
Label 3
Frames: B,
Frame
Grime P Punishment
Label
Vicient crime dediined
90s, a problem. (102)
Left: Illustration of our approach to generating indicative summaries for long discussions. The main steps are (I) unit clustering, (2) generative cluster labeling, and (3) multi-label frame assignment in order of relevance. Right: Conceptual and exemplary presentation of our indicative summary in table of contents style. Frames act as headings and the corresponding cluster labels as subheadings.
Figure I:
2 Related Work
2.2 Grouping- Summarization
Previous approaches to generating discussion sum- maries have mainly focused on generating extrac- live summaries, using two main strategies: extract- ing significant units (e.g., responses, paragraphs, or sentences), or grouping them into specific cate- gories, which are then summarized. In this section, we review the relevant literature.
This grouping- grouping-then--uunggg wwwtt-hsummnizaatiionn paradigm has been primarily applied to multi-document sum- marization of news articles (Radev et al., 2004). Follow-up work proposed cluster link analysis (Wan and Yang, 2008), cluster sentence ranking (Cai et al., 2010), and density peak identification in clusters (Zhang et al., 2015). For abstractive multi- document summarization, Nayeem et al. (2018) clustered sentence embeddings using a hierarchical agglomerative algorithm, identifying representative sentences from each cluster using TextRank (Mihal- and Tarau, 2004) the induced sentence graph. Similarly, Fuad al. (2019) clustered sentence em- beddings and selected subsets of clusters based on importance, coverage, and variety. These subsets are then input to a transformer model trained on the CNN/DailyMaii dataset (Nallapati et al., 2016) to generate summary. Recently, Ernst et al. (2022) used agglomerative clustering of salient statements to summarize sets of news articles, involving pervised ranking of clusters by importance.
2.1 Extractive Summarization
Extractive approaches use supervised learning or domain-specific heuristics to extract important en- tities from discussions as extractive summaries. For example, Klaas (2005) summarized UseNet newsgroup threads by considering thread structure and lexical features to measure message impor- tance. Tigelaar et al. (2010) identified key sen- tences based on author names and citations, fo- cusing on coherence and coverage in summaries. Ren et al. (201 1) developed a hierarchical Bayesian model for tracking topics, using random walk al- gorithm select representative sentences. Ranade al. (2013) extracted topic-relevant and emotive sentences, while Bhatia et al. (2014) and Tarn- pradab et al. (2017) used dialogue acts to summa- rize question-answering forum discussions. Egan et al. (2016) extracted key points using dependency parse graphs, and Kano et al. (2018) summarized Reddit discussions using local and global context features. These approaches generate informative summaries, substituting discussions without back- referencing them.
For Wikipedia discussions, Zhang et al. (2017) proposed the creation of dynamic summary tree to ease subtopic navigation at different levels of de- tail, requiring editors to manually summarize each

---

node's cluster. Misra et al. (2015) used sum- marization to identify arguments with similar as- pects in dialogues from the Internet Argument Cor- pus (Walker et al., 2012). Similarly, Reimers et al. (2019) used agglomerative clustering of contextual embeddings and aspects to group sentence-level ar- guments. BBar-Haim et al. b2020a,b) examined the mapping of debate arguments to key points written by experts to
3 Indicative Discussion Summarization
Our indicative summarization approach takes the sentences of a discussion as input and generates summary in the form of a table of contents, as shown in Figure 1. Its three steps consist of clus- tering discussion sentences, cluster labeling, and frame assignment to cluster labels.
summaries.
3.1 Unit Clustering
Our approach clusters discussion units, but in- stead of a supervised selection of key cluster mem- bers, we use vanilla LLMs for abstractive summa- rization. Moreover, our summaries are hierarchical, using issue-generic frames as headings (Chong and Druckman, 2007; Boydstun et al., 2014) and gener- ating concise abstractive summaries of correspond- ing clusters subheadings. Thus our approach is unsupervised, facilitating a scalable and generaliz- able summarization of discussions.
Given a discussion, we extract its sentences as dis- cussion units. The set of sentences is then clustered using the density-based hierarchical clustering al- gorithm HDBSCAN (Campello et al., 2013). Each sentence is embedded using SBERT (Reimers and Gurevych, 2019) and these embeddings are then mapped to a lower dimensionality using UMAP (McInnes et al., 2017).5 Unlike previous ap- proaches that rank and filter clusters to generate informative summaries (Ernst et al., 2022; Syed et al., 2023), our summaries incorporate all clus- ters. The sentences of each cluster are ranked by centrality, which is determined by the À value HDBSCAN. A number of central sentences per cluster are selected as input for cluster labeling by abstractive summarization.
2.3 Cluster Labeling
Cluster labeling involves assigning representative labels to document clusters to facilitate clustering exploration. Labeling approaches include compar- ing term distributions (Manning et al., 2008), select- ing key terms closest to the cluster centroid (Role and Nadif, 2014), formulating key queries (Gollub et al., 2016), identify keywords through hypernym relationships (Poostchi and Piccardi, 2018), and weak supervision to generate topic labels Popa and Rebedea (2021). These approaches often select a small set of terms as labels that do not describe a cluster's contents in closed form. Our approach overcomes this limitation by treating cluster label- ing zero-shot abstractive summarization task.
Meta-sentence filtering Some sentences in a dis- cussion do not contribute directly to the topic, but reflect the interaction between its participants. Ex- amples include sentences such as "I agree with you." or "You are setting up a straw man." Pilot experiments have shown that such meta-sentences may cause summarization approach to include them in the final summary. As these are irrele- vant to our goal, we apply a corpus-specific and channel-specific meta-sentence filtering approach, respectively. Corpus-specifie filtering is based on a small set of frequently used meta-sentences M large corpus (c.g., on Reddit). It is bootstrapped during preprocessing, and all sentences in it are omitted by default.
2.4 Frame Assignment
Framing involves emphasizing certain aspects of a topic for various purposes, such persuasion (Ent- man, 1993; Chong and Druckman, 2007). Frame analysis for discussions provides insights into dif- ferent perspectives on topic (Morstatter et al., 2018; Liu et al., 2019). It also helps to identify biases in discussions resulting, e.g., from word choice (Hamborg et al., 2019b,a). Thus, frames can serve as valuable reference points for organiz- ing long discussions. We use predefined inven- tory of media frames (Boydstun et al., 2014) for discussion summarization. Instead of supervised frame assignment (Naderi and Hirst, 2017; Ajjour et al., 2019; Heinisch and Cimiano, 2021), we use prompt-based LLMs for more flexibility.
Our pilot experiments revealed that some tences in discussions are also channel-specific (c.g.. for the ChangeMy Miew View Subreddit). Therefore, we augment our sentence clustering approach by adding a random sample M' C M to the set of sen- tences D of each individual discussion before clus- tering, where |M max 300, DD}. The maxi- mum value for the number of meta-sentences |M'
IImllementation details are given in Appendix B. 'The set used like typical stop word list, only for sentences.

---

is chosen empirically, maximize the likelihood that channel-specifie meta-sentences are clustered with corpus-specific ones. After clustering the joint set meta-sentences and discussion sen- tences D M', we obtain the clustering C. Let = C n M denote the number of meta- sentences and de (CnD) DJ the number of discus- sion sentences in cluster C' E C. The proportion of meta-sentences in cluster then estimated as C) me de: cluster classified as meta-sentence clus- |M'
ter if C(M > 0- 0-M(M",, where whereP(MM") [ assumes that meta-sentences are independent of others in discussion. The noise threshold 0 was chosen empirically. Sentences in discussion that either belong to meta-sentence cluster or whose nearest cluster is considered to be one omitted. In evaluation, an average of 23% of sentences are filtered from discussions. Figure illustrates the effect of meta-sentence filtering on discussion's s set of sentence.
(a) Joint clustering of discussion and meta- sentenceeeess DUM'.
Curated Noise Set
3.2 Generative Cluster Labeling
Most cluster labeling approaches extract keywords or key phrases as labels, which limits their fluency, These approaches may also require training data acquisition for supervised learning. We formulate cluster labeling as unsupervised abstractive sum- marization task. We experiment with prompt-based large language models in zero-shot and few-shot settings. This enables gencralization across multi- ple domains, the elimination of supervised learning, and fluent cluster labels with higher readability in comparison to keywords or phrases.
(b) The sampled meta-sentences M' C highlighted gray.
We develop several prompt templates specifi- cally tailored for different types of LLMs. For encoder-decoder models, we carefully develop ap- propriate prompts based PromptSource (Bach et al., 2022), a toolkit that provides a comprehen- sive collection of natural language prompts for var- ious tasks across 180 datasets. In particular, we analyze prompts for text summarization datasets with respect to (1) descriptive words for the gener- ation of cluster labels using abstractive summariza- tion, (2) commonly used separators distinguish instructions from context, (3) the position of in- structions within prompts, and (4) the granularity level input data (full text, document title, or sen- tence). Since our task is about summarizing groups of sentences, we chose prompts that require the full text as input to ensure that enough contextual
(c) Classification of mela-sentence clusters to be omitted.
Figure Effect of meta-sentence filtering: and b) A discussion's 's sentences 1) are jointly clustered with sample of meta-sentences M' C M. (c) Then each cluster is classified as a meta-sentence cluster based on its proportion of meta-sentences and its neighboring clusters. Meta-sentence clusters are omitted.
information is provided (within the limits of each model's 's input size), Section 4.1 provides details on the prompt engineering process.

---

3.3 Frame Assignment
Frame Inventory
Any controversial topic can be discussed from dif- ferent perspectives. For example, "the dangers of social media" can be discussed from a moral or health perspective, among others. In our indicative summaries, we use argumentation frame labels as top-level headings to operationalize different per- spectives. An argumentation frame may include one or more groups of relevant arguments. We assign frame labels from the issue-generic frame inventory shown in Table (Boydstun et al., 2014) to each cluster label derived in the previous step.
Capacity & Resources
Fairness Fquality
Constitutionality & Jurisprudence
Health & Safety
Morality
Crime & Punishment
Policy Prescription & Evaluation Political
Cultural Identity
Economic
Public Opinion
External Regulation & Reputation
Quality of Life
Security Defense
Table I: Inventory of frames proposed by Boydstun et al. (2014) to track the media's 's framing policy issues.
We use coompt-based models in both zero-shot and few-shot settings for frame assignment. In experiments with instruction-tuned models, we designed two types of instructions, shown in Figure 10, namely direct instructions for models trained on instruction-response samples, and dia- log instructions for chat models. The instructions are included along with the cluster labels in the prompts. Moreover, including the citation of the frame inventory used in our experiments has a posi- tive effect the effectiveness of some models (sec Appendix D.1 for details).
Model Variants
Description
Pre- InstructGPT
Tθ
Encoder-dceoder model trained datasets transformed as task-specifie prompts.
vanilla BLOOM vanilla
A muitlingual autoregressive model with 176B parameters for prompt-based text completion.
GPT-NeoX
Open source alternative to GPT-3.
20B
OPT
Autoregressive model with similar effective- GPT-3, but more efficient data collec- tion and training.
66B
Direct Instruction
3.4 Indicative Summary Presentation
LLaMA-COT vanilla
LLaMA-30B fine-tuned on chain-of- of-thuugt and reasoning samples (Si and Lin, 2023).
Given the generated labels of all sentence clusters and the frame labels assigned to each cluster label, our indicative summary groups the cluster labels by their respective frame labels. The cluster label groups of each frame label are then ordered by cluster size. This results in a two-level indicative summary, as shown in Figures and 4.
Alpaca
LLaMA-7B fine-tuned based on 52k self- instruct responses (Wang et al., 2022).
7B
OASST
LLaMA-300 1BB fine-tuned on the OpenAssistant Conversations dataset (Köpf al., 2023) using reinforcement learning.
vanilla
Pythia 12B
Suite of LLMs trained on public data to in- vestigate the effects of training and scaling various model properties.
OpenAl models GPT3.5 (text-davinci-003), ChatGPT (gpr-3.5- and GPT4.
4 Prompt Engineering
3.5, Chat, 4
Using prompt-based LLMs for generative clus- ter labeling and frame assignment requires model- specific prompt engineering as a preliminary step. We explored the 19 model variants listed in Table 2. To select the most appropriate models for our task, consulted the HELM benchmark (Liang et al., 2022), which compares the effectiveness of differ- ent LLMs for different tasks. Further, we have in- cluded various recently released open source mod- els (with optimized instructions) as they were re- leased. Since many them were released during our research, we reuse prompts previously opti- mized prompts for the newer models.
Dialogue Instruction
LLaMA 30B, 65B
Suite of open-source I.LMs from Mcta AI trained on public datasets.
Vicuna 7B, 13B
LLaMA models fine-tuned using conversa- tions collected by ShareGPT (htups//sharegpt. com)
Baize 7B, 13B
Open source model trained on alogues generated by letting ChatGPT (GPT 3.5-turbu) talk itself.
Falcon 40B, 408-Instruct
Trained on the RefinedWeb corpus (Penedo et al., 2023). which was obtained by filtering and deduplication of public web data.
Table 2: LLMs studied for cluster labeling frame as- signment. Older models are listed by Pre- InstructGP1 (prior to GPT3.5) and newer models are listed by their respective prompt types investigated Direct
FFor detailed label descriptions see Table the Appendix. "Sce Appendices C. and for details.
Dialogue See Appendices and for details.

---

tings. In the zero-shot setting, formulated three prompts containing (1) only frame labels, (2) frame labels with short descriptions, and (3) frame labels with full text descriptions (see Appendix D.2 for details). For the few-shot setting, we manually an- notated up to two frames from the frame inventory of Table for each of the 300 cluster labels gen- erated by the best model GPT3.5 in the previous step. We included 42 examples (3 per frame) in the few-shot prompt containing the frame label, its full-text description, and three examples. The re- maining 285 examples were used for subsequent frame assignment evaluation. Our evaluation in Section shows that GPT4 performs best on this task. Figure 3 (bottom) shows its best prompt.
G for Generative Cluster Labeling ing
Generate single deseriptive phrase that deseribes following debate in very simple language, without talking about debate the author. Debate: """ [ mnn
GPT4 for Frame Assignneet
The following (input typef" contains available media frames as defined in the work from Jauthors]: [frames] For every input, you answer with three these media frames corresponding to that input, order importance.
"A list of frame labels JSON with frame labels and their descriptions.
The best performing instructions for cluster labeling and frame assignment. For frame assignment, citing the frame inventory using the placeholder (au- thors) has positive impact on the effectiveness of some models (see Appendix D.1 for details).
Figure 3:
5 Evaluation
To evaluate our approach, we conducted automatic and manual evaluations focused on the cluster la- beling quality and the frame assignment accuracy. We also evaluated the utility of our indicative sum- maries in purpose-driven user study in which participants had the opportunity to explore long discussions and provide with feedback.
4.1 Cluster Labeling
The prompts for the encoder-decoder model TO are based on the PROMPTSOURCE (Bach et al., 2022) toolkit. We have experimented with differ- ent prompt templates and tried different combina- tions of input types (c.g. "text", "debate", "dis- cussion", and "dialogue") and output types (e.g. "title", "topic", "summary"" "theme", and "thesis"). The position the instruction within a prompt was also varied, taking into account prefix and suffix positions. For decoder-only models like BLOOM, GPT-NeoX, OPT-66B, and OPT, we experimented with hand-crafted prompts. For GPT3.5, we fol- lowed the best practices described in OpenAl's API and created a single prompt.
5.1 Data and Preprocessing
We used the "Winning Arguments" corpus from Tan et al. (2016) as data source for long dis- cussions. It contains 25,043 discussions from the ChangeMy View Subreddit that took place be- tween 2013 and 2016. The corpus was prepro- cessed by first removing noise replies and then meta-sentences. Noise replies are marked in the metadata of the corpus as "deleted" by their spective authors, posted by bots, removed by moderators. In addition, replies that referred the Reddit guidelines or forum-specific modera- tion were removed using pattern matching (see Ap- pendix A for details). The remaining replies were split into set of sentences using Spacy (Honni- bal et al., 2020). To enable the unit clustering (of sentences) as described in Section 3.1, the set of meta-sentences M is bootstrapped by first cluster- ing the entire set of sentences from all discussions in the corpus and then manually examining the clus- ters to identify those that contain meta-sentences, resulting in |M| = 955 meta-sentences. After fil- tering out channel-specific noise, the (cleaned) sets of discussion sentences are clustered as described.
Prompts were evaluated on manually annotated set of 300 cluster labels using BERTScore (Zhang et al., 2020). We selected the most effective prompt for each of the above models for cluster labeling. Our evaluation in Section 5 shows that GPT3.5 performs best in this task. Figure 3 (top) shows the best prompt for this model."
4.2 Frame Assignment
For frame assignment, models were prompted to predict maximum of three frame labels for a given cluster label, ordered by relevance. Experiments were conducted with both direct instructions and dialogue prompts in zero-shot and few-shot set- "ChatGPT and GPT4 were released after our evaluation.
Evaluation Data From the preprocessed discus- sions, 300 sentence clusters were randomly se-

---

Model
Mean Rank
# First
Length
Model
Zero-Shot
Few-Shot
Min
Max
Mean
short
full
GPT3.5
225
27
9.44
Alpaca-70
39.1
39.5
28.4
20.6
BLOOM
2.95
33
37
8.13
Baize-78
34.2
34.6
39.1
30.9
GPT-NeoX
3.20
20
34
7.42
Baize-138
42.4
48.1
42.0
39.5
OPT
12
30
8.27
BLOOM
26.7
31.7
25.5
Te
3.72
28
18
3.10
ChatGPT
60,9
58.0*
58.8
63.4
Fal-oon485
46.5
46.5
46.1
38.3
Table 3: Results of qualitative evaluation of gen- erative cluster labeling. Shown (1) the mean rank of a model from four annotators and (2) the number of times a model was ranked first by an annotator. GPT3.5 (fext-davinci-003) performed better than other models and generated longer labels on average.
Falcon-485 -08--stt..nt.
51.4
44.4
32,9
28.4
GPT3.5
53.5
60.9
58.0
53.9
GPT4
63.4
60.5*
65.4
67.1
GPT-NeoX
19.3
25.1
31.3
31.3
LLaMA-30B
45.7
41.2
39.1
LLaMA-COT
46.9
54.34
49.8
57.2
LLaMA-65B
53.14
50.6
39.5
OASST
48.6
48.1
53.5
47.7
lected. Then, manually created cluster label and up three frame labels for each cluster. Due the short length of the cluster labels, up two frames per label were sufficient. After excluding 57 examples with ambiguous frame assignments, we obtained a reference set of 243 cluster label samples, each labeled with up to two frames.
OPT
16.0
13.2
14.8
Pythia
31.7
33.3
30.5
29.6
Te
48.6
54.3*
55.64
49.8
Vicuna 7B
28.4
36.2
35,4
20.2
Vicuna-
44.0
40.7
42.0
38.3
Table 4: Results of an automatic evaluation of 19 LLMs (sorted alphabetically) for frame assignment, indicating the five best models each setting. Shown are the per- centages of samples where the first frame predicted by a model is one of the reference frames. The three zero- shot columns denote the prompt type: frame label only, label with short description, and label with full descrip- tion. Model types are also indicated: Pre- pre-Instruc tGP
5.2 Generative Cluster Labeling
The results of the automatic cluster labeling evalu- ation using BERTScore and ROUGE are shown in (Appendix) Tables 7 and 8, respectively. We find that ChatGPT performs best. To manually evaluate the quality of the cluster labels, we used ranking- based method which four annotators scored the generated cluster labels against the manually anno- tated reference labels of each of the 300 clusters. To provide additional context for the cluster con- tent, the five most semantically similar sentences to the reference label from each cluster were included, well as five randomly selected sentences from the cluster. To avoid possible bias due the length of the cluster labels by different models, longer la- bels were truncated to 15 tokens. To determine an annotator's s model ranking, we merged the prefer- ence rankings for all clusters using reciprocal rank fusion (Cormack et al., 2009). Annotator agree- ment was calculated using Kendall's W for rank correlation (Kendall, 1948), which yielded value of 0.66, indicating substantial agreement.
Direct Dialogue Missing values are model infer- ences that exceed our computational resources.
rior effectivenss in generating high-quality cluster labels. ranked first in 225 out 300 comparisons, with an average score of 1.38 by the four annotators. The cluster labels generated by GPT3.5 were longer on average (9.4 tokens) and thus more informative than those generated by the other models, which often generated disjointed or incomplete labels. In particular, TO generated very short labels on aver- age (3.1 tokens) that were generic/non-descrriiivee
5.3 Frame Assignment
In the zero-/few-shot frame assignment settings de- scribed in Section 4.2, we prompted the models to predict three frames per cluster label in order of relevance. Using the manually annotated refer- ence set of 243 cluster labels and their frame labels, evaluated the accuracy of the frames predicted for each cluster label that matched the reference frames. The results for the first predicted frame
The average ranking of each model is shown in Table 3 along with the length distributions of the generated cluster labels." !I GPT3.5 showed supe-
Fiiuuee in Appendix shows the annotation interfaces. As newer models were published after our manual evalua- tion, we show an automatic evaluation of all models using human and GPT3.5 2-7.3.-bbar b-baeed reference labels in the Appendix
in Tables 7 and 8.

---

are shown in Table 4. In most cases, GPT4 out- performs all other models in the various settings, with the exception of the zero-shot setting with a short prompt, where GPT3.5 narrowly outperforms GPT4 with 60.9% accuracy versus 60.5%. Among the top five models, the GPT* models that follow direct user instructions perform consistently well, with the LLaMA-/6SB/-CoT and TO models show- ing strong effectiveness among the open-source LLMs. Conversely, the OPT model performs con- sistently worse in all settings. The few-shot setting shows greater variance in results, suggesting that the models are more sensitive to the labeled exam- ples provided in the prompts. Including a citation to the frame inventory paper in the instructions (see Figure 10) significantly improved the effectiveness of Falcon-40B (12%) and LLaMA-65B (9%) in the zero-shot setting (see Appendix D.1 for details).
With respect to the exploration task, the five annotators agreed that our summaries outper- formed the two baselines in terms of discovering arguments from different perspectives presented by participants. The inclusion of argumentation frames proved to be valuable tool for the annola- tors, facilitating the rapid identification of different perspectives and the accompanying cluster labels showing the relevant subtopics in the discussion. For the participation task, three annotators pre- ferred the original web page, while our summaries and the search engine were preferred by the remain- ing two annotators (one each) when it came to iden- tifying the appropriate place in the discussion to put their arguments. In post-study questionnaire, the annotators revealed that the original web page was preferred because of its better display of the various response threads, a feature not comparably reimplemented in DISCUSSION EXPLORER. The original web page felt "more familiar." However, we anticipate that this limitation can be addressed by seamlessly integrating our indicative summaries into a given discussion forum's 's web page, creating consistent experience and a comprehensive and effective user interface for discussion participation.
Results
5.4 Usefulness Evaluation
addition assessing each step of our approach, we conducted a user study to evaluate the effec- tiveness of the resulting indicative summaries. In this study, we considered two key tasks: explo- ration and participation. With respect to explo- ration, our goal was to evaluate the extent to which the summaries help users explore the discussion and discover new perspectives. With respect par- ficipation, we wanted assess how effectively the summaries enabled users contribute new argu- ments by identifying the appropriate context and location for a response.
5.5 DISCUSSION EXPLORER
Our approach places emphasis summary presen- tation by structuring indicative summaries into a table of contents for discussions (see Section 3). To demonstrate the effectiveness of this presenta- tion style in exploring long discussions, we have developed an interactive tool called DISCUSSION EXPLORER.. 12 This tool illustrates how such sum- maries can be practically applied. Users can par- ticipate in discussions by selecting argumentation frames or cluster labels. Figure presents indica- tive summaries generated by different models, pro- viding a quick overview of the different perspec- tives. This two-level table of contents-like sum- mary provides effortless navigation, allowing users to switch between viewing all arguments in a frame and understanding the context of sentences in a cluster of the discussion (see Figure 5).
We asked five annotators to explore five ran- domly selected discussions from our dataset, for which we generated indicative summaries using our approach with GPT3.5. To facilitate intuitive ploration, we developed DISCUSSION EXPLORER (sce Section 5.5), an interactive visual interface for the evaluated discussions and their indicative summaries. addition to our summaries, two baselines were provided to annotators for compar- ison: (1) the original web page of the discussion on MiangeMyView, and (2) a search engine in- terface powered by Spacerini (Akiki et al., 2023). The search engine indexed the sentences within discussion using the BM25 retrieval model. This allowed users to explore interesting perspectives by selecting self-selected keywords as queries, as opposed to the frame and cluster labels that our summaries provide. Annotators selected the best of these interfaces for exploration and participation.
Conclusion
We have developed an unsupervised approach to generating indicative summaries of long discus- sions to facilitate their effective exploration and navigation. Our summaries resemble tables of con-
Phttps///dscusssoo wew.riiooneeepeeeeeeeeeeeeo web webis.de/

---

CMV: The "others have it worse' argument is terrible and should never be used in actual conversation with depressed person
GPT-4
LLAMA-30B- 200-SUPERCOOT
equality
Health and safety
Acknowlldggnng personal struggl es recogrizing others!
Depression complex mental that severity and treetment options. (96)
hardshipa (17]
Health and safety
Understanding and managing depression as complex mental state
Impact of depression and how to help those affected. [3S]
[ou]
Personal journey of overcerning depress and finding happiness.
Importance of gratitude for heppiness and mental health [19]
Impeect feffent approwcees supporting depressred
Morality
[35]
Gratitude and appreciation for the ltttle thinge in life can help improve happinees and perspective, (3%)
Controll ing reprog remering thought patterne through
awareness and rational evaluation of remotions. [22]
Perspective importance
Policy preacription and ovaluation
Positive self-takk and growth mindser [21]
Ertecilveness of advice depends on individual and context. [2G]
prescription
Quality of life
Comparing situations others' can be helpful harmful. [97]
Gaining penspective for perconal growih understanding. [22]
Focusing positive mindowt and selfgrowihh (21]
situstione
Overooming challlnges and finding happiness through personal
Psychology and the power the brain thought patterns, (22)
growth and chenge (17)
CHATUPT
FALCON 408 INSTRUCT
Capacity and resources
Health and safety
brain restructure tzelf through focused practice and mindul awarencss, (22]
The complexity of depression and Its treatment [90]
Perspective on suffering [97]
Falmess and equality
Using comparison to combat compleints about others having it better
effactiveness talling depressed person stop being
be effective
for
and be grateful,
Health and
The effectiveness the advice focus on the present moment [25]
Differences sadness and
misndersstanding sumounding the tarm dippassaion (0R)
The power of the brain to restructure itself through focused practice. (22)
Talking about self impact copecially depressed. (35)
Morality
Finding happiness by faing physioal pain and removing tosic
The importance of gratitude and appreciation in life. [39]
people
Perspective is powerful tool for parsonal growth and
Comparing pain sulfering eeperiencass [97]
understanding [22]
Quality offife
the positives and regetive
imporianie
and appreciation 139)
Life is worth living despite challenges. [17]
Caining perspective can help with managing boundaries, taking
responsibility apprecieting
Tips for cultiesting a positive mindset and cetting achievoble gnals.
Figure 4: DISCUSSION EXPLORER provides concise overview of indicative summaries from various models for a given discussion. The summary is organized hierarchically: The argument frames act as heading, while the associated cluster labels act as subheadings, similar table of contents. Cluster sizes are also indicated. Clicking frame lists all argument sentences in discussion that assigned to that frame, while clicking on cluster label shows the associated argument sentences that discuss a subtopic the context of the discussion (see Figure 5).
tents, which list argumentation frames and concise abstractive summaries of the latent subtopics for comprehensive overview of discussion. By analyzing 19 prompt-based LLMs, we found that GPT3.5 and GPT4 perform impressively, with LLaMA fine-tuned using -f-otuugth being the second best. A user study of long discussions showed that our summaries were valuable for ex- ploring and uncovering new perspectives in long
discussions, otherwise tedious task when rely- ing solely the original web pages. Finally, we presented DISCUSSION EXPLORER, interactive visual tool designed to navigate through long dis- cussions using the generated indicative summaries. This serves as a practical demonstration of how indicative summaries can be used effectively.

---

7 Limitations
Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xian- gru Tang. Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. Promptsource: An integrated develop- ment environment and repository for natural language prompts.
We focused on developing a technology that facili- tates the exploration of long, argumentative discus- sions on controversial topics. We strongly believe that our method can be easily generalized to other types of discussions, but budget constraints pre- vented us from exploring these well. We also in- vestigated oftic-oo--hh-art language models to sum- marize these discussions and found that commer- cial models (GPT3.5, GPT4) outperformed open- source models (LLaMA, TO) in generating indica- tive summaries. Since the commercial models are regularly updated, it is important to note that the results of our approach may differ in the future. Although one can define a fixed set of prompts for each model, our systematic search for the optimal prompts based on an evaluation metric is intended to improve the reproducibility of our approach as newer models are released regularly.
Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kan- tor, Dan Lahav, and Noam Slonim. 2020a. From ar- guments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 4029 4039, Association for Computational Linguis- tics.
Roy Bar-Haim, Yoav Kantor, Lilach Eden, Roni Fried- man, Dan Lahav, and Noam Slonim. 2020b. Quanti- tative argument summarization and beyond: Cross- domain key point analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 39-49. Association for Com- putational Linguistics.
To evaluate the effectiveness of the generated summaries, we conducted user study with five participants that demonstrated their usefulness in exploring discussions. Further research is needed on how to effectively integrate summaries of this type into discussion platform interfaces, which was beyond the scope of this paper.
Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions can dia- logue acts of individual messages help? In Proceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2127- 2131, Doha, Qatar. Association for Computational Linguistics.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
Acknowledgments
This work was partially supported by the Eu- ropean Commission under grant agreement GA 101070014 (OpenWebSearch.eu)
Sid Black, Leo Gao. Phil Wang. Connor Leaby, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow.
References
Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argu- mentation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-JJCNLP), pages 2915-2925, Hong Kong, China. Association for Com- putational Linguistics.
Amber E. Boydstun, Dallas Card, Justin Gross, Paul Resnick, and Noah A. Smith, 2014. Tracking the de- velopment of media frames within and across policy issues.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voo Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.
Christopher Akiki, Odumayo Ogundepo, Aleksandra Piktus, Xinyu Zhang, Akintunde Oladipo, Jimmy Lin, and Martin Potthast. 2023. Spacerini: Plug-and- play search engines with pyserini and hugging face. CoRR, abs/2302. 14534.
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea San- tilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang. Jason Alan Fries,

---

Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
Tanvir Ahmed Fuad, Mir Tafseer Nayeem, Asif Mah- mud, and Yllias Chali. 2019. Neural sentence fusion for diversity driven abstractive multi-document sum- marization. Comput. Speech Lang., 58:216-230.
Tim Gollub, Matthias Busse, Benno Stein, and Matthias Hagen. 2016. Keyqueries for clustering and labeling.
Xiaoyan Cai, Wenjie Li, Ouyang You, and Hong Yan. 2010. Simultaneous ranking and clustering of sen- tences: A reinforcement approach to multi-document summarization. In COLING 2010, 23rd Interna- tional Conference on Computational Linguistics. Pro- ceedings of the Conference, 23-27 August 2010, Bei- jing, China, pages 144-142. Tsinghua University Press.
In Information Retrieval Technology 12th Asia In- formation Retrieval Societies Conference, AIRS 2016, Beijing, China, November 30 - December 2, 2016, Proceedings, volume 9994 of Lecture Notes in Com- puler Science, pages 42-55. Springer.
Maarten Grootendorst. 2022. Bertopie: Neural topic modeling with a class-based TF-IDF procedure. CoRR, abs/2203.05794.
Ricardo G. B. Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based -base clustering based on hi- erarchical density estimates. In Advances in Knowl- edge Discovery and Data Mining, 17th Pacific-Asia Conference, PAKDD 2013, Gold Coast, Australia, April 14-17, 2013, Proceedings, Part II, volume 7819 of Lecture Notes in Computer Science, pages 160- 172. Springer.
Felix Hamborg, Anastasia Zhukova, and Bela Gipp. 2019a. Automated identification of media bias by word choice and labeling in news articles. In 19th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2019. Champaign, IL, USA, June 2-6, 2019, pages 196-205. IEEE
Felix Hamborg, Anastasia Zhukova, and Bela Gipp. 2019h. Illaal aliens or undocumented immigrants? towards the automated identification of bias by word choice and labeling. In Information in Contemporary Society- 14th International Conference, iConference 2019, Washington, DC, USA, March 31 April 3, 2019, Proceedings, volume 11420 of Lecture Notes in Computer Science, pages 179-187. Springer.
Wei-L.in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Dennis Chong and James N. Druckman. 2007. Framing theory, Annual Review of Political Science, pages 103- 126.
Philipp Heinisch and Philipp Cimiano. 2021. A multi- task approach to argument frame classification at vari- able granularity levels. it Information Technology. 63( 1::59722.
Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, pages 758-759. ACM.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Ryuji Kano, Yasuhide Miura, Motoki Taniguchi, Yan- Ying Chen, Francine Chen, and Tomoko Ohkuma. 2018. Harnessing popularity in social media for extractive summarization of online conversations. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 1139-1145. Brussels, Belgium. Association for Com- putational Linguistics.
Charlie Egan, Advaith Siddharthan, and Adam Wyner. 2016. Summarising the points made in online politi- cal debates. In Proceedings of the Third Workshop on Argument Mining (ArgMining20166,, pages 134 143, Berlin, Germany, Association for Computational Linguistics.
Maurice George Kendall. 1948. Rank correlation meth- ods.
Robert M Entman. 1993. Framing: Towards clarifica- tion of fractured paradigm. McQuail's reader in mass communication theory, 390:397.
Mike Klaas. 2005. Toward indicative discussion fora summarization. UBC CS TR-2005, 4.
Ori Emstt Avi Caciularu, Ori Shapira, Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-leve clustering for multi- document summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 1765-1779, Seat- tle. United States. Association for Computational Linguistics.
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens. Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi. Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri, Andrew Maguire. Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations de- mocratizing large language model alignment. CoRR. abs/2304.07327.

---

Percy Liang, Rishi Bommasani, Tony Lee. Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang. Christian Cosgrove, Christopher D. Man- ning. Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto. Thomas Icard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models. CoRR, abs/2211.09111
20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11 11-12, 2016, pages 280-290. ACI..
Mir Tafseer Nayeem, Tanvir Ahmed Fuad, and YI- lias Chali. 2018 Abstractive unsupervised multi- document summarization using paraphrastic sentence fusion. In Proceedings of the 27th International Conference on Computational Linguisties. COLING 2018, Santa Fe. New Mexico, USA, August 20-26, 2018, pages 1191-1204, Association for Computa- tional Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul E. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Siyi Liu, Lei Guo, Kate K. Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news fram- ing trends surrounding U.S. gum violence. In Pro- ceedings of the 23rd Conference on Computational Natural Language Learning, CONLL 2019, Hong Kong, China, November 3-4, 2019, pages 504-514, Association for Computational Linguistics.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.00116.
Christopher D. Manning, Prabhakar Raghavan, and Hin- rich Schütze. 2008. Introduction to information re- Irieval. Cambridge University Press.
Hanieh Poostchi and Massimo Piccardi. 2018. Cluster labeling by word embeddings and wordnet's 's hyper- nymy. In Proceedings of the Australasian Language Technology Association Workshop 2018, Dunedin, New Zealand, ALTT 2018, December 10-12, 2018, pages 66-70.
Leland McInnes, John Healy, and sk Steve Astels. 2017. hdbscan: Hierarchical density based clustering. J. open-source Softw,, 2(11):205.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Cristian Popa and Traian Rebedea. 2021. BART-TL: weakly-supervised topie label generation. Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, FACII 2021, Online, April 19 -23, 2021, pages 1418-1425. Association for Com- putational Linguistics.
Amita Misra, Pranav Anand, Jean F. Fox Tree, and Mar- ilyn Walker. 2015. Using summarization to discover argument facets in online idealogical dialogue. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 430 440, Denver, Colorado. Association for Computational Linguistics.
Dragomir R. Radev, Hongyan Jing, Magorzata Sty, and Daniel Tam. 2004. Centroid-based summariza- tion of multiple documents. Inf. Process. Manag.. 40(6):119995
Sarvesh Ranade, Jayant Gupta, Vasudeva Varma, and Radhika Mamidi. 2013. Online debate summariza- tion using topic directed sentiment analysis. In Pro- ceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, WISDOM 2013, Chicago, IL, USA, August 11, 2013, pages 7:1-7:6. ACM.
Fred Morstatter, Liang Wu, Uraz Yavanoglu, Steven R. Corman, and Huan Liu. 2018. Identifying fram- ing bias in online news. ACM Trans, Soc. Comput., 125:5:11-::18.
Nona Naderi and Graeme Hirst. 2017. Classifying frames at the sentence level in news articles. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 536-542, Varna, Bulgaria. INCOMA Ltd.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese nettons.. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to- sequence mns and beyond. In Proceedings of the
Nils Reimers, Benjamin Schiller, Tilman Beck, Jo- hannes Daxenberger, Christian Stab, and Tryna

---

Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 567- 578, Florence, Italy, Association for Computational Linguistics.
Frame-orienter summarization of argumentative dis- cussions. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2023, Prague, Czechia, September 11 15, 2023, pages 114-129. Association for Computational Linguisties.
Zhaochun Ren, Jun Ma, Shuaiqiang Wang, and Yang
Chenhao Tan, Vlad Niculae, Cristian Danescu- Niculescu-Mizil. and Lillian Lee. 2016. Winning ar- guments: Interaction dynamics and persuasion strate- gies in good-faith online discussions. In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 15, 2016, pages 613-624. ACM.
Liu. 2011. Summarizing web forum threads based on a latent topic propagation process. In Proceedings of the 20th ACM International Conference on Informa- tion and Knowledge Management, CIKM '11, page 779, New York, NY. USA. Association for Com- puting Machinery.
François Role and Mohamed Nadif. 2014. Beyond cluster labeling: Semantic interpretation of clusters' contents using a graph representation. Knowl. Based Sywt, 56: 141-155.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction following lamaa model. https://github. batttttsu-aa///taoodd...oo alpaca.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen. Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICIR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Sansiri Tampradab, Fei Liu, and Kien A Hua. 2017. Toward extractive summarization of online forum discussions via hierarchical attention networks. In The Thirtieth International Flairs Conference.
Almer S. Tigelaar, Rieks op den Akker, and Djoerd Hiemstra. 2010. Automatic summarisation of discus- sion fora. Nat. Lang. Eng., 16(2):61-192
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR. abs/2302.1397
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter acen-sceess multilingual language model. CoRR, abs/2211.05101 1.001000.
Marilyn A. Walker, Jean E. Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Proceed- ings of the Fighth International Conference on Lan- guage Resources and Evaluation, LREC 2012, Is- tanbul. Turkey, May 23-25, 2012, pages 812-817 -17. European Language Resources Association (ELRA).
Xiaojun Wan and Jianwu Yang. 2008 Multi-document summarization using cluster-based link analysis. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008 Singapore, July 20-24, 2008, pages 299-306. ACM.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR. abs/2212.10560.
Qingyi Si and Zheng Lin. 2023. Alpaca-cot: An in- struction fine-tuning platform with instruction data collection and unified large language models inter- face. https://github.o http:://giiabb.oom///eeebuaaaaap wow./PhoebbsSSialaacaaaaca COT.
Canwen Xu. Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open- open-source chat model with parameten-efffcient tuning self-chat data. arXiv preprint arX(h:2304.01116.
Shahbaz Syed, Timon Ziegenbein. Philipp Heinisch, Henning Wachsmuth, and Martin Potthast. 2023.

---

Amy X. Zhang, Lea Verou, and David R. Karger. 2017.
B Clustering Implementation
Wikum: Bridging discussion forums and wikis us- ing recursive summarization. In Proceedings of the
We employed HDBSCAN, a soft clustering algo- rithm (Campello et al., 2013) to cluster the contex- tual sentence embeddings from SBERT (Reimers and Gurevych, 2019). As these embeddings are high dimensional, we follow Grootendorst (2022) and apply dimensionality reduction these em- beddings via UMAP (McInnes al., 2017) and cluster them based on their euclidean distance. Most parameters were selected according to official recommendations for UMAP, and HDBSCAN.
2017 ACM Conference on Computer Supported Co- operative Work and Social Computing, CSCW 2017. Portland, OR, USA, February 25 March 1, 2017, pages 2082-2096. ACM.
Susan Zhang. Stephen Roller. Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01066
UMAP Parameters
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
metric
We set this to "cosine" because this is the natural metric for SBERT embeddings.
neighbors We set this to 30 instead of the de- fault value of 15 because this makes the reduction focus more on the global structure. This is impor- tant since the local structure is more sensitive noise.
Yang Zhang. Yunging Xia, Yi Liu, and Wenmin Wang. 2015. Clustering sentences with density peaks for multi-document summarization. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Denver, Colorado, USA, May 31 June 2015, pages 1262- 1267. The Association for Computational Linguistics.
n e_components
We set this value to 10.
min min_dist We set this value to 0 because this al- lows the points be packed closer together which makes separating the clusters easier.
A Preprocessing
HDBSCAN Parameters
metric
We set this to "eucl idean" because this the target metric that UMAP uses for reducing the points.
Deleted posts were matched using: "[deleted]", "[removed]", "[WWii[Cooell///DllBBoo""" "[His- tory]". To remove posts from moderators, we used:
method We set this value to "leaf". An alternative choice for this options is "eom". This option has the tendency to create unrea- sonably large clusters. There are instances where creates only two or three clusters even for very large discussions. The "leaf" method does not suffer from this problem but is more dependent on the mnin_cluster_ssee
cluster selection clustersselection
"hello, users of cmv! this is a footnote from your moderators"
"comment has been remove"
"comment has been automatically removed"
"if you would like to appeal, please message the moderators by clicking this link."
parameter.
cluster_siii This parameter is the most im- portant one for this approach. It is also not straight forward to find value for this since the sizes of the main subtopics of discussion depend the size of the discussion. To find good value, we sampled 50 discussion randomly and 50 discus- sion stratified by discussion length from all dis- cussions. We compute the clustering for all 100
"this comment has been overwritten by an open source script to protect"
"then simply click on your username on red- dit, go to the comments tab, scroll down as far possibe (hintuse res), and hit the new overwrite button the top."
"reply their comment with the delta sym- bol"
http::///wmmp.eceeeeeeeeeeedd//s "ttps://hdbscan hwtpaa/mww.ccccccooooooo//////aa selection.html

---

chatcucluuttrrsteee Exploring the Discussion via an Indicative Summary
SipeOor
CSNC The wittert have worder arguenent e and stuudd nover be used in actual COn expressed person
rerrs
LAt rucso mous pacie atu cow sctecisaaco with rapartac peppl by sang chaa bad ELAat agg Whar nat' AOGA tour
bru wul fas warky by sure
bod
Jugh
on of dowenton ned too nele these Nooeed 198)
prsoani pa aney it wossarin sigrocca seat
apordds roat pm very to thee aight the rn the 20e prodiir an and cifficiies diy mural retheiig
wemiow nens pargustttoo sistie
apuradiaiin mingii thel
be leteeiteri sowaery cood meie lnesuig hea byy per peroectir Can my me taid tha winiide inwwering
Fnrsactvand ta camporeeeeen tne D13
Pudd uatca wuch woll (rr)
Compaiing riutions ataeti Iecotaa canarna
poed we
Tootiom revi sorile tvellin
bod
Chanter
beriin acc medecttin fork tromdd now oi maaee sacce denyy cherical bueecc te fest
lescramard mecre sconatoers nowar
yoar Kray bo soiat
tnapping
the crppiig ontire the ty's det spenion
Acoeis teme loakd extect fach popee to pronn mece Bi holy que on fnae
inte wit sconstio
pelp
that bocck
coee vi the tre sntwe ard oepwaric out mi boin
Irr cae mohe temers for worene menersec soma yousee Sesttie the sol chamae
cardi as repior poa
derminn on as kis Jom lesee the be with what dent snd fit noult by roco retirrs priss crcat
Figure 5: An exploratory view provided by DISCUSSION EXPLORER quickly navigate long discussion via an indicative summary. On the left, clicking on a cluster label lists all its constituent sentences. On the right, a specific sentence from the chosen cluster is presented in the context of the discussion. Softly highlighted are the sentences from other clusters that surround the selected sentence. Users thus easily skim a discussion with several arguments for relevant information using the indicative summary in this exploratory view.
discussion for different values for min_ pouster_size and manually determine a lower and upper bound for min hrn__cluter__sie size that give a good clustering. We computed a regression model using the follow- ing function family as a basis: f(x/aa, b) = a x The input variable x is the number of sentences in the discussion and the output variable is the aver- age of the upper and lower bound. This yields the following function for computing
30
nir.
20
10
f(x) = 0.421 . 0.539 Figure 6 visualizes upper and lower bounds as well as the found model.
200
1000
A3400
2000
2300
2009
3500
number of sentercces scuesion
Generative Cluster Labeling
Figure 6: Blue vertical bars show the upper and lower bound for ciii_cesttrr___ size that yield a good clustering for the corresponding discussion. The red curve shows the optimal fit for the regression.
Model Descriptions Given the large number of models investigated in the paper for both the tasks, we categorized them based on their release time- lines. Models older than GPT3.5 are listed under Pre-Ins tructtPP such as TO, BLOOM, GPT-NeoX, and OPT. The Direct and Dialogue labels refer to models released after GPT3.5 which differ in their prompt styles as shown in Figure 7. Best prompts for the manually evaluated models (Section 5.2) are shown in Figure 8.
cluding summarization, and surpasses GPT-3 in some tasks despite being much smaller. It was trained on prompted datasets where supervised datasets were transformed into prompts.
2. BLOOM (Scao al., 2022) is an autoregressive L.M with 176B parameters, which specializes in prompt-based text completion for multiple languages. It also supports instruction- based
1. TO (Sanh et al., 2022) is a prompt- based encoder- decoder model, fine-tuned on multiple tasks in-

---

task completions for previously unseen tasks.
Direct Instruction for Cluster Labeling
3. GPT-NeoX (Black et al., 2021) is an open- source, general-purpose alternative the GPT-3 model (Ouyang al., 2022) containing 20B pa- rameters.
Every input the content dehate. every input, you generate single descriptive phrase that deseribes that input in very simple language, without talking about the debate the author.
4. OPT (Zhang al.. 2022) is an autoregressive LLM with 66B parameters from the suite of decoder-only pre-trained transformers. These models offer similar performance and sizes as GPT-3 while employing more efficient practices for data collection and model training.
Dialogue Instruction for Cluster Label ing
chat between curious user and an artificial intelli- gence assistant. 'The user presents debate and assistant generates single descriptive phrase that describes the dehate in very simple language, without talking about the debate or the author.
5. GPT3.5 (Brown et al., 2020; Ouyang et al.. 2022) is an eestructionfoollwwee LLM with 175B parameters that outperforms the GPT-3 model across several tasks by consistently ad- hering to user-provided instructions and gener- ating high-quality, longer outputs. We used the text-davinci-0O) variant. In contrast to the other open-source models, it is accessible exclusively through the ApenAl API.
Direct and dialogue-style instructions for gen- erative cluster labeling prompts. The best prompts for each model are shown in Figure 8.
Figure 7:
annotation interface used to collect the rankings for cluster label quality.
D Frame Assignment
Model Descriptions We categorize the models according to the instruction style followed for fine- tuning and generation. Instructions for each type are shown in Figure 10. The best prompts for each model are listed in Figure 11.
Prompt Descriptions We investigated several prompt templates for each model and selected the best performing one. All the prompts investigated for the encoder-decoder TO model are shown in Ta- ble 11. Prompt templates for the decoder-only Pre- InstructGPT models (BLOOM, ОPT, GPT-NcoX) are listed in Table 12. Prompt templates for the cestruction-oolloinn LLMs are listed in Table 13.
Direct Instruction Models
1. LLaMA-COT is a finetuned model on datasets inducing chain-of-thought and logical deduc- tions (Si and Lin, 2023).
Automatic Evaluation For the sake of comple- tion, we automatically evaluated the recently re- leased (at the time of writing) cnstruction-folloiiin models. To adapt them to generative cluster label- ing, devised two instructions (Figure 7) similar the direct and dialogue style instructions used for frame assignment (Section 3.3). Next, we com- puted BERTScore and ROUGE against two sets of references: (1) manually annotated ground-trum truth labels for 300 clusters, and (2) cluster labels from GPT3.5 5 which was the best model per our man- ual evaluation (Section 5.2, Table 3). Complete results for BERTScore along with length distribu- tions for the generated cluster labels shown in Table 7, while results for ROUGE are shown in Table 8.
2. Alpaca (Taori et al., 2023) is finetuned from the LLaMA 7B model (Touvron et al., 2023) using 52K self-instructed instruction-follooing examples (Wang et al., 2022).
OASST is finetuned from LLaMA 30B on the OpenAssistant Conversations dataset (Köpf et al., 2023) using reinforcement learning.
4. Pythia (Biderman et al., 2023) is a suite of LLMs trained on public data to study the impact of training and scaling on various model prop- erties. We used the 12B variant finetuned on the OpenAssistant Conversations dataset (Köpf et al., 2023).
GPT includes models such as text-davinci-033, gpt-3.5-turbo (ChatGPT), and GPT-4 (Bubeck
Manual Evaluation Table 9 shows the guideline provided to the annotators. Figure 9 shows the
hwtpss//huugggiggaaee///oauobbs wttp::/wwwwww..so//////oaa/a Baeeaaaa----uupppp 30b-superool
"https:/huggiingiace wotp:/ooooooosssstaattt
oasst-rlhf- 2-11a 1lllm-lm300 30h-7k--toppeee stess-p

---

Best Prompts for Generative Cluster Labeling
TO
Read the following context and answer the question.
Context: [texr]
Wiic parvcee
Question:
What is the title the discussion?
Answer:
consbecrns creecs
BLOOM
Al assistant: I am expert Al assistant and I am
very good in identifying titles of debates. DEBATE START (texr] DEBATE END
Al assistant: The title of the dehate between the two participants is"
divbl Cys
GPT-NeoXX
should taxt
DISCUSSION START [tear) DISCUSSION END
Q: What is the topic of the discussion?
A: The topic of the discussion is"
Nasting iell
OPT-66B
DEBATE START [text) DEBATE END
Q: What is the topic of the debate?
A: The topic of the dehate is
c0m
GPT3.5 (lext-davinci-003)
Generate a single descriptive phrase that describes following debate in very simple language, without
X
Cluster
talking about debate or the author.
Condall Eesermes frem Cinttr
Debate: """ [
fur ano weere stilyy
Figure 8: Best prompts for generative cluster labeling for each model. These prompts were chosen based on the automatic evaluation of several prompts for each model against 300 manually annotated cluster labels.
Gatter
b0ctut ceter ree sasee eaedy the nedhhee gatt
cnllly
iie by
et al., 2023) from the OpenAl API. These models are not open-source but have demon- strated stateoofthe--ae performance across vari- ous tasks.
Figure 9: Annotation interface for ranking-based quali- tative evaluation of cluster labels.
4. Falcon 19 is trained on the Refined dataset (Penedo al., 2023), which is derived through extensive filtering and deduplication of publicly available web data. It is currently the of the-art (at the time of writing) on the open-llm- leaderboard." 20 We utilized the 40B and 40B- Instruct variants of this model.
Dialogue Instruction Models
1. LLaMA (Touvron et al., 2023) is a suite of open- source LLMs trained on public datasets. We utilized the 30B and 65B variants.
2. Vicuna (Chiang et al., 2023) is finetuned from LLaMA using user-shared conversations col- lected from ShareGPT. 18 It has shown competi- tive performance when evaluated using GPT-4 as a judge. We used the 7B and 13B variants of this model.
D.1 Citation Impact on Frame Assignment
We conducted additional experiments to evaluate the impact of providing the citation of the media frames corpus paper by Boydstun et al. (2014) as additional information in the instructions shown in Section 3.3. This piece of information was pro- vided after the substring "defined by" in the prompt template. Table 5 shows the results. We note that providing the citation information has positive
3. Vicuna (Xu et al., 2023) is an open-source chat model trained on 100k dialogues generated by allowing CharGPT (GPT 3.5-turbo) to converse with itself. We used the 7B and 13B variants of this model.
"https//shareept.com/
19 "hntps//falcoolllm.ti..k 20 "htps://hugginggfac wtwp:/ngwwww.e..oocc////ggggggno ooce Ilm. leaderboard

---

Direct Instruction for Frame AsSi grment
"legality, constitutionality and jurisprudence",
The following [inpw[_yyef" contains available media frames as defined the work from {authors]: (frames] For every imput, you answer with three these media frames corresponding to that input, order importance.
"policy prescription and evaluation",
"crime and punishment",
"security and defense".
"health and safety", "quality of life", "cul tural identity", "public opinion",
"A list of frame labels JSON with frame labels
and their descriptions.
"political", "external regulation and reputation" ]
Dialogue Instruction Frame Ass ignment
D.2.2 Zero-Shot
A chat between a curious user and an artificial intelli- gence assistant. The assistant knows all media frames as defined by [franes). The assistant answers with three of these media frames corresponding to the user's text, in order of importance.
"economic": {
"description" "costs, benefits, or other
financiaa implications"
cappccty and resources":
"description": "availability of physical, human or financial resources, and capacity of current systems"
Figure 10: Best performing instructions for frame as- signment. Providing the citation for the frame inventory via the placeholder [authors] positively affects the per- formance some models (Appendix D.1).
"morality": "ddescriptin"": "religious
ethical implications" }
"fairness and equality": {
"description": "balance distribution of
Prompt
Falcon-40B ChatGPT LLaMA-6SB
rights, responsibilities, and resources"
Cite.
Cite.
Cite.
"legality, constitutionality and jurisprudence":
Zero-Shot
46.5
34,2 60.9 60.1 53.1
44.4
"description": "rights, freedoms, and authority individuals, corporations, government"
Zero-Shot (short)
46.5
42.8 58.0 57.2 50.6
42.4
Zero-Shot (full)
46.1
46.5
58.8 60.9 39.5
39.1
Few-Shot
38.3
39.1 63.4 64.6
"policy prescription and evaluation": {
Table Analysis of the impact of providing citation of the media frames corpus paper as additional information in the instructions for the frame assignment. Providing citation information (Cite.) shows up 12% improve- ment for Falcon-40B and 9% for LLaMA-65B under zero-shot setting (with only frame labels in the prompt).
"description": "discussion of specific
policies aimed at addressing problems"
"crime and punishment":
"description": "effectiveness and implications laws and their enforcement"
"security and defense": {
"description" "threats to welfare of the individual, community, or nation"
impact on the performance of the models. The im- provement is up to 12% for Falcon-40B and 9% for LLaMA-65B under zero-shot setting (only frame labels without descriptions in the prompt). This improvement be attributed the models be- ing trained on a large text corpus, with the citation serving as strong signal for generating more ac- curate labels. However, ChatGPT only slightly affected.
}.
"health and safety": (
"description": "health care, sanitation, public safety"
}.
"quality of life":
"description": "threats and opportunities for the individual's wealth, happiness, and well-being"
"cul tural identity":
"description": "traditions, customs,
D.2 Zero-Shot and Few-Shot Prompts for Frame Assignment
values of social group in relation to policy issue"
D.2.1
Zero-Shot (short)
public opinion":
[
"description": "atti tudes and opinions of the general public, including polling and demographics"
"economic",
"capacity ty and resources", "morrlity",
"fairness and equality",
political" {

---

"description": "considerations related to politics politicians, including lobbying, elections, attempts sway voters"
"description": "Specific policies practice their enforcement, incentives, and implications. Includes stories about enforcement and interpretation of laws by individuals and enforcement, breaking laws, loopholes, fines, sentencing punishment. Increases or reductions in crime."
-
-
),
"external regulation reputation": "description", "international reputation or 4 foreign policy of the U.S."
}.
"security and defense":
"description": "Security, threats to
D.2.3 Zero-Shot (full)
security, and protection of one's person,
family, in-group, nation, etc. Generally
[
"economic": [
an action or a call to action that can be
"descripte" iption"::on "The costs, benefits, or
taken to protect the welfare of a person,
monetary/Financial implications of the issue (to an individual, family, community, ty, or to the economy as whole)."
group, nation sometimes from not yet
manifested threat."
}.
"health and safety":
"descr ipssoription:: "Heal thcare access and effectiveness, illness, disease, sanitation, obesity, mental health effects, prevention of or perpetuation of violence, infrastructure and building safety."
).
"capacity and resources";
"description": "The lack of or availability of physical, geographical, spatial, human, and financial resources, or the capacity of existing systems and resources to implement carry out policy goals."
"qullity of life":
"description": "The effects of policy individuals' wealth, mobility, access resources, happiness, social structures. of day-to-day routines, quality of community life, etc."
g.
"morality:
"deccription": "Any perspective policy objective or action (including proposed action) that is compelled by religious doctrine interpretation, duty, honor, righteousness any other sense ethics social responsibility."
"cultural identity":
"description": "The social norms, trends, values and customs constituting culture(s), as they relate specific policy issue."
3.
"fairness and equality":
"description": "Equality or inequality with
" which laws, punishment, rewards, and resources are applied or distributed
"public opinion":
among individuals or groups. Also balance between the rights or interests
"description": "References to general social attitudes, polling and demographic information, as well as implied or actual consequences diverging from or Vggetting ahead of\" public opinion or polls."
- of one individual or group compared to
another individual or group."
3.
"legality, constiture constitutionality and jurisprudence": {
"political": {
"description": "The constraints imposed on or freedoms granted to individuals, government, and corporations via the Constitution, Bill of Rights and other amendments, or judicial interpretation.
"ddesrriptti iption"::""" "Any political considerations surrounding an issue. Issue actions or efforts or stances that are political, such as partisan filibusters, lobbyist involvement, bipartisan efforts,
This deals specifically with the
authority ty of government to regulate, and the authority of individuals/corpora atins
deal makiing and vote trading, appealing to one s base, mentions of political maneuvering. Explicit statements that policy issue is good bad for particular political party."
to act independently of government."
).
"policy prescription and evaluation":
"description": "Particular policies proposed far addressing an identified problem, and figuring out if certain policies will work, if existing policies are effective."
"external regulation and reputation": { "description": "The United States' external relations with another nation; the external relations of state with another; or relations between groups. This includes trade agreements and outcomes, comparisons of policy outcomes desired policy outcomes."
0.
ccrim and punishment":
}

---

D.2.4 Few-Shot
}.
"economic": [
"legality, constitutionality and
"description": "The costs, benefits, monetary/financial implications of the issue (to an individual, family, community, ty,
4. jurisprudence":
"description": "The constraints imposed on or freedoms granted to individuals, government, and corporations via
the economy as
whole).".
Constitution, Bill of Rights and other amendments, or judicial interpretation. This deals specifically with the authority ity of government to regulate, and the authority ity of individuals/ cow.aaaa/coooratiiiiiii porations to act independently of government."
"examples:
Necessity minimum wage laws and their effects on the market." "Consequences unregulated capitalism the potential of libertarian - society.",
"examples": [
"Risk-based insurance premiums determined - by complex modeling of probability and cost factors."
"Guns acquired through legal and illegal
- channels for criminal use.", "Importance of the 2nd amendment and the implications of gun ownership in democracy." ..
1
}.
capacity and resources"; {
"Relevance of sexual history in rape cases."
"description". "The lack of or availability 4 of physical, geographical, spatial, human, and financial resources, or the capacity of existing systems and resources to implement or carry out policy goals.",
"policy prescription and evaluation": {
"description": "Particular policies proposed for addressing an identified problem, and figuring out if certain policies will work, or if existing policies are effective.",
eexamples:
"Potential of biofuels as alternative to Fassil fuels.",
"xxmples":
"Physical fitness tests measure upper body strength and running ability for military service.",
"Religious scientists making major contributions to the world despite majoritty of scientists being agnostic atheists.",
"Physical strength and endurance needed for madern combat."
"Pros and cons of voluntary registration." "Collective ownership of production the betterment society, with workers profiting from the sale their labor."
1
g.
"morality;
desccrptiion: "Any perspective policy
objective action (including proposed action) that is compelled by religious doctrine interpretation, duty, honor, righteousness any other sense of ethics social responsibility."" "examples";
}.
"crime and punishment":
"description": "Specific policies practice and their enforcement, incentives, and implications. Includes stories about enforcement and interpretation of laws by individuals and law enforcement, breaking laws, loopholes, fines, sentencing and punishment. Increases or reductions in crime.".
"Fighting for weak vulnerable despite the odds.", "Victim-blaming debate on police brutality."
"Potential corruption some native canadian bands and the need for transparency."
"examples": [
"Complexities of police shootings and < race.",
}.
"Men are more likely to commi violent crimes than women.".
Fairrees and equality": (
idescription". "Equality or inequality with - which laws, phmint, rewards, and
"Punishment as a response to crime debated, with consideration of morality, severity, and aims."
+ resources are applied or distributed
among individuals or groups. Also the balance between the rights or interests of one individual or group compared to another individual or group.",
"security and defense":
"description": "Security, threats to
exammles":
security, and protection of one's person,
"Differences between humanism and feminism 4 and their respective goals.", "Disparities in scholarship oppor tunities for minority students.".
family, in-group, nation, etc. Generally
call to action that can be taken to protect the welfare of person, group, nation sometimes from not yet manifested threat.".
an action
"Violent suppression of native american ican populations for centuries leading to lack of advocacy and rights."
"examples": [
"Protective physical self-defense fight.",

---

"Powerful military technology making infantry obsolete war. "Protection of infants mentally disabled through social policy."
iptson: "Any political considerations surrounding an issue. Issue actions efforts or stances that are political, such as partisan filibusters, lobbyist involvement, bipartisan efforts,
]
0,
deal-making and vote trading, appealing to one's base, mentions political maneuvering. Explicit statements that a policy issue good or bad for particular political party.".
"health and safety": {
"description". "Heallhcarr access and - effectiveness, illness, disease, - sanitation, obesity, mental health effects, prevention of or perpe petatioon of gun violence, infrastructure and - building safety.",
"examples": [
"Differences between right-wing and 4
left-wing politics.",
eexamples
"Complexities of anarchy.", "Power struggle between branches of
"Complexities of food choices and their effects on health.",
4 government."
"Potentially fatal consequences of taking tao much acetaminophen."
}.
"Encouraging healthy habits without shaming or pressuring people to lose weight."
"external regulation and reputation": { "description": "The United States' external relations with another nation; the external relations of one state with another; or relations between groups. This includes trade agreements and outcomes, comparisons of policy outcomes desired policy outcomes.",
]
).
quallity of life":
"deccription": "The effects of policy individuals' wealth, mobility, access to resources, happiness, social structures, ease of day-to-day routines, quality of community ty life, etc.".
"examples": [
"Implications of us involvement in nato and its allies.".
examples":
Differences between adults and children in terms of understanding and perception.".
"Potential consequences of us intervention in ukraine."
"Conflitting opinions involvement foreign affairs."
Impor of extracurriculars and academics for college admissions."" "Appropriate times yell service workers."
}
}.
"cultural identity"; {
"description": "The social norms, trends, + values customs constituting
culture(s), as they relate to specific policy issue.",
"examples::
"Rapid shift in acceptance of homosexuality
+ in the u.s.",
"Collective action necessary for social
- progress and change."
"Complexities of gender identity and expression."
1
}.
"public opinion":
"descriptine iption"::" "References to general social attitudes, polling and demographic information, as well as implied actual consequences of diverging from \ggetiin ahead of\" public opinion or polls.",
"examples":
"Gender roles and expectations are socially constructed and changing.", "Pros and of the 48-hour work week.", "Potential appeal of political candidate."
]
).
"political":

---

{ "role": "user", Best Prompts for Frame Assignment
Alpaca-7B (Direct Instruction)
Below instruction that describes task, paired with an input that provides further context. Write response that appropriately completes request.
#*# Instruction:
[instruction]
### Input:
[input]
##H Response:
Vicuna-7B, 13B (Dialogue Instruction)
[instruction]
USER: [input]
ASSISTANT:
Pythia, OASST (Direct Instruction)
LLaMA-30B, 65B (Dialogue Instruction)
(instruction)
USER: [input]
ASSISTANT:
LLaMA-CoT (Direct Instruction)
Below instruction that describes task, paired with an input that provides further context. Write response that appropriately completes request.
#ie Instruction:
(instruction)
### Input:
(input)
### Response:
Falcon-40BB Instruct (Dialogue Instruction)
(instruction]
USER: [input]
ASSISTANT: ["
Baize-7B, 13B (Dialogue Instruction)
(instruction)
[ [ (input]
[AA]]]
GPT3.5 (Direct Instruction)
(instruction]
Input:
(input]
Answer:
ChatGPT (Direct Instruction)
"role": "system",
"content"; "(instruction)"
"content": "(input3"
GPT4 (Direct Instruction)
"role": "system", "content": "(instruction)"
{
"role": "user", "content": "(input3"
Figure 11: Best prompts for frame assignment for each model. The direct and dialogue instruction to be used with each prompt is shown Figure 10.

---

Frame
Description
Capacity & Resources
The lack of or availability of physical, geographical, spatial, human, and financial resources, or the capacity of existing systems and resources to implement or carry out policy goals.
Constitutionality Jurisprudence
The constraints imposed corporations via the Constitution, Bill of Rights and other amendments, judicial interpretation. This deals specifically the authority government regulate, and the authority individuals/copprraiions
freedoms granted individuals, government. and
independently of government.
Crime & Punishment
Specific policies in practice and their enforcement, incentives, and implications. Includes stories about enforcement and interpretation of laws by individuals and law enforvement, breaking laws, loopholes, fines, sentencing and punishment. Increases or reductions in crime.
Cultural Identity
The social norms, trends, values customs constituting culture(s), they relate to specific policy issue.
Economic
The costs, benefits, monetary/financia implications of the issue (to individual, the cconomy whole).
family, community
External Regulation & Reputation
A country's 's external relations with another nation; the external relations of one state with another; or relations between groups. This includes trade agreements and outcomes, comparisons of policy outcomes or desired policy outcomes.
Fairness & Equality
Equality inequality with which laws, punishment, rewards, and resources are upplied distributed among individuals groups. Also the balance between the rights interests of one individual or group compared another individual group.
Health & Safety
Healthcare access and effectiveness, illness, discase, sanitation, obesity, mental health effects, prevention of or perpetuation of gun violence, infrastructure and building safety.
Morality
Any perspective- or policy objective or action (including proposed action) is compelled by religions doctrine or interpretation, duty, honor, rightcousness any other sense of ethics social responsibility.
Policy Prescription & Evaluation
Particular policies proposed for addressing identified problem, and figuring out if certain policies will work, if existing policies effective.
Political
Any political considerations surrounding an issue. Issue actions or efforts or stances that are political, such as partisan filibusters, lobbyist involvement, bipartisan efforts, deal- deal-making makkimaking and vote trading, appealing to one's 's base, mentions of political maneuvering. Expliiit statements that policy issue is good or bad for a particular political party.
Public Opinion
References general social attitudes. polling and demographic information, well implied actual consequences diverging from getting ahcad of public opinion or polls.
Quality of Life
The effects of a policy, an individual's actions or decisions, on individuals wealth, mobility, access resources. happiness, social structures, ease of day-to-day rou- tines, quality of community life, etc.
Security & Defense
Security, threats to security, and protection one's person, family, in-group, nation, etc. Generally an action or call action can taken to protect the welfare person, group. nation sometimes from a not yet manifested threat.
Other
Any frames that do not fit into the above categories.
Table 6: Descriptions of frames as per Boydstun et al. (2014). For zero-shot prompts, we experimented with providing (1) list of frames, (2) frames with relevant aspects from the descriptions (e.g. the economic frame has the aspects "costs", "benefits", "financial implications"), and (3) frames with complete descriptions additional context.

---

Model
Reference
GPT3.5
Length
R
F1
P
F1
Min
Max
Mean
Alpaca-78
0.20
0.15
0.17
0.31
0.28
0.29
7.92
Baize-138
0.17
0.15
0.16
0.33
0.32
0.32
8.47
Baize-7B
0.22
0.19
0.20
0.38
0.38
0.38
46
10.73
BLOOM
0.15
0.09
0.22
0.20
8.13
Falcon-40B
0.12
0.09
0.10
0.17
0.17
0.17
57
9.57
Falcon-40B-Inst.
0.22
0.18
0.20
0.34
0.32
0.33
33
9.34
ChatGPT
0.23²
0.24
0.23'
0.39
0.43
0.41
11.10
GPT4
0.21
0.19
0.20
0.37
0.36
0.37
4
18
7.50
GPT-NeoX
0.19
0.07
0.12
0.24
0.17
0.20
7.42
LLaMA-308
0.12
0.06
0.08
0.19
0.17
0.17
9.58
LLaMA-COT
0.24
0,21
0.22
0.41
0.39
0.40
29
8.45
LLaMA-658
0.02
0.14
0.14
10.27
OASST
0.22
0.21
0.21
0.392
0.40a
0.40
31
10.15
OPT
0.16
0.09
0.12
0.22
0.19
0.20
30
8.27
Pythia
0.13
0.31
0.29
Te
0.15
0.00
0.06
0.15
0.03
0.09
18
3.10
GPT3.
0.23
0.20
0.21
27
9.44
Vicuna-138
0.21
0.21
0.21
0.36
0.39
0.37
11.87
Vicuna-7B
0.20
0.19
0.19
0.34
0.37
0.35
2
42
11.47
Table Complete results of automatic evaluation via BERTScore for the cluster labeling task of all LLMs. We compared them against the manually annotated reference and GPT3. 5, the best model from our manual evaluation. The top three models are indicated for each metric. Similar to the ROUGE evaluation, we see a strong performance ChatGPT and LLaMA-COT. Also shown the statistics of the length of the generated cluster labels (in number of tokens).
Model
Reference
GPT3.5
R-1
R-2
R-LCS
R-1
R-2
R-LCC
Alpaca-7B
13.89
3.10
12.65
19.98
6.08
18.05
Baize-13B
14.44
2,28
13.02
24.59
8.23
22.53
Baize-7B
17.40
2.88
14.95
26.35
9.43
23.89
BLOOM
12.52
2.52
11.34
13.10
3.74
12,20
Falcon-408
14.30
13.26
13.08
11.92
Falcon-40B-Inst.
17.59
3,97
1
21.72
7.66
19.57
ChatGPT
20.15
4.88
17.42
29.52
10.992
25.75
GPT4
16.43
2.84
14.42
27.76
9.573
24.80
GPT-NeoX
12.93
2.37
11.72
11.67
2,41
10.72
LLaMA-30B
12.30
2.60
11.19
12.07
2,70
11.14
LLaMA-COT
18.91
4.50²
16.83
28.94
11.69
26.38
LLaMA-65B
10.25
1.93
9.40
10.81
2.49
9.95
OASST
18.28
3.58
16.15
27.13
23.88
OPT
11.67
2.68
10.87
10.56
2.17
9.55
Pythia
14.78
2,99
13.23
21.64
6.44
19.67
9.80
2.01
9.61
7.64
1.70
GPT3.
16.82
2.96
14.61
Vicuna-138
16.90
3.02
14.81
25.32
8.66
22.66
Vicuna-7B
17.04
2.62
14.81
23.88
20.87
Table 8: Complete results of automatic evaluation via ROUGE for the cluster labeling task of all 19 LLMs. We compared them against the manually annotated reference and GPT3. 5, best model from our manual evaluation. The top three models indicated for each metric. We see that ChatGPT and LLaMA-COT perform strongly across the board.

---

Guideline for judging the quality of the clustering
Task: Given a reference text and set hypotheses, rank the hypotheses based on how similar they are to the reference text.
How similar are the small texts to the reference text?
Drag and drop the boxes with texts on the left bring them in your preferred order on right. The most preferred text on the top and less you prefer text, the lower should the ranking.
Similarity is less in sense of exact meaning but much rather in a meaning of is there some relation between the reference and hypotheses.
To get better understanding of the meaning of the reference, the title of the original discussion and central sentences from the cluster provided (click the "show cluster" button the reference). The central sentences selected based on how central they their similarity the reference and hypotheses. So these not perfectly representative to the cluster, but they can help you get better understanding of some hard understand meanings.
the original cluster and
Recommended Strategy for judging:
The relation between the reference and hypotheses is understandable:
+ only read reference the hypotheses
The reference is bit weird:
+ read the title to get hetter idea in what context the reference is used
The hypotheses are hard to understand:
+ read the central sentences from the cluster for more context
The relation between the reference and hypotheses are not clear:
read the central and random sentences from cluster
Note: We are looking for a label that sufficienlly describes the content of cluster of sentences. It is important to understand that the reference is not the perfect label but rather strongly representative of the cluster.
When lot hypotheses talk about something that is not the reference, is sensible to include this information in the reference (implicitly) to make it "complete" while ranking the hypotheses.
Example:
Reference: responsibilities between employee and employer
Majority of the given hypotheses mention: "the service industry"
Updated Reference: responsibilities between employee and employer in the service industry In the end we are looking for the central meaning of the cluster and it very likely that at least one model got the central meaning right and the task is to guess what model got the central meaning best based on what the reference suggests the best central meaning is.
Table Guideline for judging the quality of the clustering.

---

Model
Zero-Shot
Zero-Shot (short)
Zero-Shot (full)
Few-Shot
top 1
top
top
top
top 2
top 3
top
top 2
top 3
top
top
top
Alpaca-7B
39.1
64.2
39.5
51.0
28.4
37.4
57.2
26.7
49.4
BLOOM
26.7
46.5
53.5
31.7
52.7
57.6
25.5
51.9
60.1
Baize-13B
42.4
53,5
58.4
48.1
59.3
63.4
42.0
53.5
60.5
39.5
46.5
49.4
Baize-7B
34.2
44.4
52.7
34.6
46.9
56.8
39.1
46.5
53.9
30.9
38.3
45.7
Falcon-40B
46.5
68.3
72.0
46.5
67.5
75.7
46.1
56.8
64.2
38.3
53.5
68.3
Falcon-48B-Inst
51.4
64.6
72.8
44.4
56.4
68.3
32,9
44.9
57.6
28.4
49.4
63.8
ChatGPT
60.9
76.1
86.4
58.0
78.6
88.5
58.8
76.1
84.8
63.4
80.2
90.1
GPT-4
63.4
82.3
91.8
60.5
84.4
90.1
65.4
83.1
90.5
67.1
84.8
88.5
GPT-NeoX
19.3
50.6
25.1
31.3
31.3
36.6
50.2
39.5
49.0
LLaMA-30B
45.7
63.0
70.8
41.2
57.2
65.4
39.1
58.0
66.3
40.7
70.0
77.8
LLaMA-CoT
46.9
73.3
84.0
54.3
75.7
85.6
49.8
71.2
82.3
57.2
70.0
77.0
LLaMA-65B
53.1
81.9
50.6
70.8
39.5
64.6
78.6
OASST
48.6
72.8
82.3
48.1
66.3
76.5
53.5
73.7
82.7
47.7
65.0
79.8
OPT-66B
16.0
18.9
43.2
13.2
16.5
45.3
14.8
18.1
45.7
Pythia
31.7
44.0
52.3
33.3
43.6
30.5
44.9
34.2
38.7
T@++
48.6
58.4
64.2
54.3
60.1
65.4
55.6
59.7
63.8
49.8
52.3
53.5
GPT3.5
53.5
74.1
81.9
60.9
65.4
66.7
58.0
58.8
59.7
53.9
57.6
58.0
Vicuna-138
44.0
52.7
62.1
40.7
55.1
67.1
42.0
53.1
64.6
38.3
50.2
60.1
Vicuna-7B
28.4
34.6
50.2
36.2
48.1
61.3
35.4
42.8
55.1
20.2
24.3
46.1
Table 10: Complete results of automatic evaluation for the frame assignment task. Shown the % of examples where the first, second, and third predicted frames by a model are one of the reference frames. For the zero-shot setting, values are shown for each the prompt type: only frame label, label with short description, and label with full description. Missing values model inferences that exceeded computational resources.
Prompt Templates for TO
prefix
What (output_type) would you choose for the Sinput_type) below?
(text)
postfix
(text)
What (output_type) would you choose for the Einput_type) above?
prefix-postfix
What (output_type) would you choose for the (input_type] below? (text)
What (output_type) would you choose for the (input_type] above?
short
(input_type):
(text)
(output_type)):
explicit
(input_type) START (text)
(input_type] END (output_type) OF THE (input_type):
question answering
Read the following context answer the question.
Context:
(text)
Question: What (output_type) the (input_type}?
Answer:
Table 1l: Prompt templates investigated for TO model for generative cluster labeling.

---

Prompt Templates for BLOOM, GPT-NeoX, OPT, GPT3.5
dialogue
AI assistant: am expert AI assistant. How can I help you?
Human: Can you tell what the (output_type) of the following (input_type) is? (input_type) START (text)
(input_type) END
AI assistant: The toutput_type) of the (input_type) is
explicit (input_type] START (text) (input_type] END
(output_type) of the (input_type):
assistant solo
assistant: am an expert assistant and very good identifying > (output_type) of debates.
(input__type) START (text)
(input_type) END
AI assistant: The (output_type) of the (input_type) between two participants
question answering (input_type) START (text)
(input_type) END Q: What is the (output_type) of the (input_type)? A: The (output_type) of the (input_type) is
GPT3.5
Generate a single descriptive phrase that describes the follawing debate in very simple language, without talking about the debate or the author.
Debate: """(text)"""
Table 12:
Prompt templates investigated for generative cluster labeling with the four decoder-only models. The input_type is either "debate" or "discussion" and the sutput_type is either "title" or "topic".

---

Prompt Templates for Instruction-folloiingg LLMs
GPT3.5
(instruction)
Input: ""((inutt"""
Answer:
Alpaca-7B, LLaMA-CoT
Below is instruction that describes task, paired with input that provides further context. Write response that appropriately completes the request.
### Instruction:
(instruction)
ta# Input:
(input)
###l Response:
Baize-13B, Baize-7B
(instruction)
[[Human[][input]]
[[A[]]
BLOOM, Falcon-40B, Falcon-40F-Instrucc, GPT-NeoX, LLaMA-30B, LLaMA-65B, OPT-66B, Vicuna-13B, Vicuna-7B (instruction)
USER: (input)
ASSISTANT:
OASST, Pythia
T0++
(instruction)
Input: (input)
Table 13: Prompt templates used for experiments with -nstructionnfollloiing model.
CMV: The "others have worse" argument terrible and should be used in actual tion with depressed person Indicative Summary (LLaMA-CoT)
Health & Safety
Depression is complex mental health issue that varies in severity and treatment options. [98] (Policy
Prescription & Evaluation)
Impact depression how help those affected. [35] (Morality)
Personal journey of overcoming depression and finding happiness. [17] (Quality Life)
Morality
Gratitude and appreciation for the little things in life can help improve happiness and perspective, [39]
(Quality of Life)
Perspective and its importance in life. [22] (Fairess & Equality)
Positive self-talk and growth mindset [21] (Fairness & Equality)
Policy Prescription & Evaluation
Comparing situations to others' can be helpful harmful. [97] (Morality)
Effectiveness of advice in different situations [25] (Capacity & Resources)
Psychology and the power of the brain to reprogram thought patterns. [22] (Morality)
Table 14: Indicative Summary from LLaMA-CoT.

---

CMV: The "others have worse" argument is terrible and should never be used in an actual conversa- tion with a depressed person Indicative Summary (GPT3.5)
Fairness & Equality
Complexities of comparing one's own struggles to those of others. [97] (Quality of Life)
Advice be helpful unhelpful depending on how it used. [25] (Morality)
Focusing on personal goals and eliminating negative self-talk create growth mindset. [21] (Quality of Life)
Health & Safety
How to help those with depression. [35] (Quality l.ffe)
Morality
Differences between sadness and depression. [98] (Quality of Life)
Reflecting on blessings and practicing gratitude increase happiness. [39] (Quality Life)
Mindful awareness and reprogramming of thought patterns to take charge of emotions. [22] (Quality of
Life)
Gaining perspective to appreciate life and understand how one's actions affect others. [22] (Fairess
Equality)
A journey of self-discovery growth through difficult times. [17] (Quality Life))
Table 15: Indicative Summary from GPT3.5.
CMV: The "others have worse" argument terrible and should never be used in an actual conversa- tion with a depressed person Indicative Summary (GPT4)
Fairness & Equality
Acknowledging personal struggles while recognizing others' hardships [97] (Quality of Life)
Health & Safety
Understanding and managing depression as a complex mental state [98] (Quality of Life)
Importance gratitude for happiness mental health [39] (Quality of Life)
Impact of different approaches to supporting depressed individuals. [35] (Quality of Life)
Controlling and reprogramming thought patterns through mindful awareness and rational evaluation
emotions. [22] (Quality of Life)
Policy Prescription & Evaluation
Ffffcctiveness of advice depends on individual and context. [25] (Quality of Life)
Quality of Life
Gaining perspective for personal growth and understanding. [22] (Morality)
Focusing on positive mindset and self-growth [21] (Health Safety)
Overcoming challenges and finding happiness through personal growth and change. [17] (Morality)
Table 16: Indicative Summary from GPT4,

---

CMV: Today is the best time period in human history to be alive for the vast majority of people.
Indicative Summary (LLaMA-CoT)
Capacity Resources
The importance of having private space for studying and building projects. [33] (Quality of Life)
Crime & Punishment
Crime rates have changed over time. [60] (Sccurity Defense)
Cultural Identity
Nostalgia for the 90s [82] (Quality of Life)
Economic
Housing affordability is complex issue with many factors at play. [203] (Capacity & Resources)
Global poverty has decreased significantly over the past few decades. [135] (Capacity & Resources)
Global trends and perspectives [48] (Policy Prescription & Evaluation)
Health & Safety
AIDS pandemic was more fatal than the current one. [97] (Capacity & Resources)
Current mental health epidemic and its causes. [32] (Capacity & Resources)
Policy Prescription Evaluation
Climate change a serious issue that needs to be addressed. [113] (Economic)
Concentration military and economic power in history. [47] (Economic)
Quality of Life
Best time period in human history to be alive. [72] (Economic)
Quality of Life Expectations: Happiness Dehate [68] (Other)
Progress and improvement in society and culture [51] (Cultural Identity)
Impact of technology on human connection and fulfillment. [39] (Cultural Identity)
Middle Ages Modern Times: Quality Life Comparison [29] (Cultural Identity)
Security Defense
Statistics and data points in a debate about safety and progress [43] (Health & Safety)
Table 17: Indicative Summary from LLaMA-CoT.

---

CMV: Today is best time period human history alive for the majority of people. Indicative Summary (GPT3.5)
Crime & Punishment
Violent & Equality)
has significantly decreased since 1990s, but still remains issue. [60] (Faimess
Cultural Identity
A constant flow of information and socictal changes causing crisis of meaning. [39] (Quality of Life)
Economic
Catastrophic climate change leading to economic and ecological collapse. [113] (HHealth & Safety)
Fragmented global economic and military power. [47] (Sccurity Defense)
Fairness & Equality
Housing prices have skyrocketed in the past decade, making difficult for the average American to ufford a home. [203] (Economic)
Decrease global poverty and hunger since 90s, with a majority of world population still living in poverty. [135] (Capacity Resources)
Differenees between the 90s and the 2000s, and the effects of time periods different generations.
[82] (Quality of Life)
Making progress towards a better world for future generations. (51] (Quality of Life)
Throwing around statisties without meaning and misusing pereentages. [43] (Policy Prescription
Evaluation)
Room study and compete in the job market. [33] (Capacity & Resources)
A comparison of the lifestyles lower-class people in the Middle Ages and modern times. [29]
(Quality of Life)
Health & Safety
Effects of pandemies population growth and life expectancy, with comparison to the Bubonic Plague. [97] (Quality of Life)
Morality
Mental health crisis the modern world and its potential causes. [32] (Quality of Life)
Political
Strong hias towards American perspoctive on glohal issues. [48] (Cultural Identity)
Quality of Life
Best time period in human history to be alive. [72] (Faimess & Equality)
Balance between quality of life, expectations, and happiness, and how they relate each other. [68] (Fairness Equality)
Table 18: Indicative Summary from GPT3.5.

---

CMV: Today is the best time period in human history to be alive for the vast majority of people.
Indicative Summary (GPT4)
Crime Punishment
Violent crime rates have decreased since the 90s. [60] (Security & Defense)
Cultural Identity
Nostalgia for 90s differing opinions [82] (Quality of Life)
Assuming most users American [48] (Public Opinion)
Economic
Housing affordability crisis in various locations [203] (Quality of Life)
Reduced global poverty hunger rates [135] (Fairness Equality)
Concentration of military and economic power in history [47] (Security & Defense)
Health & Safety
Climate change and worsening effects on Earth humanity. [113] (Quality Life)
Comparing pandemics and death rates throughout history [97] (Quality of Life)
Mental health awareness and treatment in modern society. [32] (Quality of Life)
Policy Prescription & Evaluation
Acknowledging progress while recognizing room for improvement [51] (Quality of Life)
Quality of Life
Best time to be alive debate [72] (Economic)
Happiness influenced by expectations and quality of life. [68] (Fconomic)
Misunderstanding and misuse of statistics [43] (Policy Prescription & Evaluation)
Crisis of meaning and disconnection modern society [39] (Cultural Identity)
Importance of personal space for productivity and success [33] (Economic)
Simple life in the Middle Ages vs modern lower class life [29] (Economic)
Table 19: Indicative Summary from GPT4,

---

CMV: There shouldn't be anything other than the metric system.
Indicative Summary (LLaMA-CoT)
Capacity & Resources
Boiling and freezing points water [154] (Quality Life)
Economic
Use of different size bottles in the dairy industry. [87] (Capacity & Resources)
The cost of switching to the metric system is too high. [59] (Capacity & Resources)
Health & Safety
Temperature ranges and weather conditions [86] (Quality Life)
Temperature range and clothing suggestions [63] (Quality of Life)
Policy Prescription & Evaluation
Merits of Celsius Fahrenheit temperature scales [283] (Constiuutonnalit & Jurisprudence)
Merits of the imperial and metric systems [196] (Constitutionality & Jurisprudence)
Use of miles and feet in measuring distances [140] (Quality of Life)
Mcrits of different systems measurement [106] (Fconomic)
Base 12 is better than base 10 for certain calculations. [104] (Capacity & Resources)
Precision of measurements in inches millimeters [75] (Quality Life)
Use of feet and inches for measuring height [72] (Constitutionality Jurisprudence)
Importance of precision in measurements [64] (Capacity & Resources)
Metric vs. Imperial: Which system is better? [52] (Constitutionality & Jurisprudence)
Merits of different counting systems [49] (Faimess & Equality)
Merits of decimal time system [48] (Economic)
Merits of different scales and their practicality [46] (Capacity Resources)
Merits of different systems [42] (Economic)
Table 20: Indicative Summary from LLaMA-CoT.

---

CMV: There shouldn't be anything other than the metric system. Indicative Summary (GPT3.5)
Capacity & Resources
Over intuitive systems and their advantages. [106] (Quality of Life)
For a more efficient counting system. [104] (Policy Prescription Evaluation)
Usefulness of different measurements for everyday [87] (Quality of Life)
Economic
Legacy system rooted in society with benefits for everyday use and practical applications, but costly transition away from. [196] (Fairesss & Equality)
Counting systems and their relative merits. [49] (Faireess Equality)
Fairness & Equality
Comparing the practicality of Celsius and Fahrenheit for everyday use, with no clear advantage either. [283] (Quality of Life)
Temperature scale based on water's 's freezing and boiling points. [154] (Constitutinnality & Jurispru- denec)
Advantages and disadvantages of using inches and centimeters for measurements. [75] (Constitutionality & Jurisprudence)
Usefulness of feet and inches for measuring human height. [72] (Quality of Life) A wide range of temperatures from chilly to hot, requiring different levels of clothing. [63] (Quality Life)
Costly transition international standardization with little net benefit average American. [59] (Economic)
Advantages and disadvantages of the metric system. [52] (Constiutiinnaiity & Jurispradence)
Advantages and disadvantages of different scales. [46] (Capacity Resources)
Pros and of diffcrent systems. [42] (Policy Prescription Evaluation)
Health & Safety
Extremely cold temperatures ranging from -50C to +50C across the globe. [86] (Quality of Life)
Constitutionality & Jurisprudence
Use of miles, yards, feet, and kilometers for measuring distances. [140] (Policy Prescription & Evaluation)
Precision and accuracy in measurement, [64] (Policy Prescription & Evaluation)
Complexities measuring time. [48] (Policy Prescription Evaluation)
Table 21: Indicative summary from GPT3.5.

---

CMV: There shouldn't be anything other than the metric system.
Indicative Summary (GPT4)
Capacity & Resources
Water freezing and boiling points discussion [154] (Heellh Safety)
Base 12 system advantages [104] (Economic)
Mcasurement units their precision various contexts [75] (Quality Life)
Cultural Identity
Preference for miles over kilometers in everyday language and distances [140] (Quality of Life)
Preference based on familiarity and upbringing [106] (Quality Life)
Preference for feet and inches in measuring height [72] (Quality of Life)
Economic
Costly and challenging transition to new system. [59]
(Cupacity & Resources)
Using different counting systems and their efficiency in various situations. [49] (Capacity & Resources)
Health & Safety
Temperature range discussion and its effects daily life [86] (Quality of Life)
Temperature and clothing preferences [63] (Quality of Life)
Quality of Life
Comparing Celsius and Fahrenheit for everyday use [283] (Capacity & Resources)
Imperial system vs. Metric system debate [196] (Cultural Identity)
Metric and imperial measurements daily life and their usefulness. [87] (Capacity Resourees)
Misunderstanding precision and accuracy in measurements [64] (Health & Safety)
Metric system advantages and precision debate [52] (Policy Prescription & Evaluation)
Alternative time measurement systems [48] (Cultural Identity)
Usefulness and subjectivity of different scales [46] (Fairness & Equality)
Old system versus new system everyday life [42] (Economic)
Table 22: Indicative summary from GPT4,
CMV: Shoe sizes should be the same for both men and women
Indicative Summary (LLaMA-CoT)
Fairness & Equality
Men and women's 's feet are different size and shape. [73] (Policy Prescription & Evaluation)
Differences between men's and women's shoes and impact of unisex shoes. [44] (Policy Prescription
& Evaluation)
Shoe sizes vary by sex due to differences in foot shape. [30] (Quality of Life)
Women with broad but small feet struggle to find shoes that fit. [19] (Quality Life)
Differences clothing proportions for and women [17] (Policy Prescription & Evaluation)
Policy Prescription & Evaluation
Pros and cons of standardizing shoe sizes [64] (Economic)
Use of different measurement systems [21] (Economic)
Table 23: Indicative summary from LLaMA-COT.

---

CMV: Shoe sizes should be the same for both men and women
Indicative Summary (GPT3.5)
Fairness & Equality
Men and women's 's feet are differendy shaped. [73] (Cultural Identity)
Multiple shoe sizing systems causing confusion. [64] (Quality of Life)
Gender-speeifie shoe design and comfort. [44] (Quality of Life)
Different shoe sizes for men women based on width and length. [30] (Quality of Life)
Different measurement standards for length. [21] (Constitutonnaiity & Jurisprudence)
Wide feet struggle to find shocs that fit properly. [19] (Quality of Life)
Clothing designed differently men and women. [17] (Cultural Identity)
Table 24: Indicative summary from GPT3.5.
CMV: Shoe sizes should be the same for both men and women Indicative Summary (GPT4)
Economic
Shoe durability and gender differences in footwear preferences [44] (Quality of Life)
Fairness & Equality
Standardizing shoe sizes for everyone [64] (Quality of Life)
Differences in clothing proportions for men and women [17] (Cultural Identity)
Health & Safety
Differences men's 's and women's 's feet [73] (Quality of Life)
Quality of Life
Shoe sizes differ for men and women due to width and shape differences in feet. [30] (Fairness Equality)
Different measurement systems for sizes [21] (Cultural Identity)
Finding shoes for wide and small feet [19] (Fairness & Equality)
Table 25: Indicative summary from GPT4,
CMV: Social media is the most destructive addiction in society Indicative Summary (LLaMA-CoT)
Economic
Role of money in society and its impact humanity. [23] (Capacity Resources)
Role of capitalism in society. [20] (Policy Prescription & Evaluation)
Costs of running systems and offsetting those costs. [19] (Capacity & Resources)
Fairness & Equality
Discrimination lawsuit against Amazon founder [25] /Constiutionality & Jurisprudence)
Health & Safety
The impact of opioid addiction on individuals society devastating. [107] (Morality)
Morality
Social media addiction vs opioid crisis [38] (Capacity & Resources)
Policy Prescription & Evaluation
Pros and of social media and its impact society. [48] (Morality)
Measuring impact of technology society [35] (Economic)
The importance of education and community for a better world. [26] (Capacity & Resources)
The impact of social media society [26] (Public Opinion)
The impact of social media on mental health is debated. [24] (Health & Safety)
Table 26: Indicative summary from LLaMA-CoT.

---

CMV: Social media the most destructive addiction in our society Indicative Summary (GPT3.5)
Economic
Complexities money as a social construct. [23] (Faimess Equality)
Costly infrastructure needed run systems. [19] (Capacity Resources)
Fairness & Equality
Importance of education, societal injustices, and the consequences of comparing oneself to others. [26]
(Quality of Life)
Powerful man accused of denying bathroom access to employecs. [25] (Constitutionality Jurispru-
denec)
Effects of capitalism human behavior. [20] (Economic)
Health & Safety
Effects social media on mental health. [24) Qualitty Life)
Morality
Devastating consequences of opioid addiction leading to death and destruction. [107] (Health & Safety)
Effects social media opioid use on mental health. [38] (Hlealt & Safety)
Negative effects of social media outweigh the positives, leading a lack of critical thinking and moral panie. [26] (Quality of Life)
Public Opinion
and cons of social media. [48] (Cultural Identity)
Quality of Life
Mcasuring societal impact through quality of life and direction of society. [35] (Cultural Identity)
Table 27: Indicative summary from GPT3.5
CMV: Social media the most destructive addiction in our society Indicative Summary (GPT4)
Economic
Money a social construct and tool for exchange [23] (Quality Life)
Capitalism and human nature discussion [20] (Fairness & Equality)
Costs and responsibilities of using resnurces and services [19] (Capacity & Resources)
Health & Safety
Opioid crisis and its impact individuals and society [107] (Quality of Life)
Social media and opioid addiction relationship [38] (Quality of Life)
Social media's 's impact on mental health and potential link suicide rates. [24] (Quality Life)
Constitutionality & Jurisprudence
Lawsuit against Bezos for denying bathroom access [25) (Heallh & Safety)
Quality of Life
Social media as tool connection and learning [48] (Capacity & Resources)
Measuring impact through quality of life and societal direction [35] (Fairesss & Equality)
Improving society through better education and empathy. [26] (Faimesss & Equality)
Impact social media on society and individuals [26] (Cultural Identity)
Table Indicative summary from GPT4,


Reviews:
[{"review_id": "YG1F0uoWVt", "reviewer": "Reviewer_RYVk", "summary": "", "questions": "1. What happens if the discussion is single topic - where all users discuss the same topic ? Did authors find such discussions how did your approach tackle those \n2. How much does the corpus based filtering important - an ablation study would have provided more details about this filtering step as this makes the approach corpus dependent\n3. While user study evaluates whole discussion the offline evaluation is only based on clusters ? Is it possible to rather select N discussions and clusters in them to understand how the performance works - this will provide a better understanding like the quality of clusters formed, with-in cluster accuracy and across cluster accuracy of a method", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. The idea of using sentence topics is certainly intriguing. However, with discussions spanning hundreds of sentences, predicting multiple topics might complicate the summary presentation. Evaluating the topic model on a sentence level without any ground truth adds to the complexity. Our search engine lets users explore their own queries without sifting through an extended list of topics. Nonetheless, we'll explore ways to integrate this suggestion practically in the future.\n\n1. “__What happens if the discussion is single topic - where all users discuss the same topic ? Did authors find such discussions how did your approach tackle those__”\n\nIn cases of single-topic discussions, we've noticed a dominant frame, yet it still contains around two or three clusters with related subtopics due to automatically identifying semantically related but slightly divergent sentence groups. For instance, for the topic \"Corporations support the LGBT for money, not love,\" we observed three frames: \"Economics\" (241 sentences), \"Fairness and Equality\" (21 sentences), and \"Public Opinion\" (25 sentences). The \"Economics\" frame includes subtopics like \"Corporations support LGBTQ+ for profit and image,\" \"Corporations prioritize money over emotions,\" and more. \n\n2. “__How much does the corpus based filtering important - an ablation study would have provided more details about this filtering step as this makes the approach corpus dependent__”\n\nCorpus-based filtering actually stands as a valuable component of our approach. It enables the use of a small noise sample per corpus/domain/platform to filter numerous emerging discussions. This process is made easier by the preliminary clustering, which easily groups meta comments due to their semantic similarity, aiding straightforward noise sampling. Unlike a generic preprocessing plan with several heuristics, our approach easily addresses distinct platform styles and content nuances, such as slang/terminology variations and content moderation policies (e.g., Reddit vs Discord vs Debate portals). Moreover, crafting rules for exclusion manually might prove ineffective without semantic similarity detection. We believe our clustering-based filtering offers a simpler, customizable solution adaptable to any corpus.\n\n3. “__While user study evaluates whole discussion the offline evaluation is only based on clusters ? Is it possible to rather select N discussions and clusters in them to understand how the performance works - this will provide a better understanding like the quality of clusters formed, with-in cluster accuracy and across cluster accuracy of a method__”\n\nWe actually did manually inspect 50 to 80 random clusters (with 50 to 100 sentences) from multiple discussions and were very satisfied with the cluster quality. This is also (albeit indirectly) reflected in the generated cluster labels as coherent inputs (good clusters) resulted in high quality cluster labels. Appendix B presents more details on the clustering setup. We acknowledge that this could be quantified better and will do so in future.\n\nAlso, as we employed only one clustering algorithm (HDBSCAN), we didn’t utilize any automatic metrics (such as Dunn Index or Sum of Squared Distances) to evaluate cluster quality since these (usually) make sense only when compared to other clustering algorithms. We actually experimented with these but could not meaningfully interpret the quality just by the scores. Moreover, as we did not have ground truth cluster labels beforehand, computing accuracy would be impossible (e.g. using V-measure or Fowlkes-Mallows Index).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written with comprehensive literature\nThe paper explains the preprocessing steps in details with example sentences for reader to understand well\nThe clustering, frame assignment and labels are explained in proper details\nIt is interesting to see the users prefer summary over other techniques.", "reasons_to_reject": "The baselines for user study could be better - like just showing topics of sentences / or color each sentence based on their topics. Is it possible to run the study with more participants.", "questions_to_authors": "1. What happens if the discussion is single topic - where all users discuss the same topic ? Did authors find such discussions how did your approach tackle those \n2. How much does the corpus based filtering important - an ablation study would have provided more details about this filtering step as this makes the approach corpus dependent\n3. While user study evaluates whole discussion the offline evaluation is only based on clusters ? Is it possible to rather select N discussions and clusters in them to understand how the performance works - this will provide a better understanding like the quality of clusters formed, with-in cluster accuracy and across cluster accuracy of a method", "paper_topic_and_main_contributions": "The paper handles long discussion summarization by clustering sentences in the discussion and applying cluster labels using LLMs provide topic wise summary to the user.  They did a evaluation on 300 clusters in their system with a user study with 5 participants", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9TsGxPEW02", "reviewer": "Reviewer_UYXG", "summary": "", "questions": "Please respond to (W1)-(W4) above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and questions. We answered them to the best of our ability below. \n\n### W1: Task definition and novelty are not clear\n\nThanks for pointing this out.  We acknowledge that the clarity of our task definition and novelty could have been improved. We provide a more detailed explanation below and will integrate it into the paper:\n\n__Task definition__\n\nIndicative summaries serve the purpose of \"suggesting\" the content within a discussion, aiding readers in determining whether they should delve deeper into the details. In contrast, informative summaries aim to (partially) \"replace\" the discussion by encapsulating the key information. We draw inspiration from Klaas (2005) “Toward Indicative Discussion Fora Summarization\" (referenced in our paper) to establish a similar conceptual framework.\n\n__Novelty__\n\nOur approach introduces a new and completely unsupervised approach to generating indicative summaries that align with the discussion's structure, akin to a Table of Contents. These summaries guide readers through discussions instead of replacing them. We found no prior instances of generating such summaries for online discussions in existing literature. Moreover, our systematic prompt optimization based on established subtasks (summarization, paraphrasing, title generation), an extensive quantitative and qualitative evaluation of 19 LLMs, and a summary purpose-oriented user study are key contributions. \n\n### W2: Evaluation results do not verify the end-to-end task performance\n\nWe evaluated end-to-end performance, focusing on utility of indicative summaries. Our manual user study (Section 5.4) compared intuitive baselines.\n\nWe chose search-based baselines to reflect common online reader behavior—employing (in-browser) keyword searches for finding relevant discussion content. A search engine allows users to find pertinent sentences and arguments using their own queries. \nAdditionally, we provided the original web page of the discussion (from Reddit) for its presentation of reply threads to ensure study subjects have a familiar user experience. \n\nWe avoided extractive summaries due to potential restrictions such as low coverage of interesting arguments, redundancy of similar arguments, and ambiguity in determining the optimal summary length for a given discussion. However, he search-based baselines mimic extractive summaries at the sentence level for the top-k most relevant results, providing for flexible summary lengths and avoiding these concerns. \n\n“__And the results in 5.4 do not really support that the proposed solution is a better indicative summarization method than other summarization methods.__”\n\nAnnotators unanimously preferred our approach (lines 538 - 542) over baselines for uncovering new perspectives in conversations. Unlike other methods that generate _informative_ summaries to partially replace the discussions, our approach provides exploratory indicative summaries. \n \n### W3: Design choice is not well-justified; Technical novelty and significance are not clear\n\nThe decision to assign frames after clustering stems from careful consideration of practical issues:\n\n___Handling Noise___: Online discussions often contain noisy sentences like meta comments, leading to irrelevant frame assignments when content lacks clear information. This could deteriorate summary quality due to subpar cluster labeling during abstractive summarization, especially if meta comments and noise dominate a cluster. Frame assignment after clustering and filtering such noise mitigates this issue.\n\n___Efficient Cluster Labeling___: Focusing on assigning a frame for the cluster label itself, rather than hundreds of sentences, reduces model inferences. This streamlining enhances our approach's practical feasibility. \n\nOur indicative summaries mimic the table of contents in books, where \"chapters\" and \"sections\" outline hierarchy: frame labels serve as \"chapters\" and cluster labels as \"sections\".\n\n__Technical Novelty and Significance__\nWe substantiate the novelty and importance of our contributions as follows:\n\n1. ___Distinctive Summary Style___. The task of indicative summarization in general has received little attention compared to informative summarization of news or scholarly documents (as outlined in our related work). For a user wanting to follow up or participate in a discussion, the generated table of contents help them to quickly navigate the various arguments and learn about different perspectives. Moreover, such summaries have been mainly single texts (informative summaries) aimed at replacing the entire discussion rather than letting users explore them to discover new perspectives. \n\n2. ___New Unsupervised Approach and Extensive Proof-of-Concept Evaluation for Indicative Summarization___.  We present a new approach to indicative summarization that is unsupervised and thus does not require large-scale labeled datasets. To show that both, the notion of indicative summaries and our approach are feasible, we conduct an extensive proof-of-concept evaluation in an important application domain. We deliberately choose to ground our approach in existing approaches rather than in addition also developing new ones, as this shows feasibility of the task of indicative summarization, and that this is achievable using the state of the art. Moreover, there was little evidence that state-of-the-art models were inappropriate; rather, our evaluation shows for the first time where indicative summarization can be improved in future work.  Finally, our evaluation of 19 LLMs to find the best models for this task, established the first and a reliable baseline for future work.\n\n3. ___Instrumental Methods: Systematic Prompt Engineering, Noise Filtering, and Purpose-driven Evaluation___. Part of our approach includes original algorithmic contributions instead of reusing state-of-the-art approaches: We introduced a systematic prompt identification method (Section 4) for generative cluster labeling and frame assignment across different model classes (dialog vs. instruction-based), streamlining resource-intensive LLM investigations for the research community. Our clustering-based noise filtering offers an adaptable solution for diverse corpora. This avoids manually crafting several noise filtering patterns by manually looking at a large number of examples. Finally, our evaluation is contextually grounded, involving real-world tasks, unlike many other summarization assessments relying solely on automatic metrics. \n\n### W4: Comparisons of LLMs with no clear implications\n\nWe beg to differ. Clear conclusions are drawn in lines 455 - 460 for cluster labeling, lines 485 to 493 for frame assignment, and lines 570 to 573 in conclusion, about which models perform best for the task of indicative summarization. As stated in the paper, closed-source GPT models (ChatGPT, GPT4) perform best, followed by the LLaMa model combined with Chain-of-Thought prompting performing best among the open source LLMs. This implies that the research community can spare costly computational resources to investigate the other 16 LLMs for this task. \n\nThe best models in Table 4 are indicated in bold for zero-shot and few-shot frame assignment. We will mention this in the table caption. \n\n“__The paper does not offer training or validation datasets__.”\nRegarding training and validation sets, our approach is completely unsupervised in that it does not need any training or validation sets. The discussion titles used for qualitative evaluation of usefulness and their corresponding summaries from the three best models are already provided in the appendix. The complete data used for evaluation and preprocessing will be made publicly available.\n\n “__The reported results can be this task & dataset-only and may not be generalized__.”\nIn this paper, we first thoroughly tackled the task on one specific and a highly relevant genre (Reddit discussions) and demonstrated via proof-of-concept that the approach is feasible and effective. To conclude generalizability across several genres/datasets would effectively require a significant amount of systematic evaluation (i.e., user studies, manual quality assessments, and error analyses tailored to each domain) that would not fit into the length restrictions of a typical research paper. However, follow up works targeted at  different domains can easily adapt the approach and evaluation design established in this paper.\n\n“__Again, it’s difficult to judge only from the results__.”\nWe do not completely understand what is implied by this. \nWith regards to our evaluation, we did not rely solely on automatic metrics to determine the best models (as is usually the case in summarization), but rather carefully designed, fine-grained, and resource-intensive annotation experiments for robust qualitative evaluation of the resulting summaries. In this regard, concrete suggestions as to what is still missing to draw conclusions would be useful.\n\n__Regarding low reproducibility score__:\nWe have already included the complete code in the supplementary material, along with the annotation interfaces and the web application utilized for judgment collection. The appendix thoroughly explains all parameter values. The data will be made publicly available post-review, alongside the code. All of this has been checked for reproducibility and is easy to re-execute. If any crucial component has been inadvertently omitted, please inform us, and we will make every effort to address it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(S1) The paper tackles a multi-document summarization task from a slightly uncommon angle by formulating as indicative summarization. \n(S2) The paper conducts comprehensive experiments using a variety of LLMs.\n(S3) The paper is generally well-written and easy to follow.", "reasons_to_reject": "(W1) (Task definition and novelty are not clear.) The notion of indicative summarization is not clear. According to Footnote 2, it is not clear what is different from extreme summarization (e.g., single-document such as XSum and multi-document such as TLDR). The TIFU Reddit dataset [Ref 1] is not cited or mentioned in this paper. \n\n(W2) (Evaluation results do not verify the end-to-end task performance) The overall performance evaluation is missing or has some issues. Now the task consists of multiple components: (1) sentence clustering, (2) label assignment (which can be considered summarization), (3) frame assignment.\n\nIn 5.4, the author(s) compared with search-based solutions but summarization methods should also be compared as baselines. It is obvious that any summarization methods are better than search-based methods for exploration. And the results in 5.4 do not really support that the proposed solution is a better indicative summarization method than other summarization methods. \n\n(W3) (Design choice is not well-justified; Technical novelty and significance are not clear) Similar to (W1) and (W2), it is not clear sentence clustering before frame assignments. Can frame assignment be done directly to each sentence before clustering? The lack of design choice discussion and ablation study makes it difficult to judge the justification of the design choice for the problem.  The paper used existing LLMs and existing media frames to tackle the problem. Technical novelty and significance are not clear from the paper in its current form. \n\n(W4) (Comparisons of a variety of zero-shot LLMs with no clear implications) Although the paper conducts comprehensive experiments by using a variety of LLMs, the learnings are not clear. Specifically, Table 4 does not tell which LLMs we should use for the purpose (in this case, frame assignment). The paper does not offer training or validation datasets. The reported results can be this task & dataset-only and may not be generalized. Again, it’s difficult to judge only from the results. \n\n\n### Minor comments\n\n- LL269-272, the author(s) claim that it is novel to apply abstractive summarization models to clutter labeling. However, the problem itself is actually multi-sentence summarization, which the author(s) referred to as cluster labeling and I do not agree with this claim. (Also, extractive-abstractive can be considered clustering labeling models then.) I leave this comment as a minor comment as it is not directly related to the main claim(s). \n- To me, the task looks closer to Argument Mining rather than Summarization. In any case, the paper should further clarify the differences against Argument Mining/Discussion Summarization.", "questions_to_authors": "Please respond to (W1)-(W4) above.", "paper_topic_and_main_contributions": "This paper tackles the task of summarizing online forum discussion by making an abstractive summary from multiple sentences in the same group and assigning media frame(s) to each group, which the paper refers to as indicative summarization. The paper develops a multi-stage pipeline that first clusters multiple sentences, which are converted into concise summaries (called labels in the paper), and media frame(s) are assigned to each of the labels. Finally, the assigned frames are used to display summaries to the user. The paper evaluated the quality of cluster labeling and frame assignment.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- [Ref 1] Byeongchang Kim, Hyunwoo Kim, Gunhee Kim, Abstractive Summarization of Reddit Posts with Multi-level Memory Networks, NAACL 2019. https://aclanthology.org/N19-1260/"}, "extra_text": {}}, {"review_id": "msyLDKrXIo", "reviewer": "Reviewer_oFS8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and pointing to the typos. We try to address your main concerns below.\n\n__Annotator Selection Choices.__\nAnnotator selection for evaluating the generated cluster labels (4 annotators) and the final user study (5 annotators) was intentionally kept separate to prevent bias stemming from exposure to high-quality labels in the initial evaluation. To ensure neutrality, we engaged 5 new annotators for the user study, presenting the broader task of exploring lengthy discussions through a dedicated interface. We will provide further clarification on this distinction in the paper.\n\n__Figure Clarifications.__\nThe number in parentheses is the number of sentences belonging to a given cluster(label). This was supposed to be clarified by the gray parenthesis “(replies; click label to view)” from a user’s perspective. In hindsight, this was too subtle. We will clarify this in the figure caption. \n\n__Usage of BERTScore.__\nWe chose BERTScore due to its consideration of semantic similarity between generated cluster labels and human labels through contextual embeddings, a feature not captured by lexical metrics like ROUGE or BLEU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Experimenting with 19 LLMs is remarkable, this allows comparison over a wide range of state-of-the-art models. \n- Applying corpus-specific filtering and discussion-specific filtering is a reasonable approach, as this helps eradicate the less relevant sentences.\n- The annotation interface for ranking the cluster-based quality seems useful and user-friendly.", "reasons_to_reject": "- A few missing details in the paper: there should be a brief explanation as to how the annotators were selected. In section 5.2, four annotators are used for cluster labeling, however in section 5.4, we see five annotators. Are they the same annotators, what is the explanation for adding an extra annotator for this task? \n- There seems to be a lot going on in Figure 1, it would be helpful to perhaps include what the number in parenthesis means or take it out.\n- There are numerous metrics for automatic evaluation, why was only BERTScore used?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a prompt-based zero-shot method to generate indicative summaries for long discussions. The authors introduce a three-step approach of clustering argument sentences by subtopics, generating a short summary that captures the gist of each cluster known as the cluster label, and then assigning a frame to each cluster label. Their results show that GPT3.5 outperforms all the other open-source models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There are a few small typos in the paper. In section 3.2, there is a typo in the third sentence – it should say “been” and not “be”. Also, there is a repetition in the caption for Table 4."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an unsupervised method, based on prompting and LLM, for making "indicative summaries" from forum discussions. Sentences are clustered and cluster labels are defined, which serve as a guide for reading the forum. Different language models are tested and an evaluation with human annotators is made.
Decision: Accept-Main