Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy
Authors: Tunazzina Islam, Dan Goldwasser


Reviewer paper summaries: This paper proposes a framework called "LLMs-in-the-Loop" to automatically extract latent arguments from social media messaging on contentious topics. The approach uses large language models (LLMs) in an iterative process to cluster messages, summarize clusters, generate arguments, and map messages to arguments. The authors demonstrate the effectiveness of their method on two case studies: climate change campaigns and COVID-19 vaccine campaigns on Facebook. They evaluate the quality of argument extraction and mapping, use the extracted arguments for downstream stance classification, and analyze how arguments vary across demographics and in response to real-world events.
Summary of strengths: 1. The paper presents an LLMs-in-the-loop method that leverages the capabilities of LLMs to automate argument mining from social media data, addressing limitations of existing supervised and unsupervised techniques.

2. The authors conduct thorough evaluations of their framework, including human validation, comparisons to baselines, and analysis of argument cohesion.

3. The study demonstrates the utility of the extracted arguments for stance classification and provides insights into demographic targeting and shifts in messaging over time.
Summary of weaknesses: 1. Given the data corpus, the authors assume that the themes are predefined. However, this might not be the case in the real-world. How would the framework work in this scenario where no predefined themes are available? It is important to note that LDA is applicable in such scenarios.

2. The point of mapping instances back to the arguments using cosine similarity between their embeddings is unclear. Each argument is generated from an instance (before merging redundant arguments), which already creates an association between them. In addition, creating the mapping this way would not leave any instances unassigned. So why bother mapping them using the distance-based approach again? 

3. The novelty of the proposed framework, comparing to the baseline of directly prompting the LLM to generate the argument given a text instance and a theme (e.g. ClimateSolution), is two-fold: (a) the authors first identify sub-themes within a theme, and they generate the argument in the context of the more granular sub-theme, instead of the original theme;  (b) they represent the sub-theme in the LLM prompt using the textual summary of the sub-theme, instead of the top-k instances verbatim within the sub-theme. In the ablation study (Table 2 in Section 4.3), it indicates that the second component does not add significant value to the framework, at least in terms of argument coverage. In that sense, I’m curious to see how much value the first component would have. Therefore, the authors should (1) indicate the number of sub-themes identified under each theme, (2) compare the complete framework to the aforementioned baseline. If it turns out that there are just one or two sub-themes under the majority of themes, or that the baseline performs closely to the complete framework, then the contributions of the paper would be significantly limited.

4. For argumentative cohesion comparison (Figure 2), the authors only show a subset of identified arguments. However, how these subset of arguments are chosen is not clear. In addition, it is difficult to interpret the difference between the three methods. More quantitative results should be added.

5. In stance detection, the authors show that adding the arguments help stance detection using Longformer. However, whether they help when using an LLM as the classifier remains unclear, which could be an advanced LLM (GPT-4) or a finetuned open-source LLM (Llama-3).
Comments, suggestions and typos: Major concerns
1. Lines 364-365. I don’t quite understand the reasoning behind running a second iteration. Is the expectation that the unassigned instances from different sub-themes would form a set of new sub-themes, for each theme? If this is the case, would it imply that the identified sub-themes in the 1st iteration were not good enough?

2. Section 3.3. The authors claim here that the sub-theme summaries are used as the context of identifying the argument by the LLM. However, in Figure 4, in the prompt, it says “theme”. I’m not sure if this theme refers to the summary of the sub-theme, or the original coarse-grained theme (such as GovDistrust). Please clarify.

3. The authors do not show any examples of the summaries of the identified sub-themes within each theme. In addition, more statistics should be added. How many sub-themes are identified under each theme? How large are the sub-themes? For each theme, how many arguments are identified from each sub-theme? Perhaps the authors can add a third column in Table 8 indicating the summaries of the sub-themes and their associated arguments.

Typos and Presentation Suggestions
1. Figure 1 is very difficult to interpret. Instead of just showing the actions taken at each step, it is also important to indicate what the data looks like before and after each step. It is better to represent the data using the boxes and place the actions taken beside the arrows. For textual data, it is recommended to show the prefix of the text in the boxes.

2. Lines 111-125. It is very difficult to follow all the steps in the framework without reading through the whole paper.

3. Section 3.3.1. “Refinement” does not sound like a good title. Consider changing it to “Argument Aggregation” or “Argument Redundancy Check”.

4. Line 473, by “cluster”, do you mean “sub-cluster” (for a sub-theme)?

5. Figure 2. First, consider using two different colors for positive and negative correlations. Second, the fonts are too small to be visible.
Overall assessment scores: 2.5