USDC: A Dataset of $\underline{U}$ser $\underline{S}$tance and $\underline{D}$ogmatism in Long $\underline{C}$onversations

Authors: mounika marreddy, SUBBA REDDY OOTA, Venkata Charan Chinni, Manish Gupta, Lucie Flek
Primary area: datasets and benchmarks
Keywords: large language models, annotators, user opinions, stance, dogmatism, human-llm alignment, open-source llms, closed-source llms

Abstract:
Although prior studies have explored Stance and Dogmatism in user conversations, their datasets are constructed at the post level, treating each post as independent and randomly sampling posts from conversation threads. Thus, Stance and Dogmatism labels in these datasets cannot capture the user's opinion fluctuations expressed throughout the entire conversation context. However, identifying user's opinion fluctuations in long conversation threads on various topics can be extremely critical for enhanced personalization, market research, political campaigns, customer service, conflict resolution, targeted advertising, and content moderation. Hence, training language models to automate this task is critical. However, to train such models, gathering manual annotations has multiple challenges: 1) It is time-consuming and costly; 2) Conversation threads could be very long, increasing chances of noisy annotations; and 3) Interpreting instances where a user changes their opinion within a conversation is difficult because often such transitions are subtle and not expressed explicitly. Inspired by the recent success of large language models (LLMs) for complex natural language processing tasks, we leverage Mistral Large and GPT-4 to automate the human annotation process on the following two tasks while also providing reasoning: i) User Stance classification, which involves labeling a user's stance of a post in a conversation on a five-point scale; ii) User Dogmatism classification, which deals with labeling a user's overall opinion in the conversation on a four-point scale. The majority voting on zero-shot, one-shot, and few-shot annotations from these two LLMs on 764 multi-user Reddit conversations helps us curate the USDC dataset. USDC is then used to finetune and instruction-tune multiple deployable small language models for the 5-class stance and 4-class dogmatism classification tasks. Additionally, human annotations on 200 test conversations achieved inter-annotator agreement scores of 0.49 for stance and 0.50 for dogmatism, indicating a reasonable level of consistency between human and LLM annotations. We make the code and dataset publicly available [https://anonymous.4open.science/r/USDC-0F7F].

Decision: reject
Reviewer scores: 6, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730188161397

thread_type:
official_review

<thread_object>
creator: reviewer_hg57

<content>
summary:
Previous datasets on stance and dogmatism in user conversations are constructed at the post level. However, these datasets fail to capture fluctuations in users' opinions throughout entire conversational contexts. This paper introduces USDC, a dataset focused on user stance and dogmatism in long conversations. Inspired by the recent success of large language models (LLMs) in complex natural language processing tasks, the authors leverage Mistral Large and GPT-4 to annotate the training data for USDC. To ensure data quality, they manually labeled 200 test conversations for evaluation. They also conducted experiments to assess the performance of different models on the test set.

strengths:
1. This paper introduces a novel dataset, USDC, which focuses on user stance and dogmatism in multi-user conversations.
2. It leverages large language models (LLMs) for efficient, scalable annotation of complex conversational data.
3. Extensive experiments are conducted using a variety of small language models.

weaknesses:
In the experimental section of the paper, the authors often merely list the results without providing in-depth analysis of the underlying reasons. For more details, refer to the "Questions".

questions:
Q1: Line 407 "2) For both tasks, the majority voting labels as ground truth has a relatively high performance, scoring above 50% weighted F1-score across several models." This expression is not very accurate. As shown in Table 1, for the Dogmatism Classification task, only 2 out of 14 models achieved an F1-score above 50% when using majority voting labels as ground truth. In contrast, when GPT-4 FS labels were used as ground truth, 11 out of 14 models outperformed the corresponding F1-scores obtained with majority voting labels. To enhance clarity and accuracy, I recommend that the authors revise these statements to reflect the results more precisely.

Q2: Line 410 "4) For GPT-4 annotations, in most cases, SLMs finetuned with few-shot annotations outperform those trained with zero and one-shot annotations. For Mistral Large annotations, SLMs finetuned with one-shot annotations perform the best." For the Mistral Large annotations, why do SLMs fine-tuned with one-shot annotations outperform those fine-tuned with corresponding few-shot annotations? Could the authors provide a hypothesis that  might explain this counterintuitive result?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620332179

thread_type:
official_review

<thread_object>
creator: reviewer_sf39

<content>
summary:
In this paper, the authors introduce a new dataset, named USDC.  It is as a dataset focusing on user stance and dogmatism in long conversations. Different from the previous methods based on labeling individual posts, this dataset can track changes in user opinions throughout the entire conversation, making it particularly useful for user personalization, market research, political campaigns, and content moderation. However, this paper lacks of more llms to be involved, only using Mistral Large and GPT-4, making the dataset has risk to be restricted by the knowledge in Mistral Large and GPT-4.

strengths:
1. The article is well-written and easy for readers to understand.  

2. It contributes a novel dataset, which is a unique collection focusing on user stance and dogmatism in long conversations.  

3. Extensive experiments demonstrate that the annotations generated by LLMs are comparable to those generated by humans.  

4. Fine-tuning and instruction-tuning multiple small language models and proved the effectiveness.

weaknesses:
1. What I am particularly concerned about is that you only used two LLMs for data annotation, which poses a risk of missing knowledge from other regions and fields, especially knowledge that other models might possess.  
2. I am also worried that in some cases, the system prompts may not be clear enough, leading to confusion in the large language models during annotation, such as inaccuracies in recognizing the author's stance.  
3. Furthermore, the models face difficulties in identifying intermediate positions and ambiguous attitudes, which can easily lead to misunderstandings of user opinions. These issues will be even more pronounced in machine-annotated data, as you may not be able to provide sufficient information.

questions:
Please use more experiments to convince me that the weaknesses not exist.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670666934

thread_type:
official_review

<thread_object>
creator: reviewer_qaks

<content>
summary:
It is very costly to have humans annotate Reddit threads with multiple posts to label the stances and dogmatism of multiple users. 
Previous work considers posts independently, however, that is not the nature of the interaction. 

This paper looks at if LLMs can encapsulate the nuances to understand user opinions and whether their opinions shift through the conversation. 

LLMs are used to classify (1) user stances and (2) user dogmatism and express their reasoning for their classification. Every sample is annotated six times (two models x zero-shot, one-shot, and few-shot) and then the majority vote is taken. 

The paper introduces the USDC dataset that includes 1528 dogmatism samples (user-level) and 9618 stance samples. It is able to capture contextual and opinion shifts. As such it can be used as an instruction-tuning dataset or evaluation benchmark. The authors also instruction-tune and fine-tune LLMs on the dataset.

strengths:
1. The paper is well-motivated and easy to read. 
2. The approach is straightforward and makes sense. 
3. The added qualitative analysis is very important and nice to read.

weaknesses:
1. The majority voting conflict makes me wonder why Mistral is used at all if, in cases of conflict, the decision maker is GPT4 (which is quite a costly model)? 
2. Majority voting labels are used as ground-truth. It would be good to add experiments on what would happen if we train on unaggregated labels, as subjectivity is important in such a task.

questions:
Why is it the case that instruction-tuning is better for stance and fine-tuning better for dogmatism?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730825204809

thread_type:
official_review

<thread_object>
creator: reviewer_wjm4

<content>
summary:
This paper introduces a dataset for detecting stance and dogmatism in each turn of a conversation.
The dataset is based on Reddit conversations, where each turn is a reply within a thread.
To label the stance and dogmatism of each speaker—a task with inherent subjectivity—the authors use multiple prompting schemes across different models, finalizing each label through majority voting.

strengths:
The dataset provides valuable insights into stance and dogmatic expressions in Reddit conversations, contributing a unique resource for analyzing opinion and belief expression in online discourse.

weaknesses:
1. Moderate Inter-Annotator Agreement: The inter-annotator agreement between human and LLM annotations could be improved.

2. Fragmented Conversations: By selecting only the top two authors’ comments, the dataset lacks conversational continuity. The two selected authors’ comments are scattered across the thread, rather than forming a cohesive conversation.

questions:
1. The observed inter-annotator agreement between LLM-human and human-human highlights the subjectivity of this task. If these labels are treated as ground truth in experiments, might this reduce the robustness of the results?

2. Are there any qualitative examples demonstrating cases with high, moderate, and low inter-annotator agreement? These could provide insight into where the labeling approach is most and least effective.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731964539230

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our responses and revision

comment:
*We are grateful to all reviewers for their strong positive feedback, time and their constructive suggestions, which will further strengthen the impact of our work.*

**Summary of Reviewer Strengths:**

1. The paper is well-written, clear, and effectively motivated **(QAKS, sf39)**.
2. This paper introduces a novel contribution through the USDC dataset, which focuses on user stance and dogmatism in multi-user conversations **(WjM4, QAKS, sf39, hG57)**.
3. The methodology is straightforward and includes qualitative analysis, which adds depth to the findings **(QAKS, sf39)**
4. Extensive experiments, including fine-tuning and instruction-tuning of multiple small language models, validate the effectiveness of the proposed approach **(QAKS, sf39, hG57)**.
5. The work leverages large language models (LLMs) for efficient and scalable annotation, demonstrating that LLM-generated annotations are comparable to human annotations **(hG57)**.

**Additional changes to the draft during the rebuttal process**

We have updated the main manuscript and the appendix to address these following comments. The changes made in the manuscript are highlighted in blue color. The major additional changes are listed below.

1. **Qualitative examples demonstrating cases with high, moderate, and low inter-annotator agreement** (Reviewer WjM4, sf39): We have included qualitative examples in Appendix R of the revised manuscript, demonstrating cases with high, moderate, and low inter-annotator agreement (IAA) for the Stance and Dogmatism tasks, as illustrated in Figs. R.1–R.6. These examples highlight where the labeling approach is most effective, moderately effective, and least effective, showcasing consistent stance labels in high-agreement cases and discrepancies, particularly with Mistral Large, in low-agreement cases.

2. **Robustness analysis of Human-LLM Annotations** (Reviewer WjM4): We have included a heatmap in Appendix Q, Fig. 21, comparing human-annotated labels with majority voting labels from LLMs, highlighting class-specific agreement for the Stance and Dogmatism tasks. The significant mismatch in intermediate stance classes, particularly "Somewhat Against" in stance detection and "Open to Dialogue" in dogmatism, likely accounts for the moderate inter-annotator agreement (IAA) observed between human and LLM-generated labels.

3. **Weighted Cohen's Kappa score: IAA between human labels and LLM-generated labels** (Reviewer WjM4): Appendix S, Fig 22 presents weighted Cohen’s Kappa across eight settings, and highlights that the weighted Cohen’s Kappa metric improves the IAA between human annotations and the majority voting approach to 0.55.

We hope these revisions will satisfactorily address the concerns raised by the reviewers and elevate the overall quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work investigates Stance and Dogmatism in conversations, by using LLMs to automate the human annotation process on user stance classification and user dogmatism classification over 764 multi-user Reddit conversations. As a result, this work constructs the USDC dataset. Specifically, 200 test conversations are annotated by human. 

Strengths:
1. The paper is well-written, clear, and well-motivated.
2. The paper constructs a new dataset for studying Stance and Dogmatism in conversations.
3. Experimental results validate the effectiveness of the proposed approach.

Weaknesses:
1. The reliability of the constructed dataset is questioned by all the reviewers, including the subjectivity (moderate inter-annotator agreement/majority vote as ground truth), unnatural conversations (fragmented discussion thread), only two LLMs for annotations, etc.
2. The experimental results lack in-depth analysis. Some are solved during the rebuttal, but some are still not satisfied by the reviewer.
3. There are only 200 human-annotated test conversations, which may not be sufficient for robust evaluation. 

Overall, as a resource paper, the quality and reliability of the contributed dataset should be the key. However, according to the comments from the reviewers, despite the good quality of the writing and motivation of the problem, there are still many concerns on the curated dataset itself. Therefore, it would be better to further enhance the quality validation and even the size and diversity of the created dataset.