The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

Authors: Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, Stefan Harmeling
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: large language models, stance detection, data augmentation, active learning, online political discussions

Abstract:
Stance detection holds great potential to improve online political discussions through its deployment in discussion platforms for purposes such as content moderation, topic summarisation or to facilitate more balanced discussions. Typically, transformer-based models are employed directly for stance detection, requiring vast amounts of data. However, the wide variety of debate topics in online political discussions makes data collection particularly challenging. LLMs have revived stance detection, but their online deployment in online political discussions faces challenges like inconsistent outputs, biases, and vulnerability to adversarial attacks. We show how LLM-generated synthetic data can improve stance detection for online political discussions by using reliable traditional stance detection models for online deployment, while leveraging the text generation capabilities of LLMs for synthetic data generation in a secure offline environment. To achieve this, (i) we generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated synthetic data can substantially improve the performance of stance detection, while remaining interpretable and aligned with real world data. (ii) Using the synthetic data as a reference, we can improve performance even further by identifying the most informative samples in an unlabelled dataset, i.e., those samples which the stance detection model is most uncertain about and can benefit from the most. By fine-tuning with both synthetic data and the most informative samples, we surpass the performance of the baseline model that is fine-tuned on all true labels, while labelling considerably less data.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730519878119

thread_type:
official_review

<thread_object>
creator: reviewer_si6z

<content>
summary:
Yet another synthetic data paper that shows modest improvements but doesn't quite nail why or how to make synthetic data actually useful. Some interesting ideas buried under conventional methodology.

Look, I've seen enough "let's use LLMs to generate synthetic data" papers to last several conferences. What makes this one interesting - barely - is the political stance detection angle and the somewhat novel SQBC approach. But let's be real here: you're essentially using an LLM to generate slightly different versions of existing viewpoints, then acting surprised when this helps... a little bit.

The authors show that their approach improves F1 scores from ~0.69 to ~0.72 with synthetic data alone, and up to ~0.75 with their full pipeline. Sure, that's positive, but is it worth the computational cost of running Mistral-7B for hours to generate the synthetic data? (And don't get me started on the economic/environmental impact in general - though I suppose that's not this paper's specific sin since those models are small and this used only 1 A100 GPU.)

The most interesting part is actually buried in Section 5.1, where they show that using Mistral-7B directly for stance detection fails miserably. This suggests something important about synthetic data that the authors don't fully explore: it's better at generating plausible variations than at making decisive judgments. This deserved more analysis.

What's missing here is any real investigation into what makes synthetic data actually useful. Are we just doing expensive interpolation between existing data points? Where's the analysis of entropy and diversity in the generated samples? The visualizations in Figure 3 are pretty, but they also show that the synthetic data mostly just fills in obvious gaps rather than introducing genuinely novel perspectives.

The active learning component feels tacked on, though I'll admit the SQBC approach is clever. Using synthetic data as a reference distribution for selecting informative samples is neat, but again - why does this work? The paper handwaves at "ambiguous samples" without diving deeper into the theoretical foundations.

One thing I'll give the authors credit for: they did their homework on the translation pipeline. Using NLLB-330M and actually caring about the quality of the German-English-German round trip is more than many papers bother with. The samples in Table 8 show reasonable quality political discourse generation.

SUGGESTIONS FOR IMPROVEMENT:

- Add analysis of entropy/diversity metrics for synthetic data
- Provide theoretical justification for why synthetic data helps beyond just "more data"
- Compare computational costs vs. benefits more explicitly
- Explore what makes certain synthetic samples more useful than others
- Consider alternative methods for introducing genuine novelty into synthetic data

NITPICKS:

- The abbreviation "SQBC" is used before it's properly defined
- Figure 4 is information-dense to the point of being hard to parse
- Some ablation studies feel perfunctory rather than insightful

CONCLUSION:

This paper is fine. It's not going to revolutionize either synthetic data generation or stance detection, but it makes a modest contribution to both. The experimental work is solid if unexciting, and the results are positive if not earth-shattering. The biggest missed opportunity is not diving deeper into what makes synthetic data actually useful beyond simple interpolation.

The paper should be marginally accepted because it advances the field incrementally and might give others ideas for more innovative approaches. But let's not pretend this is more than a small step forward in a very crowded research space. I don't like these kind of papers

I'd love to see a follow-up that really digs into the entropy question and provides proper theoretical foundations for synthetic data generation in political stance detection. Until then, this feels like another "it works (a bit) but we're not quite sure why" paper.

strengths:
Clean experimental methodology with proper ablation studies
Good visualization of how synthetic data aligns with real data distributions
Actually bothered to translate German political content properly instead of using Google Translate
Reasonable baseline comparisons and honest reporting of limitations
The SQBC approach is somewhat novel, even if not revolutionary

weaknesses:
Limited theoretical justification for why synthetic data helps beyond "moar data good"
Doesn't address the entropy/diversity problem in synthetic data generation
Results are modest (~2-3% improvements) for considerable computational overhead
Heavy reliance on a specific dataset (X-Stance) limits generalizability claims
The "active learning with synthetic data" angle feels like two papers duct-taped together

questions:
Were there any computational/economic reasons for not scaling up your compute? I'm sympathetic to this as I understand the burden of even a single A100 - but if you do have more resources, why not use them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607261202

thread_type:
official_review

<thread_object>
creator: reviewer_cufq

<content>
summary:
This paper proposes to use LLM-generated synthetic data to augment the training of stance classification models. It also proposes a synthetic data-based active learning method that uses synthetic data to facilitate the selection of unlabelled data for human annotation. Experiments are conducted on the German subset of the X-stance dataset (with the help of machine translation). The results demonstrate that including synthetic data in training can improve stance prediction. The synthetic data-based active learning method, however, is not clearly better than a random selection-based baseline active learning method.

strengths:
- The proposed method is sound. I do not see any major issue with the method.
- Although the idea of using synthetic data to augment models is not entirely new, it probably has not been widely explored for stance prediction.
- The authors conducted extensive experiments to evaluate the method, including varying the size of the synthetic dataset, comparing with meaningful baselines, and the further experiments that compare with a LLM zero-shot baseline.

weaknesses:
- The experiments are conducted using a German dataset, but translation into and back from English is used in order for the method to work (probably because of limited German language understanding and generation capabilities of the Mistral model that is used?) There is no explanation of why the authors do not evaluate the method using an English dataset.
- The novelty and impact of the work is still limited. (1) Using synthetic data to augment models is not new. Although applying the idea to stance prediction might be new, it is one of many NLP tasks. The way synthetic data is generated and used during training in this paper is also standard, hence there is limited technical contribution. (2) The idea of using synthetic data for active learning is very interesting and is novel based on my knowledge. However, its effectiveness is limited based on the experiments. Therefore, overall, although the work is very solid in general, its novelty and impact may not meet the standard of this conference.
- There is room for improvement in terms of presentation. In particular, the active learning method proposed can benefit from first presenting an overview of the high-level intuition behind the method before describing the method itself.

questions:
- It would be very helpful to explain why only a German dataset is used for the experiments. Also, if German text is used, have the authors considered using a different LLM that has good German language processing capabilities for the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645578583

thread_type:
official_review

<thread_object>
creator: reviewer_kmpk

<content>
summary:
The paper tries to improve transformer-based stance detection models by fine-tuning on LLM generated data. They compare the real-world data with the synthetic data to identify difficult samples from unlabelled data (active learning) to further improve the model. They show that both these steps improves the performance of the transformer-based baseline.

strengths:
- The paper leverages the data augmentation capabilities of LLMs to improve transformer based models which are better suited for online deployment as they are more reliable. 
- The presented method can be adapted to other text classification tasks and hence is a significant contribution.
- It is well written and easy to follow, except for few instances mentioned in the comments.

weaknesses:
- It’s possible I’m missing some key context here, but I’m having trouble following the ablation study in Section 5.2. To test whether the performance gains come from dataset size or the generated content itself, the authors “shuffle” instances, apparently misaligning the posed questions with synthetic data. If the synthetic data consists of single text instances with labels, this shuffling wouldn’t seem to affect outcomes. Perhaps the authors mean they’re using different proportions of synthetic data in each run while keeping the total instance count constant, but this explanation feels somewhat unclear.
- Even though authors acknowledge this as a limitation, fine-tuning a separate model for each question doesnot seem to be a scalable approach, especially when the main motivation for the research was in line with training robust models for online deployment. 
- The X-stance dataset is described as having around 48k annotated comments on various questions. However, an overview of the dataset’s statistics—such as the number of comments per question—would greatly enhance readability. When you mention selecting 10 questions from the test set, it would be helpful to specify how many comments correspond to each question. While I see some statistics are included in the Appendix, a high-level summary within the main text would improve clarity and context for readers.
- Section 4.2, General setup: Please review this section for more readability. Currently, it is a bit difficult to get a picture of what models are being tested and how the methods differ between them.

questions:
- Why choose translation over adapting prompts directly? Is Mistral unable to generate responses in German, or were other multilingual models considered?

- In the ablation study for "Content vs. Size", I am not sure I understand why you call the shuffled dataset "misaligned". Could you please explain the reasoning behind this? 

- In the generated dataset, did you find any instance where the LLM failed to generate the requested content? For instance, generate statements not in favor when requested for "in favor" content or LLM refusing to generate any relevant content at all.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731764483245

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision

comment:
Dear reviewers,

we want to thank you for your thorough and detailed reviews. We believe your suggestions have improved our paper. In the following, we present the changes made to the revision which are marked in Blue. We have also responded to each of your concerns under your respective reviews. 

* We have made some slight rewrites in the introduction to strengthen the connection between fine-tuning and SQBC. 
We have restructured the experimental setup page, added information about the X-Stance dataset(and why we use it), and included a new table with the exact test data splits.
* We have rewritten the (Q2) and (Q3) in the results section (Section 4.3) to better reflect our insights regarding as to what synthetic data does to the model and regarding the behaviour of the active learning methods. We hope this addresses the request for more technical and theoretical insights.
* We have added a new Figure 5 and a new paragraph to Section 5.2. We now analyse the synthetic data by plotting the entropy distribution of the synthetic and the real data, while also calculating the average entropy and comment length for the interquartile ranges of the entropy distribution. Furthermore, we also show comment samples for both synthetic and real data in Appendix C1 for the interquartile ranges of the entropy distributions. The idea behind this is to show together with Figure 3 that the BERT model is largely invariant to comment structure and is more sensitive to the content (i.e. topic) of the data.
* We therefore renamed the "content vs size" paragraph to "content vs structure: Is per topic fine-tuning necessary?". We did this to better highlight the idea behind this ablation, which is to show together with the synthetic data analysis that the effectiveness of fine-tuning with synthetic data is largely due to the content of the comments.
* Finally, we have added two paragraphs to the discussion section. First, we added "Potential impact.", where we highlight the benefits of using our analysis for practitioners in the are of stance detection in online political discussions. Second, we added a section about future work to better highlight open problems arising from our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533607959

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We want to thank all reviewers again for their feedback and are glad that the changes made to the paper were received positively!

We have uploaded another revision, improving minor nitpicks such as improving figure and table readability (improving captions) and correcting typos.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

The authors explore the efficacy of using LLM-generated synthetic training data for automated stance detection. They primarily test against a Bert baseline where individual models are trained for different stance detection question types. Varying levels of synthetic data are used to finetune this model, as well as previously unlabeled human data collected using active learning that makes use of nearest neighbor synthetic examples. Results indicate performance gains from synthetic data (72.3 vs 69.3 F1) and synthetic data mixed with active learning labeled data (75.4 F1). 

**Strengths:**

- The authors' research questions are well-defined and the results seem to support use of synthetic data for stance detection 

- While the findings are unsurprising given recent literature around effectiveness of synthetic data on other tasks like content moderation, the experimentation is very solid and there are unique aspects to their approach such as using synthetic nearest-neighbors to facilitate active learning on human-derived data. 

**Weaknesses:**

- The experimental results are not very comprehensive and only consider the effect on a Bert baseline. The authors also test against Mistral, but only in a zero-shot vs fine-tuned setting. Given the sensitivity of LLMs to prompt phrasing, much more could be explored here including few-shot demonstrations in the prompt or specifying the desired output format. 

- The authors don't seem to conduct any analysis to ensure the synthetic comments are consistent with the data labels used in prompting. 

- The paper would benefit from statistically significance testing, in particular when comparing baseline vs. synthetic only results.

I think the paper is technically sound enough to be accepted and commend the authors on their efforts, however I question the impact the paper's appearance in conference proceedings will make on the broader research community.