The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

## Abstract

Abstract Stance detection holds great potential for enhancing the quality of online political discussions, as it
has shown to be useful for summarizing discussions, detecting misinformation, and evaluating opinion distributions.
Usually, transformer-based models are used directly for stance detection, which require large amounts of data.
However, the broad range of debate questions in online political discussion creates a variety of possible scenarios that the model is faced with and thus makes data acquisition for model training difficult.
In this work, we show how to leverage LLM-generated synthetic data to train and improve stance detection agents for online political discussions:
(i) We generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated
synthetic data can substantially improve the performance of stance detection.
(ii) We examine the impact of combining synthetic data with the most informative samples from an unlabelled dataset. First, we use the synthetic data to select the most informative samples, second, we combine both these samples and the synthetic data for fine-tuning.
This approach reduces labelling effort and consistently surpasses the performance of the baseline model that is trained with fully labeled data.
Overall, we show in comprehensive experiments that LLM-generated data greatly improves stance detection performance for online political discussions.

## 1 Introduction

With the recent advent of powerful generative Large Language Models (LLMs) such as ChatGPT(^1^11[https://chat.openai.com/](https://chat.openai.com/)),
Llama (Touvron et al., ) and Mistral (Jiang et al., ), investigating how synthetic data
from LLMs can be leveraged for downstream tasks has become a very important and relevant topic of research.
It is well known that Natural Language Processing (NLP) tasks based on single-label or
multi-label classification such as stance and sentiment detection
benefit immensely from large amounts of labelled data (Mehrafarin et al., ). However, in many cases, labelled data is scarce, especially in
the realm of online political discussions: one issue is that gathering labelled data may not be allowed for privacy reasons.
Another issue is that especially in stance detection for online political discussions comments and their stances are intrinsically
linked to a question. There are a potentially infinite number of specific questions open for debate for which different types of comments (and their stances) arise.
This can be a problem, since the samples may not be varied or pertain to only one class, hence usually yielding an unbalanced dataset.
For instance, performance drops severely for questions in the test dataset, that are not present in the training dataset (Vamvas and Sennrich, ).

An obvious approach to stance detection might be to apply LLMs *directly*, which has been tried with mixed results (see Section 2).
Furthermore, LLMs are not yet fast enough at inference time for smaller real world interactive deployment in online discussions, unless adequate hardware infrastructure is provided.
Also, using public hosting infrastructure may not be feasible when use cases require anonymity and privacy.

Figure: Figure 1: We investigate the use of LLM-generated synthetic data for stance detection in online political discussions. (A) We generate synthetic data for specific questions using a Mistral-7B model. The synthetic data is then used to fine-tune the stance detection model. We show that fine-tuning with synthetic data improves the performance of the model, since the synthetic data is roughly faithful to the real data’s underlying distribution. However, some real world samples cannot be captured by the synthetic data. (B) We therefore use the synthetic data to identify the most informative samples in the unlabelled real data pool. That is, we identify the samples that are on the boundary of the two classes of the synthetic data which are better off labelled by human experts. We show that combining the synthetic data with the manually labelled most informative samples improves the performance of the model even further.
Refer to caption: x1.png

While LLMs might not be directly useful for stance detection, we show in this paper, that they can be used to generate synthetic training data for more specific (and smaller) NLP models, such as BERT (Devlin et al., ), to create powerful stance detection for online political discussions that potentially work for any type of question.
How exactly synthetic data from an LLM should be combined with a small pool of unlabelled training data will be investigated along the following questions:

#### Question 1 (Q1): Does fine-tuning with synthetic data improve the stance detection model? Answer: Yes, and the more the better!

One problem of stance detection datasets for online political discussions is the lack of (balanced) data for specific questions (sometimes also called issues).
While fine-tuning with synthetic data is possible for BERT-based models, it was previously found to be challenging (Mahmoudi et al., ). Using more powerful LLMs, we can show that fine-tuning with synthetic data does improve stance detection performance in online political discussions. This also holds when augmenting existing labelled data.

#### Question 2 (Q2): Can the labelling effort be reduced by controlling what example to manually label using active learning with synthetic data? Answer: Yes, even if we don’t have real true labels initially.

Another challenge in stance detection for online political discussions
is labelling data, which is expensive and time consuming.
Active learning can help us to identify the most informative samples that are then labelled manually,
even if there are initially no labelled samples. For this we use the synthetic samples which have labels by construction, to select the most ambiguous (and thus most informative) samples in the unlabelled dataset.

#### Question (Q3): Can fine-tuning using the most informative samples plus the synthetic data outperform fine-tuning with a completely manually labelled dataset? Answer: Yes, it can!

Our study reveals that the most effective way to improve stance detection performance is to combine both synthetic data and the most informative samples obtained via active learning.
Surprisingly, this is even better than using a full dataset that has been completely manually labelled.

## 2 Background

#### Stance Detection for Online Political Discussions.

Stance detection is a sub-task of sentiment analysis (Romberg and Escher, ) and opinion mining (ALDayel and Magdy, )
that aims to automatically detect the stance (e.g., *favor*, *against*, or *neutral*) of the author of a contribution towards a discussed issue,
or target. In online political discussions, stances can be defined as the *side* that the author is on, e.g., *for* or *against* an increase in
value added tax. Research has identified stance detection as an important yet challenging task; it is important because the automated detection of stances can,
e.g., improve discussion summarisation (Chowanda et al., ), facilitate the detection of misinformation (Hardalov et al., ),
and provide more comprehensive evaluations of opinion distributions in online political discussions and participation processes (Romberg and Escher, ).
Stance detection is also implemented as a feature in many recommender systems and discussion platforms (Küçük and Can, ).
Still, due to its dependency on context, stance detection is a highly challenging task. In online political discussions, for example,
detecting the stance of a contribution towards a question requires a proper identification of both the question under discussion and the position of the contributor
towards it. However, users regularly deviate from the original question and discuss multiple topics in the same thread (Ziegele et al., ),
resulting in a high number of potential questions with often very few instances for training purposes.
Fine-tuning transformer-based models (Vaswani et al., ) to solve stance detection is a common practice,
but training these models requires a large amount of annotated data, which for the large variety of questions in online political discussions is
unfeasible to acquire. We therefore in our work, show how to improve stance detection for online political discussions
with synthetic data.

#### Active Learning.

The aim of *active learning* is to minimize the effort of labelling data,
while simultaneously maximizing the model’s performance.
This is achieved by selecting a *query strategy* that chooses the most interesting samples
from a set of unlabelled data points, which we refer to as *most informative* samples.
These samples are then passed to, e.g., a human annotator for labelling.
The usual setup consists of iteratively training a model on a small subset of manually labelled data that stems from a pool of unlabelled data.
There exist many different query strategies such as Query By Comittee (QBC, (Seung et al., )), Minimum Expected Entropy (MEE, Holub et al. () or Contrastive Active Learning (CAL, Margatina et al. ()).
By actively choosing samples and asking for the correct labelling, the model is able to learn from few labelled data points, which is advantageous especially when annotated datasets are not available.
Within the domain of political text analysis, many different tasks lack large amounts of annotated data. It has been already shown in the past that these tasks can benefit from the active learning: e.g., stance detection (Kucher et al., ), topic modeling (Romberg and Escher, ), speech act classification (Schmidt et al., ) or toxic comment classification (Miller et al., ).
In this work, we examine how LLM-generated synthetic data can be used instead of real labelled data to select the most informative samples to be manually labelled.

#### Using LLM-generated synthetic data for fine tuning.

Recent work has shown that synthetic data generated from LLMs can be used to improve the performance of a model on downstream tasks.
Møller et al. () showed that synthetic data can be used to improve the performance of a model on downstream classification tasks by comparing
the performance of a model finetuned on LLM-generated data to crowd annotated data. In many cases the model finetuned on LLM-generated data
outperforms the model finetuned on crowd annotated data. Mahmoudi et al. () study the use of synthetic data for data augmentation in stance-detection.
The authors use GPT-3 to generate synthetic data for a specific topic with mixed results due to the inability of GPT-3 to generate good data samples.
In our work,we use a newer LLM model, Mistral-7B, which generates better synthetic data samples and show that we can generate synthetic data
that matches the real data distribution.
Veselovsky et al. () analyse in which ways synthetic data is best generated for tasks like sarcasm detection and sentiment analysis. The authors reach the conclusion that grounding the prompts to generate the synthetic data
to real samples helps improve the quality of the synthetic data. Similarly, Li et al. () argue that subjectivity in the classification task determines whether synthetic data can be used effectively.

#### Using LLMs directly for stance detection.

It has been shown that LLMs can be used directly for stance detection such as (Cruickshank and Xian Ng, ), (Burnham, ).
However, the general conclusion of these studies is that while LLMs are competitive with other transformer models such as BERT, especially for edge cases, they exhibit replication issues.
(Burnham, ) also discuss the posibility of pre-training models on more specific data to improve the generalisation capability of the model.
We therefore focus on using LLMs to generate synthetic data to solve key challenges in stance detection such as the lack of available data for specific topics and labelling large amounts of data,
rather than using LLMs directly for the task.

## 3 Method

In line with our declared contributions, we present our core ideas to improve the performance of the stance detection model:
(i) To fine-tune the model with synthetic data, we first define the synthetic dataset and show our pipeline to generate it. The baseline model is then further fine-tuned
on the synthetic data. (ii) We then present our synthetic data-based approach to identify the most informative samples from a batch of unlabelled data.
We propose a synthetic extension to the QBC (Query by Commitee, (Seung et al., )) method, where the synthetic data act as an ensemble of experts.

### 3.1 Preliminaries

Political discussions are typically centered around questions $q\in\mathcal{Q}$ (sometimes also called issues or targets). For stance detection, we usually have for each of these questions $q$ a set of labelled data $\mathcal{D}^{(q)}=\{(x^{(i)},y^{(i)})\}_{i=1}^{I}$ where $x^{(i)}\in\mathcal{X}$ is a statement (or comment) and $y^{(i)}$ is the stance of the statement,
with $y^{(i)}\in\{0,1\}=\mathcal{Y}$. Note, that we use the notation $\mathcal{D}^{(q)}$ for labelled and for unlabelled datasets (then the labels are ignored).
We view the stance detection model as a binary classification function $f:\mathcal{Q}\times\mathcal{X}\rightarrow\mathcal{Y}$, where we included the question as input to provide context.
The classifier $f$ can be split into two parts, $f=h\circ g$:
the first part is a pre-trained language model $g:\mathcal{Q}\times\mathcal{X}\rightarrow\mathcal{E}$ such as BERT (Devlin et al., ) or RoBERTa (Liu et al., ). The second part is a classifier $h:\mathcal{E}\rightarrow\mathcal{Y}$ on top of the pre-trained model $g$.
The pre-trained language model $g$ outputs an embedding $e^{(i)}\in\mathcal{E}$ given a question $q$ and an input $x^{(i)}$. The classifier $h$ is usually a linear layer or a multi-layer perceptron, which takes the embedding $e^{(i)}$ from $g$ as input and outputs a prediction $\hat{y}^{(i)}$.
The stance detection model is *fine-tuned* by minimizing the cross-entropy loss between the predicted labels $\hat{y}^{(i)}=f(q,x^{(i)})$ and the actual labels $y^{(i)}$.

### 3.2 Generating Synthetic Data for Stance Detection

To generate synthetic samples, we employ a quantized version of the Mistral-7B-instruct-v0.1 model(^2^22[https://huggingface.co/turboderp/Mistral-7B-instruct-exl2](https://huggingface.co/turboderp/Mistral-7B-instruct-exl2)) to generate comments on a specific question $q$, using the following prompt:

A user in a discussion forum is debating other users about the following question: [q] The person is in favor about the topic in question. What would the person write? Write from the person’s first person perspective.

where "[q]" must be replaced with the question $q$. Similarly, to generate a negative sample, we replace "is in favor" with "is not in favor".
As in the X-Stance dataset (Vamvas and Sennrich, ), we assign the two labels 0 and 1. We denote the question-specific synthetic dataset as:

$$ $\mathcal{D}_{\text{synth}}^{(q)}=\big{\{}(x_{\text{synth}}^{(m)},1)\big{\}}_{m =1}^{M/2}\cup\big{\{}(x_{\text{synth}}^{(m)},0)\big{\}}_{m=1+M/2}^{M}$ (1) $$

where half of the $M$ synthetic data samples have *positive* labels, i.e., are comments in *favor* for the posed question, while the other
half is *against*. Since the dataset is in German, we translate the questions $q$
with a "NLLB-300M" (NLLB Team et al., ) translation model.
The English answers from the Mistral-7B model are then translated back to German using the translation model.

We show the translation pipeline in Figure 9 along with translations of the questions chosen for our experiments in Table 2.
We also attach the synthetic generated data in both English and German to the paper.
In our experiments, this translation pipeline worked well even for complex topics in online political discussions, making
our method applicable to any language where a good translation model exists. Overall, the generated dataset $\mathcal{D}_{\text{synth}}^{(q)}$ will be used in two ways: (i) to augment the existing dataset $\mathcal{D}^{(q)}$ in order to increase the amount of training data, and (ii) to detect the most informative samples in the unlabelled data pool, which is explained next.

### 3.3 Synthetic Data-driven Query By Comittee (SQBC)

In the typical active learning scenario, we have a set of unlabelled data,
where the goal is to choose the most informative samples to label.
Our approach is inspired by the two methods Query by Comittee (QBC) (Seung et al., ) and Contrastive Active Learning (CAL, Margatina et al. ()):
instead of using QBC’s ensemble of experts and the KL-divergence based information score in CAL, we directly use the synthetic data and its labels to produce classification scores for the unlabelled data using $k$ nearest neighbors. The most informative samples are then the data points with the most indecisive scores. Since we query a synthetic dataset instead of an ensemble of experts (as in QBC), we call our method *Synthetic data-driven QBC* or short SQBC.

Our method performs three steps:

#### (1) Generate the embeddings.

Given some embedding function $g:\mathcal{Q}\times\mathcal{X}\rightarrow\mathcal{E}$, we generate embeddings for the unlabelled data,

$$ $E=\big{\{}e^{(i)}\big{\}}_{i=1}^{I}=\big{\{}g(q,x^{(i)})\big{\}}_{i=1}^{I}$ (2) $$

and for the labelled synthetic data

$$ $E_{\text{synth}}=\big{\{}e_{\text{synth}}^{(m)}\big{\}}_{m=1}^{M}=\big{\{}g(q, x_{\text{synth}}^{(m)})\big{\}}_{m=1}^{M}.$ (3) $$

Note that $q$ is the question for which we generate the synthetic data and for which we want to detect the most informative samples. If obvious from the context, we often omit the superscript $(q)$.

#### (2) Using the synthetic nearest neighbours as oracles to score the unlabelled data.

For the $i$-th unlabelled embedding $e^{(i)}$ let $\text{NN}(i)$ be the set of indices of the $k$ nearest neighbours (wrt. to the embeddings using the cosine similarity) among the labelled embeddings $E_{\text{synth}}$. The score for each unlabelled data point counts the number of labels $y_{\text{synth}}^{(m)}=1$ among the nearest neighbours, i.e.,

$$ $s(i)=\sum_{m\in\text{NN}(i)}y_{\text{synth}}^{(m)}\in\{0,\ldots,k\}.$ (4) $$

For our experiments, we choose $k=M/2$ which worked well across all experiments (other values for $k$ are possible, but did not lead to significantly better results).

#### (3) Choosing the most informative samples.

The scores take values between 0 and $k$. For 0, the synthetic nearest neighbours all have labels $y_{\text{synth}}^{(m)}=0$, for value $k$, all have labels $y_{\text{synth}}^{(m)}=1$. The *most informative* samples have a score around $k/2$.
We thus adjust the range of the scores so that values in the middle range have the smallest scores (close to 0). We do this by subtracting $k/2$ from the score and taking the absolute value,

$$ $s^{\prime}(i)=|s(i)-k/2|.$ (5) $$

The $J$ most informative samples $\mathcal{D}^{(q)}_{\text{MInf}}\subset\mathcal{D}^{(q)}$ among the unlabelled samples are the $J$ samples with the smallest scores.
In the experiments we vary $J$ to study the impact of manually labelled most informative samples.

#### (4) Labelling the most informative samples.

This step must be performed manually, i.e., the most informative samples selected in the previous step are labelled by a human expert. Attempting to assign labels automatically using majority vote from the synthetic samples did not work well, since the most informative samples are exactly at the boundary between the classes, leading to random and thus noisy labels.

## 4 Experiments

We now describe the datasets, the experiments and the results.
Our experiments follow the structure of the questions Q1, Q2 and Q3 posed in the introduction. Apart from presenting numerical results for these questions, we also provide further insights by visualizing the embeddings of the synthetic and real world datasets and by showing the selection behaviour of the active learning methods.

### 4.1 Datasets

#### X-Stance dataset.

To evaluate our methods, we use the X-Stance dataset (Vamvas and Sennrich, ), a multilingual stance detection dataset, which contains a total of $67,000$ annotated comments on $194$ policy-related questions, answered by Swiss election candidates.
X-stance includes comments in German ($48,600$), French ($17,200$) and Italian ($1,400$). For this paper we only included the German comments, but as mentioned above, other languages can be easily included by automatic translation. The data has been extracted from smartvote, a Swiss voting advice platform.
The comments are labelled either as being in *favor* (positive) or *against* (negative) the posed question, which is a typical labelling scheme for stance detection datasets.
The X-Stance dataset is split in training and testing questions, i.e,. a question in the training dataset does not appear in the test dataset. Furthermore, for each question $q$ from the training data, there are several annotated comments, which form the dataset $\mathcal{D}_{\text{train}}^{(q)}$. Similarly,
for each question $q$ in the test data, the set of annotated comments is written as $\mathcal{D}_{\text{test}}^{(q)}$. To refer to the whole training dataset we write $\mathcal{D}_{\text{train}}=\cup_{q\in\mathcal{Q}}\mathcal{D}^{(q)}_{\text{
train}}$.

To limit computation time, we selected 10 questions from the test dataset to evaluate our method. The selection was done to best reflect the variability of the data (see Appendix D.1): we
choose questions with small amount of comments, unbalanced comments and also balanced comments. Furthermore, for 5 of the questions the majority class is *favor* and for the other 5 the majority class is *against*.

#### Synthetic dataset.

To test the synthetic data-augmentation and active learning based on SQBC (see Section 3.3) we generate sets of varying sizes $M=\{200,500,1000\}$ for each of the $10$ questions.
The synthetic data follows the same structure as the data from the X-stance dataset, where for a specific question $q$ we have $M$ comments and $M$ labels.
Each set contains $M/2$ positive labels and $M/2$ negative labels, i.e., the synthetic data is balanced.

### 4.2 Experimental Setup

#### General setup.

For all experiments, the starting model is the pre-trained BERT base(^3^33[https://huggingface.co/dbmdz/bert-base-german-uncased](https://huggingface.co/dbmdz/bert-base-german-uncased)) model. We adapt the model to the stance detection task by training it on the X-Stance training dataset $\mathcal{D}_{\text{train}}$ (across all questions). We call this the Baseline, and it is characteristic for BERT-based stance detection (e.g., Vamvas and Sennrich ()).
For fine-tuning and testing we evaluate the given model separately on 10 chosen questions from the test dataset of X-Stance for all experiments. For each question $q$ we split $\mathcal{D}_{\text{test}}^{(q)}$ into a $60/40$ train/test split (repeated with 5 different seeds to get error bars) and use the train split for fine-tuning to the given question and the test split for evaluation.

#### Evaluated methods.

Next, we briefly describe the different evaluated methods. Table 1 shows an overview of which datasets are used for each method/variant:

Baseline: this is the default model only trained on $\mathcal{D}_{\text{train}}$, (e.g., Vamvas and Sennrich ()).
True Labels: this variant assumes that all true labels for the train split of $\mathcal{D}_{\text{test}}^{(q)}$ are available
and thus performs *no active learning*. This is the case $\mathcal{D}^{(q)}_{\text{MInf}}=\mathcal{D}^{(q)}_{\text{test}}$ for each $q$ in the test set.
SQBC: we apply our active learning approach to get the most informative samples $\mathcal{D}_{\text{MInf}}^{(q)}$.

CAL: we apply Contrastive Active Learning (CAL, Margatina et al. ()) to get the most
informative samples $\mathcal{D}_{\text{MInf}}^{(q)}$.
Random: as an ablation, we randomly select the most informative samples $\mathcal{D}_{\text{MInf}}^{(q)}$.
Note that the "+Synth" variants additionally fine-tune the model on the synthetic data $\mathcal{D}_{\text{synth}}^{(q)}$.

#### Evaluation.

Our main results report the average F1 score over 10 selected questions from the test dataset evaluated on the comments from the test split. The error bars represent the average standard deviation over the 10 questions for 5 runs with different seeds. More detailed results per question are shown in Appendix E.

### 4.3 Experiments

For each question proposed in Section 1, we designed a specific experiment to test the impact of LLM-generated synthetic data augmentation for stance detection in online political discussions:

- •
Q1: The two models Baseline and True Labels are augmented with synthetic data of varying sizes.
- •
Q2: We compare the 3 active learning strategies to determine the most informative samples: CAL, SQBC and Random.
To study reduction in labelling effort, we define 4 different budgets of data samples ($75\%,50\%,25\%$ and $10\%$ of $\mathcal{D}^{(q)}_{\text{test}}$) to be selected for manual labelling. These variants are also compared against Baseline and True Labels.
- •
Q3: We combine fine-tuning with the synthetic data (Q1) and the (manually labelled) most informative samples (Q2). We apply the same testing procedure as in Q2, where we test the performance for different labelling budgets, while
also comparing to Baseline and True Labels.

### 4.4 Results

Figure: Figure 2: Q1: Fine-tuning the model with synthetic data improves performance for increasing dataset size: Shown are the F1-Score of Baseline (left) and True Labels (right) for increasing synthetic dataset. Even if a dataset has been fully labelled, augmenting it with synthetic data proves equally as effective.
Refer to caption: x3.png

Figure: Figure 3: Q2: SQBC selects the best samples for manual labelling and improves upon the baseline model. Increasing the synthetic data size for the selection of most informative samples only improves performance marginally, which shows that uncertain samples can already be identified with a small synthetic dataset.
Refer to caption: x4.png

Figure: Figure 4: Q3: Fine-tuning with most informative samples and synthetic data surpasses the model fine-tuned with all true labels: We label less samples manually, while outperforming the model with all true labels. The reason for the performance increase can be attributed on the effects from Q1 and Q2: (i) the synthetic data helps the model learn the underlying distribution. (ii) The most informative samples improve the model where the synthetic data distribution is not expressive enough.
Refer to caption: x5.png

#### Q1: Fine-tuning with synthetic data reaches on average the performance of the model with all true labels.

Figure 2 shows the effectiveness of fine-tuning with synthetic data by fine-tuning Baseline and True Labels with different amounts of synthetic data $M=\{200,500,1000\}$.
The baseline model with $1000$ synthetic data samples almost reaches the average performance of the model with all true labels. Thus, fine-tuning with synthetic data attains similar performance to labelling all data, without having to label any real data sample.
Moreover, even if a dataset has been fully labelled, augmenting it with synthetic data improves stance detection performance even further.

#### Q2: Most informative samples selected by SQBC are more informative than CAL and random selection and do improve upon the baseline model.

In Figure 3, we show that we can reduce labelling effort by using synthetic data to select the most informative samples. In this setting, we only use
the synthetic data for active learning and do not fine-tune with the synthetic dataset.
On average, SQBC is bettern than CAL and Random.
We observe, that fine-tuning only with the most informative samples might not be as effective as fine-tuning with synthetic data, since Baseline+Synth (M=1000) is slightly better than SQBC. However, the advantage of using the most informative samples is that the model is grounded on real data.

#### Q3: Fine-tuning with the most informative samples and additionally with synthetic data surpasses the model trained with all true labels.

In Figure 4, we show the performance of fine-tuning the baseline model with the manually labelled most informative samples and at the same time with synthetic data. This combines the experiments for Q1 and Q2. We see that on average with as little as $25\%$ of the unlabelled data, we can consistently achieve better performance than the model trained with all true labels (True Labels).
For $M=1000$ synthetic samples, we observe that surprisingly random sampling outperforms the other query strategies (Random+Synth). We will discuss this in the next section.

### 4.5 Interpreting the role of the synthetic data

The synthetic data is at the core of our work.
To get a better understanding we analyse the role of synthetic data when using it for data augmentation and for the selection of the most informative samples.
To study whether the synthetic data matches the real data, we visualize embeddings of the synthetic data together with the embeddings of the real world data using
T-SNE projection (van der Maaten and Hinton, ) in Figures 5, and 6. To gain insights into the selection mechanism for different query strategies, we add the T-SNE embeddings of the selected samples in Figures 8 and 7. Again, we discuss our insights along the questions Q1, Q2 and Q3.

#### Q1: The synthetic data matches the distribution of the real world data.

We visualize the positive and negative synthetic data samples together with the means (cluster centers) of the real world data in Figures 5 and 6.
We observe that the synthetic data extends the real world data, which we believe is the reason why fine-tuning with synthetic data is effective in online political discussions. Also, the larger the synthetic dataset size,
the more the synthetic data matches the distribution of the real world data. Furthermore, the positive and negative samples are well separated, which we attribute to having pre-trained the BERT-model on $\mathcal{D}_{\text{train}}$ of the X-Stance dataset, providing prior knowledge about the stance detection task.

#### Q2: SQBC selects samples that are not contained in the synthetic data distribution.

We visualize the selected samples of the SQBC and compare them to the selected samples by CAL and Random in
Figures 7 and 8. We see that SQBC selects the unlabelled samples that are mostly in between
the two classes of the synthetic data. This is the expected behaviour since we select the samples where the classification score is ambiguous.
For random selection, the range of selected samples is broad: some similar samples between the two classes like SQBC are selected, but also within class samples that are not covered by the synthetic data set. This explains why random selection works well with a large synthetic dataset, since it further extends the decision boundary of the model.
Finally, CAL selects samples similar to SQBC, but mostly tends to select samples from only one class, resulting in worse performance.

#### Q3: The synthetic data extends the unlabelled real data, while the most informative samples improve classification by selecting samples not well covered by the synthetic data.

Combining the insights from Q1 and Q2, we see that the synthetic data effectively extends the real world data thus providing a better estimate of the underlying distribution. When adding the most informative samples from the active learning methods, we add
samples that are not in the more general synthetic data distribution thus extending the decision boundary of the classifier. We argue that this is why the combination of most informative samples and synthetic samples is so effective.

## 5 Discussion

#### Limitations and future work.

One limitation of our approach is the fact that we have to fine-tune a separate model for each question. While this leads to excellent results, a common approach is to fine-tune a single (and thus more general) model
for several questions (like Baseline). Having observed that the synthetic data distribution differs (sometimes greatly) for each question, we feel that the extra training is justifiable and necessary.
The computing time is not a concern in our case, since fine-tuning the BERT model takes less than a minute even with a synthetic dataset size of $M=1000$ on a reasonable GPU (NVIDIA A100).
Future work, could study whether the underlying distribution of the synthetic data can serve as a proxy to combine real unlabelled data
from different questions with similar (compatible) underlying distributions. This might enable us to train capable *general* stance detection models.

#### Conclusion.

In this work, we presented how to improve stance detection models for online political discussions utilizing LLM-generated synthetic data:
(i) we showed that fine-tuning with synthetic data related to the question improves the performance of the stance detection model. After analysing the embeddings of the synthetic data, we attribute this to the LLM-generated synthetic data aligning well with the real data for the given question.
(ii) We reduce the labelling effort by selecting the most informative samples informed by synthetic data with SQBC. The synthetic data are separated by construction into the positive and negative classes, which makes finding the most informative samples more straightforward, since we can count the labels of the k-nearest synthetic neighbours to determine classification ambiguity.
(iii) Fine-tuning with synthetic data can be further enhanced by adding the most informative samples, outperforming the method that uses all true labels.
Concluding, our findings demonstrate the potential of synthetic data to significantly improve the efficiency and effectiveness of stance detection in online political discussions.

## Appendix A Broader Impact

Our work has the potential to improve the efficiency and effectiveness of stance detection in online political discussions.
This can be beneficial for social media platforms, news outlets, and political campaigns, where the detection of stance can help to understand the sentiment of the users.
The use of synthetic data in this case is not critical for the user, since the synthetic data is only used to improve the model’s performance.
Stance detection could also be used to for bad purposes, e.g., to manipulate the public opinion.
However, our work does not contribute to this, since we only focus on improving the model’s performance, i.e., the model’s ability to detect stance.

## Appendix B Visualizations

### B.1 Visualizing the synthetic data

We visualize the synthetic data together with the real world data for $M=1000$ and $M=200$ in Figures 5 and 6.
We plot the data points of the synthetic data in blue and red for the positive and negative samples, respectively. The means of the real world data are plotted as a regular polygon with 8 sides.
We observe that the synthetic data extends the real world data, which we consider a factor as to why fine-tuning with synthetic data is effective in online political discussions. Also, the larger the synthetic dataset size,
the more the synthetic data matches the distribution of the real world data since for $M=200$ (see Figure 6) the mean are not as well aligned with the synthetic data. Furthermore, the positive and negative samples are well separated, which we attribute to having pre-trained the BERT-model on $\mathcal{D}_{\text{train}}$ of the X-Stance dataset, giving the prior knowledge about the stance detection task.

Figure: Figure 5: Visualization of synthetic data with train data means for $M=1000$ synthetic data. For a larger synthetic dataset size, the means of the synthetic data are well aligned with the real world data and the positive and negative samples are well separated. The synthetic data thus extends the real world data, which we consider a factor as to why fine-tuning with synthetic data is effective in online political discussions.
Refer to caption: extracted/5675382/figures/sqbc_vis_train/train_vis10001.png

Figure: Figure 6: Visualization of synthetic data with train data means for $M=200$ synthetic data: For a smaller synthetic dataset size, the means of the synthetic data are not as well aligned with the real world data as for $M=1000$. However, the positive and negative samples are still well separated.
Refer to caption: extracted/5675382/figures/sqbc_vis_train/train_vis2001.png

### B.2 Visualizing the query strategies of the active learning methods

We visualize the selected samples of SQBC, CAL and Random query strategies for $M=1000$ and $M=200$ in Figures 7 and 8.
We plot the selected samples of the unlabelled data in green. The positive and negative synthetic data samples are plotted in blue and red, respectively. The selected samples are highlighted in orange.
We observe that SQBC selects the unlabelled samples that are mostly in between the two classes of the synthetic data. This is the expected behaviour since we select the samples where the classification score is ambiguous.
For Random, the range of selected samples is broad: some similar samples between the two classes like SQBC are selected, but also within class samples that are not covered by the synthetic data set.
This explains why random selection works well with a large synthetic dataset, since it further extends the decision boundary of the model. For the smaller synthetic dataset $M=200$, the random selection is not as effective, since the selected samples
are spread out over the whole data space and not necessarily in between the two classes as with the larger synthetic dataset.
Finally, CAL selects samples similar to SQBC, but mostly tends to select samples from only one class.

Figure: Figure 7: Visualization of SQBC, Random and CAL query strategies for $M=1000$ synthetic data: SQBC selects the unlabelled samples that are mostly in between the two classes of the synthetic data. This is the expected behaviour since we select the samples where the classification score is ambiguous. For random selection, the range of selected samples is broad: some similar samples between the two classes like SQBC are selected, but also within class samples that are not covered by the synthetic data set. This explains why random selection works well with a large synthetic dataset, since it further extends the decision boundary of the model. Finally, CAL selects samples similar to SQBC, but mostly tends to select samples from only one class, resulting in worse performance.
Refer to caption: extracted/5675382/figures/sqbc_vis/train_selected+synth10002.png

Figure: Figure 8: Visualization of SQBC, Random and CAL query strategies for $M=200$ synthetic data: For a smaller synthetic dataset size, SQBC is still able to select the unlabelled samples that are mostly in between the two classes of the synthetic data. For Random we see that the selected samples are a bit further away from the synthetic data distribution, which is why we argue it does not perform as well as with the larger synthetic dataset.
Refer to caption: extracted/5675382/figures/sqbc_vis/train_selected+synth2002.png

## Appendix C Additional Experimental Details

### C.1 Compute and Runtime

We conduct our experiments on a single NVIDIA A100 80GB GPU and a 32 core CPU.
With this setup, for Mistral-7B, the generation of synthetic data takes approximately $3$ hours per question for a synthetic dataset size of $M=1000$.
Fine-tuning the BERT model with the synthetic data takes less than a minute. For the active learning methods, the selection of the most informative samples takes less than a minute.
Hence the largest computational effort is the generation of the synthetic data.

### C.2 Translation of the X-Stance dataset for synthetic data generation

In Figure 9 we show the pipeline for translating the X-Stance dataset for synthetic data generation.
We start with a question $q$ from the X-Stance test dataset and translate the question to English with a NLLB-330M model [NLLB Team et al., ].
Then we let the Mistral-7B model generate synthetic data, i.e., comments for the translated question.
The generated comments are then translated back to German to be used for fine-tuning the model in our experiments.

### C.3 Overview of used datasets

In Table 1 we show an overview of the datasets used in our experiments for the different methods we evaluate.

Figure: Figure 9: Overview of the pipeline for active learning with synthetic data: We start with a question $q$ from the X-Stance test dataset and translate the question to English with a NLLB-330M model [NLLB Team et al., ]. Then we let the Mistral-7B model generate synthetic data, i.e., comments for the translated question. The generated comments are then translated back to German to be used for fine-tuning the model in our experiments.
Refer to caption: x6.png

**Table 1: Synth: Synthetic Data, Aug: Augmentation. We compare different variants of active learning with synthetic data.**
| Config<br>Datasets | Manual labels<br>$D_{\text{MInf}}$ | True Labels<br>$\mathcal{D}_{\text{t}}$ | Synth Aug<br>$D_{\text{synth}}$ |
| --- | --- | --- | --- |
| Baseline |  |  |  |
| Baseline + Synth |  |  | ✓ |
| Oracle |  | ✓ |  |
| Oracle + Synth |  | ✓ | ✓ |
| SQBC | ✓ |  |  |
| SQBC + Synth | ✓ |  | ✓ |
| CAL | ✓ |  |  |
| CAL + Synth | ✓ |  | ✓ |
| Random | ✓(randomly selected) |  |  |
| Random + Synth | ✓(randomly selected) |  | ✓ |

## Appendix D Dataset

X-Stance is a multilingual stance detection dataset, including comments in German ($48,600$), French ($17,200$) and Italian ($1,400$) on political questions, answered by election candidates from Switzerland. The data has been extracted from smartvote(^4^44[https://www.smartvote.ch/](https://www.smartvote.ch/)), a Swiss voting advice platform. For the task of cross-topic stance detection the data is split into a training set, including questions on 10 political topics, and a test set with questions on two topics that have been held out, namely *healthcare* and *political system*.

### D.1 Chosen questions and their distribution

We present the $10$ chosen questions for our experiments in Table 2. We show the original questions in German and their corresponding English translations by the translation model.
Furthermore, we also show the (60 / 40 ) train/test split for each question in Figure 10. We chose 10 questions that reflect the overall distribution of $\mathcal{D}^{(q)}_{\text{test}}$. We chose unbalanced, balanced and low sample size questions to evaluate the effectiveness of our approach.

Figure: Figure 10: Distribution of the positive and negative samples for the train and test split of $\mathcal{D}^{(q)}_{\text{test}}$: We show the distribution of the positive and negative samples of the X-Stance test dataset for the questions Q1-Q10. We also show the 60/40 train/test split for the 10 questions. We chose 10 questions that reflect the overall distribution of $\mathcal{D}^{(q)}_{\text{test}}$. We chose unbalanced, balanced and low sample size questions to evaluate the effectiveness of our approach.
Refer to caption: x7.png

**Table 2: Chosen questions for stance detection in German and their English translation**
| Question in German | Question in English |
| --- | --- |
| Sollen sich die Versicherten stärker an den Gesundheitskosten beteiligen (z.B. Erhöhung der Mindestfranchise) | Should insured persons contribute more to health costs (e.g. increase in the minimum deductible)? |
| Befürworten Sie ein generelles Werbeverbot für Alkohol und Tabak? | Do you support a general ban on advertising alcohol and tobacco? |
| Soll eine Impfpflicht für Kinder gemäss dem schweizerischen Impfplan eingeführt werden? | Should compulsory vaccination of children be introduced in accordance with the Swiss vaccination schedule? |
| Soll die Aufenthaltserlaubnis für Migrant/innen aus Nicht-EU/EFTA-Staaten schweizweit an die Erfüllung verbindlicher Integrationsvereinbarungen geknüpft werden? | Should the residence permit for migrants from non-EU/EFTA countries be linked to the fulfilment of binding integration agreements throughout Switzerland? |
| Soll der Bund erneuerbare Energien stärker fördern? | Should the federal government promote renewable energy more? |
| Befürworten Sie eine strengere Kontrolle der Lohngleichheit von Frauen und Männern? | Do you support stricter control of gender pay? |
| Soll der Bund die finanzielle Unterstützung für die berufliche Weiterbildung und Umschulung ausbauen? | Should the Federal Government expand its financial support for continuing vocational training and retraining? |
| Sollen in der Schweiz vermehrt Spitäler geschlossen werden, um die Kosten im Gesundheitsbereich zu senken? | Should more hospitals be closed in Switzerland to reduce healthcare costs? |
| Eine eidenössische Volksinitiative verlangt, dass der Bundesrat direkt vom Volk gewählt werden soll. Unterstützen Sie dieses Anliegen? | A sworn federal people’s initiative demands that the Federal Council be elected directly by the people. Do you support this concern? |
| Soll die Einführung der elektronischen Stimmabgabe bei Wahlen und Abstimmungen (E-Voting) weiter vorangetrieben werden? | Should the introduction of electronic voting in elections and voting (e-voting) be further promoted? |

## Appendix E Extended Results

We present the extended results for the different synthetic dataset sizes $M=200$, $M=500$ and $M=1000$ in Figures 11, 12 and 13.
As in Figure 4, we show the results for the different active learning methods and the different configurations of the synthetic data, while varying the amount of samples that need to be labelled.
We compare all methods to True Labels, hence the horizontal line corresponds to the performance of the baseline model fine-tuned with the true labels.

Figure: Figure 11: Extended results of Figure 4 for M=200:
Refer to caption: x8.png

Figure: Figure 12: Extended results of Figure 4 for M=500
Refer to caption: x9.png

Figure: Figure 13: Extended results of Figure 4 for M=1000
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - Why choose translation over adapting prompts directly? Is Mistral unable to generate responses in German, or were other multilingual models considered?

- In the ablation study for "Content vs. Size", I am not sure I understand why you call the shuffled dataset "misaligned". Could you please explain the reasoning behind this? 

- In the generated dataset, did you find any instance where the LLM failed to generate the requested content? For instance, generate statements not in favor when requested for "in favor" content or LLM refusing to generate any relevant content at all.
soundness: 3
strengths: - The paper leverages the data augmentation capabilities of LLMs to improve transformer based models which are better suited for online deployment as they are more reliable. 
- The presented method can be adapted to other text classification tasks and hence is a significant contribution.
- It is well written and easy to follow, except for few instances mentioned in the comments.
summary: The paper tries to improve transformer-based stance detection models by fine-tuning on LLM generated data. They compare the real-world data with the synthetic data to identify difficult samples from unlabelled data (active learning) to further improve the model. They show that both these steps improves the performance of the transformer-based baseline.
weaknesses: - It’s possible I’m missing some key context here, but I’m having trouble following the ablation study in Section 5.2. To test whether the performance gains come from dataset size or the generated content itself, the authors “shuffle” instances, apparently misaligning the posed questions with synthetic data. If the synthetic data consists of single text instances with labels, this shuffling wouldn’t seem to affect outcomes. Perhaps the authors mean they’re using different proportions of synthetic data in each run while keeping the total instance count constant, but this explanation feels somewhat unclear.
- Even though authors acknowledge this as a limitation, fine-tuning a separate model for each question doesnot seem to be a scalable approach, especially when the main motivation for the research was in line with training robust models for online deployment. 
- The X-stance dataset is described as having around 48k annotated comments on various questions. However, an overview of the dataset’s statistics—such as the number of comments per question—would greatly enhance readability. When you mention selecting 10 questions from the test set, it would be helpful to specify how many comments correspond to each question. While I see some statistics are included in the Appendix, a high-level summary within the main text would improve clarity and context for readers.
- Section 4.2, General setup: Please review this section for more readability. Currently, it is a bit difficult to get a picture of what models are being tested and how the methods differ between them.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: - It would be very helpful to explain why only a German dataset is used for the experiments. Also, if German text is used, have the authors considered using a different LLM that has good German language processing capabilities for the experiments?
soundness: 3
strengths: - The proposed method is sound. I do not see any major issue with the method.
- Although the idea of using synthetic data to augment models is not entirely new, it probably has not been widely explored for stance prediction.
- The authors conducted extensive experiments to evaluate the method, including varying the size of the synthetic dataset, comparing with meaningful baselines, and the further experiments that compare with a LLM zero-shot baseline.
summary: This paper proposes to use LLM-generated synthetic data to augment the training of stance classification models. It also proposes a synthetic data-based active learning method that uses synthetic data to facilitate the selection of unlabelled data for human annotation. Experiments are conducted on the German subset of the X-stance dataset (with the help of machine translation). The results demonstrate that including synthetic data in training can improve stance prediction. The synthetic data-based active learning method, however, is not clearly better than a random selection-based baseline active learning method.
weaknesses: - The experiments are conducted using a German dataset, but translation into and back from English is used in order for the method to work (probably because of limited German language understanding and generation capabilities of the Mistral model that is used?) There is no explanation of why the authors do not evaluate the method using an English dataset.
- The novelty and impact of the work is still limited. (1) Using synthetic data to augment models is not new. Although applying the idea to stance prediction might be new, it is one of many NLP tasks. The way synthetic data is generated and used during training in this paper is also standard, hence there is limited technical contribution. (2) The idea of using synthetic data for active learning is very interesting and is novel based on my knowledge. However, its effectiveness is limited based on the experiments. Therefore, overall, although the work is very solid in general, its novelty and impact may not meet the standard of this conference.
- There is room for improvement in terms of presentation. In particular, the active learning method proposed can benefit from first presenting an overview of the high-level intuition behind the method before describing the method itself.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Were there any computational/economic reasons for not scaling up your compute? I'm sympathetic to this as I understand the burden of even a single A100 - but if you do have more resources, why not use them?
soundness: 2
strengths: Clean experimental methodology with proper ablation studies
Good visualization of how synthetic data aligns with real data distributions
Actually bothered to translate German political content properly instead of using Google Translate
Reasonable baseline comparisons and honest reporting of limitations
The SQBC approach is somewhat novel, even if not revolutionary
summary: Yet another synthetic data paper that shows modest improvements but doesn't quite nail why or how to make synthetic data actually useful. Some interesting ideas buried under conventional methodology.

Look, I've seen enough "let's use LLMs to generate synthetic data" papers to last several conferences. What makes this one interesting - barely - is the political stance detection angle and the somewhat novel SQBC approach. But let's be real here: you're essentially using an LLM to generate slightly different versions of existing viewpoints, then acting surprised when this helps... a little bit.

The authors show that their approach improves F1 scores from ~0.69 to ~0.72 with synthetic data alone, and up to ~0.75 with their full pipeline. Sure, that's positive, but is it worth the computational cost of running Mistral-7B for hours to generate the synthetic data? (And don't get me started on the economic/environmental impact in general - though I suppose that's not this paper's specific sin since those models are small and this used only 1 A100 GPU.)

The most interesting part is actually buried in Section 5.1, where they show that using Mistral-7B directly for stance detection fails miserably. This suggests something important about synthetic data that the authors don't fully explore: it's better at generating plausible variations than at making decisive judgments. This deserved more analysis.

What's missing here is any real investigation into what makes synthetic data actually useful. Are we just doing expensive interpolation between existing data points? Where's the analysis of entropy and diversity in the generated samples? The visualizations in Figure 3 are pretty, but they also show that the synthetic data mostly just fills in obvious gaps rather than introducing genuinely novel perspectives.

The active learning component feels tacked on, though I'll admit the SQBC approach is clever. Using synthetic data as a reference distribution for selecting informative samples is neat, but again - why does this work? The paper handwaves at "ambiguous samples" without diving deeper into the theoretical foundations.

One thing I'll give the authors credit for: they did their homework on the translation pipeline. Using NLLB-330M and actually caring about the quality of the German-English-German round trip is more than many papers bother with. The samples in Table 8 show reasonable quality political discourse generation.

SUGGESTIONS FOR IMPROVEMENT:

- Add analysis of entropy/diversity metrics for synthetic data
- Provide theoretical justification for why synthetic data helps beyond just "more data"
- Compare computational costs vs. benefits more explicitly
- Explore what makes certain synthetic samples more useful than others
- Consider alternative methods for introducing genuine novelty into synthetic data

NITPICKS:

- The abbreviation "SQBC" is used before it's properly defined
- Figure 4 is information-dense to the point of being hard to parse
- Some ablation studies feel perfunctory rather than insightful

CONCLUSION:

This paper is fine. It's not going to revolutionize either synthetic data generation or stance detection, but it makes a modest contribution to both. The experimental work is solid if unexciting, and the results are positive if not earth-shattering. The biggest missed opportunity is not diving deeper into what makes synthetic data actually useful beyond simple interpolation.

The paper should be marginally accepted because it advances the field incrementally and might give others ideas for more innovative approaches. But let's not pretend this is more than a small step forward in a very crowded research space. I don't like these kind of papers

I'd love to see a follow-up that really digs into the entropy question and provides proper theoretical foundations for synthetic data generation in political stance detection. Until then, this feels like another "it works (a bit) but we're not quite sure why" paper.
weaknesses: Limited theoretical justification for why synthetic data helps beyond "moar data good"
Doesn't address the entropy/diversity problem in synthetic data generation
Results are modest (~2-3% improvements) for considerable computational overhead
Heavy reliance on a specific dataset (X-Stance) limits generalizability claims
The "active learning with synthetic data" angle feels like two papers duct-taped together

Meta-review: **Summary:**

The authors explore the efficacy of using LLM-generated synthetic training data for automated stance detection. They primarily test against a Bert baseline where individual models are trained for different stance detection question types. Varying levels of synthetic data are used to finetune this model, as well as previously unlabeled human data collected using active learning that makes use of nearest neighbor synthetic examples. Results indicate performance gains from synthetic data (72.3 vs 69.3 F1) and synthetic data mixed with active learning labeled data (75.4 F1). 

**Strengths:**

- The authors' research questions are well-defined and the results seem to support use of synthetic data for stance detection 

- While the findings are unsurprising given recent literature around effectiveness of synthetic data on other tasks like content moderation, the experimentation is very solid and there are unique aspects to their approach such as using synthetic nearest-neighbors to facilitate active learning on human-derived data. 

**Weaknesses:**

- The experimental results are not very comprehensive and only consider the effect on a Bert baseline. The authors also test against Mistral, but only in a zero-shot vs fine-tuned setting. Given the sensitivity of LLMs to prompt phrasing, much more could be explored here including few-shot demonstrations in the prompt or specifying the desired output format. 

- The authors don't seem to conduct any analysis to ensure the synthetic comments are consistent with the data labels used in prompting. 

- The paper would benefit from statistically significance testing, in particular when comparing baseline vs. synthetic only results.

I think the paper is technically sound enough to be accepted and commend the authors on their efforts, however I question the impact the paper's appearance in conference proceedings will make on the broader research community.

Decision: Accept (Spotlight)