Venue: NAACL 2025

Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration

Ang Li\({}^{1,3*}\), Jingqian Zhao\({}^{1**}\), Bin Liang\({}^{1,3,4}\), Lin Gui\({}^{5}\), Hui Wang\({}^{2}\),

Xi Zeng\({}^{6}\), Xingwei Liang\({}^{1}\), Kam-Fai Wong\({}^{3,4}\), and Ruifeng Xu\({}^{1,2}\)\({}^{\dagger}\)

###### Abstract

Stance detection is critical for understanding the underlying position or attitude expressed toward a topic. Large language models (LLMs) have demonstrated significant advancements across various natural language processing tasks including stance detection, however, their performance in stance detection is limited by biases and spurious correlations inherent due to their data-driven nature. Our statistical experiment reveals that LLMs are prone to generate biased stances due to sentiment-stance spurious correlations and preference towards certain individuals and topics. Furthermore, the results demonstrate a strong negative correlation between stance bias and stance detection performance, underscoring the importance of mitigating bias to enhance the utility of LLMs in stance detection. Therefore, in this paper, we propose a Counterfactual Augmented Calibration Network (FACTUAL), which a novel calibration network is devised to calibrate potential bias in the stance prediction of LLMs. Further, to address the challenge of effectively learning bias representations and the difficulty in the generalizability of debiasing, we construct counterfactual augmented data. This approach enhances the calibration network, facilitating the debiasing and out-of-domain generalization. Experimental results on in-target and zero-shot stance detection tasks show that the proposed FACTUAL can effectively mitigate biases of LLMs, achieving state-of-the-art results.

+
Footnote †: Corresponding Author

+
Footnote †: Corresponding Author

## 1 Introduction

Stance detection aims at automatically identifying the author's opinionated standpoint or attitude (e.g., _Favor_, _Against_, or _Neutral_) expressed in the content towards a specific target, topic, or proposition [10, 8]. With the development of social media platforms, stance detection plays a pivotal role in analyzing public opinion on social media topics [9, 7, 1, 13].

Large Language Models (LLMs), such as Chat-GPT1, Bard2, and LLaMA [11], have demonstrated impressive language comprehension and task-handling capabilities by leveraging extensive corpus and knowledge. However, their data-driven nature makes them susceptible to biases and spurious correlations embedded in pretraining data. In stance detection, which requires interpreting the relationship between a sentence and a specific topic, clues from any isolated aspects could become spurious and lead to biased stances.

Footnote 1: [https://openai.com/blog/chatgtt/](https://openai.com/blog/chatgtt/)

Footnote 2: [https://bard.google.com/](https://bard.google.com/)

Our experiment identifies two primary biases in LLMs for stance detection: (1) sentiment-stance spurious correlations, where sentiment misleads stance judgment, and (2) target preference bias, where LLMs favor certain individuals or topics.

Figure 1: An example demonstrates two types of biases encountered by large language models in stance detection tasks (shown at the top and bottom) as well as unbiased stance rationale (shown in the middle).

 

Figure 1 illustrates examples of the two types of biases, as well as unbiased stance rationale. Furthermore, our results reveal a significant negative correlation between stance bias and stance detection performance, emphasizing the necessity of alleviating bias to improve the effectiveness of LLMs in stance detection.

Existing research of debiasing in stance detection largely centered on the creation of unbiased training samples and the retraining of stance detection models Kaushal et al. (2021); Yuan et al. (2022b). However, there are two core limitations to the application of these debiasing methods in LLMs. **Limitation#1**, research Luo et al. (2023) has shown that such retraining processes will undermine the generality of LLMs, potentially leading to catastrophic forgetting; not to mention that there are restrictions with certain closed-source LLMs like GPT-3.5-turbo, which can only be accessed with a restricted inference API, preventing access to internal model parameters. **Limitation#2**, existing approaches to constructing unbiased training samples typically entail the analysis of prevalent bias patterns, subsequently automating their construction based on these identified patterns, exemplified by substituting 'Men' with 'Women'. However, when dealing with stance detection tasks, our forthcoming analysis illuminates that these samples display varying bias propensities, attributable to divergences in sentiments and stance objectives. Consequently, utilizing conventional methods to create unbiased samples poses a significant challenge.

Therefore, to address the above two limitations, we propose to mitigate biases of LLMs in stance detection with a Counterfactual Augmented Calibration Network, coined as FACTUAL. We establish a trainable calibration network to approximate the inverse projection function of the bias label distribution within LLMs. This calibration network takes samples as input, including stance judgments and rationales from LLMs, and generates calibrated stance judgments. We construct counterfactual augmented data against the training data to rectify stance biases. The counterfactual samples are constructed from both causal and non-causal features, which can enhance the calibration network to yield unbiased stances and accomplish out-of-domain generalization. Through counterfactual augmented supervised training, the calibration network can capture biases present in specific samples, thereby performing debiasing. The main contributions of our work are summarized as follows:

1) We are the first to investigate the biases of LLMs on stance detection, categorizing the biases into two main types from the perspective of causality and proposing metrics to quantify these two types of biases.

2) We propose FACTUAL, a novel framework called the Counterfactual Augmented Calibration Network to mitigate biases of LLMs on stance detection.

3) A series of experiments demonstrate that our FACTUAL can effectively reduce the bias of LLMs in stance detection, improving the performance in both in-target and zero-shot stance detection tasks3.

Footnote 3: The code is available at [https://github.com/Leon-Francis/FACTUAL](https://github.com/Leon-Francis/FACTUAL).

## 2 Related Work

Biases in Large Language ModelsSome studies Goncalves and Strubell (2023) have examined the biases existing in Large Language Models (LLMs), these biases mainly include gender and religion Salinas et al. (2023), politics Jenny et al. (2023); He et al. (2023), and spurious correlations Zhou et al. (2023). The associated debiasing efforts are centered around retraining the language model with debiased samples Dong et al. (2023); Limisiewicz et al. (2023). Zheng et al. (2023) found that LLMs are vulnerable to option position changes in MCQs due to their inherent 'selection bias'. They perform debiasing by approximating the overall bias distribution. While based on our analysis in Section 3, the bias distribution varies significantly across different stance detection samples, so this method is not applicable.

Mitigating Biases in Stance DetectionCurrently, studies developed for mitigating biases in stance detection are oriented toward fine-tuned models. Kaushal et al. (2021) analyzed two biases existing in the current datasets: target-independent lexical choices and target-independent sentiment-stance correlations, and built an unbiased dataset. Yuan et al. (2022a) incorporated the stance reasoning process as task knowledge to retrain the model to reduce bias. Yuan et al. (2022b) constructed unbiased samples through counterfactual reasoning and performed adversarial bias learning. These methods involve retraining models and constructing unbiased training samples through special marks, which cannot be directly applied to LLMs.

 



## 3 Biases of LLMs in Stance Detection

### Bias Measurement

Stance bias refers to the systematic errors where models tend to choose certain stances due to the influence of specific biases and stereotypes. Inspired by Zheng et al. (2023), the standard deviation of recalls (_Rstd_) on stance labels is an excellent metric for quantitatively measuring systematic errors. The formula is as follows:

\[RStd=\sqrt{\frac{1}{K}\sum_{i=1}^{K}\left(\frac{TP_{i}}{P_{i}}-\frac{1}{K}\sum _{j=1}^{K}\frac{TP_{j}}{P_{j}}\right)^{2}} \tag{1}\]

Where \(K\) is the number of stance labels, \(TP_{i}\) is the number of true positive instances for stance label \(i\), and \(P_{i}\) is the number of instances of stance label \(i\). This measurement resists label imbalance and effectively reflects the model's bias tendency on samples (Refer to Appendix A for the validation).

### Experimental Result

Through statistical analysis of the results from LLMs, we identified two significant types of bias: **Sentiment-stance Spurious Correlations** and **Target Preference Bias**.

#### 3.2.1 Sentiment-Stance Spurious Correlations

Sentiment can influence stance judgment but is not the primary determinant. Overreliance on sentiment by the model suggests susceptibility to sentiment-stance spurious correlations, leading to biased stance assessments. To investigate stance bias across different sentiments, we first ascertain the sentiment label for each sample. In the Sem16 dataset, each sample has annotated sentiment labels, categorized as positive, neutral, or negative. For the P-Stance and VAST datasets, we utilize GPT-4 to annotate the sentiment labels. To gain a preliminary understanding of sentiment-stance spurious correlations, we first divide the dataset into three subsets based on sentiment categories. For each subset, we calculate the recall score for each stance label and normalize it by subtracting the overall recall score of the corresponding stance labels across the dataset, as shown in Figure 2. Observations indicate that LLMs tend to erroneously predict 'support' for positively-sentiment samples and 'against' for negatively-sentiment ones, indicating a deviation from expected patterns and highlighting an inherent stance bias. Hence, we identify the **S**entiment-stance **S**purious **C**orrelations (SSC) as a type of bias in LLMs on stance detection.

We calculate the average of the RStd across all sentiments as our quantification for sentiment-stance spurious correlations:

\[\textit{Bias-SSC}=\frac{1}{|S|}\sum_{s\in S}RStd(X_{s}) \tag{2}\]

where \(X_{s}\) represents instances with sentiment label \(s\), \(|S|\) denotes the number of sentiment labels, which in our experiment, is 3.

We conducted experiments in various settings: _Task-Des_ used task-related descriptions for stance judgment4, _CoT-Demo_ used the task description with 4-shot chain-of-thought demonstration, and _Debias-Instruct_ used the task description indicating that sentiment was spurious cues for stance judgment. Refer to Appendix C for the detailed prompts. The results are shown in Table 1. We can observe that in most cases, there is a negative correlation between bias-SSC and stance detection performance. See further analysis in Appendix B. Moreover, prompt engineering methods proved infectual in mitigating this inherent bias.

Footnote 4: The results presented in Figures 2 and 3 are obtained based on _Task-Des_.

#### 3.2.2 Target Preference Bias

LLMs exhibit bias towards certain individuals or topics. This bias can interfere with their ability to judge stances based on the text, leading to biased stance judgments. We refer to this bias as target preference bias. To preliminarily observe the target preference bias of LLMs, we randomly sampled some targets from different datasets, calculated the recall scores for each stance label on each target subset, and normalized it by subtracting the overall recall score of the corresponding

Figure 2: The recall score of each stance label on three sentiment subsets, normalizing by subtracting the overall recall scores of the corresponding stance labels across overall dataset, on Sem16, P-Stance, and VAST. POS for positive, NEU for neutral, NEG for negative.



stance labels across all targets, as shown in Figure 3. We observed that, on different targets, LLMs displayed markedly different tendencies in stance selection, which ultimately affected the correctness of stance judgment. Therefore, we identify the **T**arget **P**reference **B**ias (TPB) as a type of bias in LLMs on stance detection.

We calculate the average of the RStd of all targets as our quantification for target preference bias:

\[\textit{ Bias-TPB}=\frac{1}{|T|}\sum_{t\in T}RStd(X_{t}) \tag{3}\]

where \(X_{t}\) represents instances with stance target \(t\), \(|T|\) denotes the number of targets.

We conduct experiments based on _Task-Des_, _CoT-Demo_, and _Debias-Instruct_ which emphasize the need to judge the stance based on the text and not to include the inherent attitude towards the target. Refer to Appendix C for the detailed prompts. The results are shown in Table 2. We can observe that in most cases, there is a negative correlation between bias-TPB and stance detection performance. See further analysis in Appendix B, and prompt engineering fails to effectively mitigate bias-TPB.

## 4 Mitigating Bias with Calibration

Given \(\{x_{n},t_{n}\}^{N}_{n=1}\) as the labeled dataset, where \(x\) denotes the input text and \(t\) denotes the corresponding target, LLMs obtain the stance predictions \(y\) through the task instructions \(\mathcal{I}\) for stance detection: \(P_{obs}(y_{i}|\mathcal{I};x_{i},t_{i})\). Inspired by Zheng et al. (2023), we believe that it can be deconstructed into the unbiased distribution \(P_{unbiased}\) of the LLMs performing the stance detection task, and the bias distribution \(P_{bias}\) formed by confounding \(C_{i}\):

\[P_{obs}=P_{unbiased}(y_{i}|x_{i},t_{i})P_{bias}(y_{i}|C_{i}) \tag{4}\]

The confounding factor \(C_{i}\) arises from the two types of biases analyzed earlier. It is important to note that it may be unaffected, influenced by a single bias, or impacted by both types of biases. Specifically, \(P_{obs}\) denotes the stance judgment text, which can be regarded as the outcome of word probability distribution following the argmax operation, derived from LLMs. We aim to estimate the unbiased stance distribution \(P_{unbiased}\).

### Calibration Network

By estimating the bias distribution based on the overall distribution of known samples (from the training set), we can obtain unbiased outputs by multiplying the observed distribution of LLMs by the inverse of the approximated bias distribution:

\[P_{unbiased}=P_{obs}(y^{\prime}_{i}|\mathcal{I};x_{i},t_{i})\widetilde{P}_{ bias}(y^{\prime\prime}_{i}|C_{i})^{-1} \tag{5}\]

where \(y^{\prime}_{i}\) and \(y^{\prime\prime}_{i}\) represent the label distribution output by LLMs and the label distribution affected by bias, \(\widetilde{P}_{bias}\) represents the estimate of bias.

However, based on the bias analysis in Section 3, we found that for stance detection, the samples with

Figure 3: The recall score of each stance label on several target subsets, normalizing by subtracting the overall recall score of the corresponding stance labels across all targets, on Sem16, P-Stance, and VAST dataset. HC for Hillary Clinton, LA for Legalization of Abortion, AT for Atheism, JB for Joe Biden, BS for Bernie Sanders, DT for Donald Trump, CH for Christian, CL for Election, HP for Humanity Program.

 

different sentiments and stance targets have completely different stance bias distributions. Therefore, we propose employing a network to capture the bias distribution specific to each sample, called the _calibration network_\(f_{Cal}\). We use the network \(f_{Cal}\) to approximate the inverse projection function of the bias distribution:

\[f_{Cal}=P_{bias}(y^{\prime\prime}_{i}|C_{i})^{-1} \tag{6}\]

By inputting the predicted stance distribution \(P_{obs}\) from LLMs, an approximating unbiased label can be obtained:

\[P_{unbiased}(\hat{y}_{i})=f_{Cal}(P_{obs}(y^{\prime}_{i}|\mathcal{I};x_{i},t_{ i})) \tag{7}\]

Specifically, as illustrated in Figure 4, we first use the _CoT-Demo_ instruction (refer to Appendix C for the detail) to obtain the stance judgment and rationale from the LLMs (which correspond to the result of an argmax operation over the \(P_{obs}\)). Then, we input the sample, along with this stance judgment and rationale, into our _calibration network_ (using RoBERTa-base in our setup) to obtain the debiased stance output. We train the _calibration network_ using the cross-entropy loss function with ground truth label:

\[\mathcal{L}_{CE}=-\sum_{i=1}^{N}y_{i}\log(f_{Cal}(P_{obs}(y^{\prime}_{i}| \mathcal{I};x_{i},t_{i})) \tag{8}\]

### Counterfactual Data Augmentation

One challenge in supervised training is the limited representation of bias within the overall training set, and the learned bias features are difficult to generalize. To fully leverage our analysis of the existing stance biases in LLMs (Section 3) and facilitate the _calibration network_ in learning diverse bias patterns, we generate non-causal and causal **C**ounferfactual **A**ugmented **D**ata (CAD) based on the training data.

The objective of non-causal counterfactual data augmentation is to explicitly perturb bias-inducing features in the data, enabling the model to identify and mitigate biases. As illustrated in Figure 4 (a), we construct counterfactual samples by modifying non-causal features while preserving stance labels. Specifically, we construct an instruction that allows the LLMs to perturb the sentence and target. To address **sentiment-stance spurious correlations**, we perturb the text \(x_{i}\) by altering sentiment-related expressions while maintaining the stance. Similarly, to counteract **target preference bias**, we modify the target \(t_{i}\) while ensuring the stance remains unchanged. Refer to Figure 10 for the detailed prompts. This obtains the perturbed text \(x^{*}_{i}\) and perturbed target \(t^{*}_{i}\). Since we only disturbed confounding, the stance label remains unaffected. We construct cross-entropy loss on non-causal counterfactual augmented data as follows:

\[\mathcal{L}^{\text{no-can}}_{CAD}=-\sum_{i=1}^{N}y_{i}\log(f_{Cal}(P_{obs}( y^{\prime}_{i}|\mathcal{I};x^{*}_{i},t^{*}_{i})) \tag{9}\]

In contrast, causal counterfactual data augmentation directly manipulates causal features by re

Figure 4: The overall architecture of our proposed FACTUAL. (a) and (b) in the counterfactual data generation represent two ways to generate counterfactual augmentation. \(X\) donates the text, \(T\) donates the target, \(H\) donates the features of the interaction of text and target, and \(Y\) donates the stance label. \(C\) represents confounding factors, which arise from the two types of biases previously analyzed and may distort the stance prediction. \(*\) denotes the perturbation of non-causal features, and \(\sim\) denotes the perturbation of causal features.

 

versing the stance. This augmentation improves the model's capacity to focus on causal features, thereby reducing its reliance on spurious correlations that may introduce bias. As illustrated in Figure 4 (b), we make necessary alterations to text \(x_{i}\) to reverse the stance to target \(t_{i}\), thereby only perturbing the causal features. Refer to Figure 11 for the detailed prompts. This obtains the perturbed text \(\widetilde{x_{i}}\) expressing a reversed stance to target \(t_{i}\). We construct cross-entropy loss on causal counterfactual augmented data as follows:

\[\mathcal{L}^{\textit{cau}}_{CAD}=\sum_{i=1}^{N}y_{i}\log(f_{Cal}(P_{obs}(y^{ \prime}_{i}|\mathcal{I};\widetilde{x_{i}},t_{i}))) \tag{10}\]

### Training Objective

The final training objective incorporates counterfactual augmented data and performs joint training:

\[\mathcal{L}=\mathcal{L}_{CE}+\mathcal{L}^{\textit{n-cau}}_{CAD}+\mathcal{L}^{ \textit{caut}}_{CAD} \tag{11}\]

## 5 Experimental Setup

### Datasets

We conduct experiments of in-target and zero-shot stance detection on three benchmark datasets: SemEval-2016 Task6 (Sem16) Mohammad et al. (2016), P-Stance Li et al. (2021) and Varied Stance Topics (VAST) Allaway and McKeown (2020). The statistic of datasets is shown in Table 3.

### Implementation Details

For GPT-3.5-turbo, we utilize GPT-3.5-turbo-0125. For LLaMA2-70b, we utilize LLaMA2-70b-chat. We set the temperature to 1.0, top p to 1.0, max tokens to 1024, and fixed the decoding seed to ensure the reproducibility of our experiments. For our calibration network, we employ the RoBERTbase model Liu et al. (2019). For our counterfactual data augmentation, we employ GPT-3.5-turbo-0301 to generate counterfactual samples, guided by the instructions detailed in Appendix C. We use AdamW as an optimizer with a batch size of 32. Learning rate is set to 1e-5 and weight decay is set to 1e-3. All training was conducted on NVIDIA A100 4QG GPUs. We report averaged scores of 5 runs to obtain statistically stable results.

### Evaluation Metric

Across three datasets, we used the same evaluation metric established by their proposers, which was also adopted by most of the subsequent baselines. We adopt the macro-average of the F1-score as the evaluation metric. For Sem16 and P-Stance, we report \(F1=(F_{favor}+F_{against})/2\). For VAST, we report \(F1=(F_{favor}+F_{against}+F_{none})/3\). In the in-target stance detection setting, the model is trained and tested on the same set of targets. We follow the dataset splits provided by the original dataset publisher to ensure a fair comparison with related baselines. The zero-shot stance detection setting presents the model with unseen targets during testing, requiring it to generalize from known targets to infer stances toward new ones. For Sem16 and P-Stance, we take one target as the test set while splitting the remaining targets into training and validation sets in a 7:1 ratio. For VAST, which is inherently a zero-shot stance detection dataset, we use the original splits provided by the dataset publisher. These settings are consistent with the baselines to ensure fairness in comparison.

### Comparison Models

The fine-tuned model baselines include vanilla RoBERta Liu et al. (2019), domain pre-trained model: BERTweet Nguyen et al. (2020), prompt tuning method KPT Shin et al. (2020), joint contrastive learning framework: JointCL Liang et al. (2022), incorporating ConceptGraph knowledge model: KEprompt Huang et al. (2023), incorporating Wikipedia knowledge model: TarBKBERT Zhu et al. (2022) and WS-BERT He et al. (2022), incorporating knowledge from LLMs: KASD-BERT Li et al. (2023). For large language models, we compare baselines include TaskDes Zhang et al. (2022), CoT-Demo Zhang et al. (2023), the self-consistent chain-of-thought: CoT-SC Wang et al. (2023), incorporating Wikipedia knowledge for retrieval-augmented generation: KASD-ChatGPT and KASD-LLaMA-2 Li et al. (2023), fine tuning LLaMA-2-7b using LoRA with training set: LLaMA-2-7b-FT, utilizing collaborative role-infused LLM-based agents: COLA Lan

\begin{table}
\begin{tabular}{l c c c c} \hline Dataset & Target & Favor & Against & Neutral \\ \hline \multirow{8}{*}{Sem16} & HC & 163 & 565 & 256 \\  & FM & 268 & 511 & 170 \\  & LA & 167 & 544 & 222 \\  & A & 124 & 464 & 145 \\  & CC & 335 & 26 & 203 \\  & DT & 148 & 299 & 260 \\ \hline \multirow{3}{*}{P-Stance} & Biden & 3217 & 4079 & - \\  & Sanders & 3551 & 2774 & - \\ \cline{1-1}  & Trump & 3663 & 4290 & - \\ \hline VAST & - & 6952 & 7297 & 4296 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of SemEval-2016 Task6, P-Stance and VAST datasets.

 

et al., 2013) and utilizing logically consistent chain-of-thought: LC-CoT Zhang et al. (2023a)5.

Footnote 5: Since JointCL, TarBK-BERT, COLA, and LC-CoT were only proposed in zero-shot stance detection scenarios, our comparisons with them are conducted solely within the corresponding experimental settings.

## 6 Experimental Results

### In-Target Stance Detection

We perform experiments on Sem16 and P-Stance for in-target stance detection. The results are presented in Table 4. It shows that our FACTUAL outperforms all baselines based on different large language models. We can observe that _FACTUAL - w/o CAD_, which is without the counterfactual data enhancement, the calibration network trained exclusively on the training set data can still improve stance detection performance. Moreover, the application of counterfactual data enhancement furthers the model's performance. When compared to the _LLaMA-2-7b-FT_ method, which fine-tunes LLaMA-2-7b, our method attains superior accuracy in stance detection with significantly reduced computation resources.

### Zero-Shot Stance Detection

We conduct experiments on Sem16, P-Stance, and VAST for zero-shot stance detection. The results are shown in Table 5. It shows that our FACTUAL outperforms all baselines including both fine-tuned models and large language models. This indicates that our FACTUAL has strong generalization capabilities and can perform well on unseen targets. We can observe that _FACTUAL - w/o CAD_ exhibits subpar performance. This can be attributed to the constraints posed by the exclusive reliance on fine-tuning within the limited training dataset, making it an uphill task to generalize the model's debiasing capability. Conversely, employing our counterfactual data enhancement bolsters the out-of-domain generalization prowess of the model considerably, yielding impressive results in zero-shot performance. Compared to the _LLaMA-2-7b-FT_, which performs poorly on zero-shot tasks, FACTUAL demonstrates strong generalization capabilities.

### Mitigating Biases Effect Analysis

We conduct experiments to evaluate the Bias-SSC and Bias-TPB of LLMs and further assess the impact of our bias mitigation efforts. The results are shown in Table 6, which indicate that our FACTUAL can effectively alleviate Bias-SSC and Bias-TPB for both GPT-3.5-turbo and LLaMA2-70b, thus validating its effectiveness in mitigating biases. The inclusion of counterfactual data augmentation can effectively improve its debiasing ability, indicating the importance of our counterfactual data augmentation. Our findings also highlight that the integration of counterfactual data augmentation enhances the debiasing capacity of the model, thereby em

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline  & \multicolumn{8}{c}{Sem16(\%)} & \multicolumn{8}{c}{P-Stance(\%)} \\ \cline{2-11}  & HC & FM & LA & A & CC & Avg & Biden & Sanders & Trump & Avg \\ \hline

- & 80.40 & 77.10 & 80.20 & 79.23 \\ KEprompt & 77.10\({}^{\dagger}\) & 68.30\({}^{\dagger}\) & 70.30\({}^{\ddagger}\) &
- & 84.40\({}^{\ddagger}\) &
- & 83.20\({}^{\ddagger}\) &
- \\ WS-BERT-Dual & 75.26\({}^{\dagger}\) & 66.02\({}^{\dagger}\) & 70.42\({}^{\dagger}\) & 71.57\({}^{\dagger}\) & 57.31\({}^{\dagger}\) & 68.12\({}^{\ddagger}\) & 83.50\({}^{\dagger}\) & 79.00\({}^{\lx@sectionsign}\) & 85.80\({}^{\lx@sectionsign}\) & 82.77\({}^{\flat}\) \\ KASD-BERT & 77.60\({}^{\dagger}\) & 70.38\({}^{\dagger}\) & 72.29\({}^{\dagger}\) & 72.32\({}^{\ddagger}\) & 61.47\({}^{\dagger}\) & 70.81\({}^{\dagger}\) & 85.66\({}^{\dagger}\) & 80.39\({}^{\ddagger}\) & 85.35\({}^{\ddagger}\) & 83.80\({}^{\dagger}\) \\ \hline
* w/o CAD & 78.00 & 70.82 & 65.57 & 71.40 & 72.24 & 71.61 & 85.35 & 82.00 & 85.51 & 84.29 \\ \hline
* w/o CAD & 82.38 & 73.80 & 63.65 & 69.21 & 62.93 & 70.39 & 85.40 & 81.36 & 85.00 & 83.92 \\ \hline \end{tabular}
\end{table}
Table 4: In-target stance detection experiment results on Sem16 and P-Stance datasets. The results with ‡ are retrieved from Huang et al. (2023), † from He et al. (2022), † from Li et al. (2023). The best scores over the same type are in bold. Results with ✗ indicate significance of our FACTUAL over the same type baseline models at p ¡ 0.05.
 

phasizing the significance of this augmentation in our methodology.

### Ablation Study

We conduct ablation studies to examine the impact of different components in our FACTUAL: (1) "**w/o** Calibration" denotes without the calibration network, letting the LLMs directly output stance labels. (2) "**w/o** CAD" denotes without the counterfactual augmented data when training the calibration network. (3) "**w/o** non-causal" denotes without the non-causal counterfactual augmented data when training the calibration network. (4) "**w/o** causal" denotes without the causal counterfactual augmented data when training the calibration network.

The results are presented in Table 7. Note that despite utilizing the same stance reasoning, a lack of calibration can result in sub-optimal results and

\begin{table}
\begin{tabular}{l 

notable biases. Thus validating the effectiveness of our gate calibration network. Analysis in Appendix E demonstrate that our method exhibits robustness across different prompt templates.

Additionally, removing non-causal counterfactual data significantly increases bias, highlighting its crucial role in bias mitigation. Conversely, eliminating causal counterfactual data markedly reduces performance, underscoring its substantial impact on the accuracy and generalizability of the calibration network.

## 7 Conclusion

In this paper, we categorize the biases of LLMs in stance detection into two types from the perspective of causality and propose metrics to quantify these biases. Then, we propose a Counterfactual A Augmented Calibration Network, coined as FACTUAL. In which, a trainable calibration network and counterfactual data augmentation are explored to mitigate the biases of LLMs in stance detection. Experimental results on intraget and zero-shot stance detection show that our FACTUAL can effectively reduce the bias of LLMs in stance detection and contribute to improved performance.

## Acknowledgements

This work was partially supported by the National Natural Science Foundation of China 62176076, Natural Science Foundation of Guang Dong 2023A1515012922, the Shenzhen Foundational Research Funding JCYJ20220818102415032, the Major Key Project of PCL2021A06, CIPSC-SMPZHIPU Large Model Cross-Disciplinary Fund ZPCG20241119405 and Hong Kong RGC GRF No. 14206324.

## Limitations

Our framework involves using GPT-3.5 to generate counterfactual augmented data. As we discussed in Appendix F, these samples may contain errors, but overall are beneficial to the training of our calibration network. The methods of constructing counterfactual augmented data using manual annotation or other methods remain to be explored. While our study primarily focuses on stance biases--particularly sentiment-stance spurious correlations and target preference bias--our approach can be extended to mitigate other forms of bias by designing appropriate counterfactual samples.

## Ethics Statement

The datasets used in this paper are sourced from open-access datasets. The VAST dataset provides complete text data in open access. In compliance with the privacy agreement of Twitter for academic usage, the Sem16 and P-Stance were accessed using the official Twitter API6 through the Tweet IDs to fetch complete text data. We removed the information on user privacy from the data. In these datasets, we analyze the biases and stereotypes in stance detection for some sensitive targets (e.g., belief, politics, etc.). We DO NOT critique any biases and stereotypes. We focus on analyzing their impacts on stance detection and mitigating these impacts. We used the counterfactual augmented data obtained from the GPT-3.5-Turbo API service from OpenAI. We followed their term and policies. Some examples in our paper may include a stance or tendency. It should be clarified that they are randomly sampled from the dataset for better studying the dataset and task, and do not represent any personal viewpoints.

Footnote 6: [https://developer.twitter.com/en/docs/twitter-api](https://developer.twitter.com/en/docs/twitter-api)

## References

* Allaway and McKeown [2020] Emily Allaway and Kathleen R. McKeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 8913-8931. Association for Computational Linguistics.
* ICANN 2021
- 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part III_, volume 12893 of _Lecture Notes in Computer Science_, pages 269-279. Springer.
* Dong et al. [2023] Xiangjue Dong, Ziwei Zhu, Zhuoer Wang, Maria Teleki, and James Caverlee. 2023. Cos^2Spt: Mitigating bias in pre-trained language models through counterfactual contrastive prompt tuning. _CoRR_, abs/2310.12490.
* 10th International Conference of the CLEF Association, CLEF 2019,
 

Lugano, Switzerland, September 9-12, 2019, Proceedings_, volume 11696 of _Lecture Notes in Computer Science_, pages 75-87. Springer.
* Goncalves and Strubell (2023) Gustavo Goncalves and Emma Strubell. 2023. Understanding the effect of model compression on social bias in large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 2663-2675. Association for Computational Linguistics.
* He et al. (2023) Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. 2023. Inducing political bias allows language models anticipate partisan reactions to controversies. _CoRR_, abs/2311.09687.
* He et al. (2022) Zihao He, Negar Mokhberian, and Kristina Lerman. 2022. Infusing knowledge from wikipedia to enhance stance detection. In _Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, WASSA@ACL 2022, Dublin, Ireland, May 26, 2022_, pages 71-77. Association for Computational Linguistics.
* Huang et al. (2023) Hu Huang, Bowen Zhang, Yangyang Li, Baoquan Zhang, Yuxi Sun, Chuyao Luo, and Cheng Peng. 2023. Knowledge-enhanced prompt-tuning for stance detection. _ACM Transactions on Asian and Low-Resource Language Information Processing_.
* Jang and Allan (2018) Myungha Jang and James Allan. 2018. Explaining controversy on social media via stance summarization. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 1221-1224. ACM.
* Jenny et al. (2023) David F. Jenny, Yann Billeter, Mrinmaya Sachan, Bernhard Scholkopf, and Zhijing Jin. 2023. Navigating the ocean of biases: Political bias attribution in language models via causal structures. _CoRR_, abs/2311.08605.
* Kaushal et al. (2021) Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. twt-wt: A dataset to assert the role of target entities for detecting stance of tweets. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 3879-3889. Association for Computational Linguistics.
* Lan et al. (2023) Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. 2023. Stance detection with collaborative roles infused llm-based agents. _CoRR_, abs/2310.10467.
* Li et al. (2023) Ang Li, Bin Liang, Jingqian Zhao, Bowen Zhang, Min Yang, and Ruifeng Xu. 2023. Stance detection on social media with background knowledge. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 15703-15717. Association for Computational Linguistics.
* Li et al. (2021) Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayataman Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-Stance: A large dataset for stance detection in political domain. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_, volume ACL/IJCNLP 2021 of _Findings of ACL_, pages 2355-2365. Association for Computational Linguistics.
* Liang et al. (2022) Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yuan He, and Ruifeng Xu. 2022. Jointcl: A joint contrastive learning framework for zero-shot stance detection. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 81-91. Association for Computational Linguistics.
* Limisiewicz et al. (2023) Tomasz Limisiewicz, David Marecek, and Tomas Musil. 2023. Debiasing algorithm through model adaptation. _CoRR_, abs/2310.18913.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. _CoRR_, abs/2308.08747.
* Mohammad et al. (2016) Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In _Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_, pages 31-41. The Association for Computer Linguistics.
* Demos, Online, November 16-20, 2020_, pages 9-14. Association for Computational Linguistics.
* Salinas et al. (2023) Abel Salinas, Louis Penafiel, Robert McCormack, and Fred Morstatter. 2023. "im not racist but..." Discovering bias in the internal knowledge of large language models. _CoRR_, abs/2310.08780.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 4222-4235. Association for Computational Linguistics.
* Somasundaran and Wiebe (2010) Swapna Somasundaran and Jansvee Wiebe. 2010. Recognizing stances in ideological on-line debates. In _Proceedings of the NAACL HLT 2010 Workshop on 

Computational Approaches to Analysis and Generation of Emotion in Text_, pages 116-124, Los Angeles, CA. Association for Computational Linguistics.
* Stefanov et al. (2020) Peter Stefanov, Kareem Darwish, Atanas Atanosov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 527-537. Association for Computational Linguistics.
* Sun et al. (2018) Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierarchical attention network. In _Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018_, pages 2399-2409. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Yuan et al. (2022b) Jianhua Yuan, Yanyan Zhao, Yanyue Lu, and Bing Qin. 2022b. SSR: utilizing simplified stance reasoning process for robust stance detection. In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeonju, Republic of Korea, October 12-17, 2022_, pages 6846-6858. International Committee on Computational Linguistics.
* Yuan et al. (2022c) Jianhua Yuan, Yanyan Zhao, and Bing Qin. 2022c. Debiasing stance detection models with counterfactual reasoning and adversarial bias learning. _CoRR_, abs/2212.10392.
* Zhang et al. (2022) Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgtp? _CoRR_, abs/2212.14548.
* Zhang et al. (2023a) Bowen Zhang, Daijun Ding, Liwen Jing, and Hu Huang. 2023a. A logically consistent chain-of-thought approach for stance detection. _CoRR_, abs/2312.16054.
* Zhang et al. (2023b) Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023b. Investigating chain-of-thought with chatgpt for stance detection on social media. _CoRR_, abs/2304.03087.
* Zheng et al. (2023) Chujie Zheng, Hao Zhou, Fanding Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. _CoRR_, abs/2309.03882.
* Zhou et al. (2023) Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. 2023. Explore spurious correlations at the concept level in language models for text classification. _CoRR_, abs/2311.08648.
* 15, 2022_, pages 2070-2075. ACM.

## Appendix A Bias Measurement Analysis

The standard deviation of recalls (RStd) can largely mitigate the impact of label imbalance in the dataset and effectively assess model bias. This metric is commonly used to evaluate model bias and, as highlighted in relevant research [Zheng et al., 2023], "This measurement is intuitive that greater recall imbalance indicates more pronounced selection bias and is not as susceptible to label imbalance as the counting-based measurement."

To illustrate the robustness of this metric against label imbalance, we provide an example from our experiments with the Sem16 dataset. The ground truth data distribution is shown in Table 8. We observed a certain degree of imbalance in label distribution. Based on this, we sampled 200 instances in which ground truth is favor, none, and against, respectively on each sentiment. The sampled instances exhibited no imbalance. We calculated Bias-SSC on these sampled instances, obtaining a result of **27.34**, compared to **27.13** on the original dataset. Thus, we believe that the actual distribution of sentiment, target, and stance labels, does not significantly affect our measurement of stance detection bias in LLMs.

## Appendix B Bias Influence

Bias represents a macro-level error pattern. In our experiments, the macroscopic results indicate that larger bias-SSC and bias-TPB tend to lead to poorer stance detection results. To prove this, we conducted a comprehensive analysis involving the calculation of Pearson correlation coefficients between bias-SSC and stance detection F1

\begin{table}
\begin{tabular}{l|c c c} \hline (\%) & Positive & Neutral & Negative \\ \hline Favor & 9.96 & 1.99 & 12.79 \\ None & 7.52 & 2.96 & 15.32 \\ Against & 13.82 & 1.46 & 34.19 \\ \hline \end{tabular}
\end{table}
Table 8: The ground truth data distribution of the Sem16 dataset.

 

scores, as well as bias-TPB and stance detection F1 scores, across 97 groups derived from Tables 1-2, 6, 7, 10, 11-15 (with bias-SSC and bias-TPB in Tables 6, 7, 10 and their corresponding F1 values in Tables 4, 5, and bias-SSC and F1 or bias-TPB and F1 values in Tables 11-15). The results are shown in Table 9, which indicate significant negative correlations for bias-SSC and stance detection F1, and for bias-TPB and stance detection F1.

## Appendix C Prompts Setting

We present the prompt templates used in Section 3.2 and Section 4.2.

Specifically, Figure 5 shows the prompt template we use with GPT-4 to obtain sentiment labels. Figures 6, 7, and 8 show the prompt templates corresponding to the Task-Des, CoT-Demo, and Debias-Instruct prompt settings in Section 3.2.1, respectively. Similarly, Figures 6, 7, and 9 display the prompt templates corresponding to the Task-Des, CoT-Demo, and Debias-Instruct prompt settings in Section 3.2.2, respectively.

Figures 10 and 11 illustrate the prompt templates used to obtain non-causal and causal counterfactual augmented data, respectively, as discussed in Section 4.2. Figure 10 presents the constructed instruction for acquiring non-causal counterfactual augmented data, while Figure 11 shows the instruction for obtaining causal counterfactual augmented data.

 

ples, necessitating evaluators to confirm the accuracy of both the grammar and the affirmed stance. The secondary consideration pertains to achieving generating objectives, necessitating evaluators to confirm if the samples were generated as guideline instructions. Evaluators respond to these considerations with a binary "yes" or "no". Subsequently, we calculate the average ratio of affirmative responses from three evaluators for each query. The results in Table 18 show that the generated samples are of high quality, contributing substantially to our calibration network training.

## Appendix G Case Study

We conduct a case study on Sem16, P-Stance and VAST datasets, to analyze the biases of LLMs in the stance detection task and the practical effectiveness of our calibration network. The results are show in Table 19, 20 and 21. The correct analysis patterns of LLMs are marked in blue, while biased analysis patterns are marked in red. We can observe that for some samples with strong sentiment expressions, such as the examples in Table 20, LLMs are influenced by sentiment Spurious cues and result in biased stance judgments. For some controversial debate topics, such as the examples in Table 19, LLMs generate hallucinations due to their preferences, leading to biased stance judgments.

Figure 5: Prompt template of sentiment labels annotation by GPT-4. Fill the blue text with the corresponding text from the sample.

Figure 6: Prompt template with Task-Des setting. We first outline the stance detection task, then instruct the LLMs to determine the stance based on the sentence in relation to the target. Fill the blue text with the corresponding text and target from the sample.

 

Stance detection is to determine the attitude or tendency towards a certain target through a given sentence, including favor, against and neutral.
*Please read the following examples carefully and use them as references to judge the attitude of the sentence towards the target.**

[in-context examples]

Yoursentence: {sentence} Question: What is the attitude of the sentence toward "!target)"? Please select the correct answer from "favor", "against" and "neutral". Answer this question with JSON format: '...'json {{  "answer"; "your answer",  "stance": "favor" | "against" | "neutral"  }} \\ ```

Figure 8: Prompt template with SSC Debias-Instruct setting. We add explicit debiasing instructions following the task description. Fill the blue text with the corresponding text and target from the sample.

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{5}{c}{Sem16(\%)} \\ \cline{2-7}  & \multicolumn{3}{c}{Positive} & \multicolumn{3}{c}{Neutral} & \multicolumn{3}{c}{Negative} \\ \cline{2-7}  & RStud\(\downarrow\) & F1\(\uparrow\) & RStud\(\downarrow\) & F1\(\uparrow\) & RStd\(\downarrow\) & F1\(\uparrow\) \\ \hline
**LLaMA-2-70b-chat** & & & & & & \\ Task-Des & **22.20** & **59.73** & **23.65** & 58.11 & 7.56 & 65.19 \\ CoT-Demo & 26.15 & 58.24 & 27.88 & 55.18 & 28.54 & 62.04 \\ Debias-Instruct & 26.86 & 55.04 & 25.51 & **58.20** & **5.34** & **68.61** \\ \hline
**GPT-3.5-Turbo-0125** & & & & & & \\ Task-Des & 29.37 & 48.37 & 25.22 & 58.78 & 26.80 & 55.44 \\ CoT-Demo & **16.02** & **65.77** & 25.47 & **61.69** & **12.74** & **70.69** \\ Debias-Instruct & 30.52 & 46.09 & **16.31** & 60.07 & 24.43 & 54.91 \\ \hline \end{tabular}
\end{table}
Table 11: RStud of sentiment labels and macro F1-score of stance detection on Sem16 dataset. The best scores are in bold.

Figure 7: Prompt template with CoT-Demo setting. We randomly select 4 samples from the training set, provide the ground truth stance labels, and guide GPT-4 to generate chain-of-thought rationales as examples for this prompt. Fill the green text with constructed examples, and fill the blue text with the corresponding text and target from the sample.

 

Stance detection is to determine the attitude or tendency towards a certain target through a given sentence, including favor, against and neutral. **Be careful to only judge the author's attitude on the target based on the content in the sentence, and do not include your inherent attitude towards the target.**

{sentence}

Question: What is the attitude of the sentence toward "target]"? Please select the correct answer from "favor", "against" and "neutral". Answer this question with JSON format: '''json {{  "stance": "favor" | "against" | "neutral",  }} ...

Figure 10: Prompt template that allows the LLMs to rephrase the original sentence with different words and sentiments and express the target while ensuring that the semantics and the stance of the perturbed sample towards the target remain unchanged. Fill the blue text with the corresponding text, target, and stance label from the sample.

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{5}{c}{P-Stance(\%)} \\ \cline{2-7}  & Positive & \multicolumn{3}{c}{Neutral} & Negative \\ \cline{2-7}  & RStd\(\downarrow\) & F1\(\uparrow\) & RStd\(\downarrow\) & F1\(\uparrow\) & RStd\(\downarrow\) & F1\(\uparrow\) \\ \hline
**LLaMA-2-70b-chat** & & & & & & \\ Task-Des & 33.32 & **68.42** & 19.98 & 62.68 & **16.77** & 72.97 \\ CoT-Demo & **32.62** & 67.17 & **15.10** & **65.00** & 20.70 & **73.61** \\ Debias-Instruct & 34.66 & 66.49 & 22.81 & 59.14 & 17.10 & 71.82 \\ \hline
**GPT-3-5-Turbo-0125** & & & & & & \\ Task-Des & 33.79 & **68.61** & 23.48 & 62.32 & **13.88** & **75.66** \\ CoT-Demo & **32.39** & 66.99 & **19.25** & **65.09** & 16.60 & 74.44 \\ Debias-Instruct & 33.56 & 68.00 & 20.05 & 63.19 & 16.83 & 75.07 \\ \hline \end{tabular}
\end{table}
Table 12: RStud of sentiment labels and macro F1-score of stance detection on P-Stance dataset. The best scores are in bold.

Figure 11: Prompt template that makes necessary modifications to reverse the applicability of the label. Fill the blue text with the corresponding text, target, and stance label from the sample.

Figure 9: Prompt template with TPB Debias-Instruct setting. We add explicit debiasing instructions following the task description. Fill the blue text with the corresponding text and target from the sample.

 



\begin{table}
\begin{tabular}{l c



\begin{table}
\begin{tabular}{l l c c c c c} \hline  & \multicolumn{3}{c}{Sem16 (LLMaA-2-70b-chat)} & \multicolumn{4}{c}{Sem16 (GPT-3.5-Turbo-0125)} \\ \cline{2-8}  & Avg & Bias-SSC & Bias-TPB & Avg & Bias-SSC & Bias-TPB \\ \hline FACTUAL-Prompt-1 & **73.82** & **9.61** & **5.52** & **76.16** & 11.31 & **7.74** \\ w/o Calibration & 67.96 & 27.52 & 27.56 & 73.32 & 18.08 & 13.47 \\ w/o CAD & 71.61 & 15.43 & 12.07 & 70.39 & 12.92 & 11.83 \\ \^{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\^{}}_{\_}{\^{}}_{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}\_{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}{\_}\_{\_}{\_}{\_}{\_}{\_}{\_}\_\_\_\_\_\_\}\_\



\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline Text (Target: Donald Trump) & Label & LLMs Pred & LLMs Stance Rationale & Calibration \\ \hline You might as well put a sock in it. Voters see these coordinated stupid attacks are like your \#Kavanaugh symphony. \#Dems can’t win with hollow attempts to bash \#Trump. & Favor Against & The sentence uses derogatory terms like ‘stupid attacks’ and ’hollow attempts to bash \#Trump, indicating a critical perspective. Therefore, the attitude of the sentence toward ’Donald Trump’ is against. & Favor \\ \hline Text (Target: Joe Biden) & Label & LLMs Pred & LLMs Stance Rationale & Calibration \\ \hline Saying that is as absurd as saying breast cancer screening increases breast cancer. Is there any Republican who isn’t as dumb as dirt? & Favor Against & The sentence implies negative assumptions about the intelligence of Republicans. The mention of Joe Biden in the context of criticizing Republicans suggests a critical or skeptical attitude towards Joe Biden. & Favor \\ \hline \end{tabular}
\end{table}
Table 20: Case study on the P-Stance dataset.

\begin{table}
\begin{tabular}{p{143.8pt}|p{56.9pt}|p{71.1pt}|p{71.1pt}|p{71.1pt}} \hline Text (Target: Homosexual) & Label & LLMs Pred & LLMs Stence Rationale & Calibration \\ \hline Yet that same Bible gives us Christians a particular anthropology, and reveals to us the meaning of sex and marriage within the cosmic order. And herein lies the problem. If you look to a “holy” book written long, long ago and betrayening os never page for guidance on issues of cosmology, biology, sexuality and other human and natural sciences, you will inevitably be in constant conflict with social progress and, indeed, reality. There is no secular opposition to homosexuality.” & Favor Against & The sentence criticizing the use of a ’holy’ book for guidance on issues related to sexuality and natural sciences. It suggests that relying on such sources will lead to conflict with social progress and reality. Additionally, the statement asserts that there is no secular opposition to homosexuality, implying a negative view towards those who oppose it based on religious beliefs. Overall, the tone of the sentence is against homosexuality, indicating a critical stance. & Favor \\ \hline Text (Target: Gun Control) & Label & LLMs Pred & LLMs Stance Rationale & Calibration \\ \hline Military and police, yes. They’re sworn and trained to uphold the country and the laws of the country. The rest is an ego trip. & Favor Against & The sentence expresses a negative attitude towards gun control in the military and police, stating that they are sworn and trained to uphold the country and its laws. However, it is critical of the use of guns by others, referring to it as an ego trip. Therefore, the attitude of the sentence towards gun control is against & Favor \\ \hline \end{tabular}
\end{table}
Table 21: Case study on the VAST dataset.