ZeroStance: Leveraging ChatGPT for Open-Domain Stance Detection via Dataset Generation (2024)

Abstract: Zero-shot stance detection that aims to detect the stance (typically against, favor or neutral) towards unseen targets has attracted considerable attention. However, most previous studies only focus on targets from a single or limited text domains (e.g., financial domain), and thus zero-shot models cannot generalize well to unseen targets of diverse domains (e.g., political domain). In this paper, we consider a more realistic task, i.e., open-domain stance detection, which aims at training a model that is able to generalize well to unseen targets across multiple domains of interest. Particularly, we propose a novel dataset generation method ZeroStance, which leverages ChatGPT to construct a synthetic open-domain dataset CHATStance that covers a wide range of domains. We then train an open-domain model on our synthetic dataset after proper data filtering. Extensive results indicate that our model, when trained on this synthetic dataset, shows superior generalization to unseen targets of diverse domains over baselines on most benchmarks. Our method requires only a task description in the form of a prompt and is much more cost-effective and data-efficient than previous methods. Our code and data are available at https://github.com/chenyez/ZeroStance.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 13390–13405 August 11-16, 2024 ©2024 Association for Computational Linguistics
Zero-shot stance detection that aims to detect the stance (typically against, favor or neutral) towards unseen targets has attracted considerable attention. However, most previous studies only focus on targets from a single or limited text domains (e.g., financial domain), and thus zero-shot models cannot generalize well to unseen targets of diverse domains (e.g., political domain). In this paper, we consider a more realistic task, i.e., open-domain stance detection, which aims at training a model that is able to generalize well to unseen targets across multiple domains of interest. Particularly, we propose a novel dataset generation method ZeroStance, which leverages ChatGPT to construct a synthetic open-domain dataset CHATStance that covers a wide range of domains. We then train an open-domain model on our synthetic dataset after proper data filtering. Extensive results indicate that our model, when trained on this synthetic dataset, shows superior generalization to unseen targets of diverse domains over baselines on most benchmarks. Our method requires only a task description in the form of a prompt and is much more cost-effective and data-efficient than previous methods. Our code and data are available at https://github.com/chenyez/ZeroStance.

heading: 1 Introduction
text: The task of stance detection is to identify the attitude (e.g., favor, against or neutral, etc.) of a given text with respect to a specific target of interest (Küçük and Can, 2020; AlDayel and Magdy, 2020; Xu et al., 2022; Li et al., 2023a; Zhao et al., 2023; Liu et al., 2023; Arakelyan et al., 2023; Ko et al., 2023). Until recently, typical stance detection task was in-domain (Mohammad et al., 2016; Li and Caragea, 2019; Siddiqua et al., 2019; Li et al., 2021b; Upadhyaya et al., 2023; Li and Caragea,
*Both authors contributed equally to this research.
2023) in which the training and test sets share the same set of targets. Most recent works began considering a cross-domain setup, i.e., crosstarget stance detection (Augenstein et al., 2016; Xu et al., 2018; Wei and Mao, 2019; Zhang et al., 2020; Liang et al., 2021) where models are trained on labeled data of a training target and tested on a destination target that is unseen during training. However, cross-target task requires human knowledge about any destination target and how it is related to the training target (Allaway and McKeown, 2020), which limits models’ ability to generalize to a wide variety of unseen targets. Hence, there has been an emerging trend to explore zero-shot stance detection (Allaway and McKeown, 2020; Liang et al., 2022b; Li et al., 2023b), which aims to determine the stance towards unseen targets at the inference stage without requiring human knowledge about unseen targets or their relation to training targets.
Recent studies (Allaway et al., 2021; Liu et al., 2021; Liang et al., 2022a,b) often conduct zeroshot evaluations with training and unseen destination targets originating from a single or limited domains, and thus zero-shot models perform poorly on out-of-domain data. For instance, the WT-WT dataset (Conforti et al., 2020) exclusively comprises targets within the financial domain for both training and evaluation sets, leading to poor performance of models trained on it when applied to political domain data. Xu et al. (2022) investigate open-domain stance detection which aims to train a model that can generalize well to unseen targets of multiple domains. However, their approach still relies on the texts and targets of existing datasets, and thus limits its applicability to a narrow spectrum of domains.
To address the limitations of prior works, we propose a novel open-domain dataset generation approach, ZeroStance, which aims to promote model’s generalization to unseen targets of diverse domains by using external knowledge from pow-
13390
erful large language models (LLMs). Motivated by the remarkable success of LLMs such as ChatGPT in text generation (Brown et al., 2020; Ouyang et al., 2022; Min et al., 2023; Zhou et al., 2023), we present CHATStance, a synthetic dataset created through ZeroStance that leverages ChatGPT to generate high-quality, human-like targets (controversial claims) and texts spanning a variety of domains. We take into account the attribute diversity during the generation by incorporating desired attributes (including domains, geographical locations, and writing styles) as constraints in the prompts. By comparing the performance of ZeroStance with previous baselines, we observe a substantial underperformance of the latter, uncovering the effectiveness of our approach for open-domain stance detection. Table 1 shows a generated sample from our CHATStance dataset.
Our contributions are summarized as follows: • We propose a novel dataset generation ap-
proach ZeroStance for open-domain stance detection that greatly improves the zero-shot performance by improving the data diversity and requires no training data (text or target) of existing datasets.
• Extensive results on six stance datasets show that the model, when trained on our synthetic open-domain dataset, demonstrates better generalization to unseen targets of diverse domains over models trained on humanannotated datasets (with or without data augmentation).
• We present an open-domain dataset CHATStance, which is much more data-efficient and cost-effective. Notably, the model trained on only 33% (around 7k instances) of the CHATStance outperforms the model that is trained with more than 70k instances from humanannotated datasets. Moreover, the total cost of creating CHATStance is only around $3, making it over two thousand times cheaper than human annotation.

heading: 2 Related Work

heading: 2.1 Zero-shot Stance Detection
text: Zero-shot stance detection that aims to detect the stance toward completely unseen targets has drawn considerable attention in recent years. Allaway and McKeown (2020) introduce a dataset for zeroshot stance detection and propose a target-grouped
attention method to implicitly capture the relationships between targets. Later, Allaway et al. (2021) extract target-invariant features to generalize across topics using adversarial learning. Meanwhile, Liu et al. (2021) exploit the structural-level and semantic-level information to strengthen the generalization abilities of zero-shot models. Contrastive learning has also shown its effectiveness by generalizing the target-based stance features to unseen targets or improving the quality of augmented data (Liang et al., 2022a,b; Li and Yuan, 2022). Li et al. (2023b) propose a teacher-student framework that leverages generated keyphrases as augmented targets to improve the performance of zero-shot models. However, most previous works only perform training and evaluations on targets from a single domain (Mohammad et al., 2016; Conforti et al., 2020) or limited domains (Allaway and McKeown, 2020). Models trained on targets of these domains cannot generalize well to unseen targets of more diverse domains. In contrast, we tackle the task from the perspective of data quality by proposing a dataset generation method, which is orthogonal to most existing methods.

heading: 2.2 Data Augmentation
text: Data augmentation has been widely used to boost the model performance. A common data augmentation method is word replacement. Wang and Yang (2015) replace words with their top-n similar words based on pre-trained word embeddings (Mikolov et al., 2013). Similarly, Zhang et al. (2015) replace words with their synonyms in WordNet (Miller, 1995). Wei and Zou (2019) present random word replacement techniques that consist of four types of operations (e.g., synonym replacement, random deletion, etc). Wu et al. (2019) utilize segment embeddings of BERT (Devlin et al., 2019) as class indicators to generate augmented samples by predicting the masked word with the masked language
modeling (MLM) objective. Another commonly used strategy for augmentation is back-translation (Yu et al., 2018), which translates the training sample from one language (e.g., English) to another (e.g., French) and then translates it back to the original language. In stance detection, Li and Caragea (2021) generate samples by performing MLM with target and label information. Li and Yuan (2022) propose a data augmentation framework for that generates synthetic training data for unseen targets by adversarial learning and contrastive learning. Xu et al. (2022) design a masking mechanism to generate data with pre-trained language models. Li et al. (2023b) propose a target augmentation method that augments targets from the original training set and exploited a teacher-student framework to improve the model performance. However, the aforementioned data augmentation methods usually rely on existing datasets to generate augmented samples, which limits the generated samples to a restricted set of domains. Thus, zeroshot models trained with augmented data cannot generalize well to diverse domains. In this paper, inspired by the recent success of LLMs (Dai et al., 2023; Ubani et al., 2023; Zhu et al., 2023; Kuzman et al., 2023; Huang et al., 2023; Gilardi et al., 2023), we construct an open-domain dataset CHATStance that covers a wide range of domains using ChatGPT. Our proposed method requires no training data from existing datasets, but a task description of stance detection.

heading: 3 Method
text: In this section, we introduce the implementation of ZeroStance which includes three steps: target generation, text generation and data filtering. During target generation, ChatGPT is used to generate relevant targets given the category information extracted from the debate forum. For text generation, generated targets and stance labels are fed to ChatGPT in the form of prompts to generate corresponding texts. Then, we perform data filtering to remove potentially noisy and less informative data. The overall framework of ZeroStance is shown in Figure 1.

heading: 3.1 Problem Formulation
text: For zero-shot stance detection, we have two disjoint sets of targets: S for seen targets ts and U for unseen targets tu. Suppose a given training set Ds={(xsi , tsi , ysi )}Nsi=1 and a test set Du= {(xui , tui )}Nui=1, where xsi is a sequence of words, t s i is the corresponding target and ysi ∈ {Against, Favor, Neutral} is the stance label. The objective of zero-shot stance detection task is to predict the stance yui given a document xui and an unseen target t u i based on a model that is trained on documents xs and seen targets ts in Ds. Note that documents xu and unseen targets tu do not necessarily come from the same or similar domains of training set Ds.

heading: 3.2 Target Generation
text: First, we create a high-quality seed list by extracting pre-defined generic categories from kialo1, which is a structured online debate platform where users provide supporting and opposing claims for each claim related to a controversial topic. Kialo includes diverse set of controversial claims that are tagged under pre-defined generic categories such as Politics, Racism, Music, Society and Economy. The complete seed list is shown in Appendix A.
Second, motivated by the recent success of LLMs on data annotation (Zhu et al., 2023; Kuzman et al., 2023; Huang et al., 2023; Gilardi et al., 2023) and data augmentation (Dai et al., 2023; Ubani et al., 2023), we leverage ChatGPT to generate controversial claims for each category from the seed list with the prompt “List 40 controversial claims in online debate forums in [CAT] area in [GEO]”, where [CAT] and [GEO] are special tokens that are replaced with a real category from the seed list and a geographic region (America, Europe or Asia), respectively. Since controversial topics of different geographic regions are usually different for the same category (e.g., for the category Politics, the controversial topics could be “police brutality towards people of color” in America and “regulating free speech in Singapore” in Asia), we generate claims for each region to cover more diverse topics. At the end, we generate a pool of around 24k controversial claims. Examples of generated claims are shown in Table 2.

heading: 3.3 Text Generation
text: After target generation, we further apply ChatGPT for text generation. We input a stance label ysi (fa-
1https://www.kialo.com/tags
vor or against) and a generated claim tsi from the previous step into ChatGPT and prompt ChatGPT to generate the corresponding text xsi . Previous works and datasets have shown that people usually express their stance towards specific topics by providing examples (Mohammad et al., 2016), sharing individual experience (Liu and Fahmy, 2011) or discussing relevant topics (Li et al., 2021a). Therefore, we consider three different prompts for text generation, which well represent how people express their stance towards controversial topics. For example, the prompt of sharing individual experience is “Generate a few consecutive sentences that are [STANCE] the “[CLAIM]”. Please generate sentences with only personal experience.”, where [STANCE] and [CLAIM] are stance label (in favor of or against) and generated claim, respectively. All three prompts used in text generation are shown in Appendix B. We randomly sample 3k claims from the target pool for each prompt and generate 18k instances in terms of three prompts and two stance labels (against and favor). Examples of generated texts are shown in Table 3. More examples are shown in Appendix B.
Note that we only generate against and favor instances, and do not generate neutral instances with ChatGPT since ChatGPT may not well understand what a neutral stance is and it is even quite challenging for humans to write a neutral opinion towards a topic in the task of stance detection (Li et al., 2021a). Therefore, similar to previous work (Allaway and McKeown, 2020), we automatically generate neutral instances by matching generated texts with most dissimilar claims from the target pool that are obtained based on the cosinesimilarity of sentence embeddings (Reimers and Gurevych, 2019). We generate and sample 3k neutral instances for each prompt type and construct a dataset of 27k synthetic instances in total.

heading: 3.4 Data Filtering
text: Swayamdipta et al. (2020) has shown that training dynamics (i.e., variability and confidence) can be used to improve the out-of-distribution (OOD) performance. Variability is defined as follows:
σ̂i =
√ ∑Ee=1 (pθ(e)(y∗i ∣xi) − µ̂i)2
E (1)
where pθ(e) denotes the model’s probability with parameter θ(e) at the end of the eth epoch, xi and y ∗ i denote the input instance and its label, respec-
tively, E is the total number of training epochs and confidence µ̂i is defined as below.
µ̂i = 1
E
E
∑ e=1 pθ(e)(y∗i ∣xi) (2) Training instances are classified into three types: 1) High variability instances exhibit frequent fluctuations in true class probabilities, introducing ambiguity and are informative for enhancing models’ OOD performance. 2) High confidence, low variability instances indicate easy-to-learn scenarios with consistently reliable and correct predictions, offering limited value for models’ further improvement. 3) Low confidence, low variability instances represent hard-to-learn scenarios with consistent prediction errors, often due to mislabeling during annotation (Swayamdipta et al., 2020).
In this paper, to enhance model generalization to unseen domains, we adopt the approach of Swayamdipta et al. (2020) by excluding instances with low variability, covering both easy-to-learn (less informative) and hard-to-learn (potentially noisy) cases. We calculate the variability for each instance after training and remove p percent of instances with the lowest variability from the training dataset. Consequently, we develop CHATStance, a stance detection dataset for open domains. Details on CHATStance’s data statistics after filtering are available in Appendix C.

heading: 4 Experimental Settings
text: In this section, we first introduce the baseline human-annotated datasets (§4.1). Then we discuss our evaluation setup (§4.2). Last, we describe baselines used in our experiments (§4.3).

heading: 4.1 Human-Annotated Datasets
text: covid COVID-19-Stance (Glandt et al., 2021) consists of tweets related to four targets in the COVID19 domain: Anthony Fauci, Stay-at-Home Orders, Wear a Face Mask, and Keeping School Closed.
pstance The P-STANCE dataset (Li et al., 2021a) consists of tweets from the political domain. The dataset includes three targets: Donald Trump, Joe Biden, and Bernie Sanders.
sem16 The SemEval-2016 Task 6 dataset (Mohammad et al., 2016) is composed of tweet-target pairs centered around five targets, namely Atheism, Feminist Movement, Legalization of Abortion, Hillary Clinton, and Climate Change is a Real Concern.
wtwt The Will-They-Won’t-They dataset (Conforti et al., 2020) consists of a large number of annotated tweet-target pairs from the financial domain, including five merger and acquisition operations (e.g., Merger of CVS Health and Aetna).
ibm30k The IBM-Rank-30k dataset (Gretz et al., 2020) includes 30k annotated text-target pairs. Targets encompass 71 selected controversial topics (e.g., We should abolish capital punishment).
vast The VAST dataset (Allaway and McKeown, 2020) includes news comments from the The New York Times Room for Debate section that contains a large number of targets from multiple domains.
The dataset split are presented in Appendix D. We also analyze the data contamination between the CHATStance and the human-annotated datasets, as presented in Appendix E.

heading: 4.2 Evaluation Setup
text: Following previous work (Hardalov et al., 2021), we consider two evaluation setups for baselines: (1) no training, i.e., unsupervised baselines that make stance predictions directly without requiring any training data; (2) out-of-domain, i.e., leave-onedataset-out training on existing human-annotated datasets. Specifically, for six evaluation datasets, we leave one dataset out as the target dataset and take the rest five datasets as source datasets. We train and validate models using training and validation sets of source datasets and test them on the test set of the target dataset. To evaluate our pro-
posed ZeroStance, we train and validate the model on CHATStance dataset and test it on test sets of six human-annotated datasets. In adherence to the zero-shot setup, we exclude any targets (claims) from CHATStance that match targets present in the evaluation datasets. Like prior works (Allaway and McKeown, 2020; Glandt et al., 2021), we employ the macro-averaged F1 across all stance classes as our evaluation metric.

heading: 4.3 Baselines
text: Unsupervised Baselines. We consider five baselines that can work on the unsupervised scheme, which are applied directly to make stance predictions. Random Guess is a baseline that randomly predicts the stance label. GPT2 (Radford et al., 2019) infers the stance label using the document’s perplexity. BART-MNLI and RoBERTa-MNLI apply BART (Lewis et al., 2020) and RoBERTa (Liu et al., 2019) pre-trained on the MNLI dataset (Williams et al., 2018) for stance prediction. ChatGPT is a strong zero-shot baseline that directly predicts the stance based on a task description (Zhang et al., 2023), as detailed in Appendix F.
Data Augmentation Baselines. We also compare ZeroStance with previous data augmentation methods. RoBERTa represents the RoBERTalarge model without data augmentation. We then consider the following data augmentation methods applied to RoBERTa. BT (Yu et al., 2018) employs a back-translation technique where English sentences are initially translated to French and then re-translated back to English. EDA (Wei and Zou, 2019) augments the dataset using four operations: random deletion, random swap, synonym replacement, and random insertion. OpenStance (Xu et al., 2022) generates training data based on texts or targets of existing datasets. TTS (Li et al., 2023b) first performs target augmentation based on a keyphrase generation model. A teacher-student learning framework is employed to improve target diversity by assigning pseudo stance labels to the augmented targets.
In our experiments, data augmentation baselines are trained in the out-of-domain setup. We perform data augmentation based on the source datasets. Models are trained on the combination of original source data and augmented data and evaluated on the out-of-domain data to understand its adaptability to an unseen dataset.
ZeroStance. ZeroStance is our proposed ap-
proach that constructs a synthetic open-domain dataset CHATStance with ChatGPT. As discussed in §3.3, three different types of prompts (prompt 1: provide examples, prompt 2: share personal experience and prompt 3: discuss relevant topics) are used to improve the diversity of generated texts. We further filter out 1% of instances with the lowest variability scores to improve the data quality. More details on data filtering are provided in Appendix G. Finally, we train and validate the RoBERTa-large model on CHATStance and test it on six benchmark datasets. Hyperparameters adopted in our experiments are shown in Appendix H.

heading: 5 Results and Discussions
text: In this section, we first compare our approach with previous unsupervised baselines and data augmentation methods (§5.1). We then perform ablation studies to understand the effectiveness of data filtering and prompts that are used for text generation (§5.2). We also investigate the impact of the data size to ZeroStance by training models on different sizes of the CHATStance dataset (§5.3). Next, we explore the performance of augmenting existing datasets with our CHATStance dataset (§5.4) and utilizing each human-annotated dataset as the open-domain dataset (§5.5).

heading: 5.1 Comparison with Baselines
text: We compare ZeroStance with unsupervised and data augmentation baselines. Results are shown in Table 4. First, we observe that ZeroStance outperforms unsupervised baselines on most datasets. In particular, ZeroStance outperforms the best baseline ChatGPT on four out of six datasets. We observe that when used as a stance classifier, ChatGPT struggles to accurately discern stances toward certain targets, such as “Atheism” in the sem16 dataset. Specifically, a large number of against and favor instances related to “Atheism” are misclassified as neutral by ChatGPT. In contrast, our proposed ZeroStance improves the zero-shot model by harnessing ChatGPT to produce a plethora of texts and targets (such as “Christianity is intolerant of other religions and beliefs.”) from diverse domains. This suggests that, rather than directly applying ChatGPT for stance prediction, leveraging knowledge from ChatGPT to generate the dataset for strong baselines (specifically, RoBERTa in our study) proves to be more effective for open-domain stance detection.
Second, in the out-of-domain setup, ZeroStance consistently outperforms RoBERTa across all datasets, achieving an average improvement of 8.2% on six datasets. Notably, ZeroStance exhibits an improvement of 32.05% on the ibm30k dataset, which suggests that the model trained on our synthetic dataset generalizes well to domains of ibm30k, which are not well captured by other models trained on existing datasets.
Third, ZeroStance shows improvements over all data augmentation methods across all datasets, achieving an average improvement of 9.08% in average F1-macro. This result demonstrates that our proposed method, which aims to cover a wide range of domains, proves to be more beneficial in open-domain stance detection than previous augmentation methods that merely augment data from limited domains. Note that ZeroStance demonstrates superior performance with a much smaller training set (around 21k) in contrast to RoBERTa and data augmentation methods such as BT that rely on substantially larger training sets of approximately 70k and 140k, respectively. This suggests that our proposed ZeroStance is much more dataefficient in open-domain stance detection.

heading: 5.2 Ablation Study
text: We conduct an ablation study to investigate the effectiveness of different components in ZeroStance.
First, we study the impact of different prompts used in text generation. For a fair comparison, we gather an equivalent amount of data as the CHATStance dataset but using only one or two types of prompts. Second, we investigate the effectiveness of our data filtering approach by training the model on the original collected data without filtering. Results are shown in Table 5.
First, the removal of one type of prompt (e.g., “w/o prompt 1”) leads to decrease in the performance on most datasets. Similar results can be observed when we only use one type of prompt (e.g., “w/o prompt 1, 2”). This implies that our proposed approach with all three prompts can effectively guide ChatGPT to generate more diverse texts, which strengthen the generalization abilities of zero-shot models. Second, the removal of data filtering (“w/o data filtering”) results in worse performance on most datasets, which indicates that removing the data that the model is most decisive about (low-variability) can effectively improve the OOD performance, which is consistent with the observations of previous work (Swayamdipta et al., 2020). In Appendix I, we perform a human evaluation based on the filtered instances and show that our data filtering approach can effectively identify examples that are less informative and noisy.

heading: 5.3 Impact of Data Size
text: To understand the impact of data size on performance, we change the training size of CHATStance by randomly sampling subsets that comprise 16%, 33%, and 66% of the original training set and doubling the size of the dataset (200%) by generating more texts and targets. Then we train the RoBERTalarge model on each selected set. Results are shown in Table 6. We observe that performance decreases as we use a smaller training set of the CHATStance dataset. This result indicates the necessity of developing a large dataset. Besides, we observe that further increasing the size of the dataset decreases the model performance on four out of six datasets. A plausible reason is, as data size increases, the diversity of targets generated by ChatGPT may saturate due to the repetitive generation and an accumulation of similar data could potentially degrade model generalization. Interestingly, with just 7k training instances (33%) of the CHATStance dataset, our model surpasses the RoBERTa baseline (as shown in Table 4), which is trained on approximately 70k instances, on five out of six datasets. This underscores our approach’s superior data efficiency over existing human-annotated datasets.
5.4 Data Augmentation with CHATStance
In this section, we explore the effectiveness of utilizing CHATStance as an augmented set for human-annotated datasets. Specifically, models are trained on the combination of CHATStance and five existing datasets in the out-of-domain setup (ZeroStance-Aug) and evaluated using the left-out dataset. The results are shown in Table 7. We observe that ZeroStance-Aug shows improvements over ZeroStance on three datasets, but the overall improvement in average F1-macro is not significant (67.23% for ZeroStance-Aug vs. 67.12% for ZeroStance). This indicates that mixing data from specific domains with an open-domain dataset does not necessarily improve the model’s generalization to unseen domains.
5.5 CHATStance vs. Human-Annotated Datasets
We explore the potential of using each humanannotated dataset as an open-domain dataset and compare its efficacy with our CHATStance dataset. Specifically, we train and validate the RoBERTa model on training and validation sets of each dataset and then evaluate its performance on test sets of all datasets. The comparison between each human-annotated dataset and CHATStance is presented in Table 8. We observe that models trained on other datasets consistently lag behind the model trained on our CHATStance dataset across all test sets. Notably, even models trained on multi-domain datasets such as vast show worse performance than the model trained on CHATStance. This underscores the limited domain diversity of previous human-annotated datasets when compared to our CHATStance dataset.
We also perform a quality analysis on CHATStance and compute the total cost of data generation, which are shown in J and K, respectively.

heading: 6 Conclusion
text: In this paper, we propose a novel dataset generation approach for open-domain stance detection, which aims to train a model that performs well on unseen targets from all domains of interest. We leverage the ChatGPT to construct a synthetic open-domain dataset from scratch by generating controversial topics of diverse domains and the corresponding texts without training samples of existing datasets. Experimental results indicate that the model trained on our synthetic dataset shows better generalization to unseen targets of diverse domains over baselines on most benchmark datasets. Moreover, our proposed method is more cost-effective and dataefficient than previous methods.
Limitations
One limitation of our dataset is that it focuses only on America, Europe, and Asia to maintain consistency with established methodologies in prior research. However, we are keen to expand our dataset to a more diverse geographical scope, including regions like Africa in future work, enhancing the robustness and diversity of our findings. The other limitation is that we use ChatGPT for data generation, which could unintentionally introduce biases such as generating texts of fixed patterns. However, we mitigate this by considering diverse domains and attributed prompts (providing examples, sharing individual experience and discussing relevant topics) during the generation.
Ethical Statement
We gather targets and texts solely based on category names from a public debate website and our proposed prompts, ensuring ethical integrity by not including user-identifiable information and offensive content in ChatGPT’s inputs. Although OpenAI has made significant efforts to mitigate toxicity issues, we have further enhanced security by integrating Google’s Perspective API for toxicity detection in our generated content. Only after clearing this toxicity assessment are samples retained.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their insightful comments to help improve the paper. This work is supported by the National Natural Science Foundation of China (NSFC) Key Project
under Grant Number 62336006, the Pioneer and “Leading Goose” R&D Program of Zhejiang under Grant Number 2022SDXHDX0003 and the Ministry of Science and Technology of China Key Project under Grant Number 2022YFE0204900. We also thank the National Science Foundation for support from grants IIS-2107487 and ITE-2137846 which supported the research and the computation in this study. Yue Zhang is the corresponding author.

heading: A Seed Categories for Target Generation
text: The full seed list of categories for target generation is shown in Table 9. We generate the list by extracting pre-defined generic categories from kialo, which is an online debate platform that covers controversial topics from a wide range of domains.

heading: B Prompts and More Examples of Text Generation
text: Motivated by observations of previous works (Mohammad et al., 2016; Liu and Fahmy, 2011; Li et al., 2021a), we consider three types of prompts for text generation, which are shown in Table 10. We input a generated claim and a stance label (against or in favor of) to ChatGPT and prompt ChatGPT to generate the corresponding text. We observe that ChatGPT can precisely capture the difference among our prompts and is able to generate high-quality texts as required. More examples of text generation using different prompts are shown in Table 11.

heading: C Data Statistics of CHATStance
text: CHATStance includes near 27k synthetic instances that cover a wide range of domains. We split the dataset into training and validation sets in an 80/20 fashion. Data statistics of CHATStance can be seen in Table 12.

heading: D Dataset Split
text: Train, validation and test sets are used as provided for covid, pstance, ibm30k and vast datasets. For sem16, training and test sets are provided and we split the training set into training and validation sets using an 85/15 split. For wtwt, we split the whole dataset into training, validation and test sets using an 80/10/10 split.

heading: E Data Contamination
text: We evaluate our proposed method on six humanannotated stance datasets, which are pstance (Li et al., 2021a), ibm30k (Gretz et al., 2020), sem16 (Mohammad et al., 2016), vast (Allaway and McKeown, 2020), covid (Glandt et al., 2021) and wtwt (Conforti et al., 2020).
Given the possibility that ChatGPT could have been trained using existing stance detection datasets, we investigate the potential data contamination by measuring the similarity between our CHATStance dataset (D1) and each existing stance detection dataset (D2). Following the previous work (Ubani et al., 2023), we adopt the Percent Word Overlap metric to measure the similarity between D1 and D2. Specifically, first, for any two text samples originating from the datasets, we remove stop words and punctuation. Second, we obtain the word overlap score by counting the number of unique words in both samples and dividing
this number by the number of unique words in the lengthier text of the pair. Third, for each text sample within the D1 dataset, we determine its maximum word overlap score by computing its word overlap score with each sample of the D2 dataset. Finally, we calculate the average of these maximum scores to represent the overall word overlap
similarity between D1 and D2. In Table 15, we report the word overlap similarity between CHATStance and the test set of each human-annotated dataset. We also measure the similarity between the training set and the test set of each human-annotated dataset. We can observe that CHATStance exhibits the lowest word overlap similarity in four out of six human-annotated test sets. For instance, the overlap between the ibm30k training set and the sem16 test set exceeds the overlap between the CHATStance and the sem16 test set by a margin of 8%. This underscores the minimal risk of data contamination posed by the CHATStance dataset in our experiments.

heading: F Prompt for ChatGPT Baseline
text: For datasets such as ibm30k and pstance, where the stance labels are limited to ’favor’ and ’against’, we use the following ChatGPT prompt: “Question: What is the stance of the text [TEXT] towards the claim [CLAIM]? The answer should just be selected from ’favor’, or ’against’. Answer:” For datasets like vast, covid19, wtwt, and sem16, where the stance labels include ’favor’, ’against’, and ’neutral’, our ChatGPT prompt is: “Question: What is the stance of the text [TEXT] towards the claim [CLAIM]? The answer should just be selected from ’favor’, ’against’, or ‘neutral’. Answer:”.

heading: G Data Filtering with Various Ratios
text: In this section, we investigate the impact of removing instances with low variability of different ratios. Specifically, we remove instances with the lowest variability from the training set at rates of 1%, 3%, 5%, 10%, 20%, and 30%, respectively. These filtered training sets are then utilized to train the RoBERTa-large model. The hyper-parameter p (p=1) mentioned in §3.4 is selected based on the performance on the validation set. Results are shown in Table 13. First, we can observe that models trained with data filtering generally perform better than the model trained without data filtering, indicating the effectiveness of removing less informative and noisy data. Second, data filtering with the ratio of 1% shows the best performance in overall, which may suggest that an excessive data filtering of CHATStance dataset could remove informative data and skew the balance of the training data, thereby negatively affecting the model performance.

heading: H Training Settings
text: In our work, we utilized the gpt-3.5-turbo-0301 version of ChatGPT for data collection. We performed all experiments on a single NVIDIA RTX A6000 GPU. RoBERTa-large2 is used to evaluate our CHATStance dataset due to the effectiveness of RoBERTa on stance detection task (Li and Caragea, 2021; Hardalov et al., 2021; Li et al., 2021a; Liu et al., 2023). The total training time for fine-tuning the RoBERTa-large model on our dataset is less than 4 hours. The learning rate of RoBERTa-large is set to 1e-5. AdamW (Loshchilov and Hutter, 2019) is utilized as the optimizer. The mini-batch is set to 64. The model is trained for 4 epochs with early stopping and the patience is 5. Each result is the average of three runs with different initializations. Hyper-parameters are selected based on the model performance on the validation set.
2 https://huggingface.co/vinai/bertweet-large

heading: I Examples of Instances Filtered by Data Filtering
text: In this section, we perform human evaluations based on instances that are identified by our data filtering method. We observe that our data filtering method can effectively identify instances that are either noisy or less informative. We show some examples in Table 14. In the first two examples, the text and target are irrelevant, suggesting a neutral stance rather than the given stance labels. The third example presents a neutral stance toward the target. However, the target “The problem of induction” is too broad. This lack of specificity diminishes its utility for improving model generalization to new domains.

heading: J Human Evaluation
text: To understand the effectiveness of ZeroStance, we perform a quality analysis on targets and texts generated by ZeroStance. For each prompt, we randomly selected 100 instances, ensuring a balanced distribution of favor, against, and neutral labels. We then examined the (text, target, stance) consistency. The analysis was conducted by two of the co-authors with expertise in stance detection, who had been trained using samples from existing stance datasets to ensure accurate relevancy assessment. Our findings indicate that 96% of instances demonstrate consistency in (texts, target, stance). Of the remaining instances, 83% exhibited
incorrect stances towards the targets, while 17% contained texts that were irrelevant to their respective targets. This inconsistency may be attributed to the noise inherent in ChatGPT. We show some of our evaluated examples in Table 16. In the first example, the text appears to support tree planting, even though the target highlights the negative side of the practice and the stance label is against. The second example depicts a generated text that lacks relevance to the given target, where the stance label is neutral.

heading: K Cost Comparison
text: The gpt-3.5-turbo-0301 API costs $0.0005 per 1k input tokens and $0.0015 per 1k output tokens. For our target generation, with the average of 30 input tokens and 12 output tokens across 204 categories in America, Europe, and Asia, the cost is $0.0005*30*204*3/1k + 0.0015*24k*12/1k ≈ $0.43. For our text generation task, we generate favor and against instances for 9k targets (including 3 types of prompts) with an average of 74 input tokens and 66 output tokens. The cost is $0.0005*74*9k*2/1k + $0.0015*66*9k*2/1k ≈ $2.45. The total cost of constructing CHATStance is only around $3. In comparison, online crowdsourcing platforms typically charge around $0.11 for 50 tokens (Wang et al., 2021), leading to a total expense of around $6.2k, which is over 2,000 times more expensive than our approach.

heading: Future Work
text: However, we are keen to expand our dataset to a more diverse geographical scope, including regions like Africa in future work, enhancing the robustness and diversity of our findings. The other limitation is that we use ChatGPT for data generation, which could unintentionally introduce biases such as generating texts of fixed patterns. However, we mitigate this by considering diverse domains and attributed prompts (providing examples, sharing individual experience and discussing relevant topics) during the generation