C-STANCE: A Large Dataset for Chinese Zero-Shot Stance Detection
Year: 2023

Abstract:
Zero-shot stance detection (ZSSD) aims to determine whether the author of a text is in favor of, against, or neutral toward a target that is unseen during training. Despite the growing attention on ZSSD, most recent advances in this task are limited to English and do not pay much attention to other languages such as Chinese. To support ZSSD research, in this paper, we present C-STANCE that, to our knowledge, is the first Chinese dataset for zero-shot stance detection. We introduce two challenging subtasks for ZSSD: target-based ZSSD and domain-based ZSSD. Our dataset includes both noun-phrase targets and claim targets, covering a wide range of domains. We provide a detailed description and analysis of our dataset. To establish results on C-STANCE, we report performance scores using state-of-the-art deep learning models. We publicly release our dataset and code to facilitate future research.1

heading: 1 Introduction
text: Stance detection aims to automatically predict whether the author of a text is in favor of, against, or neutral toward a specific target (Mohammad et al., 2016b; Küçük and Can, 2020; ALDayel and Magdy, 2021), e.g., epidemic prevention, gasoline price, or equal rights. The stance can provide useful information for important events such as policymaking and presidential elections.
Early works focus on two types of stance detection tasks: in-target stance detection, where classifiers are trained and tested on data from the same set of targets (Hasan and Ng, 2014; Mohammad et al., 2016b; Graells-Garrido et al., 2020) and cross-target stance detection, where classifiers are trained on source targets that are related to destination targets (Augenstein et al., 2016; Wei and Mao, 2019), but destination targets are unseen during training. However, it is impractical to include
1https://github.com/chenyez/C-STANCE
all possible or related targets in the training set. More recently, zero-shot stance detection (ZSSD) has been identified as a promising direction (Allaway and McKeown, 2020) to evaluate classifiers on a large number of unseen (and unrelated) targets. ZSSD is more similar to the situations in practice and has received a lot of attention (Liu et al., 2021; Luo et al., 2022; Liang et al., 2022b).
Despite the growing interest in stance detection, the task has several limitations. First, most recent advances in stance detection are limited to English (Mohammad et al., 2016b; Allaway and McKeown, 2020; Conforti et al., 2020b; Li et al., 2021a; Glandt et al., 2021), and pay little attention to other languages such as Chinese (Xu et al., 2016) although large amounts of online data with expressions of stance are available in other languages. Second, the current ZSSD task (Allaway and McKeown, 2020) aims to detect the stance of unseen targets. However, these unseen targets come from the same domain of training targets with similar meanings, which makes the task less challenging. Third, current stance detection datasets include targets either as noun phrases (Mohammad et al., 2016b; Glandt et al., 2021) or as claims (Ferreira and Vlachos, 2016; Derczynski et al., 2017). However, in practice, stance is usually expressed toward both noun phrases and claims. Models trained only on nounphrase targets do not necessarily work well for claim targets and vice versa. Little attention is paid toward incorporating targets of both types.
In an effort to minimize these drawbacks, we present C-STANCE, the first Chinese zero-shot stance detection dataset. Our dataset is collected from Sina Weibo, one of the most popular Chinese social media sites (akin to Twitter). We consider two practical scenarios for zero-shot stance detection, i.e., target-based and domain-based ZSSD. Subtask A: target-based zero-shot stance detection. Subtask A is similar to the previous ZSSD task, where stance detection classifiers are evalu-
13369
ated using a large number of completely unseen targets. Subtask B: domain-based zero-shot stance detection. Subtask B is our newly proposed ZSSD task where stance detection classifiers are evaluated using a large number of unseen targets from completely new domains. Additionally, C-STANCE captures a more diverse set of targets including both noun-phrase targets and claim targets compared with existing datasets. An example from our dataset is shown in Table 1. As we can see from the table, the author of the microblog is in favor of the noun-phrase target “Covid-19 vaccine” and against “the conflict between doctors and patients”. The author also opposes claim target 2, whose main idea is to deny the necessity of self-protection.
Our contributions can be summarized as follows: 1) We present C-STANCE, the first large Chinese zero-shot stance detection dataset. Our dataset is composed of 48,126 annotated text-target pairs. CSTANCE is more than 2.5 times larger than the English ZSSD VAST dataset (Allaway and McKeown, 2020) and more than 16 times larger than the existing Chinese stance detection dataset by Xu et al. (2016). We provide detailed description and analysis of our dataset; 2) We include two challenging ZSSD subtasks: target-based zero-shot stance detection and domain-based zero-shot stance detection for C-STANCE; 3) We consider a more diverse set of targets including both noun phrases and claims in C-STANCE as well as multiple targets per input text (see Table 1); 4) We establish baseline results using both traditional models and pre-trained language models and show that C-STANCE is a challenging new benchmark. For example, our bestperforming model based on RoBERTa achieves only 78.5% F1macro for subtask A.

heading: 2 Related Work
text: Most previous stance detection datasets are constructed for the English language (Mohammad
et al., 2016b; Conforti et al., 2020b; Glandt et al., 2021). Particularly, VAST is the only dataset for zero-shot stance detection. Even though recent years have witnessed an emerging trend of constructing stance detection datasets of other languages (Xu et al., 2016; Taulé et al., 2017; Swami et al., 2018; Lai et al., 2020; Vamvas and Sennrich, 2020), Chinese stance detection datasets are still very scarce. Xu et al. (2016) developed the first Chinese stance dataset. The dataset focuses on in-target stance detection and only includes 3,000 examples from 6 targets. In contrast, we propose the first large-scale Chinese dataset for zero-shot stance detection. Our C-STANCE which includes 48,126 samples with 11,623 noun-phrase targets and 28,581 claim targets enables multiple stance detection tasks and covers a wide range of domains.
Besides classifying stance detection by target type (noun phrases or claims), we can also categorize the task as in-target, cross-target, and zero-shot stance detection. Most previous works focused on in-target stance detection where a classifier is trained and evaluated on the same target (Zarrella and Marsh, 2016; Wei et al., 2016; Vijayaraghavan et al., 2016; Mohammad et al., 2016b; Du et al., 2017; Sun et al., 2018; Wei et al., 2018; Li and Caragea, 2019, 2021b). However, it is usually hard to obtain sufficient annotated data for each target and conventional models perform poorly when generalized to data of unseen targets. This motivated the research on cross-target stance detection (Augenstein et al., 2016; Xu et al., 2018; Wei and Mao, 2019; Zhang et al., 2020; Li et al., 2021b), where a classifier is adapted from different but related targets. However, cross-target stance detection still requires prior human knowledge of the destination target and how it is related to the training targets. Thus models developed for cross-target stance detection are still limited in their capability to generalize to a wide range of unseen targets. Zero-shot
stance detection (ZSSD) which aims to identify the stance toward a large number of unseen targets has attracted considerable attention in recent years. Allaway and McKeown (2020) developed a dataset for ZSSD which is called VAried Stance Topics (VAST) that includes thousands of targets. Based on VAST, many ZSSD models have been developed (Liu et al., 2021; Liang et al., 2022a,b; Luo et al., 2022; Li et al., 2023). In contrast to VAST, we include two types of ZSSD subtasks in C-STANCE. The first subtask is the target-based ZSSD which is similar to the VAST setting. The second subtask is the domain-based ZSSD where classifiers are evaluated on unseen targets from completely new domains, which is a more challenging task.
Target-specific stance detection is the most common stance detection task (ALDayel and Magdy, 2021), which aims to predict the stance label toward a target, which could be a figure or controversial topic (Hasan and Ng, 2014; Mohammad et al., 2016a; Zotova et al., 2020; Conforti et al., 2020a,b). Multi-target stance detection is another type of stance detection task that aims to jointly identify the stance toward two or more targets in the same text (Sobhani et al., 2017; Darwish et al., 2017; Li and Caragea, 2021a). Unlike target-specific and multi-target stance detection where targets are usually noun phrases (phrasebased stance detection), claim-based stance detection aims to predict the stance toward a specific claim, which could be an article headline or a rumor’s post (Qazvinian et al., 2011; Derczynski et al., 2015; Ferreira and Vlachos, 2016; Bar-Haim et al., 2017; Derczynski et al., 2017; Gorrell et al., 2019). However, less attention has been paid to incorporating both noun-phrase targets and claim targets into one dataset. Comparatively, our dataset
supports data for both claim-based stance detection and phrase-based stance detection as well as captures multiple targets per input text (see examples from our dataset in Appendix A). We compare our C-STANCE dataset with previous stance detection datasets in Table 2.

heading: 3 Dataset Construction
text: In this section, we describe the creation and particularities of C-STANCE, a large comprehensive stance detection dataset composed of 48,126 annotated instances covering a wide range of domains.

heading: 3.1 Data Collection
text: We collect microblogs using the Weibo API from July 26th, 2022, to November 16th, 2022. Similar to prior works (Mohammad et al., 2016b; Glandt et al., 2021; Li et al., 2021a), our crawling is performed using query keywords. To cover a wide range of domains on Weibo, we start by using the domain names listed on the Weibo hot list page as keywords for crawling (e.g., society, education, etc.). After we get our initial set, we select the most frequent words as supplementary keywords for the next round of crawling to gradually expand our keyword set. The full list of keywords that were used is provided in Appendix B. We end up collecting 60,000 microblogs.

heading: 3.2 Keywords Selection
text: After data collection, we filter keywords that are most suitable for the task of stance detection. We perform the following steps for keyword filtering: 1) We manually detect and remove keywords that often contain advertising content (e.g., beauty, renting, motor show, etc.), which are not suitable for stance detection as the purpose of those microblogs
is not to discuss controversial topics but to promote the sales of particular products. 2) For stance detection, we show more interest in controversial topics and keywords where people may express different stances (favor, against, or neutral) toward targets related to these keywords. Otherwise, models would predict the stances based on keywords information instead of the contents of microblogs and the targets. Therefore, we filter out keywords that people tend to show uni-stances on, e.g., poverty, delicious food, traveling, camera, etc., and keywords where microblogs often express personal feelings (e.g., “my girlfriend”, “my mood”, etc). After this filtering step, we select 45 keywords that cover controversial topics. We summarize the 45 keywords into 7 domains: “Covid Epidemic” (CoE), “World Events” (WE), “Cultural and Education” (CuE), “Entertainment and Consumption” (EC), “Sports” (S), “Rights” (R), and “Environmental Protection” (EP) which can be seen in Table 3.

heading: 3.3 Preprocessing
text: We perform several preprocessing steps to ensure the quality of our dataset. 1) We remove microblogs with less than 50 or more than 200 words.
From our observations, microblogs with less than 50 words usually are either too noisy or cannot cover enough information to express stances toward multiple targets. Microblogs with more than 200 words are usually technical articles that contain little stance-related discussion. 2) We remove duplicates and reposted microblogs. 3) We keep only microblogs in Chinese. We leave the multilingual dataset as our future work. 4) We manually identify a set of phrase lexicon for advertisements (e.g., check the link below, follow our WeChat public account, scan the QR code, click to join, etc.). We filter out all microblogs containing phrases in this lexicon. 5) We remove the emojis, URLs in microblogs as they may introduce noise to the dataset. After preprocessing, our corpus reduces to around 25,000 examples. We randomly sample around 215 microblogs for each of the 45 keywords, obtaining 9,696 microblogs for annotation.

heading: 3.4 Data Annotation
text: We gather annotations using Taojinniwo,2 a Chinese crowd-sourcing company that provides annota-
2http://sjbz.itaojin.cn/
tion services for big AI companies (e.g., Baidu, JD, etc.). To ensure the annotation quality, we employ strict requirements for annotators: 1) Annotators should reside in China; 2) Annotators should have college degrees. Moreover, we randomly select 10% of each annotator’s annotations for quality checks. If an annotator has an acceptance rate of less than 90%, we discard their annotations completely and re-send them for labeling using other qualified annotators. We annotated data for nounphrase targets and for claim targets as detailed below. The label distribution for each domain is shown in Table 4.

heading: 3.4.1 Annotation for Noun-Phrase Targets
text: The annotation for noun-phrase targets is performed in two steps. In step 1, one annotator is asked to detect at least 2 targets from each microblog. Annotators are given the following instructions: “You should identify 2 or more targets in the form of noun phrases. Targets should satisfy the following requirements: 1) Targets should be the main focus of the microblog instead of the trivial details; 2) Targets should be public topics that people may take stances on; 3) Avoid selecting targets to which most people may express the same stance, e.g., illegal charge.”. In step 2, we ask three annotators to assign a stance label to each microblog-target pair. The instructions are given below: “Based on the message that you learned from the microblog, predict the stance that the author would take for the given target as “Favor”, “Against”, or “Neutral”. We take the majority vote among stance annotations from the three annotators to obtain stance labels. For 9,696 microblogs, we collected 19,038 annotated instances (around 2 targets per instance). The inter-annotator agreement measured by Krippendorff’s alpha (Krippendorff, 2011) is 0.60, and a percentage agreement of 75%. We see that while the task is challenging, annotators agree the majority of the time. We can observe from Table 4 that the “World Events” (WE) do-
main and the “Sports” (S) domain have the highest percentage in the “Neutral” class. This might be because these domains include more microblogs related to news. Moreover, people are showing a higher percentage of “Against” stances toward targets in the “Covid Epidemic” (CoE) and “Rights” (R) domains, where more contrary opinions are often expressed.

heading: 3.4.2 Annotation for Claim Targets
text: The goal of this annotation task is to identify three claims, to which the microblog takes favor, against, and neutral stances, respectively. Annotators are provided with the following instructions: “After reading the microblog, write the following three claims: 1) The author is definitely in favor of the point or message of the claim (favor); 2) The author is definitely against the point or message of the claim (against); 3) Based solely on the microblog content, we cannot know whether the author supports or is against the point or message of the claim (neutral).”. To pose challenges to the ZSSD task, we have some additional requirements: First, claims with label favor should not be a direct copy of the microblog content. Second, claims with labels against should not be the simple negation of the microblog content (e.g., adding “not” before verbs). Models may easily detect such language patterns and predict the stance without considering the content of microblog-claim pairs.
Note that our claim annotation differs from the task of rumor detection (Zubiaga et al., 2015; Derczynski et al., 2017), where claims are replies stemming from the text. Some of such claims may miss information mentioned in the text (e.g., Text: Coronavirus is made by the alien. Claim: I don’t believe that.). Our task focuses on predicting the stance toward a claim that discusses the same topic and does not omit any necessary information (e.g., Claim: I believe the Coronavirus is made by some terrorists). We collect 29,088 annotated microblog-claim pairs. For quality assurance, we hide the stance label and
ask another group of annotators to annotate the stance label for a subset of microblog-claim target pairs. The two annotation groups agree on 95% of the annotation. We observe from Table 4 that each domain has a balanced label distribution. This is because we annotate one claim for each stance label from each microblog.

heading: 3.5 Dataset Split
text: We split the annotated data into training, validation, and test sets for the target-based ZSSD (subtask A) and the domain-based ZSSD (subtask B). For subtask A, we separate the dataset following the VAST dataset (Allaway and McKeown, 2020): the training, validation, and test sets do not share any microblogs and targets with each other. We randomly select 70% of unique annotated microblogs for the training set and split the remainder evenly for the validation and the test set. The dataset distribution is shown in Table 5. We have 2,865 unique zero-shot noun-phrase targets and 4,487 unique zero-shot claims for 1,503 unique microblogs in the test set, with the average length of 4.7, 26.5, and 105.9 for noun-phrase targets, claim targets, and microblogs, respectively. We also report the average percentage of tokens in targets that overlap with tokens in microblogs (see Appendix C).
For subtask B, we use the data from six domains (source) for training and validation, and the data from the left-out domain (zero-shot) as the test set. In the end, we have 7 dataset splits for subtask B with one dataset split for each of the 7 domains where each domain in turn is used as the test set. To ensure there are no overlapping targets between the source domains and the zero-shot domain, we remove data with overlapping targets from the source domains in each split. We then split the source domains into the training and the validation set without overlapping microblogs and targets. The statistics when using the “Covid Epidemic” as the zero-shot domain are shown in Table 5. The full statistics of subtask B are shown in Appendix D.
Because of the linguistic variations in the nounphrase target expressions, we study the prevalence of LexSimTopics (Allaway and McKeown, 2020) between the training and the test set, which is defined as test targets that have more than 0.9 cosine similarities with any train targets in the word embedding space (Bojanowski et al., 2017). We observe that for subtask A, we have 11% LexSimTopics in the test set. Whereas for the “Covid Epi-
demic” domain in subtask B, we only have 7% LexSimTopics. This implies that subtask B is more challenging as the training and test targets are more different from each other. Comparatively, VAST dataset has 16% LexSimTopics in the zero-shot test set which is higher than our task.

heading: 4 Experimental Settings
text: In this section, we introduce the baselines in Section 4.1 and the training settings in Section 4.2.

heading: 4.1 Baseline Methods
text: To evaluate C-STANCE, we run experiments with the following baselines. BiCE (Augenstein et al., 2016) and CrossNet (Xu et al., 2018) predict the class label using the conditional encoding of BiLSTM models. TGA-Net (Allaway and McKeown, 2020) implicitly captures relationships between targets using generalized topic representations to assist stance classification. We also consider the base version of BERT (Devlin et al., 2019) trained using the whole word masking (wwm) on Chinese Wikipedia (Cui et al., 2020), the 12-layer RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019) pre-trained on Chinese news, Q&A, and BaiduBaike (Cui et al., 2020).

heading: 4.2 Training Settings
text: We perform experiments using an NVIDIA RTX A5000 GPU. Our experiments are conducted based on PyTorch (Paszke et al., 2019). The validation set was used to determine the hyperparameters for the models. For BiCE and CrossNet, we used the AdamW (Loshchilov and Hutter, 2019) with a learning rate of 0.001. Each model was trained for 20 epochs, with a mini-batch size of 64. For TGANet, we followed hyperparameters suggested in the previous work (Allaway and McKeown, 2020). For BERT, RoBERTa, and XLNet, we used the AdamW with a learning rate of 5e-6. Models were fine-tuned for 5 epochs using a mini-batch size of 32. The total training time is less than 3 hours.

heading: 5 Results
text: In this section, we first perform experiments on subtask A and subtask B. We then conduct experiments on cross-lingual stance detection using C-STANCE and the previous English ZSSD VAST dataset. We also study the impact of incorporating both nounphrase targets and claims targets into one dataset. Lastly, we perform the spuriosity analysis for claim
targets. Each result is the average of 4 runs with different initializations. Similar to prior works (Allaway and McKeown, 2020; Liang et al., 2022b), we use the F1 for each class and the macro-averaged F1 of all classes as evaluation metrics.

heading: 5.1 Target-based Zero-Shot Stance Detection
text: Target-based zero-shot stance detection (subtask A) aims to evaluate the classifier on a large number of completely unseen targets (Allaway and McKeown, 2020). Our experiments are performed using the full dataset with mixed targets (both noun phrases and claims), the dataset with noun-phrase targets, and the dataset with claim targets, respectively.
Experimental results are shown in Table 6. First, we can observe that transformer-based models show better performance than RNN-based models, demonstrating the effectiveness of the pre-trained transformer models. Moreover, RoBERTa and XLNet outperform BERT in most metrics, suggesting the effectiveness of additional training performed by RoBERTa and XLNet to address different limitations of BERT. Second, transformer-based models perform better on claim targets than noun-phrase targets. This might be because transformer models are better at capturing contextual information and claims are usually composed of more contextual information than noun phrases. Comparatively, BiCE and CrossNet perform worse on claim targets than noun-phrase targets, showing that claim targets are more challenging for RNN-based models. We also notice that TGA-Net achieves worse performance on claim targets. This might be because the model requires clustering based on target representations, which is more difficult for the claim targets. Third, model performance for the mixed target is higher than the noun-phrase targets and lower than the claim targets. This suggests that ZSSD models that can properly utilize both types of targets are still needed, which we leave as our future work.

heading: 5.2 Domain-based Zero-Shot Stance Detection
text: Domain-based stance detection (subtask B) focuses on evaluating classifiers using unseen topics from completely new domains. Particularly, we select one domain as the zero-shot domain and the rest six domains as source domains. We train and validate models using data from source domains and test models using data from the zero-shot domain. We have seven zero-shot domain settings (each with a different zero-shot domain). Similar to subtask A, our experiments are performed using the full dataset with mixed targets, data with noun-phrase targets, and data with claim targets, respectively.
Results are shown in Table 7. First, we can observe that among the seven zero-shot domain settings, most models show the highest performance when predicting stances for claim targets and the mixed targets from the “Environmental Protection” domain. For example, RoBERTa achieves the highest F1macro of 0.879 for the claim targets, improving its performance over the rest domains by up to 5.9%. Second, stances for noun-phrase targets
from the “Sports” and the “World Events” domains are easier to predict than the other domains, where RoBERTa and XLNet achieve the highest F1macro of 0.687 and 0.680, respectively. This might be because sports and world events are domains with a higher percentage of microblogs discussing news, which usually captures more diverse target ranges than the other domains. Moreover, we also observe that in most cases, the “Covid Epidemic” is the most difficult domain to predict for all targets, suggesting that the “Covid Epidemic” domain shares the least domain knowledge with the other domains, making it the most difficult zero-shot domain for domain-based ZSSD.
For mixed targets experiments, we also report the results for test sets of only noun-phrase targets and only claim targets separately in Appendix E (i.e., training on mixed targets and testing on noun-phrase targets or training on mixed targets and testing on claim targets).

heading: 5.3 Cross-Lingual Zero-Shot Stance Detection
text: To better understand the difference between the existing English ZSSD dataset (VAST) and our Chinese C-STANCE dataset, we perform experiments on cross-lingual zero-shot stance detection between the two datasets. Particularly, we fine-tune a multilingual transformer model BERT (mBERT) (Devlin et al., 2019). The model is pre-trained on 104 languages. We train and validate mBERT using one dataset, and test the model using the other dataset. During the test stage, we experiment with both the original test set and the test set translated into the other language using Google Translate 3.
As shown in Table 8, models trained on VAST perform poorly on the neutral class for the CSTANCE, while models trained on C-STANCE show much higher performance. The results imply that the neutral class in C-STANCE is more challenging than VAST. This is because data for the neutral class in VAST is generated by randomly permuting existing targets and texts, which may generate easy-to-detect text-target pairs. Compara-
3https://translate.google.com/
tively, for C-STANCE, targets for the neutral class are manually extracted by annotators from each microblog, which are more closely related to the microblog content. Moreover, machine-translated test sets in both languages show worse F1-macro than the original test sets, indicating that machine translation fails to generate high-quality data. This suggests the importance of developing a zero-shot stance detection dataset for Chinese, which has not been done prior to this work.

heading: 5.4 Impact of Incorporating Two Target Types
text: To analyze the impact of incorporating the nounphrase targets and the claim targets in one dataset, we evaluate models trained with noun-phrase targets using the claim targets and vice versa. Results are compared with models trained using mixed target types and evaluated by two types of targets separately. Experiments are performed for subtask A, using the best-performing XLNet and RoBERTa.
Results are shown in Table 9, where we can observe that when models are trained with claim targets and evaluated with noun-phrase targets, the performance is much worse than ones trained by the mixed targets (e.g., 0.291 vs. 0.679 for XLNet). Similar results can be observed when models are trained with noun-phrase targets. These results suggest that datasets including the uni-target type are not capable of handling other target type, which further strengthens the necessity of developing datasets including both target types.

heading: 5.5 Spuriosity Analysis for Claim Targets
text: We perform spuriosity analysis for claim targets to ensure that we cannot predict the stance based
solely on the claim. For subtask A, we conduct experiments using XLNet and RoBERTa with only the claim target as input, which are compared with experiments using both microblog and claim target. The results are shown in Table 10, where we can observe a substantial amount of performance decrease when only the claim target is used as input. Therefore, both microblogs and claim targets are required for the models to make correct stance predictions by learning the semantic relation between them.

heading: 6 Conclusion
text: In this paper, we introduce C-STANCE, the first Chinese zero-shot stance detection dataset. Our dataset includes two challenging ZSSD subtasks: target-based ZSSD (evaluating classifiers using a large number of unseen targets) and domain-based ZSSD (evaluating classifiers using a large number of unseen targets from unseen domains). Moreover, we consider both noun-phrase targets and claim targets. Our dataset is larger and more challenging compared with the previous Chinese stance detection dataset, consisting of 48,126 annotated microblog-target pairs. C-STANCE can serve as a new benchmark for ZSSD, along with VAST, and can enable future research for other stance detection tasks. We conduct experiments using state-ofthe-art deep learning models. Future work includes studying the multilingual ZSSD with the union of C-STANCE and other multi-lingual datasets.
Limitations
Our C-STANCE data is collected from social media, which may be seen as a limitation, as we may not cover all aspects of formal texts that could be used in essays or news comments. We will plan to extend this dataset with other types of text in the future. However, this is a limitation of any other datasets that focus on social media content.
Ethical Statement
Our dataset does not provide any personally identifiable information. Microblogs are collected using generic keywords instead of user information as queries, therefore our dataset does not have a large collection of microblogs from an individual user. Thus our dataset complies with Sina Weibo’s information privacy policy.

heading: Acknowledgements
text: We thank the National Science Foundation for support from grants IIS-1912887, IIS-2107487, and ITE-2137846 which supported the research and the computation in this study. We also thank our reviewers for their insightful feedback and comments.

heading: A More Examples of C-STANCE
text: In this section, we show more examples for each domain of our C-STANCE dataset in Table 11.

heading: B Query Keywords
text: The full keywords set that we used for data crawling is shown in Table 12. We generate the list by gradually extending the initial keywords set from Weibo hot list with the most frequent words.

heading: C Token Overlap
text: We also report the average percentage of tokens in targets that overlap with tokens in microblogs. The results are shown in Table 16. We observe that noun-phrase show a higher overlapping percentage than claim targets, which is because annotators tend to summarize noun-phrase targets using semantically similar tokens from the text.

heading: D Full Statistics of Subtask B
text: The statistics of the 7 dataset splits (data from six domains for training and validation, and the data from the left-out domain as the zero-shot test set) are shown in Table 13.

heading: E Evaluations on Models Trained by Mixed Targets with Noun-Phrase Targets and Claim Targets
text: In subtask A and subtask B, for experiments using mixed targets, we also test the baseline models using noun-phrase targets and claim targets separately. Our goal is to better understand how each model trained on mixed targets performs for each type of target separately. The results for subtask A and subtask B are shown in Table 14 and Table 15, respectively. We can observe that the fine-tuned transformer-based models (i.e., BERT, RoBERTa, and XLNET) show higher performance on the claim targets. For BiCE, CrossNet, and TGA-Net, stances for claim targets are more difficult to predict.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation Section after the Conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitation Section after the Conclusion.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 and 4.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We discuss the dataset that we created in Section 3 and the baseline models that we used in Section 4.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discussed our usage of baseline models in Section 4. We discussed the intended use of the dataset that we created in Section 3. We show our dataset is compatible with the original access conditions in the Ethical Statement Section.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethical Statement Section.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.
C 3 Did you run computational experiments? Section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We discussed the annotation company that we worked with and how we recruited annotators in Section 3.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3 and Section 5.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3.

heading: Future Work
text: C-STANCE can serve as a new benchmark for ZSSD, along with VAST, and can enable future research for other stance detection tasks. We conduct experiments using state-ofthe-art deep learning models. Future work includes studying the multilingual ZSSD with the union of C-STANCE and other multi-lingual datasets. Limitations
Our C-STANCE data is collected from social media, which may be seen as a limitation, as we may not cover all aspects of formal texts that could be used in essays or news comments. We will plan to extend this dataset with other types of text in the future. However, this is a limitation of any other datasets that focus on social media content