EZ-STANCE: A Large Dataset for English Zero-Shot Stance Detection (2024)

Abstract: Zero-shot stance detection (ZSSD) aims to determine whether the author of a text is in favor, against, or neutral toward a target that is unseen during training. In this paper, we present EZ-STANCE, a large English ZSSD dataset with 47,316 annotated text-target pairs. In contrast to VAST (Allaway and McKeown, 2020), which is the only other large existing ZSSD dataset for English, EZ-STANCE is 2.5 times larger, includes both noun-phrase targets and claim targets that cover a wide range of domains, provides two challenging subtasks for ZSSD: target-based ZSSD and domain-based ZSSD, and contains much harder examples for the neutral class. We evaluate EZ-STANCE using state-of-the-art deep learning models. Furthermore, we propose to transform ZSSD into the NLI task by applying simple yet effective prompts to noun-phrase targets. Our experimental results show that EZ-STANCE is a challenging new benchmark, which provides significant research opportunities on English ZSSD. We publicly release our dataset and code at https://github.com/chenyez/EZ-STANCE.

heading: 1 Introduction
text: The goal of stance detection is to automatically detect whether the author of a text is in favor of, against, or neutral toward a specific target (Mohammad et al., 2016b; Küçük and Can, 2020; ALDayel and Magdy, 2021), e.g., public education, mask mandate, or nuclear energy. The stance can reveal valuable insights relevant to events such as public policy-making and presidential elections.
Earlier research has concentrated on two types of stance detection tasks: in-target stance detection, in which models are trained and evaluated using data from the same target, e.g., both train and test sets contain data about “Donald Trump” (Hasan and Ng, 2014; Mohammad et al., 2016b; GraellsGarrido et al., 2020), and cross-target stance detection, where the models are trained on source targets
that are related to, but distinct from, the destination targets (Augenstein et al., 2016; Wei and Mao, 2019), which remain unseen during training (e.g., the destination target is “Donald Trump” whereas the source target is “Hillary Clinton”). However, it is unrealistic to incorporate every potential or related target in the training set. As such, zero-shot stance detection (ZSSD) has emerged as a promising direction (Allaway and McKeown, 2020) to evaluate classifiers on a large number of unseen (and unrelated) targets. ZSSD is more related to real-world scenarios and has consequently started to receive significant interest recently (Liu et al., 2021; Luo et al., 2022; Liang et al., 2022b).
Despite the growing interest in ZSSD, the task still exhibits several limitations. First, the VAST dataset (Allaway and McKeown, 2020) which is the only other large existing ZSSD dataset for English, contains only noun phrase targets. Yet, in realworld scenarios, stance is often taken toward both noun phrases (Mohammad et al., 2016b; Glandt et al., 2021) and claims (Ferreira and Vlachos, 2016; Derczynski et al., 2017). We notice that models trained on data with noun-phrase targets struggle to accurately predict the stance for claim-target data, and vice versa, due to the mismatch between the training and test data. The need to incorporate both types of targets for ZSSD has been relatively overlooked. Second, VAST is designed solely to detect the stance of unseen targets, but these unseen targets at the inference stage originate from the same domain as the training targets (in-domain), possessing similar semantics, which makes the task less challenging. Third, despite being instrumental for the development of zero-shot stance detection, VAST generates data for the neutral class by randomly permuting documents and targets, leading to a lack of semantic correlation between the two (we show an example from VAST from the neutral class in Table 1). Deep learning models can easily detect these patterns, consequently diminishing the
15697
complexity of the task. In an effort to address the aforementioned limitations and spur research in ZSSD, we present EZ-STANCE, a large English Zero-shot stance detection dataset collected from Twitter. In contrast with VAST, EZ-STANCE is, to our knowledge, the first large English ZSSD dataset that captures both noun-phrase targets and claim targets, covering a more diverse set of targets. By training a single model on our comprehensive dataset, we achieve comparable or superior performance than training separate models for each type of target. Moreover, EZ-STANCE includes two real-world scenarios for zero-shot stance detection, namely target-based and domain-based ZSSD. Subtask A: target-based zero-shot stance detection. This subtask is the same as the traditional ZSSD task (Allaway and McKeown, 2020), where stance detection classifiers are evaluated using a large number of completely unseen (and unrelated) targets, but from the same domains (in-domain). Subtask B: domainbased zero-shot stance detection. This subtask is our proposed ZSSD task where stance detection classifiers are evaluated using a large number of unseen targets from completely new domains (out-ofdomain). Furthermore, in EZ-STANCE, annotators manually extract targets from each tweet to form the neutral class, ensuring semantic relevance to the tweet content. We show an example from our dataset along with corresponding noun-phrase and claim targets for each stance (against, favor, and neutral) in Table 1. As we can see from the table, the noun-phrase target “zip lock bag” is relevant to the tweet but the author of the tweet holds a neutral stance towards this target.
In summary, our contributions are as follows: 1) We present EZ-STANCE, a unique large zero-shot stance detection dataset, composed of 47,316 annotated English tweet-target pairs. EZ-STANCE is
2.5 times larger than VAST (Allaway and McKeown, 2020), which is the only large existing ZSSD dataset for English. We provide a detailed description and analysis of our dataset; 2) We consider a more diverse set of targets including both noun phrase and claim targets (see Table 1); 3) We include two challenging ZSSD subtasks in EZSTANCE: target-based zero-shot stance detection and domain-based zero-shot stance detection; 4) We establish baseline results using both traditional models and pre-trained language models; 5) We propose to formulate stance detection into the task of natural language inference (NLI) by applying simple yet effective prompts to noun-phrase targets. Our results and analysis show that EZ-STANCE is a challenging new benchmark.

heading: 2 Related Work
text: Target-specific stance detection is the most prevalent type of stance detection (ALDayel and Magdy, 2021), whose goal is to determine the stance expressed in a text towards a target. Usually, the target is an entity / short noun-phrase, e.g., a political figure or controversial topic (Hasan and Ng, 2014; Mohammad et al., 2016a; Zotova et al., 2020; Conforti et al., 2020a,b), or a claim, e.g., an article’s headline or a reply to a rumorous post (Qazvinian et al., 2011; Derczynski et al., 2015; Ferreira and Vlachos, 2016; Bar-Haim et al., 2017; Derczynski et al., 2017; Gorrell et al., 2019).
Most earlier research is centered around in-target stance detection where a classifier is trained and evaluated on the same target (Zarrella and Marsh, 2016; Wei et al., 2016; Vijayaraghavan et al., 2016; Mohammad et al., 2016b; Du et al., 2017; Sun et al., 2018; Wei et al., 2018; Li and Caragea, 2019, 2021). However, the challenge often arises in gathering enough annotated data for each specific target, and traditional models perform poorly when generalized to unseen target data. This spurred interest in investigating cross-target stance detection (Augenstein et al., 2016; Xu et al., 2018; Wei and Mao, 2019; Zhang et al., 2020), where a classifier is adapted from different but related targets. However, cross-target stance detection still requires human knowledge of the destination target and how it is related to the training targets. Thus, models developed for cross-target stance detection are still limited in their capability to generalize to a wide range of unseen targets (Liang et al., 2022b).
Zero-shot stance detection (ZSSD) which aims
to detect the stance on a large number of unseen (and unrelated) targets represents a more realistic scenario, and has received significant interest in recent years (Allaway and McKeown, 2020; Liu et al., 2021; Liang et al., 2022a,b; Luo et al., 2022; Xu et al., 2022; Li et al., 2023; Zhao et al., 2023). These works tackle stance detection where the target is an entity or noun phrase, whereas other works that follow a ZSSD scenario (training and test data are collected from different events) focus on stance classification towards rumours in which the target is a longer text or a claim (Ferreira and Vlachos, 2016; Derczynski et al., 2017; Gorrell et al., 2019).
Surprisingly, despite substantial interest in ZSSD, large-scale datasets for the evaluation of this task are limited. In Table 2, we compare our EZ-STANCE dataset with existing English stance detection datasets. As we can observe from the table, VAried Stance Topics (VAST) (Allaway and McKeown, 2020) is the only existing dataset for English ZSSD that encompasses thousands of targets, whereas Emergent (Ferreira and Vlachos, 2016), RumEval17 (Derczynski et al., 2017), RumEval19 (Gorrell et al., 2019) contain only several hundred targets. We can also observe that these datasets have either noun phrase targets or claim targets, but not both. In contrast, our EZ-STANCE dataset integrates both target types into a single dataset; includes two types of ZSSD subtasks—target-based ZSSD (same as VAST) and domain-based ZSSD, a new and more challenging task in which classifiers are evaluated on unseen targets from completely new domains; contains much harder neutral examples—the targets of the neutral data in EZ-STANCE are extracted from the texts, ensuring strong semantic relevance to the text content, while data for the neutral class in VAST is generated by randomly permuting existing texts and targets, resulting in easy-to-detect patterns; and is much larger in size—EZ-STANCE is notable for
its extensive range of 40,678 targets across a comprehensive corpus of 47,316 examples (we provide more examples from EZ-STANCE in Appendix A).

heading: 3 Dataset Construction
text: Here, we detail the construction of EZ-STANCE.

heading: 3.1 Data Collection
text: Our data are collected using the Twitter API, spanning from May 2021 to January 2023. Like previous works (Mohammad et al., 2016b; Glandt et al., 2021; Li et al., 2021), we crawl tweets using query keywords. To cover a wide range of domains, we use domain names from the Explore page of Twitter as keywords for crawling (e.g., sports, education, etc.). Then we expand the keywords set for the next round by including the most frequent words as supplementary keywords. In total, we collect 50,000 tweets. Next, we perform data filtering to eliminate keywords and tweets that are not suitable for stance detection. The detailed keywords selection process, the full list of keywords for crawling, as well as our data filtering strategy are provided in Appendix B. Eventually, we select 72 keywords covering controversial topics. We summarize the 72 keywords into 8 domains: “Covid Epidemic" (CE), “World Events" (WE), “Education and Culture" (EdC), “Entertainment and Consumption" (EnC), “Sports" (S), “Rights" (R), “Environmental Protection" (EP), and “Politics" (P). Table 3 shows the domains and query keywords in each domain.

heading: 3.2 Data Annotation
text: The target and stance annotations of our dataset are gathered through Cogitotech,1 a data annotation company that provides annotation services for big AI companies (e.g., OpenAI, AWS, etc.). To
1https://www.cogitotech.com/
ensure high-quality annotations, we apply rigorous criteria: 1) Annotators must have a minimum education qualification of college graduation; 2) The annotators’ native language must be English. Moreover, we randomly sample 10% of each annotator’s annotations to perform quality checks and discard annotations from an annotator if the acceptance rate is lower than 90%. This data is re-sent to other qualified annotators for labeling. The overall stance label distribution for both noun-phrase and claim targets for each domain is shown in Table 4.

heading: 3.2.1 Annotation for Noun-Phrase Targets
text: The annotation for noun-phrase targets is performed in two steps. In step 1, one annotator is asked to identify a minimum of 2 targets from each given tweet. In step 2, we instruct 3 annotators to assign a stance label to each tweet-target pair. The instructions for the annotators are provided in Appendix C.1. After the annotations are completed, we determine the stance for each tweet-target pair by using the majority vote amongst the three annotators. The inter-annotator agreement measured using Krippendorff’s alpha (Krippendorff, 2011) is 0.63, which is higher than VAST (0.427).

heading: 3.2.2 Annotation for Claim Targets
text: We ask one annotator to generate three claims for each tweet, to which the tweet takes favor, against, and neutral stances, respectively. The detailed annotation instructions are in Appendix C.2. For quality assurance, we hide the stance labels for a subset of tweet-claim pairs and ask another group of annotators (who did not write the claims) to annotate the stance. The two groups agree on 95% of the times. This result indicates high-quality generations of the claim targets and stance labels.

heading: 3.3 Dataset Split
text: For subtask A, we split the dataset in alignment with the VAST dataset (Allaway and McKeown, 2020): the training, validation, and test sets do not share any texts (tweets) and targets with each other. The detailed split process are shown in Appendix D. For subtask B, we use the data from seven domains (source) for training and validation, and the data from the left-out domain (zero-shot) as the test set. This results in 8 dataset splits for subtask B with one dataset split assigned for each of the eight domains, wherein each domain in turn is used as the test set. We exclude data with overlapping targets between the source and zero-shot domains, and then partition the source domains into training and validation sets, ensuring no duplication of tweets and targets. The statistics of subtask A and subtask B (using “Covid Epidemic” as the zero-shot domain) are shown in Table 5. The full statistics of subtask B are shown in Appendix E.

heading: 3.4 Dataset Statistics
text: In this section, we present a statistical analysis of our EZ-STANCE dataset.
Label Distribution We can observe from Table 4 that the “Sports” (S) and the “Entertainment and Consumption” (EnC) domains have the highest percentage in the “Neutral” class. This might be because these domains include more tweets related to news. Moreover, people are showing a higher percentage of “Against” stances toward targets in the “Covid Epidemic” (CoE), “World Event” (WE), and “Politics” (P) domains, where more contrary opinions are often expressed.
Dataset Size In Table 5, we observe that EZSTANCE includes a much larger number of zeroshot targets than VAST. EZ-STANCE uniquely provides zero-shot claim targets, further expanding its coverage. Moreover, VAST only includes targetbased ZSSD, whereas EZ-STANCE also enables the more challenging domain-based ZSSD.
Text/target Lengths Table 5 indicates that the average word counts in EZ-STANCE are 2 for noun-phrase targets, 19 for claim targets, and 40 for texts. VAST features similarly lengthed nounphrase targets but longer texts (around 100 words), which are from New York Times news comments, unlike the shorter tweet-based EZ-STANCE texts.
LexSimTopics Given the linguistic variations in the noun-phrase target expressions, we investigate the prevalence of LexSimTopics (Allaway and McKeown, 2020) between the train and test sets. LexSimTopics is defined as the percentage of targets that possess more than 0.9 cosine similarities with any training targets in the word embedding space (Bojanowski et al., 2017). As shown in Table 5, in Subtask A, we have 12% and 13% LexSimTopics in the test set and the validation set, respectively, whereas for the “Covid Epidemic” domain in subtask B, we only have 10% and 11% LexSimTopics for the test and validation sets, indicating that subtask B poses more challenges as the targets
in the training and test sets exhibit more differences. In comparison, the VAST dataset has 16% and 19% LexSimTopics in the test set and validation set, respectively, which are higher than our dataset.

heading: 4 Methodology
text: We now present our approach for converting ZSSD into the natural language inference (NLI) task.

heading: 4.1 Problem Definition
text: Suppose we are given a training set Dtrain= {(xtraini , ttraini , ytraini )}Ntraini=1 and a test set Dtest= {(xtesti , ttesti )}Ntesti=1 , where xtraini is a training document (tweet), ttraini is a target and y train i is the label (or stance) ∈ {Favor, Against, Neutral}. For target-based ZSSD (subtask A), targets in xtesti do not overlap with targets in xtraini . For domainbased ZSSD (subtask B), targets in xtesti not only do not overlap with the targets in xtraini , but they also belong to a domain that is not seen in Dtrain. The objective is to predict the stance given both xtesti and t test i by training a model on the D train.

heading: 4.2 Transform ZSSD into NLI
text: Natural Language Inference (NLI) (Bowman et al., 2015; Williams et al., 2018) is a task that classifies the relationship between a premise and a hypothesis as entailment, contradiction, or neutral. Models pre-trained on NLI datasets are adept at discerning intricate relationships between sentences, a skill that is beneficial for similar NLP tasks.
To leverage the extensive knowledge of NLI pretrained models, we propose to transform stance detection into NLI. Particularly, we convert the text and target into the premise and hypothesis, respectively. The task of predicting stance labels (Favor, Against, or Neutral) is transformed into the task of predicting entailment labels (Entailment, Contradiction, or Neutral). To effectively apply
NLI pre-trained models for stance detection, we introduce prompt templates that transform nounphrase targets into sentence-like hypotheses, aligning them with the typical sentence format of NLI hypotheses. Claim targets remain the same as they already resemble hypotheses quite closely. Particularly, we design five simple yet effective prompts: "The above text entails [target]!", "The premise has an entailment relation with [target]!", "This implies an entailment relation with [target]!", "The premise has the entailment relation with the hypothesis [target]!", and "The premise entails the hypothesis [target]!". For each noun-phrase target, we randomly apply one of the five prompts. Examples in Table 6 demonstrate the re-formulation of ZSSD as NLI. We fine-tune the BART-large encoder (Lewis et al., 2020) pre-trained on MNLI (Williams et al., 2018) dataset to predict the stance.

heading: 5 Baselines and Models
text: We introduce the ZSSD baselines and our approach of utilizing NLI pre-trained models for ZSSD.

heading: 5.1 ZSSD Baselines
text: We evaluate EZ-STANCE using the following ZSSD baselines. BiCE (Augenstein et al., 2016) and CrossNet (Xu et al., 2018) predict the stance using the conditional encoding of BiLSTM. TGA-Net (Allaway and McKeown, 2020) captures implicit relations/correlations between targets in a hidden space to assist stance classification. Next, we consider fine-tuning the base version of state-of-the-art transformer-based models as strong baselines, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019). We also use LLaMA 2 and ChatGPT to directly predict
the stance label based on a task description (Zhang et al., 2023; Touvron et al., 2023).

heading: 5.2 NLI Pre-trained Models
text: To evaluate NLI pre-trained models for ZSSD, we compare the following methods. BART-MNLI-e: We fine-tune the BART-MNLI encoder on our original EZ-STANCE dataset without prompts. The BART decoder is not included due to memory constraints. BART-MNLI-ep: Same as above, except that the BART-MNLI encoder is fine-tuned on EZSTANCE with prompts applied to noun-phrase targets. Note that BART-MNLI-e and BART-MNLI-ep are identical in experiments conducted with claim targets. BART-MNLI: We use the pre-trained BARTMNLI online version with both encoder and decoder without fine-tuning on EZ-STANCE to infer the stance labels for the test set of EZ-STANCE. In addition, to verify our approach across different model architectures, we fine-tune other NLI pretrained models including BERT-MNLI, RoBERTaMNLI, and XLNet-MNLI using the original EZSTANCE dataset. We also experiment with these models using prompted noun-phrase targets, designated as BERT-MNLIp, RoBERTa-MNLIp, and XLNet-MNLIp, respectively. We show the hyperparameters used in experiments in Appendix F.

heading: 6 Results
text: In this section, we first present results for subtask A (§6.1) and subtask B (§6.2). We then compare EZ-STANCE with the VAST dataset (§6.3). Next, we study the impact of different prompt designs (§6.4). Like prior works (Allaway and McKeown, 2020), we employ the macro-averaged F1 score across all classes as our evaluation metrics.

heading: 6.1 Target-based Zero-Shot Stance Detection
text: Target-based ZSSD (subtask A) aims to evaluate the classifier on a large number of completely unseen targets. We train models using three scenarios: 1) on the full training set with both noun-phrase and claim targets; 2) on training data with noun-phrase targets only; and 3) training data with claim targets only. Each model is then evaluated in three corresponding scenarios: 1) the full test set with mixed targets; 2) the test subset with noun-phrase targets only; and 3) the test subset with claim targets only.
Results are shown in Table 7. First, we observe that models trained on noun-phrase targets and evaluated on claim targets (N→C), or the re-
verse (C→N), demonstrate much worse performance compared to those trained with mixed targets (M→N or M→C) or with consistent target types (N→N or C→C). This suggests that datasets limited to one target type struggle to correctly predict the stance of the other, highlighting the necessity for developing a dataset that includes both noun-phrase and claim targets. Second, models trained on mixed targets demonstrate similar performance on both noun-phrase and claim targets (M→N or M→C) compared to those trained solely on noun-phrase or claim targets (N→N or C→C), respectively. This underscores the advantages of our dataset: training a single model on mixed targets yields similar results compared to training separate models for each target type, demonstrating both efficiency and efficacy. Third, fine-tuning MNLI pre-trained models (e.g., BART-MNLI-e, etc.) consistently outperform traditional ZSSD baselines (that do not have NLI pre-trained knowledge) in all three settings, showing the effectiveness of transforming ZSSD into NLI. Forth, when evaluated on noun-phrase targets, MNLI models trained with prompted noun-phrase targets consistently outperform those trained with original noun-phrase targets, indicating that our proposed prompts can effectively formulate noun-phrase targets into more refined hypotheses to better leverage the MNLI model for ZSSD. Note that when trained with claim targets, where prompts are not incorporated, BARTMNLI-ep and BART-MNLI-e are exactly the same. Our results with BERT-based and XLNet-based models (provided in Appendix G) show similar observations. Next, the BART-MNLI model without fine-tuning on EZ-STANCE performs much worse than the fine-tuned BART-MNLI encoders, particularly for the noun-phrase targets. This result demonstrates the necessity of developing a large dataset for ZSSD, so that the NLI pre-trained knowledge
can be fine-tuned and better utilized. Moreover, LLaMA 2 and ChatGPT exhibit much lower performance than fine-tuned transformer-based models, indicating that our dataset is very challenging even for advanced large language models.

heading: 6.2 Domain-based Zero-Shot Stance Detection
text: Domain-based ZSSD (subtask B) focuses on evaluating classifiers using unseen topics from new domains. One domain is selected as the zero-shot domain, and the remaining seven as source domains. Models are trained and validated on source domain data and tested on zero-shot domain data, with eight different zero-shot domain settings in total.
Table 8 shows F1macro scores for various zeroshot domain settings. Models trained on the full mixed-target dataset are evaluated across three settings: 1) the full mixed-target test set; 2) the nounphrase target-only test set; and 3) the claim targetonly test set, denoted as M, N, and C, respectively. First, we notice that models show lower performance when compared with the in-domain subtask A (see results in Table 7). This is because the domain shifts between the training and testing stages introduce additional complexity to the task, making domain-based ZSSD a more challenging ZSSD task. Second, models generally perform worse on the “Covid Epidemic” (CE) and the “Politics” (P) domain, suggesting that these two domains share less domain knowledge with other domains, making them more difficult zero-shot domains. Moreover, we observe that most models show higher performance when predicting stances for the “Rights” (R) and the “Environmental Protection” (EP) domain. BERT-based and XLNet-based models exhibit worse performance than RoBERTabased models, as illustrated in Appendix G. Classspecific performance for both Subtasks A and B is shown in Appendix H.

heading: M .437 .440 .478 .464 .475 .479 .470 .442

heading: N .442 .461 .476 .463 .442 .492 .471 .468

heading: C .423 .423 .458 .443 .437 .436 .445 .414

heading: M .501 .513 .514 .500 .509 .546 .521 .512

heading: N .475 .492 .480 .483 .473 .508 .512 .486

heading: C .518 .525 .530 .513 .505 .534 .509 .516

heading: M .568 .569 .583 .580 .572 .635 .566 .577

heading: N .531 .554 .556 .551 .563 .588 .565 .555

heading: C .589 .568 .600 .602 .572 .631 .565 .582

heading: M .345 .328 .348 .342 .273 .354 .330 .328

heading: N .375 .341 .404 .391 .353 .393 .383 .345

heading: C .322 .320 .310 .314 .240 .321 .288 .314

heading: M .485 .493 .490 .497 .506 .486 .513 .491

heading: N .576 .563 .568 .586 .586 .564 .572 .564

heading: C .422 .445 .435 .437 .448 .429 .464 .438

heading: M .738 .758 .762 .753 .765 .777 .769 .755

heading: N .597 .624 .609 .606 .609 .625 .648 .620

heading: C .826 .849 .862 .851 .848 .846 .845 .838

heading: M .597 .594 .637 .632 .671 .623 .652 .591

heading: N .307 .264 .337 .332 .369 .327 .364 .315

heading: C .778 .813 .806 .800 .814 .797 .802 .776

heading: M .768∗ .789∗ .784 .777∗ .792 .783 .791∗ .777∗

heading: N .627∗ .638∗ .615 .625∗ .625 .615 .672∗ .624

heading: C .863∗ .890∗ .888 .878 .882 .873∗ .872∗ .877∗

heading: 6.3 EZ-STANCE vs. VAST
text: We compare EZ-STANCE and VAST to understand which dataset presents more challenges. We consider the following experiments: 1) cross-dataset setting: training our best-performing BART-MNLIep using one dataset and testing the model using the other dataset, and 2) in-dataset setting: training and testing the model on the same dataset. Since VAST is designed for target-based ZSSD and includes only noun-phrase targets, we ensured a fair comparison by utilizing the dataset with noun-phrase targets from subtask A of EZ-STANCE.
Per-class and overall results are shown in Table 9. First, we observe that models show higher performance for the in-dataset setting than the crossdataset setting. Second, in the in-dataset setting, the model trained on EZ-STANCE (E→E) exhibits much lower performance for the neutral class than
its VAST-trained counterpart (V→V). The result demonstrates that data from the neutral class in EZSTANCE with close semantic correlations between documents and targets are much more challenging than in VAST, where documents and targets are randomly permuted (and do not reflect the natural/realworld data for the neutral class). Third, in the cross-dataset setting, the model trained on VAST performs extremely poorly on the neutral class of the EZ-STANCE test set (V→E), while the model trained on EZ-STANCE show much higher performance on VAST, particularly for the neutral class (E→V). This indicates that EZ-STANCE test set captures more challenging real-world ZSSD data, especially for the neutral category. This reinforces our motivation to create a new, large ZSSD dataset.

heading: 6.4 Impact of Prompt Templates
text: To assess the efficacy of various prompt templates in our proposed approach, we compare the following prompt settings for noun-phrase targets: 1) applying no prompt to noun-phrase targets; 2) using one of our proposed prompts consistently across all noun-phrase targets; and 3) our approach that randomly assigns each noun-phrase target with one of five distinct prompts. The results with our bestperforming BART-MNLI-ep model are shown in Table 10. We observe that the models trained using our random-prompt approach exhibits better performance than those trained with the singular-prompt approach or no prompts at all.

heading: 7 Conclusion
text: In this paper, we present EZ-STANCE, a large English ZSSD dataset. Compared with existing English ZSSD datasets, our dataset is larger and more challenging. EZ-STANCE covers both nounphrase targets and claim targets and also comprises two challenging ZSSD subtasks: targetbased ZSSD and domain-based ZSSD. We improve the data quality of the neutral class by extracting targets from texts. We evaluate EZ-STANCE on ZSSD baselines and propose to transform ZSSD into the NLI task which outperforms traditional baselines. We hope EZ-STANCE can facilitate future research for varied stance detection tasks.
Limitations
Our EZ-STANCE data is collected from social media. This might be perceived as a drawback as it might not encompass all facets of formal texts that could be found in essays or news comments. In the future, we aim to expand this dataset to include other types of text (e.g., from social media to research articles). Yet, this restriction is not unique to our dataset, but also affects any other datasets that concentrate on social media content.
Ethical Statement
Our dataset does not provide any personally identifiable information. Tweets are collected using generic keywords instead of user information as queries, therefore our dataset does not have a large collection of tweets from an individual user. Thus, our dataset complies with Twitter’s information privacy policy.

heading: Acknowledgements
text: We thank the National Science Foundation for support from grants IIS-2107487 and ITE-2137846 which supported the research and the computation in this study. We also thank our reviewers for their insightful feedback and comments.

heading: A More Examples of EZ-STANCE
text: In this section, we show examples of tweets with noun-phrase targets and claim targets for each domain of our EZ-STANCE dataset in Table 11.

heading: B Query Keywords and Data Filtering Strategy
text: The full keywords set that we used for data crawling is shown in Table 12. We generate the list as follows. First, we use domain names from the Explore page of Twitter as keywords for crawling (e.g., sports, education, etc.). This represents our initial set of keywords. After we collect tweets using this initial set, we tokenize the tweets and remove stop words and words with part-of-speeches other than nouns and adjectives, and form words / phrases (from the contiguous words in the text) using the regular expression adjective*noun+. These words/phrases (possibly single words if no adjectives appear in front of the nouns) represent our candidate set of keywords / keyphrases. We then calculate the frequency of these candidates from the tweet collection and rank them in descending
order of their frequency. Last, we selected the top20 keywords / keyphrases as supplementary query keywords / keyphrases to collect tweets in the next iteration. We repeat this process in multiple iterations until we collect enough data.
After we collect the initial keywords set, we perform keyword filtering in the following steps: 1) We manually detect a subset of tweets crawled using each keyword and we remove keywords that are frequently associated with promotional content (e.g., YouTuber, live shopping, etc.), whose main purpose is for product/people promotion instead of addressing controversial topics; 2) Keywords that people predominantly hold single stances on are filtered out, e.g., pollution, crime, delicious food, etc. This is because models would simply learn the correlation between the keywords and the stance and predict stances based solely on keywords instead of the content of tweets and targets.
To ensure the quality of our dataset, we then perform the following preprocessing steps: 1) We remove tweets with less than 20 or more than 150 words. According to our observations, tweets with less than 20 words are either too easy or cannot include enough information to express stances toward multiple targets. Tweets with more than 150 words usually contain links to external content; 2) We remove duplicates and retweets; 3) We keep only tweets in English; 4) We filter out tweets containing advertising contents (e.g., scan the QR code, reply or DM me, sign up, etc.); and 5) We remove emojis and URLs as they may introduce noise. We randomly select around 86 tweets for each keyword, obtaining 6204 tweets for annotation.

heading: C Annotation Instruction
text: C.1 Noun-phrase Targets
For noun-phrase targets, the annotation takes two steps. For step1, annotators are given the following instructions: From each tweet, please identify at least 2 noun-phrase targets. Targets should meet the following criteria: 1) Targets should be the principal subject of the tweet rather than minor details; 2) Targets should represent widely discussed topics where different stances are exhibited; 3) Targets where people often express the same stance should be avoided, e.g., violence abuse. In step 2, we instruct 3 annotators to assign a stance label to each tweet-target pair, using the following instructions: Imagine yourself as the author of the tweet, please annotate the stance that you would
take on this given target as “Favor”, “Against”, or “Neutral”.
C.2 Claim Targets For claim targets, annotators are provided with the following instructions: Based on the message that you learned from the tweet, write the following three claims: 1) The author is definitely in favor of the point or message of the claim (favor); 2) The author is definitely against the point or message from the claim (against); 3) Based solely on the information from the tweet, we cannot know whether the author definitely supports or opposes the point or message of the claim (neutral). To make this task more challenging, we establish some extra requirements: First, claims labeled with favor must not replicate the tweet verbatim. Second, claims labeled with against should not merely negate the tweet content (e.g., adding “not" before verbs). Models could easily detect such linguistic patterns and predict stances without learning the content of tweet-claim pairs.

heading: D Split Method
text: Initially, we randomly select x% of unique tweets for the training set and the rest as the combination of validation and test set. We then move data with overlapping targets and documents from the mixture of validation and test sets to the training set.
After this step, we may introduce some additional overlapping targets during the transaction. This is because the tweets that are moved to the training set may have other noun-phrase targets that overlap with the remaining validation and test set. Therefore we repeat this transferring procedure y times until we do not have any overlapping targets and documents between the training set and the mixture of validation and test set. In our experiments, we use x=40% and y=4, because with these parameters, 66% tweets are split into our final training (similar to VAST). We then perform similar procedures to split validation and the test set. Therefore, the training, validation, and test set do not include overlapping tweets and targets with each other.

heading: E Full Statistics of Subtask B
text: The statistics of the 8 dataset splits (data from seven domains for training and validation, and the data from the left-out domain as the zero-shot test set) are shown in Table 13.

heading: F Training Details
text: Our experiments are carried out using an NVIDIA RTX A5000 GPU based on the PyTorch (Paszke et al., 2019). Hyperparameters were fine-tuned using our validation set. The BiCE and CrossNet models were trained using AdamW (Loshchilov

heading: M .707 .731 .726 .726 .735 .752 .739 .725

heading: N .570 .610 .576 .605 .598 .626 .626 .593

heading: C .798 .811 .818 .810 .810 .811 .812 .804

heading: M .726 .740 .744 .741 .754 .761 .757 .734

heading: N .578 .595 .585 .598 .593 .606 .623 .581

heading: C .825 .835 .843 .836 .841 .841 .839 .827

heading: M .723 .741 .745 .735 .753 .755 .756 .734

heading: N .575 .596 .589 .586 .599 .604 .631 .586

heading: C .822 .836 .843 .834 .838 .841 .835 .826

heading: M .722 .741 .748 .732 .745 .762 .752 .738

heading: N .594 .625 .595 .606 .599 .622 .639 .612

heading: C .806 .819 .844 .818 .822 .827 .823 .817

heading: M .738 .772 .766 .767 .775 .776 .768 .754

heading: N .580 .622 .586 .617 .606 .627 .639 .608

heading: C .846 .873 .877 .867 .864 .851 .850 .848

heading: M .749 .765 .766 .760 .775 .767 .768 .762

heading: N .600 .609 .594 .605 .605 .604 .629 .620

heading: C .850 .868 .873 .863 .865 .854 .855 .852
text: and Hutter, 2019) as the optimizer with a learning rate of 0.001. Each model was trained for 20 epochs, with each mini-batch of size 128. As for TGA-Net, we adhered to the hyperparameters as recommended in prior research (Allaway and McKeown, 2020). The AdamW optimizer with a learning rate of 2e-5 was utilized for vanilla transformer-based models (BERT, RoBERTa, XLNet), and the NLI pre-trained models (BERTMNLIp2, RoBERTa-MNLIp3, XLNet-MNLIp4, BART-MNLI-ep5), which were fine-tuned for 4 epochs using batch size of 64. The entire training process for each model was completed within 3 hours. Each result is the average of 4 runs with dif-
2https://huggingface.co/textattack/ bert-base-uncased-MNLI
3https://huggingface.co/textattack/ roberta-base-MNLI
4https://huggingface.co/textattack/ xlnet-base-cased-MNLI
5https://huggingface.co/facebook/ bart-large-mnli
ferent initializations. For ChatGPT, we utilized the gpt-3.5-turbo-0301 version. We use the following prompt to extract stance predictions using ChatGPT: “Q: What is the stance of the text ’[tweet]’ towards the target ’[target]’? The answer should be selected from ’Favor’, ’Against’, or ’None’. A:” For LLaMA 2, we utilized the Llama-2-13b-chathf version with the following prompt: “Classify the stance that the author of the text takes towards the target into favor, against, or neutral. The answer should only be one of the following three words:
’favor’, ’against’, or ’neutral’. Don’t give further explanation other than one of these three words. Text: "[tweet]". Target: "[target]".”

heading: G Evaluations on BERT-based and XLNet-based Models for Subtask A and Subtask B
text: We also evaluate BERT-based models (i.e., BERT, BERT-MNLI, BERT-MNLIp) and XLNet-based models (i.e., XLNet, XLNet-MNLI and XLNetMNLIp). The results for subtask A and subtask B are shown in Table 14 and Table 15, respectively. First, we can observe that for both subtask A and subtask B, the models with MNLI pretraining (BERT-MNLI, XLNet-MNLI) outperform the corresponding standard BERT and XLNet models. Additionally, when our proposed prompts are applied to noun-phrase targets, BERT-MNLIp and XLNet-MNLIp demonstrate enhanced performance compared to BERT-MNLI and XLNet-MNLI, especially in experiments involving noun-phrase targets. This finding underscores the effectiveness of our prompts in improving the MNLI model with BERT and XLNet architectures.

heading: H Class-specific Performance for Subtask
text: A and Subtask B
To establish more comprehensive results, for subtask A and subtask B, we report the class-specific performance. The performance is reported using F1 score for the against (Con), favor (Pro), neutral

heading: M .514 .544 .521 .498 .443 .575 .457 .558

heading: N .492 .553 .502 .436 .212 .620 .432 .607

heading: C .524 .538 .528 .520 .477 .545 .467 .523

heading: M .500 .533 .477 .478 .452 .555 .504 .549

heading: N .541 .588 .487 .472 .309 .656 .535 .627

heading: C .470 .491 .471 .480 .479 .454 .485 .474

heading: M .553 .550 .536 .545 .497 .636 .519 .584

heading: N .598 .635 .585 .545 .455 .709 .551 .662

heading: C .521 .472 .503 .545 .507 .577 .502 .510

heading: M .329 .315 .288 .248 .119 .342 .226 .326

heading: N .407 .358 .371 .359 .276 .399 .286 .383

heading: C .277 .286 .240 .192 .082 .303 .194 .282

heading: M .567 .578 .559 .523 .474 .592 .586 .603

heading: N .640 .632 .614 .594 .520 .651 .577 .685

heading: C .516 .536 .525 .482 .460 .547 .591 .537

heading: M .695 .718 .711 .706 .722 .731 .739 .732

heading: N .633 .673 .630 .608 .506 .721 .673 .708

heading: C .741 .756 .761 .755 .770 .738 .778 .755

heading: M .727 .738 .746 .736 .765 .755 .768 .748

heading: N .654 .662 .641 .614 .513 .710 .682 .701

heading: C .786 .802 .811 .800 .823 .795 .820 .794

heading: M .721 .747 .743 .733 .765 .750 .763 .745

heading: N .641 .678 .648 .608 .517 .699 .672 .696

heading: C .781 .806 .807 .798 .821 .794 .817 .793

heading: M .720 .740 .752 .713 .740 .757 .759 .742

heading: N .673 .697 .651 .614 .493 .736 .694 .711

heading: C .758 .777 .815 .768 .793 .775 .795 .772

heading: M .741 .781 .774 .769 .794 .779 .783 .766

heading: N .640 .690 .640 .625 .511 .739 .691 .711

heading: C .818 .857 .856 .844 .858 .815 .835 .818

heading: M .754 .779 .779 .766 .796 .774 .785 .775

heading: N .664 .694 .648 .625 .522 .727 .684 .730

heading: C .824 .852 .853 .838 .856 .816 .841 .818

heading: M .745 .766 .769 .752 .768 .775 .780 .765

heading: N .694 .703 .656 .625 .522 .744 .702 .731

heading: C .789 .818 .839 .819 .826 .802 .824 .798

heading: M .769 .779 .791 .777 .793 .788 .797 .785

heading: N .688 .700 .673 .626 .540 .748 .702 .726

heading: C .832 .847 .864 .853 .857 .823 .848 .840

heading: M .764 .787 .790 .779 .794 .794 .795 .789

heading: N .687 .708 .662 .623 .534 .748 .691 .730

heading: C .827 .855 .867 .854 .860 .834 .850 .842

heading: M .574 .587 .629 .627 .735 .568 .654 .562

heading: N .224 .193 .200 .200 .290 .225 .232 .236

heading: C .761 .794 .797 .776 .822 .760 .803 .754

heading: M .779 .798 .804 .780 .820 .786 .809 .799

heading: N .692 .720 .691 .638 .542 .723 .730 .738

heading: C .844 .862 .872 .859 .876 .837 .855 .857

heading: M .786∗ .807∗ .801 .775 .812 .798∗ .812∗ .801∗

heading: N .703∗ .723∗ .682 .626 .543∗ .738 .722 .738∗

heading: C .849∗ .876∗ .871 .855 .872 .849∗ .862∗ .857∗
text: (Neu). Results are shown in Table 16 (subtask A), Table 17, Table 18, and Table 19 (subtask B).

heading: C .876 .901 .898 .892 .887 .905 .865 .891∗

heading: N .521 .577 .461 .580 .659 .389 .518∗ .497

heading: M .736 .764 .747 .758 .775 .757 .729∗ .746

heading: C .879 .894 .899 .893 .891 .908 .859 .887

heading: N .530 .574 .477 .553 .681 .398 .489 .507

heading: M .740 .760 .752 .752 .785 .759 .721 .748

heading: C .764 .792 .778 .789 .783 .801 .766 .751

heading: N .235 .226 .282 .295 .214 .195 .224 .272

heading: M .576 .572 .599 .600 .567 .613 .580 .579

heading: C .880 .898 .896 .896 .871 .910 .861 .879

heading: N .479 .560 .459 .597 .625 .431 .451 .507

heading: M .729 .757 .744 .764 .755 .768 .710 .740

heading: C .879 .899 .893 .894 .870 .905 .859 .881

heading: N .540 .588 .491 .598 .652 .434 .492 .506

heading: M .741 .766 .750 .764 .764 .770 .719 .742

heading: C .868 .881 .882 .884 .866 .902 .845 .877

heading: N .461 .575 .496 .577 .660 .388 .469 .500

heading: M .719 .749 .739 .750 .763 .764 .707 .744

heading: C .875 .889 .888 .886 .872 .908 .850 .878

heading: N .510 .552 .451 .559 .614 .389 .435 .501

heading: M .727 .747 .734 .745 .750 .754 .702 .740

heading: C .871 .895 .890 .889 .867 .899 .847 .873

heading: N .503 .596 .436 .585 .636 .416 .465 .498

heading: M .721 .763 .737 .755 .756 .761 .708 .737

heading: C .857 .872 .879 .874 .854 .898 .842 .868

heading: N .495 .596 .465 .580 .634 .399 .459 .518

heading: M .716 .751 .733 .743 .748 .760 .699 .737

heading: C .864 .873 .876 .867 .854 .901 .842 .860

heading: N .492 .562 .458 .521 .628 .428 .459 .470

heading: M .715 .740 .731 .724 .743 .751 .699 .721

heading: C .863 .873 .875 .869 .858 .899 .842 .864

heading: N .487 .578 .459 .552 .643 .408 .427 .455

heading: M .715 .742 .729 .736 .751 .760 .698 .724

heading: C .860 .869 .872 .874 .847 .891 .832 .867

heading: N .492 .586 .431 .595 .643 .429 .454 .469

heading: M .713 .745 .720 .750 .746 .758 .689 .727

heading: C .342 .399 .336 .423 .411 .281 .286 .312

heading: N .498 .538 .455 .558 .598 .367 .425 .432

heading: M .405 .457 .383 .478 .491 .311 .339 .360

heading: C .224 .195 .214 .267 .142 .184 .182 .178

heading: N .199 .231 .234 .245 .140 .153 .183 .163

heading: M .215 .210 .221 .258 .141 .174 .182 .172

heading: C .670 .663 .687 .679 .631 .729 .637 .664

heading: N .431 .563 .475 .549 .586 .354 .488 .461

heading: M .579 .618 .602 .621 .610 .625 .573 .588

heading: C .606 .614 .610 .609 .575 .650 .582 .617

heading: N .465 .491 .371 .498 .573 .265 .360 .387

heading: M .544 .559 .524 .558 .575 .539 .501 .531

heading: C .556 .577 .561 .563 .539 .583 .543 .581

heading: N .465 .491 .371 .498 .573 .265 .360 .387

heading: M .496 .520 .489 .505 .552 .491 .474 .513

heading: Future Work
text: We hope EZ-STANCE can facilitate future research for varied stance detection tasks. Limitations
Our EZ-STANCE data is collected from social media. This might be perceived as a drawback as it might not encompass all facets of formal texts that could be found in essays or news comments. In the future, we aim to expand this dataset to include other types of text (e. g. , from social media to research articles). Yet, this restriction is not unique to our dataset, but also affects any other datasets that concentrate on social media content