Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection
Year: 2023

Abstract:
Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose Topic Efficient StancE Detection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier. We evaluate the method on an existing benchmark of 16 datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of 3.5 F1 points increase in-domain, and is more generalizable with an averaged increase of 10.2 F1 on out-of-domain evaluation while using ≤ 10% of the training data. We show that our sampling technique mitigates both interand per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels.

heading: 1 Introduction
text: The goal of stance detection is to identify the viewpoint expressed by an author within a piece of text towards a designated topic (Mohammad et al., 2016). Such analyses can be used in a variety of domains ranging from identifying claims within political or ideological debates (Somasundaran and Wiebe, 2010; Thomas et al., 2006), identifying mis- and disinformation (Hanselowski et al.,
2018; Hardalov et al., 2022a), public health policymaking (Glandt et al., 2021; Hossain et al., 2020; Osnabrügge et al., 2023), news recommendation (Reuver et al., 2021) to investigating attitudes voiced on social media (Qazvinian et al., 2011; Augenstein et al., 2016; Conforti et al., 2020). However, in most domains, and even more so for crossdomain stance detection, the exact formalisation of the task gets blurry, with varying label sets and their corresponding definitions, data collection protocols and available annotations. Furthermore, this is accompanied by significant changes in the topicspecific vocabulary (Somasundaran and Wiebe, 2010; Wei and Mao, 2019), text style (Pomerleau and Rao, 2017; Ferreira and Vlachos, 2016) and topics mentioned either explicitly (Qazvinian et al., 2011; Walker et al., 2012) or implicitly (Hasan and Ng, 2013; Derczynski et al., 2017). Recently, a benchmark of 16 datasets (Hardalov et al., 2021) covering a variety of domains and topics has been proposed for testing stance detection models across multiple domains. It must be noted that these datasets are highly imbalanced, with an imbalanced label distribution between the covered topics, i.e. inter-topic and within each topic, i.e. per-topic, as can be seen in Figure 2 and Figure 3. This further complicates the creation of a robust stance detection classifier.
Given the inherent skew present within the dataset and variances within each domain, we propose a topic-guided diversity sampling method, which produces a data-efficient representative subset while mitigating label imbalances. These samples are used for fine-tuning a Pre-trained Language Model (PLM), using a contrastive learning objective to create a robust stance detection model. These two components form our Topic Efficient StancE Detection (TESTED) framework, as seen in Figure 1, and are analysed separately to pinpoint the factors impacting model performance and robustness. We test our method on
13448
the multi-domain stance detection benchmark by Hardalov et al. (2021), achieving state-of-the-art results with both in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics evaluations. Note though that TESTED could be applied to any text classification setting.
In summary, our contributions are:
• We propose a novel framework (TESTED) for predicting stances across various domains, with data-efficient sampling and contrastive learning objective; • Our proposed method achieves SOTA results both in-domain and out-of-domain; • Our analysis shows that our topic-guided sampling method mitigates dataset imbalances while accounting for better performance than other sampling techniques; • The analysis shows that the contrastive learning objective boosts the ability of the classifier to differentiate varying topics and stances.

heading: 2 Related Work
text: Stance Detection is an NLP task which aims to identify an author’s attitude towards a particular topic or claim. The task has been widely explored in the context of mis- and disinformation detection (Ferreira and Vlachos, 2016; Hanselowski et al., 2018; Zubiaga et al., 2018b; Hardalov et al., 2022a), sentiment analysis (Mohammad et al., 2017; Aldayel and Magdy, 2019) and argument mining (Boltužić and Šnajder, 2014; Sobhani et al., 2015; Wang et al., 2019). Most papers formally define stance detection as a pairwise sequence classification where stance targets are provided (Küçük and Can, 2020). However, with the emergence of differ-
ent data sources, ranging from debating platforms (Somasundaran and Wiebe, 2010; Hasan and Ng, 2014; Aharoni et al., 2014) to social media (Mohammad et al., 2016; Derczynski et al., 2017), and new applications (Zubiaga et al., 2018a; Hardalov et al., 2022a), this formal definition has been subject to variations w.r.t. the label dictionary inferred for the task.
Previous research has predominantly focused on a specific dataset or domain of interest, outside of a few exceptions like multi-target (Sobhani et al., 2017; Wei et al., 2018) and cross-lingual (Hardalov et al., 2022b) stance detection. In contrast, our work focuses on multi-domain stance detection, while evaluating in- and out-of-domain on a 16 dataset benchmark with state-of-the-art baselines (Hardalov et al., 2021).
Topic Sampling Our line of research is closely associated with diversity (Ren et al., 2021) and importance (Beygelzimer et al., 2009) sampling and their applications in natural language processing (Zhu et al., 2008; Zhou and Lampouras, 2021). Clustering-based sampling approaches have been used for automatic speech recognition (Syed et al., 2016), image classification (Ranganathan et al., 2017; Yan et al., 2022) and semi-supervised active learning (Buchert et al., 2022) with limited use for textual data (Yang et al., 2014) through topic modelling (Blei et al., 2001). This research proposes an importance-weighted topic-guided diversity sampling method that utilises deep topic models, for mitigating inherent imbalances present in the data, while preserving relevant examples.
Contrastive Learning has been used for tasks where the expected feature representations should be able to differentiate between similar and divergent inputs (Liu et al., 2021; Rethmeier and Augenstein, 2023). Such methods have been used for image classification (Khosla et al., 2020), captioning (Dai and Lin, 2017) and textual representations (Giorgi et al., 2021; Jaiswal et al., 2020; Ostendorff et al., 2022). The diversity of topics (Qazvinian et al., 2011; Walker et al., 2012; Hasan and Ng, 2013), vocabulary (Somasundaran and Wiebe, 2010; Wei and Mao, 2019) and expression styles (Pomerleau and Rao, 2017) common for stance detection can be tackled with contrastive objectives, as seen for similar sentence embedding and classification tasks (Gao et al., 2021; Yan et al., 2021).

heading: 3 Datasets
text: Our study uses an existing multi-domain dataset benchmark (Hardalov et al., 2021), consisting of 16 individual datasets split into four source groups: Debates, News, Social Media, Various. The categories include datasets about debating and political claims including arc (Hanselowski et al., 2018; Habernal et al., 2018), iac1 (Walker et al., 2012), perspectum (Chen et al., 2019), poldeb (Somasundaran and Wiebe, 2010), scd (Hasan and Ng, 2013), news like emergent (Ferreira and Vlachos, 2016), fnc1 (Pomerleau and Rao, 2017), snopes (Hanselowski et al., 2019), social media like mtsd (Sobhani et al., 2017), rumour (Qazvinian et al., 2011), semeval2016t6 (Mohammad et al., 2016), semeval2019t7 (Derczynski et al., 2017), wtwt (Conforti et al., 2020) and datasets that cover a variety of diverse topics like argmin (Stab et al., 2018), ibmcs (Bar-Haim et al., 2017) and vast (Allaway and McKeown, 2020). Overall statistics for all of the datasets can be seen in Appendix C.

heading: 3.1 Data Standardisation
text: As the above-mentioned stance datasets from different domains possess different label inventories, the stance detection benchmark by Hardalov et al. (2021) introduce a mapping strategy to make the class inventory homogeneous. We adopt that same mapping for a fair comparison with prior work, shown in Appendix C.

heading: 4 Methods
text: Our goal is to create a stance detection method that performs strongly on the topics known during training and can generalize to unseen topics. The benchmark by Hardalov et al. (2021) consisting of 16 datasets is highly imbalanced w.r.t the intertopic frequency and per-topic label distribution, as seen in Figure 2.
These limitations necessitate a novel experimental pipeline. The first component of the pipeline we propose is an importance-weighted topic-guided diversity sampling method that allows the creation of supervised training sets while mitigating the inherent imbalances in the data. We then create a stance detection model by fine-tuning a Pre-trained Language Model (PLM) using a contrastive objective.

heading: 4.1 Topic-Efficient Sampling
text: We follow the setting in prior work on data-efficient sampling (Buchert et al., 2022; Yan et al., 2022), framing the task as a selection process between multi-domain examples w.r.t the theme discussed within the text and its stance. This means that given a set of datasets D = (D1, . . .Dn) with their designated documents Di = (d1i , . . . dmi ), we wish to select a set of diverse representative examples Dtrain, that are balanced w.r.t the provided topics T = (t1, . . . tq) and stance labels L = (l1, . . . lk).
Diversity Sampling via Topic Modeling We thus opt for using topic modelling to produce a supervised subset from all multi-domain datasets. Selecting annotated examples during task-specific fine-tuning is a challenging task (Shao et al., 2019), explored extensively within active learning research (Hino, 2020; Konyushkova et al., 2017). Random sampling can lead to poor generalization and knowledge transfer within the novel problem domain (Das et al., 2021; Perez et al., 2021). To mitigate the inconsistency caused by choosing suboptimal examples, we propose using deep unsupervised topic models, which allow us to sample relevant examples for each topic of interest. We further enhance the model with an importance-weighted diverse example selection process (Shao et al., 2019; Yang et al., 2015) within the relevant examples generated by the topic model. The diversity maximisation sampling is modeled similarly to Yang et al. (2015).
The topic model we train is based on the technique proposed by Angelov (2020) that tries to find topic vectors while jointly learning document and word semantic embeddings. The topic model is initialized with weights from the all-MiniLM-L6 PLM, which has a strong performance on sentence embedding benchmarks (Wang et al., 2020). It is shown that learning unsupervised topics in this fashion maximizes the total information gained, about all texts D when described by all wordsW .
I(D,W) = ∑
d∈D
∑
w∈W P (d,w) log
( P (d,w)
P (d)P (w)
)
This characteristic is handy for finding relevant samples across varying topics, allowing us to search within the learned documents di. We train a deep topic modelMtopic using multi-domain data D and obtain topic clusters C = (Ci, . . . Ct),
Algorithm 1 Topic Efficient Sampling Require: S ≥ 0 ▷ Sampling Threshold Require: Avg ∈ {moving, exp} Ensure: |C| > 0 Dtrain ← {} I ← { |C1|∑
Ci∈C Ci . . . |Ct|∑ Ci∈C Ci } ▷ Cluster Importances
for Ci ∈ C do ▷ Iterating for each cluster Ei ← {PLM(d1i ) . . . } = {e1i . . . emi } si ← max(1, S · Ii) ▷ Threshold per cluster j← 0
cent0 ← ∑ ei∈E ei
|E| ▷ Centroid of the cluster while j ≤ si do
sim = ⟨E,cent⟩∥E∥∥cent∥ ▷ Similarity Ranking sample = arg sort(sim,Ascending)[0]
▷ Take the sample most diverse from the centroid Dtrain ← Dtrain ∪ sample j← j + 1 centj ← { α · esample + (1− α) · centj−1 exp (j−1)
j · centj + esample
j moving ▷ Centroid update w.r.t. sampled data
end while end for return Dtrain
where |C| = t is the number of topic clusters. We obtain the vector representation for ∀di from the tuned PLM embeddings E = (e1, . . . em) in Mtopic, while iteratively traversing through the clusters Ci ∈ C.
Our sampling process selects increasingly more diverse samples after each iteration. This search within the relevant examples is presented in Algorithm 1. This algorithm selects a set of diverse samples from the given multi-domain datasets D, using the clusters from a deep topic modelMtopic and the sentence embeddings E of the sentences as a basis for comparison. The algorithm starts by selecting a random sentence as the first diverse sample and uses this sentence to calculate a “centroid” embedding. It then iteratively selects the next most dissimilar sentence to the current centroid, until the desired number of diverse samples is obtained.

heading: 4.2 Topic-Guided Stance Detection
text: Task Formalization Given the topic, ti for each document di in the generated set Dtrain we aim to classify the stance expressed within that text towards the topic. For a fair comparison with prior work, we use the label mapping from the
previous multi-domain benchmark (Hardalov et al., 2021) and standardise the original labels L into a five-way stance classification setting, S = {Positive, Negative, Discuss, Other, Neutral}. Stance detection can be generalized as pairwise sequence classification, where a model learns a mapping f : (di, ti)→ S. We combine the textual sequences with the stance labels to learn this mapping. The combination is implemented using a simple prompt commonly used for NLI tasks (Lan et al., 2020; Raffel et al., 2020; Hambardzumyan et al., 2021), where the textual sequence becomes the premise and the topic the hypothesis.
[CLS] premise: premise
hypothesis: topic [EOS]
The result of this process is a supervised dataset for stance prediction Dtrain = ((Prompt(d1, t1), s1) . . . (Prompt(dn, tn), sn)) where ∀si ∈ S. This method allows for dataefficient sampling, as we at most sample 10% of the data while preserving the diversity and relevance of the selected samples. The versatility of the method allows TESTED to be applied to any text classification setting.
Tuning with a Contrastive Objective After obtaining the multi-domain supervised training set Dtrain, we decided to leverage the robustness of PLMs, based on a transformer architecture (Vaswani et al., 2017) and fine-tune on Dtrain with a single classification head. This effectively allows us to transfer the knowledge embedded within the PLM onto our problem domain. For standard finetuning of the stance detection modelMstance we use cross-entropy as our initial loss:
LCE = − ∑
i∈S yi log (Mstance(di)) (1)
Here yi is the ground truth label. However, as we operate in a multi-domain setting, with variations in writing vocabulary, style and covered topics, it is necessary to train a model where similar sentences have a homogeneous representation within the embedding space while keeping contrastive pairs distant. We propose a new contrastive objective based on the cosine distance between the samples to accomplish this. In each training batch B = (d1, . . . db), we create a matrix of contrastive pairs P ∈ Rb×b, where ∀i, j = 1, b, Pij = 1
if i-th and j-th examples share the same label and −1 otherwise. The matrices can be precomputed during dataset creation, thus not adding to the computational complexity of the training process. We formulate our pairwise contrastive objective LCL(xi, xj ,Pij) using matrix P .
LCL = { e(1− ecos(xi,xj)−1),Pij = 1 emax(0,cos(xi,xj)−β) − 1,Pij = −1
(2)
Here xi, xj are the vector representations of examples di, dj . The loss is similar to cosine embedding loss and soft triplet loss (Barz and Denzler, 2020; Qian et al., 2019); however, it penalizes the opposing pairs harsher because of the exponential nature, but does not suffer from computational instability as the values are bounded in the range [0, e− 1e ]. The final loss is:
L = LCE + LCL (3)
We use the fine-tuning method from Mosbach et al. (2021); Liu et al. (2019) to avoid the instability caused by catastrophic forgetting, small-sized fine-tuning datasets or optimization difficulties.

heading: 5 Experimental Setup

heading: 5.1 Evaluation
text: We evaluate our method on the 16 dataset multidomain benchmark and the baselines proposed by Hardalov et al. (2021). To directly compare with prior work, we use the same set of evaluation metrics: macro averaged F1, precision, recall and accuracy.

heading: 5.2 Model Details
text: We explore several PLM transformer architectures within our training and classification pipelines in order to evaluate the stability of the proposed technique. We opt to finetune a pre-trained robertalarge architecture (Liu et al., 2019; Conneau et al., 2020). For fine-tuning, we use the method introduced by Mosbach et al. (2021), by adding a linear warmup on the initial 10% of the iteration raising the learning rate to 2e−5 and decreasing it to 0 afterwards. We use a weight decay of λ = 0.01 and train for 3 epochs with global gradient clipping on the stance detection task. We further show that learning for longer epochs does not yield sizeable improvement over the initial fine-tuning. The optimizer used for experimentation is an AdamW
(Loshchilov and Hutter, 2019) with a bias correction component added to stabilise the experimentation (Mosbach et al., 2021).
Topic Efficiency Recall that we introduce a topicguided diversity sampling method within TESTED, which allows us to pick relevant samples per topic and class for further fine-tuning. We evaluate its effectiveness by fine-tuning PLMs on the examples it generates and comparing it with training on a random stratified sample of the same size.

heading: 6 Results and Analysis
text: In this section, we discuss and analyze our results, while comparing the performance of the method against the current state-of-the-art (Hardalov et al., 2021) and providing an analysis of the topic efficient sampling and the contrastive objective.

heading: 6.1 Stance Detection
text: In-domain We train on our topic-efficient subset Dtrain and test the method on all datasets D in the multi-domain benchmark. Our method TESTED is compared to MoLE (Hardalov et al., 2021), a strong baseline and the current state-of-the-art on the benchmark. The results, presented in Table 1, show that TESTED has the highest average performance on in-domain experiments with an increase of 3.5 F1 points over MoLE, all while using≤ 10% of the amount of training data in our subset Dtrain sampled from the whole dataset D. Our method is able to outperform all the baselines on 10 out of 16 datasets. On the remaining 6 datasets the maximum absolute difference between TESTED and MoLE is 1.1 points in F1. We also present ablations for TESTED, by replacing the proposed sampling method with other alternatives, removing the contrastive objective or both simultaneously. Replacing Topic Efficient sampling with either Random or Stratified selections deteriorates the results for all datasets with an average decrease of 8 and 5 F1 points, respectively. We attribute this to the inability of other sampling techniques to maintain inter-topic distribution and per-topic label distributions balanced while selecting diverse samples. We further analyse how our sampling technique tackles these tasks in subsection 6.2. We also see that removing the contrastive loss also results in a deteriorated performance across all the datasets with an average decrease of 3 F1 points. In particular, we see a more significant decrease in datasets with similar topics and textual expressions, i.e. poldeb
and semeval16, meaning that learning to differentiate between contrastive pairs is essential within this task. We analyse the effect of the contrastive training objective further in subsection 6.4.
Out-of-domain In the out-of-domain evaluation, we leave one dataset out of the training process for subsequent testing. We present the results of TESTED in Table 2, showing that it is able to overperform over the previous state-of-the-art significantly. The metrics in each column of Table 2 show the results for each dataset held out from training and only evaluated on. Our method records an increased performance on 13 of 16 datasets, with an averaged increase of 10.2 F1 points over MoLE, which is a significantly more pronounced increase than for the in-domain setting, demonstrating that the strength of TESTED lies in better outof-domain generalisation. We can also confirm that replacing the sampling technique or removing the contrastive loss results in lower performance across all datasets, with decreases of 9 and 5 F1 points respectively. This effect is even more pronounced compared to the in-domain experiments, as adapting to unseen domains and topics is facilitated by diverse samples with a balanced label distribution.

heading: 6.2 Imbalance Mitigation Through Sampling
text: Inter-Topic To investigate the inter-topic imbalances, we look at the topic distribution for the top 20 most frequent topics covered in the complete multi-domain dataset D, which accounts for ≥ 40% of the overall data. As we can see in Figure 2, even the most frequent topics greatly vary in their representation frequency, with σ = 4093.55, where σ is the standard deviation between represented amounts. For the training dataset Dtrain, by contrast, the standard deviation between the topics is much smaller σ = 63.59. This can be attributed to the fact that Dtrain constitutes ≤ 10% of D, thus we also show the aggregated data distributions in Figure 2. For a more systematic analysis, we employ the two sample KolmogorovSmirnov (KS) test (Massey, 1951), to compare topic distributions in D and Dtrain for each dataset present in D. The test compares the cumulative distributions (CDF) of the two groups, in terms of their maximum-absolute difference, stat = supx |F1(x)− F2(x)|.
The results in Table 3 show that the topic distribution within the full and sampled data D, Dtrain, cannot be the same for most of the datasets. The results for the maximum-absolute difference also show that with at least 0.4 difference in CDF, the
sampled dataset Dtrain on average has a more balanced topic distribution. The analysis in Figure 2 and Table 3, show that the sampling technique is able to mitigate the inter-topic imbalances present in D. A more in-depth analysis for each dataset is provided in Appendix A.
Per-topic For the per-topic imbalance analysis, we complete similar steps to the inter-topic analysis, with the difference that we iterate over the top 20 frequent topics looking at label imbalances within each topic. We examine the label distribution for the top 20 topics for a per-topic comparison. The standard deviation in label distributions averaged across those 20 topics is σ = 591.05 for the whole dataset D and the sampled set Dtrain σ = 11.7. This can be attributed to the stratified manner of our sampling technique. This is also
evident from Figure 3, which portrays the overall label distribution in D and Dtrain.
To investigate the difference in label distribution for each of the top 20 topics in D, we use the KS test, presented in Table 4. For most topics, we see that the label samples in D and Dtrain cannot come from the same distribution. This means that the per-topic label distribution in the sampled dataset Dtrain, does not possess the same imbalances present in D.
We can also see the normalized standard deviation for the label distribution within Dtrain is lower than inD, as shown in Figure 4. This reinforces the finding that per-topic label distributions in the sampled dataset are more uniform. For complete pertopic results, we refer the reader to Appendix A.
Performance Using our topic-efficient sampling method is highly beneficial for in- and out-ofdomain experiments, presented in Table 1 and Table 2. Our sampling method can select diverse and representative examples while outperforming Random and Stratified sampling techniques by 8 and 5 F1 points on average. This performance can be attributed to the mitigated inter- and per-topic
imbalance in Dtrain.

heading: 6.3 Data Efficiency
text: TESTED allows for sampling topic-efficient, diverse and representative samples while preserving the balance of topics and labels. This enables the training of data-efficient models for stance detection while avoiding redundant or noisy samples. We analyse the data efficiency of our method by training on datasets with sizes [1%, 15%] compared to the overall data size |D|, sampled using our technique. Results for the in-domain setting in terms of averaged F1 scores for each sampled dataset size are shown in Figure 5. One can observe a steady performance increase with the more selected samples, but diminishing returns from the 10% point onwards. This leads us to use 10% as the optimal threshold for our sampling process, reinforcing the data-efficient nature of TESTED.

heading: 6.4 Contrastive Objective Analysis
text: To analyse the effect of the contrastive loss, we sample 200 unseen instances stratified across each dataset and compare the sentence representations before and after training. To compare the representations, we reduce the dimension of the embeddings with t-SNE and cluster them with standard K-means. We see in Figure 6 that using the objective allows for segmenting contrastive examples in a more pronounced way. The cluster purity also massively rises from 0.312 to 0.776 after training with the contrastive loss. This allows the stance detection model to differentiate and reason over the contrastive samples with greater confidence.

heading: 7 Conclusions
text: We proposed TESTED, a novel end-to-end framework for multi-domain stance detection. The method consists of a data-efficient topic-guided sampling module, that mitigates the imbalances inherent in the data while selecting diverse examples, and a stance detection model with a contrastive training objective. TESTED yields significant performance gains compared to strong baselines on indomain experiments, but in particular generalises well on out-of-domain topics, achieving a 10.2 F1 point improvement over the state of the art, all
while using ≤ 10% of the training data. While in this paper, we have evaluated TESTED on stance detection, the method is applicable to text classification more broadly, which we plan to investigate in more depth in future work.
Limitations
Our framework currently only supports English, thus not allowing us to complete a cross-lingual study. Future work should focus on extending this study to a multilingual setup. Our method is evaluated on a 16 dataset stance benchmark, where some domains bear similarities. The benchmark should be extended and analyzed further to find independent datasets with varying domains and minimal similarities, allowing for a more granular out-ofdomain evaluation.

heading: Acknowledgements
text: This research is funded by a DFF Sapere Aude research leader grant under grant agreement No 0171-00034B, as well as supported by the Pioneer Centre for AI, DNRF grant number P1.

heading: B Evaluation Metrics
text: To evaluate our models and have a fair comparison with the introduced benchmarks we use a standard set of metrics for classification tasks such as macroaveraged F1, precision, recall and accuracy.
Acc = TP + TN
TP + TN + FP + FN (4)
Prec = TP
TP + FP (5)
Recall = TP
TP + FN (6)
F1 = 2 ∗ Prec ∗Recall Prec+Recall = 2 ∗ TP
2 ∗ TP + FP + FN (7)

heading: C Dataset Statistics
text: We use a stance detection benchmark (Hardalov et al., 2021) whose data statistics are shown in Table 5. The label mapping employed is shown in Table 6.

heading: D TESTED with different backbones
text: We chose to employ different PLM’s as the backbone for TESTED and report the results in the Table 7. The PLMs are taken from the set of robertabase, roberta-large, xlm-roberta-base, xlm-robertalarge. The differences between models with a similar number of parameters are marginal. We can
Label Description
Positive agree, argument for, for, pro, favor, support, endorse Negative disagree, argument against, against, anti, con, undermine, deny, refute Discuss discuss, observing, question, query, comment Other unrelated, none, comment Neutral neutral
Table 6: Hard stance label mapping employed in this paper, following the stance detection benchmark by Hardalov et al. (2021).
see a degradation of the F1 score between the base and large versions of the models, which can be attributed to the expressiveness the models possess. We also experiment with the distilled version of the model and can confirm that in terms of the final F1 score, it works on par with the larger models. This shows that we can utilise smaller and more computationally efficient models within the task with marginal degradation in overall performance.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix D
C 3 Did you run computational experiments? 6
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We use standard pre-trained language models.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: While in this paper, we have evaluated TESTED on stance detection, the method is applicable to text classification more broadly, which we plan to investigate in more depth in future work. Limitations
Our framework currently only supports English, thus not allowing us to complete a cross-lingual study. Future work should focus on extending this study to a multilingual setup. Our method is evaluated on a 16 dataset stance benchmark, where some domains bear similarities. The benchmark should be extended and analyzed further to find independent datasets with varying domains and minimal similarities, allowing for a more granular out-ofdomain evaluation