TATA: Stance Detection via Topic-Agnostic and Topic-Aware Embeddings
Authors: Hans William Alexander Hanley
Zakir Durumeric
Venue: EMNLP 2023 Main (2023)
Keywords: Zero-shot
Few-shot
Stance Detection
Topic-Agnostic
Topic-Aware
TL;DR: We utilize topic-agnostic and topic-aware embeddings to perform zero-shot stance detection.

Abstract: Stance detection is important for understanding different attitudes and beliefs on the Internet. However, given that a passage's stance toward a given topic is often highly dependent on that topic, building a stance detection model that generalizes to unseen topics is difficult. In this work, we propose using contrastive learning as well as an unlabeled dataset of news articles that cover a variety of different topics to train topic-agnostic/TAG and topic-aware/TAW embeddings for use in downstream stance detection. Combining these embeddings in our full TATA model, we achieve state-of-the-art performance across several public stance detection datasets (0.771 $F_1$-score on the Zero-shot VAST dataset). We release our code and data at https://github.com/hanshanley/tata.

TATA: Stance Detection via Topic-Agnostic and Topic-Aware Embeddings
Hans W. A. Hanley
Zakir Durumeric Stanford University
Stanford University
hhanley@cs. stanford.edi
.ewiirrrssstaaffffeeeoe edu
Abstract
and on topies for which there are few training ex- amples (Few-shot stance detection). In this work, building on these insights, we create and release synthetic datasets for designing topic- embeddings and generalized lopic-agnostic/TAG stance embeddings for use in downstream stance classification tasks. Taking advantage of both the topic-aware and topic-agnostic perspectives and embeddings, our full TATA model achieves state- of-the-art performance in both the Zero-shot and Few-shot stance classification settings on the VAST benchmark dataset (Allaway and McKeown, 2020).
Stance detection is important for understanding different attitudes and beliefs on the Internet. However, given that a passage's stance toward a given topic often highly dependent on that topic. building stance detection model that generalizes to unseen topics is difficult. In this work, we propose using contrastive learning well unlabeled dataset of news arti- cles that cover a variety of different topics to train topic-agnostic/TAG and lopic-aware/TAW embeddings for use in downstream stance de- tection. Combining these embeddings in our full TATA model, we achieve state-offthe-art performance across several public stance detec- datasets (0.771 Pi-score on the Zero-shot VAST dataset). We release our code and data at https://gi thub. cww/ hwtmm//ansaaneey/eeeae tata.
Specifically, utilizing an unstructured news ar- ticle dataset (Hanley and Durumeric, 2023) and extracting topics from within particular passages using the T5-Flan language model (Chung et al., 2022), we train topic-aware/TAW embedding layer with triplet loss (Hermans al., 2017). By using triplet loss, force training elements with similar topics to have similar embeddings and train- ing elements with different topies to have dissim- ilar embeddings. By using an unlabelled dataset of news articles with a diverse and wide set of au- tomatically extracted topics when creating these topic embeddings, rather than simply relying on the set of topics with expensive and carefully la- beled stance datasets in other works (Allaway and McKeown, 2020; Liang al., 2022b), enables these topic embeddings to incorporate relationships between many different topics.
1 Introduction
Stance detection is the task of determining the at- titude of a piece of text with regard to a partieular topic or target. Stance detection is often utilized to gauge public opinion toward products, political is- sues, or public figures. For example, the statement "I think Abraham Lincoln would make a good Presi- dent" would have Pro stance concerning the topic of Abraham Lincoin. Simultaneously. the same text would be Neutral to the topic of Fruits.
Several works have built topic-specific stance classifiers that have achieved promising results for particular targets (Mohtarami et al., 2018; Xu et al., 2018); however, these same approaches have often failed to generalize to topics outside of their train- ing domain, limiting their real-world use. Recent works have sought to alleviate this issue by utilizing general stance features (Liang et al., 2022a), com- mon sense reasoning (Liu et al., 2021a), and gen- eralized topic representations (Allaway and McK- eown, 2020, 2023; Liang al., 2022b), achieving better performance in predicting stance on previ- ously unseen topics (Zero-shot stance detection)
Similarly, for our topic-agnostic/TAG embed- dings, after first extending the original VAST train- ing dataset with paraphrased versions of the origi- nal texts, we utilize contrastive learning objectives to then extract generalized stance features (Saun- shi et al., 2019). By using contrastive learning. force training instances with the same stance to have similar embeddings while simultancously forcing training examples with different stances to have different embeddings. This enables our TAG layer to model general stance features that

---

are important for classifying the stance in given example.
on previously unseen topics. Li and Yuan (2022), utilized generative models (e.g., GPT2) create synthetic data to engineer training data for unseen topics. Finally, Liang et al. (2022b) further pro- posed target-aware graph contrastive learning to learn relationships between topic- representa- tions for downstream stance classification. Constrastive Learning. Contrastive learning, which seeks to build representations by differenti- ating similarly and differently labeled inputs, has become an increasingly popular version of self- supervised learning (Liu et al., 2021b; Rethmeier and Augenstein, 2023; Gao et al., 2021). Con- trastive learning is often utilized to build robust representations that can then be utilized in down- stream tasks (Gunel et al., 2020) including gener- ating sentence embeddings (Gao et al., 2021; Wu et al., 2022), image classification (Park et al., 2020; Wang and Qi, 2022), and image captioning (Dai and Lin, 2017). As in our work, contrastive learn- ing has also increasingly been utilized to model stance features (Liang et al., 2022a,b; Mohtarami et al., 2019). For example, Liang et al. (2022b) utilized in- contrastive learning to differenti- ate between examples of stance classes while also building a graph of prototypical topies train topic- specific features for downstream stance detection.
Combining both the topic-awarc/TAW and topic- agnostic/TAG perspectives into one model TATA, we achieve state-of-the-art performance on the VAST dataset in both Zero-shot (0.771 Fi-score compared to 0.723 in prior work (Liang et al., 2022b)) and Few-shot (0.741 Fi-score compared to 0.715 in prior work) settings. Our TATA model further achieves competitive results on the SEM1 6t6 dataset (Mohammad et al., 2016) in the Zero-shot setting, By training on synthetic datasets, taking ad- vantage of stance features that are common across different topics/TAG, and deriving specific features particular to given topies utilizing rich embedding latent/TAW, our TATA model thus achieves state- of-the-art performance on current benchmarks for stance classification.
2 Related Work and Background
Stance Detetection. Biber and Finegan define stance as "the expression of speaker's attitude, standpoint, and judgment" toward topic (1988). In the most common implementation, stance detec- tion tasks consist of inputting a passage-topic pair (where the passage is a longer text and the topic is a noun phrase) and outputting the stance from among { (Pro, Against, Neutral) of that passage toward that given topic. Stance detection, in addition to understanding political attitudes (Darwish et al., 2020), has been utilized to improve fact check- ing (Dulhanty et al., 2019; Si et al., 2021; Bekoulis al., 2021), and to gauge general public opin- ion (Altraaeeff et al., 2023).
3 Topic-Aware/TAW Dataset
As previously stated, within this work, we seek build informative topic-aware/TAW embeddings that can then be utilized in downstream stance de- tection. As such to pre-train these embeddings, we collect new dataset that includes a wide range of topics. This new dataset, designed for creating topic-aware embeddings. consists of sets of quadru- plets such that Drow {ar; (pi, fu1, 42, 7:) ) where ps is a passage, tu is p corresponding topic, 1.2 a paraphrase of the topic t,,, and g: is a semantically similar passage to pi.
Given its real-world applications, several works have sought to improve upon domain-specific and baseline stance-detection methods to identify stances in Few-shot settings (where the classifier is evaluated on large number of topies for which it had few training examples) and Zero-shot sce- narios (where the classifier is evaluated on new, previously seen topics) (Liang et al., 2022a; All- away and McKeown, 2020, 2023). For example, in addition to developing one of the most varied benchmark datasets for evaluating sero-shot stance detection VAST, Allaway and McKeown (2020) also pioneered the use of generalized topic representa- tions to perform Zero-shot and Few-Shot stance detection. More recently, Zhu et al. (2022), im- proving upon prior work, utilized targeted outside knowledge to add information for stance detection
3.1 Passage Selection and Noisy Topic Identification
To first build a dataset of passages that contain wide-ranging multifaceted topics, we make use of dataset from Hanley and Durumeric (2023) of 15. 50 million news articles collected between Jan- uary 1, 2022, and May 2023, from 3,074 news sites. From these articles, we randomly subselect a set of 100,000 articles (with at most 1000 articles from any given single website). For each article in our new set, we take the first paragraph (up to

---

100 words) for inclusion as passages within our TAW dataset. In order to use this set of passages to pre-train a topic-aware encoding layer. as in Allaway and McKeown (2020), we subsequently identify the topics present within these passages. As in Allaway and McKeown (2020), for each passage Pi we define a candidate topic as a noun phrase in the passage from the constituency parse generated using the Python Spacy library. If no noun phrases are identified, we remove the passage from our dataset and randomly select another article and passage for use in our dataset. Unlike Allaway and McKeown (2020), who then heuristically select one of the noun phrases as a topic, once candidate noun phrases have been identified, we subsequently utilize the off-the-shelf T5-Flan-XL (Chung et al., 2022) model from Hug- gingface² to further refine our topic identification. The particular model that we utilize was fine-tuned to identify topies within passages using the fol- lowing prompt: Select the topic that this about: : (text) (eeext)\nnn(oopp (nnooptioss\ANy (answer) and we also utilize this prompt to identify the topic of our passages.
model (LLM) fine-tuned on semantic similarity tasks.3 Embedding all constituent 100-word pas- sages from Hanley and Durumeric (2023)'s dataset and indexing them with the FAISS library (Johnson et al., 2019), a library for efficient semantic sim- ilarity search, for each passage pi in our dataset, we subsequently identify the passage q; that had the highest semantic similarity to the given passage Pt- To ensure that each topically similar passage was stylistically different from its similar passage Pt, we only include passage pairs if they originated from two different websites. We set the minimum similarity of two passages to be topically similar at a cosine similarity threshold of 0.70 (Hanley et al., 2023; Grootendorst, 2022). Across our fi- nal dataset, each topically identified similar pas- sage qe had an average/median cosine similarity of 0.796/0.786 to its assigned passage pi in our training dataset and an average/median similarity of 0.786/0. 764 in our validation set.
We further augment this dataset with different paraphrases of the extracted topic strings. By para- phrasing these topics, we model the different ways of expressing the same topic. To make these para- phrases, we utilize another publicly available T5- based paraphraser Parrot that was trained on short texts (Damodaran, 2021). For more details on this paraphraser, see Appendix B and for example topic paraphrases see Appendix D. We note that some of the topics are particular named entities such as loca- tions, person names, eic... and thus cannot be effec- tively paraphrased. As such, we utilize the Spacy named entity recognizer to identify instances of topics that cannot be effectively paraphrased. Alto- gether, we remove entities in the following classes ('PERSON', 'GPE', 'LOC", 'TIME', 'PERCENT' 'QANNTITY', 'ORDINAL', 'MONEY', 'DATE').
To ensure a diversity of different topics within our dataset, we allow at most three passages within our dataset to have the same topics. We again re- move passages-topic pairs that did not fit this crite- rion, creating another passage-topic pair using Han- ley and Durumeric (2023)'s s dataset if necessary. Altogether, our TAW training dataset contained 97,984 unique topic noun phrases, illustrating the diversity of different topics considered. After iden- tifying 100,000 passage-topic pairs for use in our training dataset, we identify an additional 10,000 passage-topic pairs, each with a unique topic and with nonoverlapping topics with our TAW training dataset, as our validation set. We note that while we only utilize 1 110,000 pairs here, this TAW dataset could casily be extended to utilize more passages and topics.
Our final dataset Draw Pu,,ta,t 222 # consists of 110,000 quadru- plets, where the passages pt were taken from news articles (Hanley and Durumeric, 2023). the topics la were identified by T5-FLan XL, 12 are Parrot paraphrases of the 41 topics, and the semantically similar passages Oi were identified using MPNet. With permission from Hanlcy and Durumeric (2023), we release these quadru- plets and the splits upon request to form at
{x;
3.2 Topically Similar Passages
Once we identified the topics of our set of pas- sages, we subsequently identified other passages within Hanley and Durumeric (2023)'s dataset that were similar/about the same topic as our set of pas- sages. To do so, we use an off-the-shelf version of the MPNet (Song et al., 2020) large language
'https://spacy.ior io/
'nttps://huggingfacee httpe//inaaac.cc/eetttatee cor/sennence--rrasffrreerrr
htttts://huuginggfac. wttp://faaea..eea.eaeeeeeeeeeeee
all all-mpnet-bar base-v2

---

4 Augmented VAST Dataset
Apples are better thua tegetabls
(Fruii
addition developing our own dataset for train- ing TAW embeddings, we further extend Allaway and McKeown (2020)'s VAST dataset to pre-train our topic-agnostic/TAG embeddings. As previ- ously noted, the VAST dataset one of the most varied stance datasets. VAST consists of dataset Dvast = {z; (pt, f4, s:)]== where Ps pas- sage, & the topic of the passage. and s; is pas- sage pi's stance toward the topic with 8; (Pro, Neutral, Against). We augment the train- ing dataset with paraphrases of the original pas- sages and topics while pretraining our TAG em- bedding layer enabling the generation of additional cases of passages with the exact stances and lexi- cally similar topics to those in the VAST dataset.
a, (SEP)
o
O
Be Becieeik
Dustenaa Rederiing
Combriaad Tigel Berheliig:
1GM lueeeii
Scakel Dut Product Attentin
Sealed Dut Product Attuntion
Pro
Neutral
Against
Figure I: TATA Model.
layer ferwaro neural network for stance clas- sification (5.4).
To create paraphrases of the original VAST dataset's passages, we rely on the pre-trained Dipper paraphraser*. a T5-based model fine-tuned to paraphrase paragraph-Ievel texts (Krishna et al., 2023). See Appendix C for details on the hyper- parameters we utilized for Dipper and see Ap- pendix E for example Dipper paraphrases. By uti- lizing the T5-based Dipper model, we augment the original VAST dataset without changing the original meaning of the original VAST passages. As with our TAW dataset, we further augment VAST with different paraphrases of the original topic strings with the same methodology outlined Section 3.2. By paraphrasing these topics, we again model the different ways of expressing the same topic.
5.1 Topic-Aware/TAW Embedding Layer
As previously noted, we train a topic-aware embed- ding layer to extract topic-aware features for use in downstream stance classification. As in Allaway and McKeown (2020), we seek design topic rep- resentations that can be used to learn topic-specific stance features in downstream tasks. However, un- like previous work (Liang et al., 2022a,b; Allaway and McKeown, 2020), we do not use representative topic clusters as embeddings of particular topics during training. Instead, we use our TAW dataset and contrastive learning techniques develop rich general latent representation of different top- ics. Once trained, this topic latent, which implicitly contains the relationship between topics, can then be utilized to get informative generalized topic rep- resentations. After training this layer, we freeze this layer for use in the rest of our TATA model. Specifically, we utilize a triplet loss to learn similar representations for similar topics and to differentiate different topics. This is such that we embed each example o; Draw
Altogether, by paraphrasing each from the VAST dataset (paraphrasing individual texts as many as 16 times) and by extracting out as many as 10 paraphrases of each topic phrasc/word within the original VAST dataset Dy AST, we extend the original training dataset of 13,477 examples to a total of 743,644 different examples Dy ASTang
5 Methods
{a; = (pi, 41, qú, 42) using a contextual word model by inputting by [SLSSp: and [CLS)n[SEPIsSSEP and outputting the hidden states of the [CLS] tokens hp, and ba. Then using the hidden states of the [CTSS] tokens, minimize with triplet loss the Euclidean dis- tance between the hidden hp: h,, that share common paraphrased topic. We condition the hidden state hy, using a paraphrased version 422 of the topic t. to guard against our model only learn- ing to recognize moving together sentences with
We develop model (TATA) that combines topic- awarc and topic-agnostic embedding layers per- form Zero and Few-shot stance detection. This model (Figure 1) consists of the topic-aware em- beddings layer 1), the topic-agnostic embedding layer (5.2), two attention layers using the output the topic-aware embedding layer and the topic- agnostic embedding layer (5.3), and finally a two-
hhttps:/huggingfaan ace.. www.c...ocaaaae wtt..aaleeskk2211111111 di paper-paraphraser--x!

---

the exact same topic t. Putting this together, we train our embedding layer such that given the out- Na putted set of hidden vector pairs (h by J jij== =0 in batch B of size No. we treat all pairs where = as positive pairs and pairs where # j as negative pairs. Within each batch B, the triplet loss is computed across all positive pairs in the batch:
pairs where 8; A sj are considered negative pairs, Within each batch B, the contrastive loss is com- puted across all positive pairs the batch such that:
Ltag
1 Nb
Σ
(lag (hi)
B,EB
Σ ΣjEB, [os=s,] eep( h by
Ltaw
Σ
Nla (hpc bay))
(0g (h:) = log
No hay Byy EB,
EjeBy exp (xp(>IIh,
b h, 7 11by]]
glam (hp,: hay)
Σ
[max(0,
1/pbp, - b2g 112
where T is the temperature parameter.
||ا hzg 112)
5.3 TATA Attention
Finally, once our topic-agnostic/TAG and topic- aware/TAW layers are fully trained, as shown in Figure 1, we utilize them embedding layers within our full TATA model. Specifically, taking as input instances 2; such that {-; (pi, 4i si) our TATA model, uses the outputted hidden vectors how and htog in attention layers with contextual embeddings of the passage and topic. This is such that after getting the output of the TAW layer hoow and our TAG layer htog. and outputting another contextual embedding representation Ppassageltopic of the joint input of the passage-topic pair, and a contextual embedding hoopcc the topic 14, we finally utilize a scaled dot product attention (Vaswani al., 2017) between hew and htpic as well between hag and h passsnge (topic
We use a triplet loss for our TAW layer rather than a contrastive loss because normal contrastive loss pushes similarly labeled elements to have a near-zero distance between them (Xuan et al., 2020). However, for TAW encoding layer, given the noisiness of our Drow dataset, while we wish for examples with similar topics to have simi- lar embeddings, here we do not wish to train them to have the same representation. Triplet loss only ensures that negative examples are further away in the latent space than positive examples. Namely, triplet loss enables us to ensure that our noisy pairs have more similar TAW representations between themselves than with other examples in dataset.
5.2 Topic-Agnostic/TAG Embedding Layer
As noted by Allaway and McKeown (2023), em- beddings that encode general stance features have been shown to improve model's ' ability to dis- cern the stance of different passages. As in Liang et al. (2022b), we thus use contrastive learning to diffcrentiate the (Pro, Neutral, Against) stance classes and build a TAG embedding layer with our augmented VAST dataset Dv ASTTT.u After training this layer, we freeze it for use in the rest of our TATA model.
Σ an tano
Ttopic
a₂ softmax (Ahfaaw (Warn
Σ bih (1 passage ttopie topic
I'stance
b; = softmax (An (i) passageltopic (W bogg)
We train our TAG layer such that we em- bed each example I; E DV = {x; = (pr, 11,8,) N using a contextual word model, in- putting (CLSp:[SEP]J.R
where Wing E ExE and Winw E ExE are learned parameters, and X = 1/VE. This enables us to obtain a representation Ftopic that captures the relationship between the generated TAW embed- ding and our topic; in this way, we manage to better determine which aspect of the general TAW latent space representation of the topic is important in representing the topic for downstream stance clas- sification. We further get the representation l'tancce that captures how much the calculated stance fea- tures extracted by our TAG embedding apply to our given passage.
and outputting the hidden vector of the [CLS] token for each z4. Then, given a set of hidden vectors [h: where N₂ is the size of the batch, we perform contrastive Iearn- ing with that batch. This is such that for each batch B, for an anchor hidden embedding h within the batch, the set of hidden vectors h; hj C B vectors where A are considered a positive pair if their corresponding stances 8i, 8j are equivalent; other

---

Train
Validation
Test
as the optimizer (Kingma and Ba, 2015). Due computational constraints, while pretraining, we use batch size of 16. While pre-training and using a contrastive loss, we set the temperature parameter T to 0.07 as Liang et al. (2022b). For our TAG layer, we ended training after 2 epochs, and for our TAW layer, we ended training after epoch. We completed all training an Nvidia A6000 GPU. Once trained, we freeze the TAG and TAW embedding layers.
# Examples
13,477
2,062
3,006
# Unique passages
1,845
682
786
# Zeoo-shot topics
4,003
383
600
# Few-shot topics
638
114
159
Table 1: Data statistics for the VAST dataset.
5.4 Label Prediction
Once ltopic and Tstance calculated, we concate- nate them with |paseage/topic using a residual con- nection. We then feed the resulting representation into a two-layer ferd-forward network with soft- max activation to compute the output probabili- ties for Pro, Neutral, Against). We minimize cross-entropy loss while training.
6.2 Training
Training Data. While much of our pre-training data was artificially generated or was unlabeled data gathered from news websites, after we freeze our TAG and TAW layers, for training our TATA stance classifier, we use only the original VAST dataset. As previously noted, this dataset consists of 13,477 soopsagge-tpiissteee overview of this dataset in Table 1.
6 Experiments
6.1 Pre-training
triplets. We give
TAW Training Data. As pre-training data for our TAW embedding layer, we utilize our TAW Drawe dataset of passage pairs with shared topics. As pre- viously stated, this dataset includes 100,000 pas- sage pairs with shared topics training data and 10,000 passage pairs validation data.
TATA Training Details. We use similar settings training as we did for pre-training except for chang- ing to a batch size of 32. In our feed-forward layers. we utilize dropout with = 0.30. While training, we utilize early stopping with patience of 3.
TAG Training Data. As pre-training data for our TAG embedding layer, we utilize our augmented VAST dataset DV consisting of 743,644 unique passage-toiic-stance triplets. We utilize the validation split of the original VAST dataset as validation data.
6.3 Testing
Testing Data. We utilize the testing split within the VAST dataset. Following prior work (Liang et al., 2022b; Allaway and McKcown, 2020; Liu et al., 2021a), when reporting results on this dataset, we calculate the macro-averaged F1 for each stance la- bel. We present results separately for the Few-shot (where the detector is evaluated on a large number of topics for which it had few training examples) and Zero-shot (where the detector is evaluated on new, previously seen topics) settings.
We utilize DeBERTa-v3-base as our contextual model and to jointly embed our passage-topic pairs, where our passage-topic pairs are mapped to a 768-dimensional embedding using the outputted hidden state of the [CLS] token (He et al., 2022). We use DeBERTa given its larger vocabulary 110 tokens vs. 30,522 tokens in BERT (Devlin al., 2019)) and its better overall documented performance on various downstream tasks (He et al., 2022). While encoding the passage-topic pairs, we use a maximum length of 512 possible tokens as opposed to the first 200 tokens as in other works (Allaway and McKeown, 2020). Unlike in previous works (Allaway and McKeown, 2020; Liang et al., 2022b), we further do not remove stopwords and punctuation from our datasets, electing
Pre-training Details.
Baseline and Comparison Models. We com- pare our proposed TATA model with several other baselines including TGA-Net (Allaway and McK- cown, 2020), TOAD (Allaway et al., 2021), CKE- Net (Liu et al., 2021a), BERT-GCN (Lin et al.. 2021), and JointCL (Liang et al., 2022b). As an additional baseline, we further train a model based on DeBERTa-v3-bass that takes the hidden state of the [CT.S] for the combined passage- input followed by two-layer ford-forward neural net- work predict the stance. Finally, to understand the respective importance of the TAW and the TAG embedding layers, we further train a TAG archi- tecture that takes our original TATA architecture and removes the TAW embedding layer as well as a
the full unprocessed version of original texts and their topics.
While pretraining the TAG and TAW layers, we set the learning rate to x 10 and use AdamW

---

Zero-Shot VAST
Few-Shot VAST
Pro
Against
Neutral
All
Pro
Against
Neutral
All
BERT
0,546
0.584
0.853
0.661
0.554
0.597
0.796
0.646
TGA-Net
0.554
0.585
0.858
0.665
0.589
0.595
0.805
0.663
CKE-Net
0.612
0.612
0.880
0.702
0.644
0.622
0.835
0,701
BERT-GCN
0,583
0.606
0,869
0.686
0.628
0.643
0.830
0.697
TOAD
0.426
0,367
0.438
0,410
JoinCCL
0.649
0.632
0.889
0.723
0.632
0.667
0.846
0.715
DeBERTa
0,680
0.683
0,900
0.755
0.659
0.657
0.869
0.728
TAW
0.672
0.709
0,903
0.760
0.656
0.677
0.869
0.736
TAG
0,681
0.687
0,901
0.756
0.665
0.655
0.868
0.729
TATA
0.695
0.711
0.905
0.771
0.665
0.683
0.873
0.741
Table 2: Fi-scorss of models on benchmarks from the VAST dataset. We bold the highest/best score in each column. We ohtain scores BERT and TGA-Net from Allaway and McKeown (2020), scores for CKE-NNe and BERT-GON from Liu et al. (2021a), and scores for TOAD and JointCL from Liang et al. (2022b).
TAW architecture that removes the TAG embedding layer. Lastly, we note to improve the consistency and robustness of our results, we train each model with different random seed a total of five times and report the average of those five different runs in our results.
using only the input topic text. We this pri- marily to ensure that our TAW model is actually utilizing the input texts and is not relying pri- marily on the topic embeddings. To do so, for our TAW embedding layer rather than passing [CLS)p[SEP]) to the layer input, we pass in [CLS]SEP)) predicting based on the topic significantly hurts our results in the Zero-shot and Few-shot settings, with our model nearly always predicting the Neutral class. This behavior largely conforms to the defini- tion of the Neutral class; without any input text, the stance of an empty string to given topic should be neutral. This suggests that our TAW model is not only picking up on the topic but is also, as expected, heavily considering the text when predicting the stance.
As seen in Table 3,
7 Experimental Results
Our models achieve noticeable increase in perfor- mance prior work. We present results the VAST test dataset in Table 2. We observe that by using the unedited text, including stopwords. using a max token length of 512, and by finetuning a DeBERTa model, wc achieve similar results to JointCL (Liang et al., 2022b). This illustrates that utilizing a model with a larger token vocabulary (128, 100 in DeBERTa vs. 30,522 in BERT) can largely assist with the stance detection task. How- ever, we further observe that by utilizing the TAG and TAW features in our full TATA models, we can increase our performance, achieving 0.771 Fi score in the Zero-shot setting and 0.741 Fi score in the Few-shot setting.
Examining our TAG model, we only observe slight improvements over our DeBERTa model. In- deed, plotting the t-SNE (Van der Maaten and Hin- ton, 2008) the entire VAST validation dataset, as seen in Figure 2, while we do observe the forma- tion of clear stance clusters during training on the VAST validation set, these clusters are not exact. De- spite this, we note that incorporating the TAG layer embeddings to our TATA model did lead to better downstream stance classification (an increase of 0.011 Fi and 0.005 Fi in the Zero-shot and Few- shot settings respectively). This reinforces prior work (Allaway and McKeown, 2023; Liang et al., 2022b) that has found that general stance features, incorporated with other features and knowledge, can lead to better downstream stance classification.
Ablation Studies. We perform an ablation study in order to understand the relative importance of our TAG and TAW layers. As seen in Table 2, our TAW model performs better than our TAG model both Zero-shot and Few-shot scenarios suggesting the importance of the TAW topic latent predicting stance. By removing this layer from our TATA model, our model's score in the Zero-shot and Few- shot scenarios decreases by 0.015 F1 and 0.012 Fi respectively.
Given the reliance of our TAW model on the in- formative TAW topie latent for predicting stance, we perform another study to investigate whether it is simply predicting the stance of input texts
7.1 Error Analysis
Challenging Linguistic Phenomena. As in All- away and McKeown (2020), we analyze the perfor-

---

Zero-Shot VAST
Few-Shot VAST
Model
Pro
Against
Neutral
All
Pro
Against
Neutral
All
TAW
0.034
0.000
0,519
0.184
0.020
0.000
0.507
0.176
Table 3: Performing ablation study lesting whether our TAW model utilizes only the topic for determining the stance of a given text, we find that our model does utilize the passage input, with this reduced model nearly always predicting the Neutral class.
(a) Before Training
(b) One Epoch
Two Epochs
Figure 2: As we train the lopic-agnostic/AAG layer of our model on our augmented VAST training set, while not separating perfectly (illustrating the need for additional features) clear Pro, Against, and Neutral clusters appear in the t-SNE of the embeddings of the VAST validation dataset. As confirmed elsewhere (Allaway and McKeown, 2020; Liang et al., 2022a), the Neutral category of examples is the most differentiable from Pro and Agains! categories.
Qte
Sare
Imp
mlS
mlT
TATA model able to effectively utilize the TAW informative latent space of topics. We note that our TAW model, also performs second-best in each of these scenarios, besting our TATA model in per- formance cases where individual passages have topics with different non-neutral stances (mlS). In these (mIS) cases, the model must learn to not simply always predict the same stance for a given text; and as seen in Table 4, while our topic-ware models perform better these scenarios compared to all other models, across every model tested, they perform the worst in on this particular challenging linguistic phenomenon. We leave to future work to explore how to encourage models to better differ- entiate stances in passages with conflicting stances multiple topics.
Model
N=539
N=431
N=1231
N=952
N=1802
TGA-Net
0.661
0.637
0.623
0.547
0.624
BERT
0.625
0.587
0.600
0.541
0.610
DeBERTa
0.709
0.713
0.667
0.578
0.676
TAW
0.712
0.722
0.673
0.620
0.689
TAG
0.710
0.731
0.668
0.568
0.676
TATA
0.714
0.730
0.693
0.603
0.703
Table Accuracies on various challenging linguistic phenomena in the VAST test set. We give accuracies to directly compare our results against Allaway and McKeown (2020). We utilize scores for BERT and TGA-Net from Allaway and McKeown (2020).
mance of our TATA network on five challenging linguistic phenomena provided within the VAST test dataset. The phenomena within the VAST dataset for passage-topic pairs are as follows: (1) Qie: the passage contains quotations, (2) Sarc: the passage contains sarcasm, (3) Imp: the topic is expressly contained in the passage, (4) mlS: the passage is among a set with multiple topies with different non- neutral stance labels, and (5) mIT': the passage has multiple topics. As seen across the various phenom- ena, our set of models outperforms the TGA-Net and BERT baselines from Allaway and McKeown (2020), with TATA having the best overall perfor- mance. We find that our TATA model performs particularly better in cases where there are quota- tions (Qte), where there are multiple topics (mlT), and where the passage does not expressively con- tain the topic (Imp). This again suggests that our
Performance on Topics Included in TAW dataset. We note that several of the Zero-shot topies in the VAST dataset had lexically similar topics in our TAW dataset. We thus determine if the number of lexically similar topics included in the TAW dataset correlated with our TATA model's performance on those topics. In this way, we test if learning the specific tested topics within our latent space was important in later predicting a stance involving the topic. To identify lexically similar topics, in All- away and McKeown (2020), we represent each topic in the Zero-shot test set using pre-trained Glo Ve word embeddings (Pennington al., 2014) and determine the number of topics examples our TAW dataset with cosine similarity to these

---

Zero-Shot SEM16t6
Taget
Pro
Against Neutral
Donald Trump
148
200
260
DT
LA
IIC
FM
CC
A
Avg
Hillary Clinton
163
565
256
JointCL
0.505
0,495
0.548
0.538
0.397
0.545
0.505
Feminist Movement
268
511
170
TOAD
0.495
0.462
0.512
0.541
0.309
0.461
0.463
Legalization of Abortion
167
544
222
Atheism
124
464
145
DeBERTa
0.660
0.606
0.698
0.656
0.347
0.528
0.566
Climate Change
335
26
203
TAW
0.609
0.642 0.664
0.644
0.360
0.428
0.558
TAG
0.623
0.618
0.727
0.647
0.423
0.479
0.586
TATA
0.638
0.629
0.654
0.669
0.416
0.521
0.588
Table
Data statistics for the 6t6 dataset.
Table 6:
Macro average of the Pro and Against classes for the SEM16t6 dataset for our set of trained models. We bold the highest/best score in each column.
topics greater than 0.90. We do not find statis- tically significant correlation between the number of lexically similar topies in our TAW dataset and whether our TATA model predicted the correspond- ing VAST test instance's s stance correctly (p-value =0.353). This illustrates that our TAW model's la- tent was not particularly biased toward the specific topics that we included within our TAW datasct.
sults, we evaluate our model on the Zero-shot SEM1 6t6 dataset (Mohammad et al., 2016) which contains passages on six pre-defined topics: Don- ald Trump (DT), Hillary Clinton (HC), Femi- nist Movement (FM), Legalization of Abortion (LA), Atheism (A), and Climate Change (CC). Each passage's stance can again be classified in (Pro, Neutral, Against). We give dataset statis- tics for the SEM1 6t6 dataset in Table 5.
Performance in Few-shot vs. Zero-Shot Settings. Given our model's greater performance in the Zero- shot setting than the Few-shot setting, measure whether the number of lexically similar training examples with regards a given topic within VAST (e.g., where lexically similar topic is as previously defined in our paper and Allaway and McKeown (2020) as having a GloVe cosine similarity above 0.9) is correlated with whether our system correctly labels a given instance of that topic. If there is a positive correlation, this may indicate that the topics with few examples may be noisy/biasing results (as those with similar training examples all have higher F1 -ccores than those in the Few- shot setting). In the Few-shot setting, we find a small Pearson correlation (0.186, p-value =0.021) between the number of semantic similar training examples and the percentage of a given topic's test examples that are labeled correctly by our TATA model (0.2675, 0.luue=000001 for the TAG model; 0.199, 00002 for the TAW model; 0.215, p-value= 0.007 for the DeBERTa model). Together with the other models (Allaway and McKeown, 2020; Liu et al., 2021a; Liang et al., 2022b) that perform worse in the Few-shot setting, this suggests some noisiness and a tendency of our model to be biased on topics in VAST with only one or two training examples. As reported by Allaway and McKeown (2020), there was high (75%) but not perfect agreement among labelers of the original VAST, further helping to explain this phenomenon. Generalizability. As was seen in Table 2, our models have better performance in Zero-shot and Few-shot settings than other basclines on the VAST dataset. However, to further confirm these re-
We note that given that our TAG embeddings were trained on the VAST dataset that contained SEM1 6t6 topics, we retrain our TAG embedding layer on a reduced dataset that removed these SEM1 6t6 topics (30 training instances removed). Further, given that our TAG and TATA models werere implicitly trained on the rest of the VAST dataset (through the TAG embedding layer), in order give a fairer comparison, we combine the reduced VAST (i.e. without the SEM1 6t6 top- ics) and the SEM16t6 training sets when training each model. For each topic, we use the Icave-one- target-out evaluation setup (Allaway and McKe- own, 2023; Liang et al., 2022b) and report the macro average of the Pro and Against classes. As seen in Table 6, our TATA model, on average, again outperforms our other models and baseline models.
8 Conclusion
Combining pre-trained layers that learn topic- agnostic/TAG features and topic-aware/TAW fea- tures, our TATA model achieves state-of-the-art performance on the VAST test dataset. TATA per- forms better than prior works not only in Few-shot settings but also in Zero-shot settings. Our most common failure mode is cases where a given pas- sage has both positive and negative stances within it. For future work, during training, we plan in- corporate additional passages with multiple stances toward different topics.

---

Limitations
References
We note several limitations of our approach here: Use of News Articles for TAW Layer. We note that while our use of news articles from a variety of sources enables us to build an informative latent across a variety of topics, our dataset is probably largely biased against topics that are not regularly spoken about in the news such different science concepts or medical information.
Emily Allaway and Kathleen McKeown. 2020. Zero- Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), ),, pages 8913- 8931, Online, Association for Computational Lin- guistics.
Emily Allaway and Kathleen McKeown. 2023. Zero- shot stance detection: Paradigms and challenges. Frontiers in Artificial Intelligence, 5:1070429.
Use of Paraphrases for TAG and Noisy Topic Identification. We note that using paraphrases while developing our augmented VAST dataset may unintentionally change the meaning of the original text. While we guard against this by utilizing con- servative parameters of our Dipper Paraphraser, we acknowledge that it may result in incorrect in- formation in augmented VAST dataset.
Emily Allaway, Malavika Srikanth, and Kathleen Mcke- own. 2021. Adversarial learning for zero-shot stance detection social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4756-4767.
Nora Alturayeff, Hamzah Luqman, and Moataz Ahmed. 2023. A systematic review of machine learning tech- niques for stance detection and its applications. Neu- ral Computing and Applications. 35(7):5113-514
Zero-Shot vs. Few-Shot Setting Performance. We note that, as in prior work such as CKE- Net (Liu et al., 202 and JointCL. (Liang et al., 2022b), we achieve better results in a Zero setting than in a Few-shot setting. This may indi- cate partially disagrecing labels among multiple annotators present within the original training VAST dataset that may be biasing the results for the topics in the Few-shot setting.
Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. 2021. A review fact extraction and verification. ACM Computing Surveys (CSUR), 55(1):1-35.
Douglas Biber and Edward Finegan. 1988. Adverbial stance types in english. Discourse processes, 11(1)11 34.
Ethics Statement
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-iiecuueed language models. arXiv preprint arXiv:2210. 1416.
We utilize dataset of public news articles collected by Hanley and Durumeric (2023) to train topic- aware encoding layer. We ask for permission to utilize and release sections of their dataset.
We note that our work can be utilized to under- stand the stance of news articles and comments on various phenomena ranging from individual po- litical figures like Joe Biden Donald Trump to topics such as immigration and vaccines. How- ever, while our work substantially improves upon past works for stance detection, before can be utilized or deployed, at scale, the accuracy for Pro and Against classes still needs to be improved as we still only achieve near 0.70 F1 scores for those classes.
Bo Dai and Dahua Lin. 2017. Contrastive learning for image captioning. Advances in Neural Information Processing Systems, 30.
Prithiviraj Damodaran. 2021. Parrot: Paraphrase gener- ation for nlu.
Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020. Unsupervised user stance de- tection on twitter. In Proceedings of the Interna- tional AAAI Conference on Web and Social Media, volume 14, pages 141-152.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
A Acknowledgements
This work was supported in part by a gift from Google, Inc., the NSF Graduate Fellowship DGE- 1656518, a Stanford Graduate Fellowship, and the Meta Ph.D. Fellowship in Computational Social Science.
Chris Dulhanty, Jason L Deglint, Ibrahim Ben Daya, and Alexander Wong. 2019. Taking a stance on

---

fake news: Towards automatic disinformation as- sessment via deep bidirectional transformer lan- guage models for stance detection. arXiv preprint 191195191..1195.
Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang. and Ruifeng Xu. 2022a. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2738-2747.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910.
Bin Liang, Qinlin Zhu, Xiang Li, Min Yang. Lin Gui, Yulan He, and Ruifeng Xu. 2022b. Jointel: A joint contrastive learning framework for zero-shot stance detection. Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), volume I, pages 81-91. Association for Computational Linguisties.
Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with class- based tf-idf procedure. arXiv preprint arXiv:2203.0577
Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang. Jiwei Li, and Fei Wu. 2021. Bertgen: Transductive text classification by combining gnn and bert. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 1456-1462.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. In Interna- tional Conference on Learning Representations.
Hans WA Hanley and Zakir Durumeric. 2023. Machine-made media: Monitoring the mobiliza- tion of machine- machine-generated tion and mainstream news websites. arXiv preprint arXiv:2305.09820.
Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021a. Enhaneing zero-shot and few-shot stance detection with commonsense knowledge graph. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 3152-3157.
generated articles on misinforma-
Hans WA Hanley, Deepak Kumar, and Zakir Durumeric. 2023. Happenstance: Utilizing semantic search to track russian state media narratives about the russo- ukrainian war reddit. In Proceedings of the inter- national AAAI conference on web and social media, volume 17, pages 327-338.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021b. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering. 35( 11:5578876
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task Detecting stance tweets. In Proceedings of the 10th International Workshop Semantic Evaluation (omeEval-2016) pages 31-
Pengeheng Hc, Jianfeng Gao, and Weizhu Chen. 2022. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangeed embedding shar- ing. In The Eleventh International Conference on Learning Representations.
41. San Diego, California. Association for Computa- tional Linguistics.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In defense of the triplet loss for person re- identification. arXiv preprint arXiv: 170::00733777
Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluís Marquez, and Alessandro Moschitti. 2018. Automatic stance detection using end-to-end memory networks. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long Papers), pages 767-776.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEFF Transactions on Big Data, 73):533-5447.
Diederik P Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic gradient descent. In ICLR: international conference on learning representations, pages 1-15. ICLR US.
Mitra Mohtarami, James Glass, and Preslav Nakov. 2019. Contrastive language adaptation for cross- lingual stance detection. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) pages 4442 4452.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. 2023. Paraphras- ing evades detectors of ai-generated text, but re- trieval is an effective defense. arXiv preprint arXiv:2303. 13408.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. 2020. Contrastive learning for unpaired image-to-image translation. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, Augus! 23-28, 2020, Proceedings, Part IX 16. pages 319-345. Springer.
Yang Li and Jiawei Yuan. 2022. Generative data aug- mentation with contrastive learning for zero-shot stance detection. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6985-6995.

---

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Xing Wu, Chaochen Gao, Liangjun Zang. Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022. Esimose: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. Proceedings of the 29th International Conference on Computational Linguistics, pages 3898-3907.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-texi text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. sross-targed stance classification with self- attention networks. nual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 778- 783.
Proceedings of the 56th An-
Nils Rethmeier and Isabelle Augenstein. 2023. A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives. ACM Computing Surveys, 55( 55(10):1-17.
Hong Xuan, Abby Stylianou, and Robert Pless. 2020. Improved embeddings with easy positive triplet min- ing. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 2482.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022- Tenth International Conference on Learning Representa- tions.
Qinglin Zhu, Bin Liang. Jingyi Sun, Jiachen Du Lanjun Zhou, and Ruifeng Xu. 2022. Enhancing zero-shot stance detection via targeted background knowledge. In Proceedings of the 45th International ACM Si- GIR Conference on Research and Development in Information Retrieval, pages 2070-2075
Nikunj Saunshi, Orestis Plevrakis. Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019. A theoretical analysis of contrastive unsu- pervised representation learning. In International Conference on Machine Learning, pages 5628-5637.
A
T5-Flan
Rather than considering all noun phrases in the sub- ject and object sections of passages as potential top- ics in Allaway and McKeown (2020), we instead prompt T5-Flan-XI (Chung al., 2022), 11 bil- lion parameter fine-tuned large language model to identify the topic for a given article passage given a list of candidate noun phrases extracted by Spacy. The T5-Flan-XL model that we uti- lize was fine-tuned on 193 tasks present in the TO dataset (Sanh et al., 2022), which includes topic classification. Given how T5-Flan-XL was fine- tuned, we utilize the following prompt extract the topic using the noun-phrases from the text potential options: Select the topic that this about: (text))nknroppioon))nn (ann//, (answer).
PMLR.
Jiasheng Si, Deyu Zhou, Tongzhe Li, Xingyu Shi, and Yulan He. 2021. Topic-aware evidence reasoning and stance-aware aggregation for fact verification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 1612-1622, Online. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33: 16:166 16867.
B Parrot Paraphraser
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of machine learning research, 9(11).
We utilize the Parrot Paraphraser (Damodaran, 2021) to generate paraphrases of topics. We utilize the Parrot Paraphraser for our topic paraphrases as it was fine-tuned on several shorter datasets. The Parrot Paraprhaser" was trained to develop seman- tic similar and logically consistent paraphrases by fine-tuning NLU tasks. The Parrot paraphrase has input parameters for:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information proceessssroocssinnn systems, 30.
Xiao Wang and Guo-Jun Qi. 2022. Contrastive learn- ing with stronger augmentations. IEEE transac- tions on pattern analysis and machine intelligence, 45(5):5549-5561
http::///ggnnn.fee.e//rrrrr//ara P5raphraser_on_T5

---

Adequacy: Is the output semantically equiva- lent the input?
Fluency:
How fluent is the English of the para- phrase?
Diversity: How much has paraphrasing
changed the original input?
We use the default setting for the Parrot Para- phrser, only changing the fluency a value of 0.50 given that we input topie phrases rather than full sentences.
Dipper Paraphraser
We utilize the Dipper (Krishna et al., 2023) Para- phraser?, T5-based model (Raffel et al., 2020) in order paraphrase the original text in the VAST dataset. The Dipper paraphrase model was trained to give paraphrases of paragraph-level data by uti- lizing different translations of non-English nov- els into English aligned at a paragraph level. The Dipper model in addition has two parameters L. and O that correspond to the estimated lexical di- versity (using unigram token overlap) and the or- dering of words (using the Kendall-Tau correlation of tokens). This is such that L 40 value corre- sponds to 40% lexical modification of the original text and similarly for
The authors of Dipper found that using a high lexical diversity value (on scale of 0 to 100) in the Dipper model has a tendency to change the proper nouns within given paragraph. Given that for our pretraining of the TAG embedding layer, require the stance of the original text to be maintained toward the nouns within it, as recommended by Krishna et al. (2023), we use a low lexical diversity value of 20. We use an order diversity of = 40.
acc. 2c.kc//11pssss11hk228111111 di dipper paperrppaaphraser-voll

---

D Example Parrot Paraphrases
Original Topic
Paruphaae List
isruelis' retributions
the retribution of Israel. isruel's revenge aide, farm assistant, farm support
farm aide
luiing
of wealthy taxing wealthy
for another
faver for
government funding
of money, govermment grants.federal funding, government money the royal family. zoyalty
the reyals
paän pills
pain reliever, pain medication, pain relief pills
illness
medical condition, disease, ill
china's 's manufacturing
mazufactuuing china, china'ss is production
people being obese
obesity, obese, people who are fat, people who ane overweight
Table 7: Examples of different topic paraphrases generated by the Parrot paraphraser (Damodaran, 2021).

---

E
to stay out of Example Dipper Paraphrases
Original Text
Parupthase
Absolutely it's needs be defined and regulated in its currently the word "uttrral" when used on food products is totally coofusing and meuning- less. Cleearly they ane trying to imply the item is 'healthy' or possiblyy 'organic', when you see food 'maaufaccuurers like Frius- Lay or Cumpbell's is with products labelled 'natural", that alone should set off alarms that all is not what scems. 1-)
The word "natural" has absolutely to be defined and regulated its Appliad to food products, the word now so confusing and meaningless that it's useless. The mere fact that food "manufacturer" like Prito- Lay can munufacturer like Campbell Soup Company label iheir products as "nutural" should alert you to the fact that not all is as it seems. Obvioudy they are trying suggest the product "healthy," possibly "organse." Suomi Maestro!
.one must ask how mach money they mast make to demonstrate that they are among the best managed companies the planet." They must make enough money to insure that they can never fail and threaten the stability of the worlds cconomy again. That much moncy.
.. to see that they are one of the best-managed companies on the planet, one mast ask how much profit they make.' If you wish keep the stable you must never again threaten to go bankrupt. So you must make enough money to guarantee that. This much money.
had besutiful home birth with licensed midwife 2011. I inspired by the experience that became a cortified birth doula and have attended 17 births (many at home or at hirth center). All women should have opportunity chouse where they safest birthing. grateful for midwifory!
In 201 had beautiful homebbrrh with licensed midwile. enthusiastic about the whole thing that became a cortified birth doula and have been seventeen more births since then, monly at home or in birthing center. "The women should able choose where they want give birth. The gratefulness for midwifery!
"As single mother I adupted my nearly two year old daughter from China at three years later brought home my son China and nearly 13. Today he is one semester away from graduating college and she had her Bat Miteval and rising eighth grader. Okay, I am the oldest mom her class- in her school. Its not my fiolt. 'Thes are the children I was meant have and I just had to wait for them be available so could bring them home. One the things about heing an older parent one really doesn't 't care what others think.
In my forties I adopted my daughter from China. She was nearly two when to live with Three years lnter brought home my thirteen. Then she had her Bat Mitrvah and hegon the eighth grade. He was only semester her class-in her school. It is not my Buult. 'No, I am not sorry," he eried. only wish this could have own children. I have been waiting for so long far them become free no longer concerned with the opinions of others.
aged
from graluating from college. am the oldest mother
could them home.' As old parent
In considering the threat posed by Iran, it's important to remember that Iran neighbors. We have supported Iraq's invasion Iran, under Saddam and even assisted Iraq in targeting chemical weapons attacks upon Iranian forces. We also supported the Shoh, throughour his tenure. including over throwing an elected government and installing the Shah. Iran has far more reason to distrust and fear than the other way around. also to he understond that Iran presently in Iraq and Syria at
In considering the threat posed by Iran it is important to remember that Iran nover invaded any its neighbours, Iraq, under Saddam. invaded and we even helped Iraq to attack Ima with chemical weapons. Throughout the Shah's 's reign, we cuppomed him. We also supported his overthrowing an elected government and putting himself in power. In faci, we have far more resson to distrust and fear Iran than the other way around. fact, Iran is in Syria Iraq now at official invitation their respective govemments (the one in Bughdad having busically been chosen by us). while the Syrian best amhivalent about our presence there. A weak and wary nation is in fact the adversary of strong and confident local power. But the kingsom of Saudi Arabia is warse than Iran in killing and threstening Americas and Westerness. Cortainly in the slaughter of Syrin, Smudi Arahin a far worse offender than Iran.
not invaded any
invilation of the oflicial governeents of those nations (with the Iragi leader hoving basically been
by us), while the Syrian government does want us in Syria and the Iraqi government is unbivalent at hest about our presence. Waging combat through surrogates is actually the approach of weak and cautious nation. not a strong and confident regional power. In terms of Americans and Wesremers killed and threatened, Sondi Arahia is far worse actor than Iran. terms of killing Syria, Saudi Arabia is also far werse actor than Iran.
government does not want us Syria and the tragi povernment
This is one of the must intelligent articles I have read on Syria. Hollima is divil war, and only Syriams can resalve it. We need stay out of this it is not our concern.
The most inselligent article on Syria I have ever read. But it's a civil war. Only the Syrians can solve it. 'This ourr business, this.
right-
Do ANY programs make money after taking into account Amortization of physical plant including all athleric
Among the costs of a university program, you should include the depreciation of the physical plant, that is, the sports arenas and the huildings occupied exclusively by cooches, staff and athletes. Do any of these programs show profit, after you have included the depreciation? ? For all sports venues and their suroundings, maintenance, heating, sir-conditioning gardening, police, and fire protection. At the same time, the time of the manager is spent on athletic scundals, criminal athletes, spoiled coaches. Administrators and members traveling to post-sctson games, Contriutiins to higher education are likely to relatively fixed within any given of laws. Consider alsn the following. Contributions to the atbletic depariment detract from contributions to legitimate academic purposes and should therefore be regarded as a cost, not bonotit.
and huildings occupied exclusively by coaches, staff und uthletes. Maintenance, beating, cooling, groundskeep ing, police & fire protection for all atbltic facilities and the territory that surrounds them. Administrator time spent dealing with athletic scunduls, crim- inal athletes, and spoiled coaches. Subsidized travel to post season games by administrttors and board member.. tax deductibic contributions to higher ed is likely to be selatively fixed sum within any given set tax laws. Therefore athletic department contributions subiraci from contributions to legilimate academic parposes and should be counted as a cost, not a benefit.
consider:
iotal amount
Table Examples of different paraphrases of texts from the VAST dataset generated by the Dipper paraphraser (Kr-
ishna et al., 2023).


Reviews:
[{"review_id": "HS5TE8Skkr", "reviewer": "Reviewer_epNc", "summary": "", "questions": "* A. For topic-agnostic setting, all of one stance seems to shares a similar embedding space– does it mean despite the topics?  Why is contrastive loss preferred in this case especially if topics are not the same? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer epNc\n\nWe first would like to thank Reviewer epNc for their detailed feedback. For our rebuttal, we address the main issues raised by Reviewer epNc in turn, namely (1) the potential overfitting of our models on the topic-agnostic/TAG encodings, (2) implementing retraining of our models so that we may perform zero-shot stance detection on the SemEval16 dataset, (3) whether our TAW encoding layer potentially is only predicting based on the topic, (4) our choice of contrastive learning for the topic-agnostic/TAG encoding layer, and finally (5) the stylistic issues present within our original document. In addition to responding to these issues and describing several new experiments that we performed to address these issues, we further describe several additional steps that we took to bolster our work's reproducibility. \n\n\n## Overfitting on Topic Agnostic Encodings: (Zero-shot vs. Few-shot)\nWhile we acknowledge that it is possible that our TATA model is overfitting the Topic Agnostic (TAG) embeddings, we note that this would largely not explain why our Topic-Aware (TAW)  subnetwork model also has a higher score within the Zero-shot setting. We further note, as reported in our paper and in the JointCL [1] paper,  CKE-Net (0.1 F1 worse), TGA-NET (0.2 F1 worse), JointCL (0.8 F1 worse), BERT (1.5 F1 worse), as well as our DeBERTa model (2.4 F1 worse), (which is fine-tuned with a classification head on top of DeBERTa model) all perform worse within the few-shot setting compared to the zero-shot setting [1].\n\nIn order to ascertain whether the number of few-shot examples for particular topics in the VAST dataset affects the performance of our models during inference, we perform another experiment. In order to ascertain whether the number of few-shot examples for particular topics in the VAST dataset affects the performance of our models during inference, we perform another experiment; namely we measure whether the number of lexically similar training examples with regards to a topic/target within VAST (e.g. where a lexically similar topic is as previously defined in our paper and Alloway et al. [2] as having a GloVe embedding cosine similarity above 0.9) is correlated with whether our system correctly labels a given instance of that target/topic. If there is a positive correlation this may indicate that the topics with few examples may be noisy/biasing our results (as those with no similar training examples at all have higher F1-scores than those in the Few-shot setting). In the few-shot setting, we find a slight Pearson correlation (0.186, p-value =0.021) between the number of semantic similar training examples and the percentage of a given topic's test examples that are labeled correctly by our TATA model (0.2675, p-value=0.0008 for TAG model; 0.199, p-value=0.012 for TAW model; 0.215, p-value= 0.007 for deBERTa model). Together with the other models that perform worse in the Few-shot setting, this suggests some noisiness or a tendency of our model to be biased on topics in VAST that only have one or two training examples. As reported by Alloway et al.[3], there is a high, but not perfect agreement among labelers (75%), which may also help explain this phenomenon. We now include this discussion within our paper following the *Correlation with Inclusion in TAW dataset* section.\n\n\n## Zero-shot Stance Detection on the SemEval16 dataset \nWe now as suggested by Reviewer epNc, separately retrain our TAG layer without training instances that have SemEval16 topics (Hillary Clinton, Donald Trump, Legalization of Abortion, Climate Change, Atheism, Feminist Movement) in order to test our TAG and TATA models in a Zero-shot setting on the SemEval16 topics.  We note that these topics only consisted of 30 training instances (out of 13,477) in the original VAST dataset. We note that given that our TAG and TATA models are implicitly trained on the rest of the VAST dataset (through the TAG encoding layer), we now, in order to give a fairer comparison, combine the reduced VAST (i.e. without the SemEval16 topics) and the SemEval16 training sets when training each model. We give the results below.\n\n|Model|A|CC\t|LA|FM|\tHC|\tDT|\tAvg|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.528|0.347|0.606|\t0.656\t|0.698\t|0.660|\t0.566|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.479|0.423|0.618|\t0.647|0.727\t|0.623|\t0.586|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.428|0.360|0.642|0.644|0.664|0.609|0.558|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.521|0.416|0.629|0.669|0.654|0.638|0.588|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n\n\nWe again observe that our TATA model performs better within this setting compared to our DeBERTa model ( as well as previous state-of-the-art models which achieved an average score near 0.50 [e.g. JointCL] [1]).\n\n\n## Topic Aware/TAW Layer Potentially Just Utilizing the Topic/Target\nAs suggested by Reviewer epNc, we perform an additional ablation study to ensure that our TAW subnetwork does not simply utilize the topics to predict the stance of each example. As such we ran a study where for the Few and Zero-shot test sets within the the VAST dataset, we predicted the stance based on the topic. To do so, for our TAW encoding layer rather than passing [CLS] p\\_i [SEP] t\\_i [SEP] to the the layer as input, we pass in [CLS]  [SEP] t\\_i [SEP].\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|TAW|0.034,0.000,0.519,0.184 | 0.020,0.000,0.507,0.176\n|-----------|-----------|-----------|\n\nAs seen in the above table, just predicting based on the topic significantly hurts our results in the few-shot and zero-shot settings, with our model nearly always predicting the Neutral class. This behavior largely conforms to the definition of the Neutral class; without any input text, the stance of an empty string to a given topic should be neutral.  This suggests that our TAW model is not only picking up on the topic but is also heavily considering the text when predicting the stance.\n\n\n\n\n## Choice of Contrastive Learning for Topic-Agnostic/TAG Layer\nZero-shot stance detection seeks to identify the stance of texts toward targets/topics that were not seen during training [2]. One intuitive way of identifying the stance of a given text toward a topic/target is to train a model that identifies relevant generalized per-class stance features from text-target pairs that can then be utilized during inference when the target/topic was not seen during training. As found in prior works [3], contrastive learning can be utilized to create robust feature representations within classification tasks. Given our need to apply these features to topics/targets outside of our training and the use of contrastive learning for creating hard distinctions between different classes, we thus apply this methodology for creating generalized stance features which may be useful for predicting the stance of unseen topics. We thus *do* train a network where each stance class is placed in a distinct embedding space regardless of its topic so that features can be used later to train a  model that works on unseen topics.\n\n\n\n\n\n## Stylistic Improvements\nWe have fixed the errors outlined by Reviewer epNc. We have already taken several passes at our paper to ensure that we clean up the errors and stylistic problems. We further utilized Grammarly to ensure all spelling mistakes were fixed. In addition, we took the following steps to polish and fix errors within the original document:\n\n* The use of \"passage\" versus \"text\" when referring to the documents within the VAST dataset has been standardized to only use \"passage\".\n* All mentions of VAST dataset now have the same stylized text (using texttt)\n* We note that the place of h\\_taw and h\\_topic were switched within our original document on lines 414-415. This has been fixed. \n* We redid Figure 1 so that all errors have been fixed. We have removed the Microsoft images (Microsoft developed DeBERTa, hence our prior use of them), opting for a simpler and cleaner trapezoid encoder image. \n\n## Improved Reproducibility \n\nWe note that all data and models that we utilized for this work are open-sourced. Since the submission of this paper, Hanley et al. [4] have created a means (by emailing the authors) on their paper's GitHub for requesting their dataset. All other datasets that we utilized within this work (VAST, SemEval16) are public or were generated using the public Parrot and Dipper models. However, we plan to release the paraphrase datasets we utilized to train our TAG and TAW upon publication (although similar datasets can be generated using the Parrot and Dipper huggingface models). Finally, to improve the consistency and robustness of our results, we now train each model with a different random seed a total of five times and report the average of those five different runs. We present these updated results below:\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|DeBERTa|0.680, 0.683,0.900,0.755 | 0.659,0.657,0.869,0.728|\n|-----------|-----------|-----------|\n|TAG|0.681, 0.687,0.901,0.756 | 0.665,0.655,0.868,0.729|\n|-----------|-----------|-----------|\n|TAW|0.672, 0.709,0.903,0.760 | 0.656,0.677,0.869,0.736|\n|-----------|-----------|-----------|\n|TATA|0.695,0.711,0.905,0.771|0.665,0.683,0.873,0.741|\n|-----------|-----------|-----------|\n\n|Model|Qte |Sarc|Imp|mlS|mlT|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.709|0.713|0.667|\t0.578|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.710|0.731|0.668|\t0.568|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.712|0.722|0.683|\t0.585|0.693|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.714|0.730|0.693|\t0.603|0.704|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n\nWe again observe that all of our models perform better than the previous state-of-the-art JointCl (0.723 in Zero-shot and 0.715 in the Few-shot) [1]. Furthermore, our TAG and TAW models perform better than our DeBERTa model in the Zero-shot setting evidencing that our approach *does* help for stance detection on unseen topics. As seen, we again see that our TATA model performs the best in both the Zero-shot and the Few-shot settings. \n\n\n\n## References \n\n\n[1] Bin Liang, Qinlin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022b. Jointcl: A joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 81–91. Association for Computational Linguistics.\n\n[2] Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: a dataset and model using \ngeneralized topic representations. arXiv preprint arXiv:2010.03640.\n\n[3] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar.2019. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, pages 5628–5637. PMLR.\n\n\n[4] Hans WA Hanley and Zakir Durumeric. 2023. Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. arXiv preprint arXiv:2305.09820.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A combination of two models for stance detection such that one explicitly models the topic while the disregards the topic and models the target class is an interesting approach and can help improve stance detection to unseen, new topics.\n* The organization is good and it is easy to understand the narrative of the paper. \n\n", "reasons_to_reject": "* While being zeroshot better than fewshot is glossed away in the limitations section as being similar to other two works before this but in fact few shot is better than zero shot in most cases, and in cases where it is not, it is still better for some of classes- signifying an improvement. The macro average is almost always better for fewshot except for JointCL. The other models are able to improve with fewshot which means that the issue is not with the annotator disagreement, but maybe the model is probably overfit to the topic-agnostic model? While a more extensive comparison could be welcome, maybe it is good to include a discussion with results not including the topics from SemEval so that zeroshot could also be evaluated on SemEval as well, in the next version of the paper. \nHowever, in this paper, fewshot is consistently worse by 0.02-0.05 which is a strange phenomenon and there needs to be a discussion on this more than just a point on Limitations.\n* Topic-aware embeddings are trained with topics being one of the inputs (separated from text by a separator token), this could bias the TAW model to pick up on solely the topics, and topics could themselves be used to predict the stance. An attention map/ ablation of topic tokens/predicting based on topic alone from the input could be better to explore what the model focuses on. \n* The paper needs to significantly be improved stylistically. There are many grammatical and spelling errors, along with messy figures which should definitely be worked on for the next version.\n", "questions_to_authors": "* A. For topic-agnostic setting, all of one stance seems to shares a similar embedding space– does it mean despite the topics?  Why is contrastive loss preferred in this case especially if topics are not the same? \n", "paper_topic_and_main_contributions": "This paper introduces a new architecture for stance detection based on building a contrastive network that learns topic-agnostic embeddings (irrespective of topics, two inputs that have the same class of stance favor/neutral/against share similar values in embedding space) and topic-aware embeddings (two inputs that belong to the same topic are placed closer in the embedding space, irrespective of stance). This in combination with an attention network can help achieve state-of-the-art performance of stance detection.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 89: “Detection”\n* Line 157: “topic-topic” just once?\n* Line 159: t_i? Not t_i1?\n* Line 246: “datasett”\n* Line 276: comma after phrases (ambiguous otherwise)\n* paraphrase is spelled as “paraprhase” in multiple locations\n* Many other typos: spell-check recommended for the next version of the paper\n* Fig 1 caption: v_{text|topic} – is it h instead? Inconsistent use of “text” and “passage”. The Microsoft logo colors are distracting and not necessary in the diagram, and it could be made cleaner and easier to understand.\n\n"}}, {"review_id": "CeLrok39d4", "reviewer": "Reviewer_TUpb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer TUpb\nWe first would like to thank Reviewer TUpb for their valuable feedback. For our rebuttal, we begin by describing several details for why we chose contrastive learning and triplet losses for our topic-agnostic and topic-aware embedding layers. We further describe several additional steps that we took to bolster our work's reproducibility as well as some new experiments that we performed.\n\n\n## Choice of Contrastive Learning for Topic-Agnostic/TAG Layer\nZero-shot stance detection seeks to identify the stance of texts toward targets/topics that were not seen during training [1]. One intuitive way of identifying the stance of a given text toward a topic/target is to train a model that identifies relevant generalized per-class stance features from text-target pairs that can then be utilized during inference when the target/topic was not seen during training. As found in prior works [2], contrastive learning can be utilized to create robust feature representations within classification tasks. Given our need to apply these features to topics/targets outside of our training and the use of contrastive learning for creating hard distinctions between different classes, we thus apply this methodology for creating generalized stance features which may be useful for predicting the stance of unseen topics. We thus *do* train a network where each stance class is placed in a distinct embedding space regardless of its topic so that features can be used later to train a  model that works on unseen topics. We now include this reasoning in our introduction and at the beginning of section 5 (methodology section).\n\n## Choice of Triplet Loss for Topic Aware/TAW Layer\nWe note that prior works such as Alloway et al. [1] and JointCl [3] have focused on utilizing generalized topic representations by clustering the VAST dataset. These latent topic representations that they and we design, contain information about the topic relationships without requiring explicit human knowledge, allowing our approaches to extend to topics unseen in our VAST stance dataset. However, in previous approaches, by clustering the VAST dataset, these generalized representations were limited to the topics present within the dataset (i.e. those for which there were stance labels). However, by instead using an unstructured dataset and by first building a latent space where similar topics have similar representations and different ones have different representations, we manage to greatly expand the number of topics that we train on beyond the 4,641 present in the VAST training dataset to over 100,000. We further note that given our use of unstructured and unlabeled data, our approach is not limited to only these 100,000 topics. We now include this reasoning in our introduction and at the beginning of section 5 (methodology section).\n\nWe note that while training our TAW layer and while performing our contrastive learning objective to create our topic latent, we utilized as positive pairs, items that had different texts and two different paraphrased topics (e.g. 1: [CLS] Drinking cups are the great [SEP] Cups [SEP] and 2: [CLS] Drinking containers are awesome [SEP] Mugs [SEP]. We utilized different paraphrased topics, to ensure that our TAW layer did not just learn to just move together items with the same exact same topic/target (e.g. only \"cup\" and \"cup\" vs. \"cup\" and \"mug\").  Rather, as previously noted, we wished for our topic latent to learn similar representations to items with similar and rephrased topics. However, given that we utilize different but similar topics in our contrastive learning step, we wish to ensure that the similar topics only have similar embeddings rather than one single representative cluster embedding. Triplet loss, which is used for contrastive learning and has been shown to produce informative latent spaces [4], allows for this intra-class variance. This is because Triplet loss only ensures that negative examples are further away in the latent space than positive examples. As such we utilized a Triplet loss within this training step. Once trained, this topic latent, which implicitly contains the relationship between topics, can then be utilized to get informative generalized topic representations for use in training a stance classifier, which heavily depends on the features of the topic/target. \n \n## Stylistic Improvements\nWe have fixed the errors outlined by the other reviewers. We have already taken several passes at our paper to ensure that we clean up the errors and stylistic problems. We further utilized Grammarly to ensure all spelling mistakes were fixed. In addition, we took the following steps to polish and fix errors within the original document:\n\n* The use of \"passage\" versus \"text\" when referring to the documents within the VAST dataset has been standardized to only use \"passage\".\n* All mentions of VAST dataset now have the same stylized text (using texttt)\n* We note that the place of h\\_taw and h\\_topic were switched within our original document on lines 414-415. This has been fixed. \n* We redid Figure 1 so that all errors have been fixed. We have removed the Microsoft images (Microsoft developed DeBERTa, hence our prior use of them), opting for a simpler and cleaner trapezoid encoder image. \n\n## Improved Reproducibility \n\nWe note that all data and models that we utilized for this work are open-sourced. Since the submission of this paper, Hanley et al. [5] have created a means (by emailing the authors) on their paper's GitHub for requesting their dataset. All other datasets that we utilized within this work (VAST, SemEval16) are public or were generated using the public Parrot and Dipper models. However, we plan to release the paraphrase datasets we utilized to train our TAG and TAW upon publication (although similar datasets can be generated using the Parrot and Dipper huggingface models). Finally, to improve the consistency and robustness of our results, we now train each model with a different random seed a total of five times and report the average of those five different runs. We present these updated results below:\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|DeBERTa|0.680, 0.683,0.900,0.755 | 0.659,0.657,0.869,0.728|\n|-----------|-----------|-----------|\n|TAG|0.681, 0.687,0.901,0.756 | 0.665,0.655,0.868,0.729|\n|-----------|-----------|-----------|\n|TAW|0.672, 0.709,0.903,0.760 | 0.656,0.677,0.869,0.736|\n|-----------|-----------|-----------|\n|TATA|0.695,0.711,0.905,0.771|0.665,0.683,0.873,0.741|\n|-----------|-----------|-----------|\n\n|Model|Qte |Sarc|Imp|mlS|mlT|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.709|0.713|0.667|\t0.578|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.710|0.731|0.668|\t0.568|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.712|0.722|0.683|\t0.585|0.693|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.714|0.730|0.693|\t0.603|0.704|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n\nWe again observe that all of our models perform better than the previous state-of-the-art JointCl (0.723 in Zero-shot and 0.715 in the Few-shot) [3]. Furthermore, our TAG and TAW models perform better than our DeBERTa model in the Zero-shot setting evidencing that our approach *does* help for stance detection on unseen topics. As seen, we again see that our TATA model performs the best in both the Zero-shot and the Few-shot settings. \n\n## Additional Experiments\n\n\n### Few-shot Bias \nWe note that for our results, our models perform better in the Zero-shot setting. In order to ascertain whether the number of few-shot examples for particular topics in the VAST dataset affects the performance of our models during inference, we perform another experiment; namely we measure whether the number of lexically similar training examples with regards to a topic/target within VAST (e.g. where a lexically similar topic is as previously defined in our paper and Alloway et al. [2] as having a GloVe embedding cosine similarity above 0.9) is correlated with whether our system correctly labels a given instance of that target/topic. If there is a positive correlation this may indicate that the topics with few examples may be noisy/biasing our results (as those with no similar training examples at all have higher F1-scores than those in the Few-shot setting). In the few-shot setting, we find a slight Pearson correlation (0.186, p-value =0.021) between the number of semantic similar training examples and the percentage of a given topic's test examples that are labeled correctly by our TATA model (0.2675, p-value=0.0008 for TAG model; 0.199, p-value=0.012 for TAW model; 0.215, p-value= 0.007 for deBERTa model). This suggests some noisiness or a tendency of our model to be biased on topics in VAST that only have one or two training examples. As reported by Alloway et al.[3], there is a high, but not perfect agreement among labelers (75%), which may also help explain this phenomenon. We now include this discussion within our paper following the *Correlation with Inclusion in TAW dataset* section.\n\n### Zero-shot Stance Detection on the SemEval dataset \nWe now as suggested by Reviewer epNc, separately retrain our TAG layer without training instances that have SemEval16 topics (Hillary Clinton, Donald Trump, Legalization of Abortion, Climate Change, Atheism, Feminist Movement) in order to test our TAG and TATA models in a Zero-shot setting on the SemEval16 topics.  We note that these topics only consisted of 30 training instances (out of 13,477) in the original VAST dataset. We note that given that our TAG and TATA models are implicitly trained on the rest of the VAST dataset (through the TAG encoding layer), we now, in order to give a fairer comparison, combine the reduced VAST (i.e. without the SemEval16 topics) and the SemEval16 training sets when training each model. We give the results below.\n\n|Model|A|CC\t|LA|FM|\tHC|\tDT|\tAvg|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.528|0.347|0.606|\t0.656\t|0.698\t|0.660|\t0.566|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.479|0.423|0.618|\t0.647|0.727\t|0.623|\t0.586|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.428|0.360|0.642|0.644|0.664|0.609|0.558|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.521|0.416|0.629|0.669|0.654|0.638|0.588|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n\n\nWe again observe that our TATA model performs better within this setting compared to our DeBERTa model ( as well as previous state-of-the-art models which achieved an average score near 0.50 [e.g. JointCL] [3]).\n\n\n### Topic Aware/TAW Layer Potentially Just Utilizing the Topic/Target\nAs suggested by Reviewer epNc, we perform an additional ablation study to ensure that our TAW subnetwork does not simply utilize the topics to predict the stance of each example. As such we ran a study where for the Few and Zero-shot test sets within the the VAST dataset, we predicted the stance based on the topic. To do so, for our TAW encoding layer rather than passing [CLS] p\\_i [SEP] t\\_i [SEP] to the the layer as input, we pass in [CLS]  [SEP] t\\_i [SEP].\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|TAW|0.034,0.000,0.519,0.184 | 0.020,0.000,0.507,0.176\n|-----------|-----------|-----------|\n\nAs seen in the above table, just predicting based on the topic significantly hurts our results in the few-shot and zero-shot settings, with our model nearly always predicting the Neutral class. This behavior largely conforms to the definition of the Neutral class; without any input text, the stance of an empty string to a given topic should be neutral.  This suggests that our TAW model is not only picking up on the topic but is also heavily considering the text when predicting the stance.\n\n## References \n\n[1] Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: a dataset and model using \ngeneralized topic representations. arXiv preprint arXiv:2010.03640.\n\n[2] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar.2019. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, pages 5628–5637. PMLR.\n\n[3] Bin Liang, Qinlin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022b. Jointcl: A joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 81–91. Association for Computational Linguistics.\n\n[4] Hong Xuan, Abby Stylianou, and Robert Pless. 2020. Improved embeddings with easy positive triplet mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages\n903 2474–2482.\n\n[5] Hans WA Hanley and Zakir Durumeric. 2023. Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. arXiv preprint arXiv:2305.09820.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles an important problem in stance detection and proposes an effective solution.\n2. The use of contrastive learning and the incorporation of topic-aware features demonstrate the authors' innovation in addressing the challenges of generalization to unseen topics.\n3. The TATA model achieves state-of-the-art results on zero-shot and few-shot stance detection datasets, showing its effectiveness.\n", "reasons_to_reject": "1. Novelty of Topic Agnostic/TAG Encoding Layer and Topic-Aware/TAW Encoding Layer:\n\nWhile the paper introduces the Topic Agnostic (TAG) and Topic-Aware (TAW) encoding layers as novel components of the TATA model, there are a few aspects that could be further clarified or expanded upon to strengthen their novelty:\n\na. The rationale behind choosing contrastive learning for the TAG encoding layer needs more in-depth explanation. The paper briefly mentions that contrastive learning allows similar embeddings for instances with the same stance and different embeddings for instances with different stances. However, more discussion on how this approach improves stance feature modeling compared to other methods would enhance the novelty explanation.\n\nb. The use of a triplet loss in the TAW encoding layer for topic representation learning is intriguing. However, the paper could elaborate on why this approach is suitable for handling the noisy TAW dataset and how it effectively generates rich latent representations of topics. A comparison with alternative methods for topic-awareness in stance detection would provide better insight into the novelty of this approach.\n\n2. Analysis of Experiments:\n\nWhile the paper reports state-of-the-art performance on zero-shot and few-shot stance detection datasets, the analysis of the experiments could be more comprehensive。\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel stance detection mechanism called TATA (Topic-Agnostic and Topic-Aware) that addresses the challenge of generalizing stance detection to unseen topics. The authors combine contrastive learning with an unlabeled dataset of news articles to train topic-agnostic (TAG) and topic-aware (TAW) embeddings. The embeddings are then utilized in a full TATA model, achieving state-of-the-art performance on several public stance-detection datasets. The paper presents a clear methodology and achieves promising results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8qSfih5yxL", "reviewer": "Reviewer_dEgC", "summary": "", "questions": "1. Could you further explain the design of attention calculation between h_taw and h_topic, as well as the meaning of r_topic?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer dEgC\nWe first would like to thank Reviewer dEgC for their valuable feedback. For our rebuttal, we begin by describing why we decided to use two separate pre-trained layers for calculating topic-agnostic and topic-aware embeddings. We then describe in further detail the design of the attention mechanisms behind our TAW model.  We further describe several additional steps that we took to bolster our work's reproducibility as well as some new experiments that we performed.\n\n## Our Use of Two Separate Layers\nWe take a topic-agnostic and topic-aware approach given that we wish to take advantage of the two separate perspectives of determining stance (i.e. looking at generalized stance features that are common across different topics *and* trying to derive specific stance features particular to given (both seen and unseen) topics based on learned topic-related information). We now include this reasoning in our introduction and at the beginning of section 5 (methodology section). The reason why we take to developing separate embeddings for TAG and TAW is that the main crux of our approach is developing a topic latent that takes into account tens of thousands of diverse topics. As such, we cannot train these two together because we do not have stance labels for the passage attached to the topics we gather. As such we train these separately. We further note that while our TAW layer enables us to gather specific representations of each topic/target, our TAG layer in contrast enables us to design generalized stance features that are common within each stance class. By combining the generalized features and the rich topic embeddings, and training a network to utilize a combination of these perspectives, as shown by our results, we manage to further improve the generalization of our approach to unseen topics compared to past work.\n\n## Design of Topic Aware/TAW Layer\n\nWe note that prior works such as Alloway et al.[1] and JointCl [3] have focused on utilizing generalized topic representations by clustering the VAST dataset. These latent topic representations that they and we design, contain information about the topic relationships without requiring explicit human knowledge, allowing our approaches to extend to topics unseen in our VAST stance dataset. However, in previous approaches, by clustering the VAST dataset, these generalized representations were limited to the topics present within the dataset (i.e. those for which there were stance labels). However, by instead using an unstructured dataset and by first building a latent space where similar topics have similar representations and different ones have different representations, we manage to greatly expand the number of topics that we train on beyond the 4,641 present in the VAST training dataset to over 100,000. We further note that given our use of unstructured and unlabeled data, our approach is not limited to only these 100,000 topics. We now include this reasoning in our introduction and at the beginning of section 5 (methodology section).\n\nWe note that while training our TAW layer and while performing our contrastive learning objective to create our topic latent, we utilized as positive pairs, items that had different texts and two different paraphrased topics (e.g. 1: [CLS] Drinking cups are the great [SEP] Cups [SEP] and 2: [CLS] Drinking containers are awesome [SEP] Mugs [SEP]. We utilized different paraphrased topics, to ensure that our TAW layer did not just learn to just move together items with the same exact same topic/target (e.g. only \"cup\" and \"cup\" vs. \"cup\" and \"mug\").  Rather, as previously noted, we wished for our topic latent to learn similar representations to items with similar and rephrased topics. However, given that we utilize different but similar topics in our contrastive learning step, we wish to ensure that the similar topics only have similar embeddings rather than one single representative cluster embedding. Triplet loss, which is used for contrastive learning and has been shown to produce informative latent spaces [4], allows for this intra-class variance. This is because Triplet loss only ensures that negative examples are further away in the latent space than positive examples. As such we utilized a Triplet loss within this training step. Once trained, this topic latent, which implicitly contains the relationship between topics, can then be utilized to get informative generalized topic representations for use in training a stance classifier, which heavily depends on the features of the topic/target. \n\n\n\n## Design of Attention in TAW network\nWe note that the place of h\\_taw and h\\_topic were switched within our original document. This has been fixed. The purpose of the attention layer within our model is to determine the relationship between the DeBERTa representation of the topic/target t and the latent representation of the topic. In this way, we manage to better determine which aspect of the latent space representation of the topic is important in representing a particular instance of a topic/target. We thus represent this as the representation r\\_topic that captures the relationship between the topic/target t and its generalized latent space representation h\\_taw. We utilize a scaled dot product attention as proposed in Vaswani et al. [3].\n\n## Stylistic Improvements\nWe have fixed the errors outlined by Reviewer dEgC. We have already taken several passes at our paper to ensure that we clean up the errors and stylistic problems. We further utilized Grammarly to ensure all spelling mistakes were fixed. In addition, we took the following steps to polish and fix errors within the original document:\n\n* The use of \"passage\" versus \"text\" when referring to the documents within the VAST dataset has been standardized to only use \"passage\".\n* All mentions of VAST dataset now have the same stylized text (using texttt)\n* We note that the place of h\\_taw and h\\_topic were switched within our original document on lines 414-415. This has been fixed. \n* We redid Figure 1 so that all errors have been fixed. We have removed the Microsoft images (Microsoft developed DeBERTa, hence our prior use of them), opting for a simpler and cleaner trapezoid encoder image. \n\n\n## Improved Reproducibility \n\nWe note that all data and models that we utilized for this work are open-sourced. Since the submission of this paper, Hanley et al. [5] have created a means (by emailing the authors) on their GitHub for requesting their dataset. All other datasets that we utilized within this work (VAST, SemEval16) are public or were generated using the public Parrot and Dipper models. However, we plan to release the paraphrase datasets we utilized to train our TAG and TAW upon publication (although similar datasets can be generated using the Parrot and Dipper huggingface models). Finally, to improve the consistency and robustness of our results, we now train each model with a different random seed a total of five times and report the average of those five different runs. We present these updated results below:\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|DeBERTa|0.680, 0.683,0.900,0.755 | 0.659,0.657,0.869,0.728|\n|-----------|-----------|-----------|\n|TAG|0.681, 0.687,0.901,0.756 | 0.665,0.655,0.868,0.729|\n|-----------|-----------|-----------|\n|TAW|0.672, 0.709,0.903,0.760 | 0.656,0.677,0.869,0.736|\n|-----------|-----------|-----------|\n|TATA|0.695,0.711,0.905,0.771|0.665,0.683,0.873,0.741|\n|-----------|-----------|-----------|\n\n|Model|Qte |Sarc|Imp|mlS|mlT|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.709|0.713|0.667|\t0.578|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.710|0.731|0.668|\t0.568|0.676|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.712|0.722|0.683|\t0.585|0.693|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.714|0.730|0.693|\t0.603|0.704|\n|-----------|-----------|-----------|-----------|-----------|-----------|\n\n\nWe again observe that all of our models perform better than the previous state-of-the-art JointCl (0.723 in Zero-shot and 0.715 in the Few-shot) [2]. Furthermore, our TAG and TAW models perform better than our DeBERTa model in the Zero-shot setting evidencing that our approach *does* help for stance detection on unseen topics. As seen, we again see that our TATA model performs the best in both the Zero-shot and the Few-shot settings. \n\n## Additional Experiments\n\n\n### Few-shot Bias \nWe note that for our results, our models perform better in the Zero-shot setting. In order to ascertain whether the number of few-shot examples for particular topics in the VAST dataset affects the performance of our models during inference, we perform another experiment; namely we measure whether the number of lexically similar training examples with regards to a topic/target within VAST (e.g. where a lexically similar topic is as previously defined in our paper and Alloway et al. [2] as having a GloVe embedding cosine similarity above 0.9) is correlated with whether our system correctly labels a given instance of that target/topic. If there is a positive correlation this may indicate that the topics with few examples may be noisy/biasing our results (as those with no similar training examples at all have higher F1-scores than those in the Few-shot setting). In the few-shot setting, we find a slight Pearson correlation (0.186, p-value =0.021) between the number of semantic similar training examples and the percentage of a given topic's test examples that are labeled correctly by our TATA model (0.2675, p-value=0.0008 for TAG model; 0.199, p-value=0.012 for TAW model; 0.215, p-value= 0.007 for deBERTa model). This suggests some noisiness or a tendency of our model to be biased on topics in VAST that only have one or two training examples. As reported by Alloway et al.[3], there is a high, but not perfect agreement among labelers (75%), which may also help explain this phenomenon. We now include this discussion within our paper following the *Correlation with Inclusion in TAW dataset* section.\n\n### Zero-shot Stance Detection on the SemEval dataset \nWe now as suggested by Reviewer epNc, separately retrain our TAG layer without training instances that have SemEval16 topics (Hillary Clinton, Donald Trump, Legalization of Abortion, Climate Change, Atheism, Feminist Movement) in order to test our TAG and TATA models in a Zero-shot setting on the SemEval16 topics.  We note that these topics only consisted of 30 training instances (out of 13,477) in the original VAST dataset. We note that given that our TAG and TATA models are implicitly trained on the rest of the VAST dataset (through the TAG encoding layer), we now, in order to give a fairer comparison, combine the reduced VAST (i.e. without the SemEval16 topics) and the SemEval16 training sets when training each model. We give the results below.\n\n|Model|A|CC\t|LA|FM|\tHC|\tDT|\tAvg|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|DeBERTa|0.528|0.347|0.606|\t0.656\t|0.698\t|0.660|\t0.566|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAG|0.479|0.423|0.618|\t0.647|0.727\t|0.623|\t0.586|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TAW|0.428|0.360|0.642|0.644|0.664|0.609|0.558|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|TATA|0.521|0.416|0.629|0.669|0.654|0.638|0.588|\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n\n\nWe again observe that our TATA model performs better within this setting compared to our DeBERTa model ( as well as previous state-of-the-art models which achieved an average score near 0.50 [e.g. JointCL] [2]). \n\n\n### Topic Aware/TAW Layer Potentially Just Utilizing the Topic/Target\nAs suggested by Reviewer epNc, we perform an additional ablation study to ensure that our TAW subnetwork does not simply utilize the topics to predict the stance of each example. As such we ran a study where for the Few and Zero-shot test sets within the the VAST dataset, we predicted the stance based on the topic. To do so, for our TAW encoding layer rather than passing [CLS] p\\_i [SEP] t\\_i [SEP] to the the layer as input, we pass in [CLS]  [SEP] t\\_i [SEP].\n\n|Model|Zero-Shot VAST |Few-Shot VAST |\n|-----------|-----------|-----------|\n| |Pro, Con, Neutral, All|Pro, Con, Neutral, All|\n|-----------|-----------|-----------|\n|TAW|0.034,0.000,0.519,0.184 | 0.020,0.000,0.507,0.176\n|-----------|-----------|-----------|\n\nAs seen in the above table, just predicting based on the topic significantly hurts our results in the few-shot and zero-shot settings, with our model nearly always predicting the Neutral class. This behavior largely conforms to the definition of the Neutral class; without any input text, the stance of an empty string to a given topic should be neutral.  This suggests that our TAW model is not only picking up on the topic but is also heavily considering the text when predicting the stance.\n\n## References \n\n[1] Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: a dataset and model using \ngeneralized topic representations. arXiv preprint arXiv:2010.03640.\n\n[2] Bin Liang, Qinlin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022b. Jointcl: A joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 81–91. Association for Computational Linguistics.\n\n[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.\n\n[4] Hong Xuan, Abby Stylianou, and Robert Pless. 2020. Improved embeddings with easy positive triplet mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages\n903 2474–2482.\n\n\n[5] Hans WA Hanley and Zakir Durumeric. 2023. Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. arXiv preprint arXiv:2305.09820.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper constructs a new dataset and expands the existing dataset for pre-training.\n2. The proposed method achieves better performance on zero-shot stance detection.", "reasons_to_reject": "1. The idea of topic-agnostic and topic-aware features itself has been proposed in previous work, thus this work is incremental to further advance this idea.\n2. The main contributions of this work are still vague, which makes it hard to assess how solid this work is in advancing the research area.\n3. The overall presentation of the paper is problematic with many typos and mistakes.", "questions_to_authors": "1. Could you further explain the design of attention calculation between h_taw and h_topic, as well as the meaning of r_topic?", "paper_topic_and_main_contributions": "This paper mainly pre-trains two embedding layers to extract topic-agnostic and topic-aware features and utilizes these two kinds of features for zero-shot stance detection. Specifically, it constructs a new dataset including news articles and topics for pre-training the topic-aware embedding layer, and expands the original VAST dataset with paraphrases for pre-training the topic-agnostic embedding layer.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. This article has a lot of grammar errors, as well as some typos, such as Line 83 “77.9 compared to 0.723...”, Line 157 “passage-topic-topic-passage”, Line 513 “TGA=Net”, Line 576 “(4) mlS - (4)”...\n2. In Table 3, “Marijuana” should be “Abortion”.\n3. The figure of the overall method is relatively rough and there are some typos in it."}}]

Author rebuttal:

Meta-review: This paper proposes a method for decoupling topic-agnostic and topic-aware representations for stance detection. This is done via two seperate embedding layers, trained via contrastive learning. The authors further introduce a new dataset for pre-training the topic-aware representations. This method leads to favourable empirical results. However, the method proposed is incremental and there are issues with the presentation. Some reviewers also ask for additional ablation results, which have been reported in the author response.
Decision: Accept-Main