Dynamic Stance: Modeling Discussions by Labeling the Interactions
Authors: Blanca Calvo Figueras
Irene Baucells
Tommaso Caselli
Venue: EMNLP 2023 Findings (2023)
Keywords: stance
corpus
multi-lingual
cross-topic
TL;DR: We introduce dynamic stance, a new annotation schema to model debates by labelling the interactions, for which we have created a multilingual corpus.

Abstract: Stance detection is an increasingly popular task that has been mainly modeled as a static task, by assigning the expressed attitude of a text toward a given topic. Such a framing presents limitations, with trained systems showing poor generalization capabilities and being strongly topic-dependent. In this work, we propose modeling stance as a dynamic task, by focusing on the interactions between a message and their replies. For this purpose, we present a new annotation scheme that enables the categorization of all kinds of textual interactions. As a result, we have created a new corpus, the Dynamic Stance Corpus (DySC), consisting of three datasets in two middle-resourced languages: Catalan and Dutch. Our data analysis further supports our modeling decisions, empirically showing differences between the annotation of stance in static and dynamic contexts. We fine-tuned a series of monolingual and multilingual models on DySC, showing portability across topics and languages.

Dynamic Stance:
Modeling Discussions by Labeling the Interactions
BBlancc Calvo Figueras o Irene Baucells Tommaso Caselli
"Brcelonaa Supercomputing Center
"HiTZ Center - Ixa, University of the Basque Country UPV/EHU
CCLCG, University of Groningen, The Netherlands eus, rirene. baucells@bser @ t. casellllerugg.
Abstract
vestigated between pairs of texts. Key contribu- tions in this direction are the RumourEval shared tasks (Derezynski et al., 2017; Gorrell et al., 2019) 9), the Fake News Challenge (Pomerleau and Rao, 2017: Hanselowski et al., 2018) and, more recently, the Stanceosaurus corpus (Zheng et al., 2022a). Al- though different with respect to the nature of the data, in all cases systems are required to determine the stance of a message with respect to a previous claim with the goal of annotating stance as a com- ponent of fact or rumor verification. Consequently, the used labels focus on denying or supporting some facts instead of modeling the actual internal dynamics of an online exchange.
Stance detection is increasingly popular task that has been mainly modeled as a static task, by assigning the expressed attitude of text toward a given topic. Such framing presents limitations, with trained systems showing poor generalization capabilities and being strongly lopie-dependent. In this work, we propose modeling stance dynamic task, by focus- ing the interactions between message and their replies. For this purpose, we present a new annotation scheme that enables the cate- gorization of all kinds of textual interactions. As result, we have created a new corpus, the Dynamic Stance Corpus (DySC), consisting of three datasets in two middle-resourced lan- guages: Catalan and Dutch. Our data analysis further supports modeling decisions, empir- ically showing differences between the anno- tation of stance in static and dynamic contexts. We fine-tuned series of monolingual and mul- tilingual models on DySC, showing portability across topies and languages.
Our contributions In this work, we present annotation scheme to model stance dynami- cally way account for the evolution of online exchange between two users. We introduce Dynamic Stance detection as a new independent task, which captures different insights from Static Stance detection. To this end, we have manually annotated a new multilingual corpus with short exchanges between pairs of users from different social media platforms. We have worked with two middle-resoureed languages, namely Catalan and Dutch, and conducted an extensive set of experi- ments investigating the portability of the trained models across topics and languages. We also show the relation between Dynamic Stance and Static Stance labels.
1 Introduction
Stance detection consists of identifying opinions, perspectives, or attitudes expressed in texts. The task has been found to be relevant as compo- nent for fact verification, argument mining, and media analysis (Boltužić and Snajder, 2014; Baly al., 2018; Bauwelinck and Lefever, 2020; Guo al., 2022), but also on its own (Hardalov al., 2022). The standard setting for stance detection the adoption of a static perspective, where different texts are labeled with a three-class label approach, i.e., being in favour, neutral or against a given topic (Somasundaran and Wiebe, 2009; Moham- mad et al., 2016; Conforti et al., 2020; Clark et al., 2021, amog others). One of the limits of this ap- proach is its strong dependence on topics, resulting poorly portable systems (Reuver et al., 2021). Other works have taken a different direction: rather than keeping a fixed topic, stance in-
The remainder of the paper is structured as fol- lows: in Section 2, we present a critical overview of previous work to highlight the differences and innovations of our proposal. Section 3 introduces the Dynamic Stance task and presents annota- tion scheme. The Dynamic Stance Corpus (DySC) is discussed in Section focusing on the data col- lection, the annotation process, and analyzing the relation between Static and Dynamic Stance. Our experiments and results are presented in Section 5.

---

Finally, Section 6 summarises our findings and dis- cusses future work.
of labels when compared RumourEval (support- ing, refuting, discussing, querying, and irrelevant). Stanceosaurus is the largest available corpus for stance with respect to claims. Besides the size of the corpus, the authors obtain a macro Fl score of 0.53 when testing on unseen claims,' with perfor- mances dropping to 0,40 for zero-shot cross-lingual transfer in Hindi and Arabic.
2 Related Work
Stance has been modeled extensively as a static task in which a text is tagged with respect to its attitude towards specific topic or target (Mo- hammad et al., 2016; Küçük and Can, 2020; Cao et al., 2022). Methods for stance classification have evolved alongside the developments in Natural Lan- guage Processing. moving away from feature- based approaches up to fine-tuning and prompting pre- trained models (Ferreira and Vlachos, 2016; Aker et al., 2017: Fang et al.. 2019; Zhang et al., 2020; Allaway and McKeown, 2020; Zheng et al., 2022h, among others). While performances on bench- mark data have seen improvements, issues such as balance among classes and portability of systems across topics and languages remain open research questions.
The foeus of these two studies was to use inter- actions between texts verify claims. Therefore, they use labels such as support and deny, which entail that the departing message must be claim, which is the case in RumourEval and Stancesaurus. However, in online interactions, most of the mes- sages we encounter are not claims but rather opin- ions and perspectives, which can not always be supported or denied, but rather agreed upon or dis- agreed.
When it comes to language- resources for stance detection in Catalan and Dutch, previ- ous work is quite limited and focused on modeling Static Stance. For Catalan, stance is investigated in Twitter and only with respect to the "Catalan In- depedence" topic (Taulé al., 2017, 2018; Zotova et al., 2020). For Dutch, the only available dataset is Wang et al. (2020), where stance annotated targeting face masks and social distance measures.
Stance detection has only been approached as a relation between two texts on a few occasions. As already mentioned, remarkable contributions are the Fake News Challenge (Pomerleau and Rao, 2017), RumourEval (Derczynski et al., 2017; Gorrell et al., 2019), and the Stanceosaurus cor- pus (Zheng et al., 2022a).
In the Fake News Challenge, the authors aim at using disagreements between news pieces to de- tect false information. They design an annotation scheme based on Ferreira and Vlachos (2016) to estimate the stance of a body text from a news ar- ticle relative to a headline in English. The labels they define are agrees, disagrees, discusses, and unrelated. While the headline and the content of an article are very often in agreement, in online de- bates this is not frequent. Therefore, more granular labels are needed to model messages engaging with each other in an exchange of opinions.
The corpus comprises messages from different plat- forms (Twitter, Reddit, and Nu.nl). Additionally, the CoNTACT corpus (Lemmens et al., 2022) iden- tifies stance towards vaccines using Twitter and Facebook posts, but their data is not publicly avail- able.
Our work departs from previous studics by fo- cusing on modeling online exchanges intrinsically. The goal of this approach is to be able to capture the relation of text with respect to disputed claim as well as any other dialogic interaction in online forums micro-blogginn platforms. This will be relevant for tasks such as fact verification, but also for controversy detection (Hessel and Lee, 2019; Figueras et al., 2023), argument mining (Lawrence and Reed, 2020; Dutta et al., 2022; Ruiz-Dolz et al., 2022), detecting previously verified claims (Nakov et al., 201 8; Shaar et al., 2022; Nakov et al., 2021), and discourse analysis.
In Eval,rEval, the goal is verify a rumor in English by modeling the discourse around it. From this perspective, the authors designed an annotation scheme that focuses on making explicit whether the replies to rumor on Twitter support or deny the given claim. Four labels are available: support, deny, comment, and query. The Stanceosaurus corpus (Zheng et al., 2022a) follows a similar ap- proach. In this work, the authors provide specific claims and look for Twitter messages related to those claims. The corpus is multilingual, covers multiple topics, and uses a more grained set
Fine-tuning BERT-large model (Devlin et al., 2019) FFinc-tuning a mBERT model (Devlin et al., 2019)

---

3 Dynamic Stance Annotation
Agree:
The reply agrees with the parent mes-
Inspired by this previous work, we hypothesize that the stance of a message towards a previous mes- sage (Dynamic Stance) can show more linguistic patterns than the static approach, as it will be less topie-dependent than modeling agreement over topic. While Dynamic Stance is an independent task, for the purpose of this work, we also anno- tated Static Stance. Having access to both annota- tion levels is essential empirically investigate the interactions between these two schemes, showing similarities and differences.
sage.
The reply disagrees with the par- ent message. This can include questioning making fun of the parent message with clear dislike towards it. If the reply questions the parent message and it shows a clear dis- agreement with it (c.g. use of irony, rhetorical questions, etc.), the reply will be labeled as Disagree.
Disagree:
The reply is not against the par- ent message, but it expresses additional opin- ions/information with which we do not know if the parent message would agree.
Elaborates:
Static Stance As highlighted in Section 2, pre- vious work has mainly targeted stance annotation in isolation, i.e., with respect to given topic. An- notation efforts Static Stance are very similar each other, mainly differentiating for the topics and text types (Conforti et al., 2020). To maximize compatibility and reusability of data from previous work, we inherit the labels and definitions proposed by Mohammad et al. (2016). In particular, we have a four-label annotation scheme:
Query: The reply will be annotated Query if the message questions or requires more ar- gumfens/prooflevidenee about the parent mes- sage. It will not be annotated as Query if the question is rhetorical, ironic, or has clear stance towards the parent message.
The reply is on the same topic as the purent message but does not express any posi- tion or opinion about it. If the parent message was a genuine question with no opinion (also not ironic or rhetorical), the reply should be labeled as Neutral.
Neutral:
Favour: The message expresses a positive
stance opinion with respect the topic;
Against: The message expresses negative
stance or opinion with respect to the topic;
The reply is not on the same topic and it has no relation to the parent message.
Unrelated:
Neutral: The message is about the topic but does not hold any recognizable stance, e.g., it can report facts about the given topic;
NA: One or both of the messages in the pair are unintelligible, e.g., URL
NA (Not Applicable): The message may or may not express stance, but content does not address the target topic; or the message is intelligible.
In our guidelines, we emphasize that, in the cases in which the parent message does not hold an opin- ion, the label of the reply must be assigned on the basis of the content. Therefore, if the parent mes- sage is a descriptive text with no opinion of its own (e.g. "EMA approves another COVID vaccine"). the stance of the reply toward the content of the text should still be considered, and a reply such as "More poison for us." should be labeled as Disagree (i.e. the reply disagrees with the facts described in the parent message but not necessarily with its author, who did not express any opinion). 3
Dynamic Stance To capture the dynamic dimen- sion of stance, the target message must always be put in relation to its direct parent, that means, the message that the target message is replying to. Dy- namic Stance annotation inherently a two-step process: first, the annotators have to read the parent and the reply messages, secondly, they must assign them a label by answering the following question: "What is the stance or opinion of the reply with respect to its parent?". To properly answer this question, the set of labels must be richer but, the same time, manageable, i.e., easy to understand and remember. To define this set we have departed from the labels in Pomerleau and Rao (2017), and extended them to the granularity of Zheng et al. (2022a). We have identified seven labels:
Figure 1 illustrates example of these schemes. From Static Stance perspective, the parent mes- sage is against vaceines, while the reply is in favor. At the Dynamic layer, the parent and reply stand
"The full guidelines can be found here: https: t/githbb. com/pr con//pojeecte--o aina- aina/dynamiiaccccccc.a hetaaadddamiisstancaaa stance- analysis/tree/ main/create, dateeeeatttteeeaaaaaaaaaaaaaaeee wwwp//tnoonnoooo/////o wowpo//gudddeeeeeeeeeeeeeeee

---

Dynamic Stance
Parent message
Static Stance
Escandalós! La Generalitat proposa el carnet de vaccinació. Als no vacunats ens posaran en camps de concentració potser?
Against
Vaccines
Outrageous! The government proposes a vaccination card. Are they going to put unvaccinated in concentration camps?
Disagree
Reply
Això és tret de context. Han copiat el model britànic. Potser serà necessari per culminar pla de vaccinació massa laxa.
Favour
Vaccines
This is out of context. They just copied the British system. Maybe will be necessary culminate vaccination plan that has been too lax.
Figure 1: Example the two annotation schemes. The Dynamic Stance scheme (left) assesses the relation between the reply and the parent message, while the Static Stance scheme (right) identifies the relation of each message with the topic, in this case, vaccines. Messages are extracted from the DySC corpus.
in disagreement relationship. we will show in Section 4.2, the dynamic label cannot always be inferred from the static ones.
campaign; (ii) the regulation of rent prices; (iii) the expansion of Barcelona's Prat airport: (iv)
the legalization of surrogate pregnancy; and (v)
the rigging in a TV music contest that decided the Spanish candidate for Eurovision." 6 We used the Twitter Academic API to collect the messages. To maximize the number of messages that contain a reply, have developed pipeline that first finds whether a message contains any of the target key- words for a thread during the relevant time period, and subsequently checks the target message has a parent message. If so, both the parent and the tar- get messages are retrieved and stored. In this way, we collected 11,078 unique messages organized in 6,673 pairs, which have been manually annotated.
4 The Dynamic Stance Corpus
The Dynamic Stance Corpus (DySC) is multi- lingual, and multi-topic corpus in Catalan and Dutch. This section will illustrate the methods that were used for collecting and annotating the data the two languages. We will also analyze the resulting labels.
4.1 Data Collection
Catalan Data Collection For Catalan, col- lected data from two online platforms: Twitter and the Racó Català forum. Given the different nature of the platforms, we have kept the data separated.
For the Catalan Stance and Emotion Racó dataset (CaSERa), 7 we followed slightly different ap- proach. We collected messages belonging to ran- dom online exchanges compliant with the follow- ing requirements: (a) exchanges must have mini- mal length of two messages and maximum length of four messages; (b) up to two branches of the same conversation can be collected; and (c) all mes- sages in the exchange have to be between 300 and
The Catalan Stance and Emotions Twitter dataset (CaSET)² has been obtained by extracting messages belonging to five different threads. Each thread identifies a controversial topic in Catalan soci- cty and it has been identified by means of key- words and limited to specific time periods (see Ap- pendix A for the full list of keywords and time periods per thread). We targeted the following discussion threads: (i) the COVID-19 vaccination
"hups://elpais " com/ielevissoom/2022.000 21w.02202222000-l--piiaa-a 01-30-e--publico- cuntr--eljurrdd las reacciones-ante-la- -ea.aatt-----iiiiiiii-----eaat en-el-benidorm-fest hem
hnttps:/www.rrcccastt cat/ "This dataset has also been annotated with emotion labels, but this level of annotation not reported in this paper. http:///hugiinaaae.eaaaaaassss/p co/daataset//pojjcte-iiin https://huggingfate
"This dataset has also been annotated with emotion labels, but this level of annotation is not reported in this paper. Find the data in https://huuginge co//attssessprojecceeaaa LasERa-catall stancetanee--ee emotions-race
Find the data in
emotions- wwttoonssttwtttttt twitter DaSET-catalan -ttalannnsttte---ee stance-emations

---

15 characters. The requirements prevent the col- lection of very long forum posts, which may have contained many opinions, and very long threads, which may have over-represented some topics in the dataset. We obtained 14,000 pairs, which have been manually annotated.
The CaSERa dataset has been annotated for Dy- namic Stance only, lacking any specific topic for the Static Stance. We have followed the same an- notation procedure as in CaSET and employed the same annotators. Similarly to the scores obtained in CaSET, the average Cohen's 's kappa is 0.58. The judge was involved in 18.17% of the cases.
Dutch Data Collection We have collected a com- parable Twitter dataset for Dutch. The Dutch Stance Twitter dataset (DuST) aims at testing the feasibility of cross-lingual transfer of Dynamic Stance. For this reason, we limited the collection of the data common online platform, i.c., Twitter, and common topic, namely the COVID-19 vacci- nation campaign. Focusing on the same online plat- form and common topic a strategy to minimize language variety factors that may negatively affect the cross-lingual transfer learning approach (Ram- poni and Plank, 2020). To maximize the similar- ities with CaSET, we have used the same pipeline approach. Wherever possible, we have used trans- lations of the Catalan keywords for the COVID-19 vaccination. However, we integrated the keyword sets with others that mirrored culturally specific aspects of the vaccination in the Netherlands and Belgium (c.g.. 3G, or coronabewijs). We obtained 14,840 messages of which 2,485 pairs have been manually annotated.
For DuST, we have followed a slightly different annotation procedure. A team of three students performed the annotations as part of their thesis project. Similarly to the expert annotators, students received training and went through calibration ses- sions. As in CaSERa, we have annotated the mes- sages only for Dynamic Stance. The final label was assigned using majority vote (two out of three). In cases in which the majority threshold could not be reached, the annotators held discussion sessions to reach a unanimous agreement. The average Co- hen's kappa is 0.52. Although lower, the scores are comparable to CaSET.
Tables 1 and 2 summarize the annotated data for CaSET, while Table 3 reports the Dynamic Stance annotations for CaSERa and DuST. Concerning the Static Stance labels of the CaSET dataset, we can observe that there is a relatively balanced pres- ence of all the labels, but the NA label takes al- most a third of the data. By contrast, the Dynamic Stance labels of the CaSET dataset (Table 2) have a very unbalanced distribution, although we barely find messages labeled Unrelated (1.52%) or NA (0.38%). Given that the messages the same in both datasets, this difference comes from model- ing the stance toward a static, prefixed topic, or modeling it toward other messages thus capturing a variety of attitudes. As claimed in Section 2, the empiric results show that the Agree label is not common in online debates.
4.2 Data Annotation and Analysis
The data annotation has been conducted by three separate teams. For Catalan, expert annotators have been recruited and trained.
For CaSET, a total of six annotators plus two judges were involved. The judges had the role of resolving disagreements when unanimous an- notations could not be reached. The annotation of the Static and Dynamic Stance has been con- ducted in parallel. A team of two annotators and one judge worked on the Static Stance, while we had a team of four annotators plus one judge for Dy- namic Stance. The annotation of Dynamic Stance is more challenging than the static one: for Dy- namic Stance, the average Cohen's 's kappa between independent annotators was 0.57, while this score jumps to 0.83 for the static annotation. This means that the judges were involved in 1,284 cases for the Dynamic Stance annotation (18.95 95% of the in- stances), and 2,087 for the Static Stance annotation (18.84% of the instances).
The distribution of the Dynamic Stance labels across the three datasets is quite similar. Agree, Query and Unrelated are infrequent, and most in- teractions are labeled with Disugree, Elaborate or Neutral. CaSET is the dataset with more disagree- ments, while CaSERa the one with more neutral interactions. This makes sense since we specifi- cally selected controversial topies for CaSET. The different nature of the sources (i.c. micro-blogging and forums) can also have an impact on the differ- ent distribution. As for DuST, the debate around vaccines seems to spark slightly fewer disagree- ments and more reinforcements of the same opin- ions than in Catalan, with the most frequent class
All annotators had a regular contract and received a fair payment in line with Spanish labor laws.

---

Rent regulation
Airport expansion
Surrogate pregnancy
TV show rigging
Label
Vaccines
Total
Favour
982 (22.78%)
990 (36.26%)
91 (4.83%)
160 (10.84%)
75 (10.9%)
2,298 (20.74%)
Against
532 (12.34%)
405 (14.84%)
474 (25.31%)
712 (48.24%)
27 (3.92%)
2,150 (19,41%)
Neutral
1,806 (41.8%%
433 (15.86%)
487 (26.0%)
293 (19.85%)
242 (35.17%)
3,261 (29.44%)
NA
991 (22.99%)
902 (33.04%)
821 (43.83%)
311 (21.07%)
344 (50.0%)
3,369 (30.41%)
Total
4,311
2,730
1,873
1,476
688
11,078
Table I: Statisties of the Static Stance annotations in CaSET. All the numbers refer to individual messages, which can correspond to parent messages or to replies. The percentages refer to the frequency of the label in the topic.
Rent regulation
Airport expansion
Surrogate pregnancy
TV show rigging
Label
Vaccines
Total
Agree
52 (1.81)
35 (2.25%)
20 (1.86%)
(1.04%)
3 (0.75%)
119 (1.76%)
Disagree
1,154 (40.21%)
620 (39.85%)
447 (41 47%%)
466 (53.69%)
125 (31.17%)
2,812 (41.52%)
Elaborate
771 (26.86%)
621 (39,91%)
419 (38.87%)
288 (33.18%)
183 (45.64 %)
2,282 (33. 70%)
Query
157 (5.47%)
26 (1.67%)
28 (2.6%)
21 (2.42%)
8 (2.0%)
240 (3.54%
Neutral
699 (24.36%)
192 (12.34%)
148 (13.73%)
78 (8.99%)
74 (18.45%)
1,191 (17.58%)
Unrelated
17 (0.59%)
61 (3,92%)
11 (1.02%)
6 (0.69%)
8 (2.0%)
103 (1.52%)
NA
20 (0.7%)
(0.06%)
5 (0.46%)
26 (0.38%)
Total
2,870
1,556
1,078
868
401
6,773
Table 2: Statisties of the Dynamic Stance annotations in CaSET. All the numbers are based message pairs. The percentages refer to the frequency of the label the topic.
being Elaborate.
Parent + Reply
Fav. Agai.
Agai. Fav.
Fav. Fav.
Agai. Agai.
Neut.
Neut.
Agree
0
32
19
27
Label
CaSERa
DuST
Disagree
313
169
110
59
234
Agree
218 (1.55%)
29 (1.17%)
Elaborate
27
33
244
204
221
Disagree
3,183 (22.74%)
822 (33.08%)
Query
8
10
49
Elaborate
3,992 (28.52%)
884 (35.57%)
Neutral
12
11
30
7
215
Query
859 14%%)
170 (6.84%)
Unrelated
0
0
0
Neutral
5,564 (39.75%)
574 (23.1%%
NA
0
0
0
0
Unrelated
175 (1.25%)
5 (0.2%)
TOTAL
360
216
426
292
746
NA
9 (0.06%)
(0.04%)
Total
14,000
2,485
Table 4: CaSET distribution of Static Stance labels per parent and reply message pair against their Dynamic Stance annotations.
Table 3: Label distribution for CaSERa and DuST.
Restricting the analysis to the topic distribution in CaSET, the highest rate of disagreements is in the discussion around the legalization of surrogate pregnancy. This topic is also the one with the high- est number of messages against it. Rent regula- tion is the topic with more messages in favor. For the COVID-19 vaccination, the difference between messages in favor and against smaller than in the other politically controversial topics, with Neutral being the largest class. Finally, half of the messages about the TV show rigging not on the rigging itself, but mostly other topies surrounding the involved artists. The observations raised by simply comparing the two annotation levels of this data are not very intuitive. For this reason, we analyzed these relations further.
Static vs. Dynamic Stance A closer look at the relation between the two levels of annotation can be seen in Table 4, in which we report the Dynamic Stance values by pairs of Static Stance values for the parent and the reply messages. While the pairs with different Static Stance (either Favour-Against or Aguinst-Favour) are mostly labelled as Disagree, many pairs of messages with the same Static Stance labels (e.g. Favour-Favour) are also labelled as Disagree.
Looking at these cases observe that they are messages that in fact disagree with each other. but without being in disagreement with the central topic (in the following examples, vaccination). Ex- amples (1) and (2) illustrate two of these instances.

---

The examples have been extracted from CaSET and translated into English.
base (He et al., 2021) for sequence classification. We used learning rate of 1-e5, a batch size of 8. a warmup of 0.06, and trained for epochs with an Adam optimizer and cross-entropy loss. We kept the best checkpoint on the development set. To deal with the issue that some labels in the Dy- namic Stance annotation have very few instances, merged the label Agree with Elaborate, Query with Neutral, and Unrelated with NA. That way, we had 4 labels for both Dynamic and Static Stance.
(1) Favourrarent Ravour -iippyy-DiigggreDynammtt
a.
Parent: How many people will die and infect new people every day that passes with vaccination stopped? 10,000(vac- cine emoji)= emoji).
b. Reply: Covid is the discase to fight. A vaccine is the shield to eradicating it, and it is given to a healthy person. If any type of incident is recorded and is as serious as death, it must be investigated.
(2) Pauutal Reply -iisaa-----eeeeee Dynamic
All topics experiments For our monolingual ex- periments, we generated fixed train, development, and test sets from CaSET, For the Static Stance, we have 9,073 instances in train and maintained 1,000 instances each for development and test. For the Dynamic Stance, we used 4,771 instances for train- ing and 1,000 instances each for development and test. The CaSERa data has been used as additional training materials, increasing the training set to 18,771.
a. Parent: On the one hand, no one can force them to get vaccinated. On the other hand, no one can forced allow people who refuse to be vaccinated into their business. Which rights must be respected? Serious question. In the USA, unvaccinated children are not allowed to go to public school. b.
Reply: That's a lic. They don't 't vaccinate chil- dren here nor in the USA.
The fact that these disagreements would be missed by the Static Stance annotation scheme, and the fact that these examples are very common (31.37% of the NeutralParent Neeiiarrnn--uuurallleee Reply pairs and 25.88% of the Parent-PPPPat-aaaaa
Table 5 shows the resulting macro Fl score for the models. In both cases, the monolingual model achieves the best scores with RoBERTa-large. For the Dynamic Stance, we observe a positive impact of the additional training materials from CaSERa, even if it comes from a different platform and with specific topic. Although not directly compara- ble, the higher scores for Static Stance are in line with the TAA scores, which show that Statie Stance is an casier task to model.
Reply ones), shows that the Static Stance not enough to model the in- teractions of a debate. For tasks such as con- troversy detection, this phenomenon might arise from disagreements in specific issues of the topic. which might not be reflected by the Against/Favont scheme. In addition, for monitoring the discourse around a topic, being able to classify the interac- tions would provide deeper knowledge of the dis- cussion.
Static Stance Detection (CaSET)
Dynamic Stance Detection (CaSET) (CaSET-CaSERa)
Model
5 Automatic Stance Detection
RoBERTa-I-ca
0.71
0.62
0.65
mDeBERTaV3
0.66
0.43
0.48
We developed several baseline models for both Static and Dynamic Stance detection. We investi- gate the abilities of pre-trained language models to model Static and Dynamic Stance with two set- tings: one where the topic is seen at training time and another where the topic is excluded. Lastly, we investigate the cross-lingual transfer abilities of multilingual model on DuST.
Table 5: Results RoBERTa-I-ca and mDeBERTaV3 models for Static and Dynamic Stance in Catalan with all topies on the training set. Scores correspond to macro F1. The best scores per type of stance are in bold.
In Table 6 we report the results of the best sys- tem per label for both Static and Dynamic Stance. While labels that clearly express a stance (Favour, Against, and Neutral) obtain comparable results for both phenomena, the largest difference affects the NA label. This is likely due to differences in the training instances, with the Dynamic Stance having very few NA cases.
Experimental setup For our experiments, we fine-tuned both the Catalan language model RoBERTa-large-ca-22 (Armengol-Estte et al., 2021) and the multilingual model mDDBERTa-v3-
haap::://uggingfaae wttp:/acecccpopcceeeeee.c. aina/ roberta- taatttabbbe----tee
htagss:://ugggngte htt.ugginnaace.ccccirrrrsffff mdeberta-v3- tabbbrat----bseee

---

Stance Type
Label
Label F1
Macro F1
covering the vaccine topic, thus maximising the compatibility with DuST. In all of our experiments, we used mDeBERTaV3. investigate the benefit of cross-lingual data for Dynamic Stance classifi- cation, we first developed two baselines using the Catalan and the Dutch data only. For Catalan, we used training set of 1,868 instances and devel- opment and test sets of 500 instances each. For Dutch, we have 1,483 instances in the training, and 500 each for development and test. For the cross- lingual experiments, we concatenated the training materials.
Favour
0.66
Against
0.65
Static
0.71
Neutral
0.55
NA
0.88
Disagree
0,68
Elaborate
0.67
Dynamic
0.65
Neutral 0.58
NA
0.56
Table 6: Results per label of the models with best per- formance (in bold in Table 5) for Static and Dynamic Stance in Catalan.
Table 8 reports the results. In both languages, the use of the language-specifie training data ob- tains limited results, especially for the Dutch data. However, when concatenating the languages, results improve, with Dutch being the one benefit- ing the most.
Zero-shot topic experiments One of the moti- vations for creating Dynamic Stance annotation scheme is the possibility to improve the cross-topic portability of models. To test this, we conducted a series of zero-shot topic experiments which exclude one topic time from the training set and use it the test set. Therefore, the sizes of the test and training change in each of the experiments, the details about these sets can be found Appendix B.
Train - Test
Macro
Catalan Catalan
0.43
Catalan+Dutchh Catalan
0,47
We have experimented only with RoBERTa-large since it is the model that obtained the best results when using all the topics. The results are illustrated in Table 7.
Dutch Dutch
0.19
Catalan+Duttch Dutch
0.37
Table 8: Results of the cross-lingual experiments for Dynamic Stance detection. The best scores per test language are in bold.
We observe that, while the cross-topic perfor- mance of Dynamic Stance drops bit from the model that had seen all topics, it still learns. This is the case with the Static Stance cross-topic models, which have below-chance performance most topics. Although not directly comparable, these results suggest that Dynamic Stance can be modeled across topics in an easier way than Static Stance. It has to be noted that the addition of the CaSERa data to augment the dynamic training set docs not have much effect on the results.
Conclusion and Future Work
In this work, have investigated the task of stance detection with focus on dynamic interactions between parent messages and their replies. Our approach addressed the limitations of traditional Static Stance detection models, which heavily rely on specific topics and exhibit poor generalization capabilities. By introducing a novel annotation scheme and creating the Dynamic Stance Corpus (DySC), provided new perspective the cate- gorization of textual interactions, which captures different insights from the data. Through our data analysis, we demonstrated empirical differences between the annotation of stance in static and dy- namic contexts. Furthermore, DySC is fully devel- oped for non-English languages, namely Catalan and Dutch.
Static Stance Detection (CaSET)
Dynamic Stance Detection (CaSET) (CasET+CaSERa)
Test Topic
Vaceines
0.10
0.43
0.47
Rent regulation
0.14
0.40
0.44
Airport expansion
0.18
0.49
0.54
Surrogate pregnancy
0.41
0.47
0.48
TV show rigging
0.24
0.43
0.43
Table 7: Results of the zero-shot topic experiments in Catalan. "Test Topic" indicates the topic used at test time and excluded from the training set. Scores correspond macro F1.
Using generic monolingual pre-trained model for Catalan (RoBERTa-ca--) we achieve macro Fl score of 0.65 for Dynamic Stance using all topies on training. We also show that, while Dynamic Stance models exhibit some learning on the zero-
Cross-lingual experiments For the cross- lingull experiments, we employed only data from CaSET

---

shot topic scenario (with an average score of 0.47), Static Stance models go from a macro Fl score of 0.71 with all topics on train, to an average score of 0.21 when in the zero-shot scenario. We therefore conclude that Dynamic Stance is easier to model in cross-topic scenarios. As for the cross-lingual experiments, the results show that there is some knowledge transfer between languages. While the results leave room for improvement, they are close to the ones obtained by similar previous work such as the Stancesaurus (reported in Section 2).
lines to validate the annotations and identify lower- bound thresholds. The use of general language pre- trained models, RoBERTa-ca-I and mDeBERTaV3, may have had an impact on the results, consider- ing the different language varieties of DySC. Future work will need to address this issuc.
Acknowledgements
We acknowledge Racó Català for the release of their data for this project. This work has been par- tially funded by Generalitat de Catalunya, within the framework of Projecte AINA. Blanca Calvo is funded by Disargue (TED2021 113300010C21 funded by MCIN/AEI /10.13039/50 10001 1033 and by European Union NextGenerationEU/ PRTR, and by a PIF UPV/EHU 2023 doctoral grant.
This study represents an important initial step to- ward effectively modeling the task of stance detec- tion in dynamic textual interactions. Future work will address some of the pending issues by expand- ing the annotated data to other languages and in- vestigating new methods (including the use of in- context learning) to improve the portability of the Dynamic Stance models.
References
Limitations
Ahmet Aker, Leon Derezynski, and Kalina Bontcheva. 2017. Simple open stance classification for rumour analysis. Proceedings of the International Con- ference Recent Advances in Natural Language Pro- cessing, RANLP 2017. pages 31-39, Varna, Bulgaria. INCOMA Ltd.
We have used a manually-curated selection of key- words for our data collection. While we have en- sured to cover relevant keywords (including syn- onyms) for all of the topics we have targeted, the list of keywords may not be exhaustive and contain potential bias. We aim to expand the data collection by using few-shot in-context learning approaches to extend the list of keywords. Furthermore, we have to acknowledge some intrinsic limitations of the Twitter API, which prevents the collection of all potentially relevant messages.
Emily Allaway and Kathleen McKeown. 2020. Zero- Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8913- 8931, Online, Association for Computational Lin- guistics.
Jordi Armengol-Estappe Casimiro Pio Carrino, Carlos Rodriguez-Penate Penagos, Ona de Gibert Bonet, Carme Armentano-Ollee Aitor Gonvalez-Agiree, Maite Melero, and Marta Villegas. 2021. Are multilin- gual models the best choice for moderately under- resourced languages? A comprehensive assessment for Catalan, Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 4933 4946, Online. Association for Computational Linguistics.
The data collection and annotation have been conducted with all possible human resources avail- able and applying fair treatment to all parties in- volved. We leave the collection of additional mes- sages and the expansion to other languages for fu- ture work. Our guidelines are made available and so is the code we have used to collect the mes- sages." !l thus offering opportunities to other inter- ested researchers to expand DySC and set up larger benchmark for Dynamic Stance.
Ramy Baly, Mitra Mohtarami, James Glass, Lluis Mårquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact check- ing in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pages 2 21-27, New Orleans, Louisiana. Association for Computational Linguistics.
In our experiments, we did not use all of the fine- grained classes we have identified for the annota- tion. This does not mean that these labels should be discarded. Our decision was motivated by the relatively few instances we were able to identify.
Our models leave room for improvement. Our experiments represent an implementation of base-
Nina Bauwelinck and Els Lefever. 2020. Annotating topics, stance, argumentativeness and claims in Dutch social media comments: A pilot study. In Proceed- ings of the 7th Workshop on Argument Mining, pages
1| Find hoth in
dynamic-stance stance-nallyiis

---

8-18. Online. Association for Computational Lin- guistics.
William Ferreira and Andreas Vlachos. 2016. Emer- gent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1163-1168, San Diego, California. Associa- tion for Computational Linguistics.
Filip Boltušić and Jan Snajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumen- lation Mining, pages 49-58, Baltimore, Maryland. Association for Computational Linguistics.
Blanca Calvo Figueras, Asier Cuttéérrez-Faddine and Marta Villegas. 2023. Anticipating the Debate: Predicting Controversy in News with Transformer- based NLP. Procesamiento del Lenguaje Natural, 70(0):123-133 Number: 0.
Rong Cao, Xiangyang Luo, Yaoyi Xi, and Yaqiong Qiao. 2022. Stance detection for online public opin- ion awareness: An overview. International Journal of Intelligent Systems,
1944-1 1965. _eprint: 100p///000000000022333.111
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ah- met Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-20119 Task 7: Ru- mourEval, Determining Rumour Veracity and Sup- port for Rumours. Proceedings of the 13th Inter- national Workshop on Semantic Evaluation, pages 845-854, Minncapolis, Minnesota, USA. Associa- tion for Computational Linguisties.
Thomas Clark. Costanza Conforti, Fangyu Liu, Zaiqiao Meng, Ehsan Shareghi, and Nigel Collier. 2021. Inte- grating transformers and knowledge graphs for Twit- ter stance detection. In Proceedings of the Seventh Workshop on Noisy fiser-generated Text (W-NUT pages 304-312, Online, Association for Com- putational Linguistics.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won' t-they: A very large dataset for stance detection on Twitter. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational L Linguistics, pages 1715- 1724, Online. Association for Computational Linguis- tics.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206.
Andreas Hanselowski, Avinesh PVS, Benjamin Schiller. Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Con- ference on Computational Linguistics, pages 1859- 1874, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Leon Derezynski, Kalina Bontcheva, Maria Liakata. Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for ru- mours. In Proceedings of the 11th International Workshop on Semantic Evaluation (S0m pages 69-76, Vancouver, Canada. Association for Computational Linguistics.
Momchil Hardalov, Arnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance de- tection for mis- and disinformation identification. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1259-1277, Seattle, United States. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3; 3: Improving deberta using clectra-style pre- training with gradient-disentanllee disentangied embedding shar- ing.
Subhabrata Dutta, Jeevesh Juneja, Dipankar Das, and Tanmoy Chakraborty. 2022. Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining? In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7774 7786, Dublin, Ireland. Association for Computational Linguistics.
Jack Hessel and Lillian Lee. 2019. Something's Brew- ing! Early Prediction of Controversy-causin Posts from Discussion Features. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages (648-1659, Minneapolis, Min- nesota. Association for Computational Linguistics.
Wei Fang. Moin Nadeem, Mitra Mohtarami, and James Glass. 2019. Neural multi-task learning for stance prediction. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 13-19, Hong Kong, China. Association for Computa- tional Linguistics.
Dilek Küçük and Fazli Can. 2020. Stance detection: A survey, ACM Comput. 53(1.
John Lawrence and Chris Reed. 2020. Argument Mining: A Survey. Computationnal Linguistics. 45(4):765-818.

---

Jens Lemmens, Jens Van Nooten, Tim Kreutz, and Wal- ter Daelemans, 2022. CoNTACT: A Dutch COVID- 19 adapted BERT for vaccine hesitancy and argumen- tation detection. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 6837-6885, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.
all previously fact-checked claims in a document. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 2069-20888 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
S Swapna Somasundaran and Janyce Wiebe. 2009. Rec- ognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Confer- ence Natural Language Processing of the AFNLP, pages 226-234, Suntec, Singapore. Association for Computational Linguisties.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2011 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-201) 2016). pages 31- 41, San Diego, California. Association for Computa- tional Linguistics.
Mariona Taulé, M Antonia Martí, Francisco M Rangel, Paolo Rosso, Cristina Bosco, Viviana Patti, et al. 2017. Overview of the task on stance and gender de- tection in tweets on catalan independence at ibereval 2017. In CEUR Workshop Proceedings. volume 1881, pages 157-177. CEUR-WS.
Preslav Nakov, Alberto Sarrón-Cedeñne Tamer Elsayed,
Reem Suwaileh, Lluís Mårquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Ver- ification of Political Claims. In Experimental IR Meets Multilinguality, Multimodality, and Interac- tion, Lecture Notes in Computer Science, pages 372- 387, Cham. Springer International Publishing.
Mariona Taulé, Francisco M Rangel Pardo, M Antônia Martí, and Paolo Rosso. 2018. Overview of the task multimodal stance detection in tweets catalan# loct referendum. In IberEval@ SEPLN, pages 149- 166.
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed. Alberto Barrón- Cedeñño,, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated fact-checking for assisting human fact-checkers, In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence, IJCAI-21, pages 4551 -45884 International Joint Con- ferences on Artificial Intelligence Organization. Sur- vey Track.
Shihan Wang, Marijn Schraagen, Erik Tjong Kim Sang, and Mehdi Dastani. 2020. Public sentiment gov- ermmental COVID-19 measures Dutch social me- dia. In Proceedings of the 1t Workshop on NLP for COVID-19 19WID-19/A (Part 2) at EMNLP 2020, Online, Associ- ation for Computational Linguistics.
Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xi- aofei Xu, and Kuai Dai. 2020. Enhancing cross- target stance detection with transferable semantic- emotion knowledge. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 3188 3197, Online. Association for Computational Linguistics.
Dean Pomerleau and Delip Rao. 2017. Fake news chal- lenge. Exploring how artificial intelligence tech- nologies could be leveraged to combat fake news. url: https://www. faw.eneaeeeeeehhlleneeee vrg/tvisited on 03/13/2020).
Jonathan Zheng, Ashutosh Baheti, Tarek Naous, Wei Xu, and Alan Ritter. 2022a. Stanceosaurus: Classi- fying stance towards multicultural misinformation. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 2132 2151, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics
Alan Ramponi and Barbara Plank. 2020. Neural unsu- pervised domain adaptation in N survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages (838-6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Myrthe Reuver, Suzan Verberne, Roser Morante, and Antske Fokkens. 2021. Is stance detection topic- independent and cross-topic generalizable? a repro- duction study. In Proceedings of the 8th Workshop on Argument Mining, pages 46-56, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Kai Zheng, Qingfeng Sun, Yaming Yang, and Fei Xu. 2022b. Knowledge stimulated contrastive prompting for low-resource stance detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages (668-118,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Elena Zotova, Rodrigo Agerri, Manuel Nuñez, and Ger- man Rigau. 2020. Multilingual stance detection in tweets: The Catalonia independence corpus. In Pro- ceedings of the Twelfth Language Resources and Evaluation Conference, pages 1368-1375, Marseille, France. European Language Resources Association.
Ramon Ruiz-Dolz, Stella Heras, and Ana Garefa-Fornes. 2022. Automatic Debate Fivaluation with Argumen- tation Semantics and Natural Language Argument Graph Networks.
Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022. Assisting the human fact-checkers: Detecting

---

1st August A Keywords and time frames for data collection
Topic
Keywords
Time frames
vacunes, vacuna, vaccí, vaccines, vaccinat, vacunades, COVID-19 pfizer, astrazeneca, vacuna moderna, passaport covid, tercera dosis, efectes, secundaris, dosi reforg, vacunats, tercera dosis, vacuneu-vos, vacunem-nos, vacunet, vacune't
1st January 31st December 2021
vaccines
regulació rent, preus de rent, regularrents, regulació dels preus, okupes, RegulemEIssentsJA, desnonaments, desnonament, fons voltor, fons voltors, seguretat als propietaris, propietaris immobiliaris, immobiliari, blackstone, lobbies immobiliaris, Ilei de vivenda
Rent
1st January 31st December 2020
regulation
ampliació airport, #hoampliaciosirport, 19S, airport delta, airport el prat, sostenible airport, més verd del món, delta Ilobregat, soroll avions, airport internacional, ricarda, ambiental airport
Airport expansion
-31st October 2021
30th January
6th February 2017
gestació surrogate, ventre de rent, mares portadores, inscripció de menors, dona gestant, gestaciosurrogate, paternitat surrogate, ventres de rent, comprar nens, embaràs subrogat, inscripció de nens, gestant, comprar criatures, pares biològics, comprar' bebes, filiar un nadó, explotació reproductiva, maternitat surrogate
1t August
16th November 2017
Surrogate pregnancy
1st February
24th April 2019
30th August
6th September 2020
26th February
27th March 2022
TV show
benidorm fest, tanxungueiras, chanel, rigoberta, ay mama
29th January
rigging
hai frontciras, slomo, rtve jurat, curovisión, tongo
15st February 2022
Table A.I: Keywords and time frames used retrieve Twitter messages in CaSET.
Topic
Keywords
Time frame
vaccins, vaccin, vaccinatie, mRNA, pfizer, astrazeneca, sputnik,
COVID-19 moderna, 2G, 3G, vaccinatiebewijs coronatoeganbewij..
1st September
gevaccincerd, vaccineren, prikken, prik, inenten, vaccineer, eerste prik, tweede prik, booster, booster vaccinatie, derde prik. boostervaccinatie, coronabewijs, coronavaccins, coronavaccinatie
vaccines
31st December 2021
Table A.2: Keywords and time frame used to retrieve Twitter messages in DuST.

---

B Zero-shot topic experiments
(a) Dynamic Stance
Test
Train
Train
Dev
Test
topic
(CaSET only)
(CaSET+CaSERaa)
vaccines
3,403
17,403
500
2,870
rent
4,717
18,717
500
1,556
airport
5,195
19,195
500
1,078
surrogate
5,405
19,405
500
868
benidormfest
5,872
19,872
500
401
(b) Static Stance
Test topic
Train
Dev
Test
(CaSET only)
vaccines
6,267
500
4,308
rent
7,844
500
2,731
airport
8,678
500
1,897
surrogate 9,138
500
1,437
benidormfest 9,873
500
702
Table B.I: Size of the sets in the Zero-shot topie experiments.


Reviews:
[{"review_id": "nSuGYP1bXp", "reviewer": "Reviewer_yDAp", "summary": "", "questions": "l09: describe what you mean by parent messages or use another term\n\nl233ff: wrt the Query label -- this label seems to be solely driven by the grammatical structure of the sentence, instead of its function in the discourse. Reading the first sentence of the description `expressing doubts about the content' goes somewhat against that because it's more general. \n\nl256ff: The example oversimplifies the stance here: 'More poison for us' is not a disagreement with the previous message (rather with the implied content that more Covid vaccines is a good thing). It's rather `expressing doubts about the content'. The characterisation of 'Disagree' lists dislike of the parent message in the case of rhetorical questioning, which is not the case in the example here. --> tighten up the descriptions of the labels\n\nSection 4.2. - given stance annotation might be quite subjective, it'd be good to know in how many cases adjudication had to be performed to arrive at the gold labels. \n\nExample 1b: I don't see how in the static stance, this is Favour_Reply... Maybe adjust the English translation in 1a, it's not really clear what it means...\n\nl490: merging the labels 'Query' with 'Neutral' seems counterintuitive based on the descriptions above...\n\nl545ff: What do you think is the reason for the improvement? \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R1, thank you for the review and comments. Concerning your comments:\n- LREC paper: We disagree with this comment. We present a new task (see also R3) in the Sentiment Analysis track and a new resource. The paper fits EMNLP’s criteria expressed in the CfP.\n- “parent message”: We agree that this wording can be confusing before the framework is explained in Section 3. We will reword as follows in the Abstract: “interaction between a message and their replies”. Furthermore, in Section 3, we will add to L207 the following clarification: “To capture the dynamic dimension of stance, the target message must always be put in relation to its direct parent, that means, the message that the target message is replying to”.\n- Query definition: The label “Query” in the Dynamic Stance framework is not grammatical, but rather discursive. In our annotation guidelines, the label is defined as follows: “A reply will be annotated as Query if the message questions or requires more arguments/proof/evidence about the parent message. It won’t be annotated as Query if the question is rhetorical, ironic, or has a clear stance towards the topic of the parent message.”  The definition of the label Disagree in the annotation guidelines also specifies that “If the reply questions the parent message and it shows a clear disagreement with it (e.g. use of irony, rhetorical questions, etc.), the reply will be labeled as Disagree.” Additionally, the guidelines contain examples that show the difference between Query and Disagree. We will include these clearer and more specific definitions in the paper and the full guidelines will be published with the final version.\n- L248: Following your observations, we have realized that the sentence preceding the “More poison for us” example is not specific enough. We will change it to “In our guidelines, we emphasize that, in the cases in which the parent message does not hold an opinion, the label of the reply must be assigned on the basis of the content.\" Therefore, “More poison for us” exemplifies a reply to a parent message with no stance associated (“EMA approves another COVID vaccine”), in which, as emphasized in the paragraph, the label must be assigned precisely based on the content of the parent message. In this case, the reply disagrees with the content of the parent message (“these vaccines are poison, and this new vaccine is another poison”), i.e., it shows a clear dislike of the content of the parent message.\n- Section 4.2 role of the judge: Thanks for this comment. We will add the following sentence after line 343: “That means that the judges were involved in 1,284 cases for the Dynamic Stance annotation (18.95% of the instances), and 2,087 for the Static Stance annotation (18.84% of the instances).” And, in line 356, we will add: “The judge was involved in 18.17% of the cases.”\n- Example 1b: We have checked and this is not an annotation error, but rather an interpretation error (R2 also had issues with this example). Here is our full-fledged explanation: the Reply does not express a disagreement against the vaccination policy in general (“A vaccine is the barrier to eradicating it”), rather it is a call of caution and for an investigation of suspicious deaths (which is exactly what the EU did with the AstraZeneca vaccines). However, this call to be cautious (which is still in favor of vaccines as they are “the barrier to eradicating the disease”) disagrees with the original message which requests a fast pace with the vaccination program. When seen in isolation, all our annotators labeled this message as being in Favour of vaccines. However, when looking at the dynamic stance, two annotators labeled the interaction as Disagree, and two of them labeled it as Neutral. The judge, however, rightly assessed that the messages were disagreeing. It is a non-trivial example, but it perfectly illustrates our task, the difficulty of the annotation, and the difference between static stance and its dynamic counterpart. Given that two reviewers misread this example, we will extend the explanation of this case.\n- Merge of “Query” and “Neutral”: When it comes to the stance of the reply towards the parent message, “Query” and “Neutral” labels do not express any clear stance, thus merging them is not counterintuitive.\n- Improvement in cross-lingual experiments: For us, this is clearly due to a larger amount of training instances, together with the fact they are on the same topic and use the same annotation scheme.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is merit in the fact that the authors annotate both static and dynamic stance in order to show the relations between the two levels of analysis. I also like the crosslinguistic approach in Catalan and Dutch. ", "reasons_to_reject": "The paper has a couple of gaps that I address below in the more detailed comments. Overall I think this is rather an LREC paper, given its focus. ", "questions_to_authors": "l09: describe what you mean by parent messages or use another term\n\nl233ff: wrt the Query label -- this label seems to be solely driven by the grammatical structure of the sentence, instead of its function in the discourse. Reading the first sentence of the description `expressing doubts about the content' goes somewhat against that because it's more general. \n\nl256ff: The example oversimplifies the stance here: 'More poison for us' is not a disagreement with the previous message (rather with the implied content that more Covid vaccines is a good thing). It's rather `expressing doubts about the content'. The characterisation of 'Disagree' lists dislike of the parent message in the case of rhetorical questioning, which is not the case in the example here. --> tighten up the descriptions of the labels\n\nSection 4.2. - given stance annotation might be quite subjective, it'd be good to know in how many cases adjudication had to be performed to arrive at the gold labels. \n\nExample 1b: I don't see how in the static stance, this is Favour_Reply... Maybe adjust the English translation in 1a, it's not really clear what it means...\n\nl490: merging the labels 'Query' with 'Neutral' seems counterintuitive based on the descriptions above...\n\nl545ff: What do you think is the reason for the improvement? \n\n", "paper_topic_and_main_contributions": "The paper describes two new datasets for dynamic stance annotation in two languages, Dutch and Catalan. It's shown that a cross-lingual approach to stance detection helps performance for Dutch, for which there is only a smaller dataset available. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6IZtpD28UH", "reviewer": "Reviewer_Q591", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R2, thank you for the review and comments. Concerning your comments:\n- Research question: As stated in line 180 and also highlighted by R3, we present Dynamic Stance detection, a new stand-alone task with an accompanying annotated dataset. In Section 2, we motivate our task and identify the differences with the state of the art (lines 12-16; lines 137-146; lines 161-166). Additionally, as stated in line 182, we annotate Static Stance to draw the relation between these two frameworks, as the latter has been the most used approach for stance detection in the last few years. Our main claim from the comparison is that, while Static Stance has the issue of being difficult to generalize to new topics, Dynamic Stance can do this easier. Therefore, this new and independent way of modeling stance can be used for new topics for which we do not have any annotated data. We will make this aspect clearer in the Introduction and Conclusion. We will also acknowledge that these two independent tasks capture slightly different insights from the input. \n- Static Training / Dynamic Training: We agree that the wording is not optimal, although it only affects Table 5 and Table 7. In Section 5, we present the experiments as Static Stance Detection and Dynamic Stance Detection. In the final version, we will change the titles in Table 5 and Table 7 to “Static Stance Detection” and “Dynamic Stance Detection”. In both cases, this is a text classification task and the models were trained by fine-tuning Transformer models. In the Static Stance detection model, the input was the target text. In the Dynamic Stance detection model, the input was the parent message paired with the reply.\n- Test set split: The test subsets used for Static and Dynamic Stance Detection are the same. However, given that the Dynamic approach is modeled by pairs of messages, there are fewer instances in both the training and the test set. \n- Claim not supported by example: As we have answered to R1, we disagree. We have checked and this is not an annotation error, but rather an interpretation error. Here is our full-fledged explanation: the Reply does not express a disagreement against the vaccination policy in general (“A vaccine is the barrier to eradicating it”), rather it is a call of caution and for an investigation of suspicious deaths (which is exactly what the EU did with the AstraZeneca vaccines). However, this call to be cautious (which is still in favor of vaccines as they are “the barrier to eradicating the disease”) disagrees with the original message which requests a fast pace with the vaccination program. When seen in isolation, all our annotators labeled this message as being in Favour of vaccines. However, when looking at the dynamic stance, two annotators labeled the interaction as Disagree, and two of them labeled it as Neutral. The judge, however, rightly assessed that the messages were disagreeing. It is a non-trivial example, but it perfectly illustrates our task, the difficulty of the annotation, and the difference between static stance and its dynamic counterpart. Given that two reviewers misread this example, we will extend the explanation of this case.\n- The use of the term “Dynamic” is motivated to differentiate the standard stance detection tasks (what we call “Static”), in which texts are labeled towards a fixed topic or claim, from the new proposed task, in which every text is labeled towards a previous text and can be about any topic. Therefore, the reference claim becomes dynamic i.e. it can be a different one in every instance.\n- Finally, thank you for pointing out the two typos in the paper, they have already been changed for the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Useful resource: The paper presents an annotated dataset in two languages and on several topics, while ensuring cross-lingual comparability.\n- Baseline results: The paper presents basic experiments, including on cross-topic and cross-lingual transfer, that will be useful as a reference for future work.", "reasons_to_reject": "Unfortunately, the presentation of the research questions and the results is not very clear, and especially the relation to “static” stance detection remains unclear: Does this paper propose an improvement over previous approaches, or does it propose an entirely new task?\n- Unclear presentation: The experiments with automatic stance detection (Section 5) focus on a comparison between “Static Training” and “Dynamic Training”. However, the two concepts are not formally defined. Is “Dynamic Training” multi-task learning on dynamic and static stance detection? If so, the use of different subsets used in the experiments (L496–501) make it difficult to see whether this objective has a positive effect. Or is “Dynamic Training” just learning to classify the relation to the parent text, independent of the static stance? In that case, it is not clear in what respect this approach “address[es] the limitations of traditional Static Stance detection” (L569), since it would appear to be a new task.\n- Claim not supported by example: A main claim of the paper (L420f.) is that static stance and dynamic stance are not equal: Two statements can have the same stance w.r.t a topic but still disagree with each other. While Table 4 provides quantitative support for this claim, Example 1 (L437–446) does not seem to be a valid illustration. In fact, Reply (b) seems to be critical of the COVID-19 vaccination campaign, and not in favor. I would encourage the authors to check the example and make sure it does not contain an annotation error.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about classifying whether a reply agrees or disagrees with the preceding text. The authors propose to differentiate this task (“dynamic” stance detection)  from the task of “static” stance detection, which is independent from any preceding text. (It is unclear whether the authors conceptualize dynamic stance detection as a stand-alone task, or as an objective that should be learnt jointly with static stance detection.)\nThe authors annotate social media posts on several controversial topics in Catalan and Dutch. They then present some automatic classification results for the proposed task, based on fine-tuning a pre-trained encoder model.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Thanks for the paper, it was interesting to read. Personally, I found it challenging to develop an intuition for “dynamic stance”, because “dynamic” usually means something that is changing over time. Sections 3 and 4 provide more clarity, but maybe intuition could also be helped by an alternative word choice.\n- L206: “intelligible” -> “unintelligible”\n- L392: “Disagre” -> “Disagree”"}}, {"review_id": "Jj4OWEUPbR", "reviewer": "Reviewer_m8Kb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear R3, thank you for the review and comments. Concerning your comments:\n- Annotation analysis: We do agree that the two annotation frameworks are different tasks that capture different insights. This is mentioned in line 180, but we will make it clearer since R2 also had doubts regarding this. As stated in line 181, we annotate both levels to investigate the interactions in a descriptive way, which in turn exemplifies that Dynamic Stance can be used in many discussions, even if they are not on the specified topic. We will make it clear that the frameworks have different goals and thus capture different information. \n- Automatic stance detection: Regarding the test subsets used for Dynamic and Static Stance, I can confirm that they are the same subsets. However, given that the Dynamic approach is modeled by pairs of messages, there are fewer instances in both the training and the test set.  Regarding the “All topic experiments”, we have added a table in the final version that addresses the distribution of topics in these subsets.\n- Thanks for pointing out the contribution by Joshi et al. (2020). Our wording was not the best. Given that Joshi et al. (2020) consider Dutch as a good example of the \"Underdog\" class (lots of unsupervised data, few labeled data), and given the results of the EU project European Langauge Equality 2 of both Catalan and Dutch, we will reword the presentation of both languages as middle-resourced languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a new way of modeling the tasks of stance detection dynamically instead of statically. I see this as a promising new way of framing the task.  \n\nTogether with the new task, the paper presents a new resource in Catalan and Dutch annotated for static and dynamic stance detection. This is a rich new dataset that allows for analyzing static and dynamic stance annotations. \n\nThe paper presents initial experiments with language models that give valuable first insights into the generalizability of the two annotation frameworks.", "reasons_to_reject": "I did not see real reasons to reject the paper. There are, however, a couple of points that should be clarified: \n\nAnnotation analysis: I am really not so sure about the comparison of static versus dynamic stance annotations and their consequences for the data (Section 4.2). The annotations resulting from both annotation frameworks are considered side by side and it is discussed what aspects of a debate they can capture. My problem with this comparison is that the two annotation frameworks essentially capture a different task; one (static) assesses a text with respect to a claim or topic (predefined, explicit) while the other (dynamic) assesses a text with respect to another text. In the latter, the claim in the first text remains entirely implicit. While the overall topic of a discussion may be related to the topic of covid vaccines, the first text may contain a claim that is entirely unrelated. This is not really an insight that arises from the data. Rather, it is inherent to the design of the two different frameworks. I think it is valid to show the consequences of these differences in the data, but I think it is important to be clear about the fact that the frameworks have different goals and thus capture different information. \n\nUpdate after rebuttal: Thank you for the clarification. I am confident that the authors can make the necessary changes in the final version of the paper. \n\nAutomatic stance detection: Please clarify whether the train and test splits for dynamic and static stance detection are the same. I understand that there is a difference in the number of instances (and labels) due to the different annotation frameworks, but right now, it is not clear whether the texts are the same. It would be great to have a bit more information about topic overlap between train and text in the ‘all topics experiments’. \n\nUpdate after rebuttal: Thank you for the clarification!\n\nTopic independence is a huge plus of the dynamic way of modeling stance. I think it would be good to make this (and its consequence for automatic approaches) more prominent in the framing of the paper. \n\nMinor remark: I don’t think that Dutch can really be called a low-resource language with respect to NLP resources. ‘Low resource’ may be true with respect to stance resources, but not in terms of general NLP resources (see for instance Joshi et al. 2020). Perhaps check this. \n\nReference\nJoshi, P., Santy, S., Budhiraja, A., Bali, K. and Choudhury, M., 2020, July. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 6282-6293).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a new way of modeling the task of stance detection accompanied by an annotated dataset in Dutch and Catalan encompassing several different topics and initial experiments testing the potential for topic and language transfer. In contrast to traditional stance detection, the new task models stance as a dynamic relationship between two texts (in the case of the current dataset between a target and a parent post in an online discussion). The underlying motivation behind this approach is that dynamic stance detection is more generalizable across topics than static stance detection, which always relies on a pre-defined topic (or claim) with respect to which texts are assessed. Initial experiments with language models confirm that the new model of the task does indeed lead to better topic transfer. \n\nI found the paper well-written, well-argued, and easy to follow. I think it makes a valuable contribution to the field in presenting a new way of modeling stance detection together with a new resource and experiments and should be accepted to the conference. I do have some doubts about the comparison between static and dynamic stance annotation (see comments below),  but I think they can most likely be clarified in the final version of the paper and should not prevent acceptance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes two new datasets for dynamic stance annotation in two languages, Dutch and Catalan. It's shown that a cross-lingual approach to stance detection helps performance for Dutch, for which there is only a smaller dataset available.

The reviewers were generally in agreement that the use of annotations of both static and dynamic stance to show relations on different levels of analysis was a strong aspect of the paper, and a framework that could be useful in future work. The cross linguistic approach was also very well received. Further the paper offers a useful resource to the community and provides strong baseline results. 

The reasons to reject that concern me most (which are not resolved in the rebuttal/discussion period) include:
* the suggestion that the paper may be more suited for LREC rather than a conference focusing on empirical analyses
* the unclear presentation of the thesis of the paper (which was discussed in the rebuttal, but I do see the point the reviewer makes)

I feel this paper is strong and could be a useful resource to the community. I do also feel that the authors should address the issues brought up by reviewers and adapt the paper accordingly before acceptance.
Decision: Accept-Findings