Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign
Authors: Nicolas Stefanovitch
Jakub Piskorski
Venue: EMNLP 2023 Main (2023)
Keywords: persuasion techniques
annotation
inter-annotator agreement
data quality
IAA

Abstract: In this paper we report on the complexity of persuasion technique annotation in the context of a large multilingual annotation campaign involving 6 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans to annotate and elaborate on our findings on the causes of this phenomenon.
We introduce Holistic IAA, a new word embedding-based annotator agreement metric and we report on various experiments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metrics. However, given somewhat limited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between annotators tasked to annotate different documents and in different languages, for which classical IAA metrics can not be applied.

Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign
Nicolas Stefanovitch
Jakub Piskorski
European Commission Joint Research Centre
Institute for Computer Science Polish Academy of Sciences Warsaw, Poland jpiskorski@gmail com
Text and Data Mining Unit
Ispra, Italy
nicolas. www.sssttaaviiiiccceeeeeeeee europa. eu
Abstract
in the SemEval 2023 Task 3: Detecting the Cate- gory, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup (Piskorski et al., 2023b). The primary objective of the work reported in this paper is threefold, namely:
this paper we report on the complexity of persuasion technique annotation in the context large multilingual annotation campaign involving 6 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans annotate and elaborate of this phenomenon. We introduce Holistic IAA, a new word embedding- based annotator agreement metric and we report on various ex- periments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metries. However, given somewhat lim- ited and loose interaction between annotators, ie., only few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find good proxy for TAA between an- notators tasked to annotate different documents and different languages, for which classical IAA metries can not be applied.
share some lessons learned from this large multi-lingual annotation campaign that might be beneficial for other researchers planing sim- ilar tasks,
findings the causes
present a detailed analysis of the disagree- ments between annotators and potential causes thercof and try to mcasure the com- plexity the annotation task, and
propose a new concept of measuring Inter- Annotator Agreement (IAA) multilingual set-up, to overcome the limitations the clas- sical TAA metrics in such scenario.
We first highlight the techniques that appear be difficult for humans annotate using the classi- cal Cohen's 's (McHugh, 2012), and Krippendorf''s a (Krippendorff, 2009).
1 Introduction
In the recent years we have observed an emergence automated tools for facilitating online media analysis for better understanding of the presented narratives around certain topics across countries, and to identify manipulative, deceptive and propa- gandistic content. Developing such tools requires annotated data of high quality.
Classical IAA measures impose certain limita- tions. First, they only capture the coherence of the annotations in texts written in the same language. Secondly, considering annotations done for single language, there tating totally different subsets of documents. The classical IAA metrics are computed using a tiny fraction of the whole dataset: the one where the annotators annotated the same articles, despite the fact that the exact same text could be annotated in different articles by different annotators. Finally, the classical IAA measures only capture agreement at the time of the annotation, but do not tell any- thing about the coherence and quality of the final curated dataset.
many annotators, but anno-
We report on the complexity of annotating such manipulative devices, i.e., persuasion techniques, in the context of a large annotation campaign in- volving 6 languages and approximately 40 annota- tors, whose details are described in (Piskorski et al., 2023c). The persuasion technique taxonomy used in the campaign is an extension of the taxonomies used in different shared tasks, contains 23 tech- niques, and includes i.a., the techniques appealing to emotions, justifications and some forms logi- cal fallacies. The resulting dataset has been used
In order overcome the aforementioned lim- itations, we introduce Holistic IAA, a new mul- tilingual word embedding based LAA metric and

---

we report on various experiments using and its correlation with the traditional IAA metrics, How- ever, given somewhat limited and loose interaction between annotators, i.e., only a few annotators an- notate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between annotators tasked to annotate different documents and in different languages. We present our pre- liminary results on this research problem with an ultimate goal of establishing a mechanism that al- lows to compare all annotators no matter which document they annotated, and to detect diverging annotations across languages. Our contributions can be summarized as follows: (i) we measure how confusing were the persuasion technique labels for different groups of annotators; (ii) assess the coherence of the dataset using standard IAA mea- sures; (iii) we introduce a new mutlilingual pan- corpus IAA measure based on semantic similarity; (iv) we exploit this new measure on the raw and curated annotations of the annotators, and com- pare the resulting ranking of annotators to the one obtained by standard TAA measurements; (v) we comment on the self-coherence of the annotators using the new measure, well of the dataset language-wise.
Cedeno et al. (2019) presented a corpus of tagged either as propaganda or non con-propaganda)
In parallel, other efforts focused on the detec- tion of specific persuasion techniques. Habernal et al. (2017, 2018) presented a corpus annotated with 5 fallacies that directly relate to propaganda techniques. A more fine-grained analysis was done by Da San Martino et al. (2019b), who developed corpus of English news articles labelled with 18 propaganda techniques at span/sentence level. Somewhat related work on detection of use of pro- paganda techniques in memes is presented (Dim- itrov et al., 2021a), the relationship between propa- ganda and coordination (Hristakieva et al., 2022), and work studying COVID-19 related propaganda in social media (Nakov et al., 2021a,b). Bonial et al. (2022) reported the creation of annotated text snippet dataset with logical fallacies for Covid-19 domain. Sourati al. (2022) presents three-stage evaluation framework of detection, coarse-graines and fine-grained classification of logical fallacies through adapting existing evaluation datasets, and evaluate various state-of-the-ar models using this framework. Jin et al. (2022) proposed the task of logical fallacy detection and new dataset of log- ical fallacies found in climate change claims. All the persuasion techniques and logical fallacy tax- onomies introduced the aforementioned works do overlap to a very high degree, but are structured differently, and use different naming conventions.
This paper focuses primarily on the annotation agreement and complexity, whereas the description of the resulting dataset is kept to the minimum nec- essary for understanding the content. For further details please refer to (Piskorski et 2023c).
The paper is organized as follows. Section re- ports on the related work. Section 3 introduces the persuasion technique taxonomy and describes the annotation process. Next, Section 4 reports on the annotation coherence computed using traditional IAA metrics and highlights the hard-to-annotate techniques. Subsequently, Section 5 introduces new word embedding- based annotator agreement metric and reports on various experiments using and correlating it with the traditional IAA metrics. We end with some concluding remarks in Section 6.
Various related shared tasks on the detection of persuasion techniques were organized recently, and various taxonomies were introduced (Da San Mar- tino et al., 2020, 2019a; Dimitrov et al., 2021b; Alam et al., 2022; Piskorski et al., 2023b).
Related work IAA which explores going beyond the limitation of standard measures was reported in (Passomneau and Carpenter, 2014), proposing an idea similar to our that they are able to compare all annotators between themselves, however, this comparison is done statistically on label distribution while we look at actual content of the annotate textd. Moreover, they are interested in assessing the gold label uncertainty, which is similar concern our effort of capturing the la- bel definition difficulty. However, they treat in statistical fashion, while we provide simple de- scriptors. It would be an interesting future work to explore the combination of both approaches.
2 Related Work
Persuasion detection in text is related to work on propaganda detection. The work in the latter area initially focused on document-level analysis and predictions, e.g., Rashkin et al. (2017) reports on prediction of classes (trusted, satire, hoax and propaganda) of documents, whereas Barrón-

---

3 Persuasion Technique Annotation
document was annotated by at least two annotators independently. On a weekly basis reports were sent to annotator pairs highlighting complementary and potentially confiicting annotations in order to con- verge to a common understanding of the task, and regular meetings were held with all annotators to align and to discuss specific annotation cases. Annotations were curated in two steps. In the first step (document-level curation) the independent annotations were jointly discussed by the annota- tors and a curator, where the latter was a more experienced annotator, whose role was to facili- tate making a decision about the final annotations, including: (a) merging the complementary anno- tations (tagged only by one annotator), and (b) re- solving the identified potential label conflicts. In the second step (corpus-level curation) global consistency analysis was carried out. The rationale behind this second step was to identify inconsisten- cies that are difficult to spot using single-document annotation view and do comparison at corpus level, e.g., comparing whether identical or near-identical text snippets were tagged with the same or a sim- ilar label (which should be intuitively the case in most situations). The global consistency analysis sketched above proved to be essential ensure the high quality of the annotations.
3.1 Taxonomy
The taxonomy used in annotation endeav- our is an extension of the taxonomy introduced in Da San Martino et al. (2019b.c). At the top level, there are 6 gaarse types of persuasion tech- niques. namely: Attack on Reputation, Justification, Distraction, Simplification, Call, and Manipula- tive Wording, whose full definitions are provided in Appendix A. These core types are further sub- divided into 23 fine-grained techniques. The 5 new techniques vis-a-vis the taxonomy presented in Da San Martino et al. (2019b,c) are: Appeal to Hypocrisy, Questioning the Reputation, Appeal to Values, Consequential Oversimplification, and Appeal To Time. The main drive beyond introduc- ing these 5 new techniques is due to their frequent presence in news articles based on our empirical observations. The full two-tier taxonomy, includ- ing short definitions, and examples of each fine- grained technique are provided in Figure 3 and 4 in Appendix A respectively.
3.2 Annotation Process
Our annotation task consisted of annotating persua- sion techniques in a corpus consisting of circa 1600 news articles revolving around various globally dis- cussed topics in six languages: English, French, German, Italian, Polish, and Russian, using the taxonomy introduced earlier. A balanced mix of mainstream media and "alternative" media sources that could potentially spread mis/disinformation were considered for the sake of creating the dataset. Furthermore, sources with different political orien- tation were covered as well.
The annotation resulted in annotation of approx. 1600 documents with ca. 37K text spans annotated. The dataset is highly imbalanced. The class distri- bution and some statistics are provided in Annex B
Annotation Coherence & Complexity
4.1 Traditional IAA
We measured the Inter-Annotator Agreement (IAA) using Krippendorff's 's a, achieving value of 0.342. This is lower than the recommended threshold of 0.667, but we should note that this value repre- sents the agreement level before curation, and such, is more representative of the curation dif- ficulty rather than of the quality of the final con- solidated annotations. We used the IAA during the campaign to allocate curation roles and to remove low-performing annotators.
The pool of annotators consisted of circa 40 per- sons, all native near-native speakers of the lan- guage they annotated. Most of the annotators were either media analysts or rescarchers and experts in (computational) linguistics, where approximately 80% of the annotators had prior experience per- forming linguistic annotations of news-like texts. A thorough training was provided to all annotators which consisted of: (a) reading 60-page anno- tation guidelines (Piskorski et al., 2023a) - an excerpt thereof is provided in Appendix C). (b) participating in online multi-choice question-like training, (c) carrying out pilot annotations on sam- ple documents, and (d) joint sharing experience with other annotators and discussions with the or- ganisers of the annotation task. Subsequently, each
We further studied the TAA by ranking the an- notators by their performance with respect to the ground truth on the subset of documents they anno- tated. We split then the annotators into two groups: top and low based on subjective assessment by the curators after the end of the curation campaign, this assessment was then further confirmed numerically

---

(see Annex E for details). Their respective average a were 0.415 and 0.250. Finally, we considered the a of the group of the curators, in order to make an approximate estimation of the coherence of the curated dataset, as we expect these curators to con- sistently curate the data with at least the same coher- ence they had when annotating documents. There are only two such curators, whose a is of 0.588, which is lower but close the recommended value.
for all their overlapping annotations the percentage of disagrecing annotations for a given label divided by the total number of annotations between them with that label. Here, annotations of two annotators are considered overlapping if one is at most 10% longer or shorter than the other one, taking into account the exact position of the annotations in the text. We report these numbers in Table 1.
In order to interpret the results, it is also im- portant to take into account that the 2 sub gro-gr namely, top and low, also do interact with each other. We consider the following indicator of com- plexity: for each of the group if the disagreement is above a given threshold e that we fixed for illustra- tion purpose at 0.25 in the table, the corresponding values are boldfaced. We also divide the techniques in the table (column 'difficulty") ') into four general annotation complexity classes based on the overall disagreement: very casy (all S .1, in light green), easy (all I .25, in green), moderate (all .4, in orange), and difficult (all .4, in red).
4.2 Confusion matrix
In Figure/ I we present the confusion matrix be- tween the annotations of annotators. A high count denotes both frequent class and tendency to confuse the given pair of labels.
One can see that Loaded Language (MW:LL)
is the single label that is most confused with any other label, and the Name Calling (AR:NCL) is the label with which it co-occurs most, and indeed, these two labels have very similar definition. The same applies to the pair Casting Doubt (AR:D) and Questioning the Reputation (AR:
Additionally, consider the following indica- tor: if top all or if top low (the techniques for which this applies are marked with an asterisk in the table).
-2
ARGA
10
10
18
12
161
COUTT
One can see that high low value does not nec- essarily mean that the label is actually hard, for instance, the label False Dilemma is very well understood by the top group. High low value and low top value denotes label whose understand- ing is not straightforward but does not pose special learning problem, in such case improving anno- tations for this label requires simply insisting on more basic training.
0.2K
10
12
0:00
L.
Indad
arras
Iabel
Figure I: Confusion matrix between single annotations of annotators, thereby denoting tendencies in confusion between the given pairs of labels. Values lower than 10 are blanked.
On the contrary, when the top value is higher than the others (techniques marked with aster- isk), it means that at least one of the groups agrees more with the other group than top group with itself, meaning that there is an inconsistent under- standing of the label within the group. This could indicate difficult label requiring additional clarifi- cation to be made to all annotators, or potential
4.3 Techniques' Annotation Complexity
In order to study which persuasion techniques are more difficult to annotate we again divided the an- notators in 3 groups: all which contains all the an- notators, top which contains half of the annotators whose performance are the highest as measured by their average Cohen's 's k agreement, and low which contains the rest of the annotators.
inconsistency in the label definition. This is, for in- stance, the case for the label Repeti tion, which is indeed inconsistent it includes two very different definitions of repetition.
For each of these groups, and for each of the per- suasion techniques, we measured how annotators in a given group tend to disagree with each other- irrespective of the actual ground truth. More pre- cisely, we compute for each pair of annotators and
The overall picture of the annotation complexity classes resembles the per-label performances of classifier systems reported in (Piskorski et al.,
2023c), where with a few exceptions the "casiest" labels are the ones with the highest F, score. It is

---

important to note that these values are computed on the annotated labels before any curation had taken place, and as such do not reflect the quality of the final dataset, but are more and indication of the intrinsic difficulty of the labels for new annotators.
disregarding small nuances in the definition of Loaded Language and Name Calling we noticed that disagreements and annotation or non-annotation of some instances were due to subjective perception linked to cultural differ- ences, which was apparent when comparing annotations across languages,
The class Doubt has one of the best reported Fi scores, however, it has difficult annotation complexity, the reason being that it one of the most confused classes, as it is often a subpart of other techniques.
some annotators had problems with the Jus- tification techniques, including, in particular. Appeal to Popularity, Appeal to Values, Ap- peal to Authority due to not understanding upfront that one subjective opinions on what is considered a value or an authority does not play role for definition of these techniques, and not considering the role of negation, e.g.. not understanding that making reference something not being popular falls per defini- tion under Appeal Popularity too,
Some hard labels remain challenge even for top annotators, and as such selecting 'reliable' an- notators solely based on their overall 1AA might not be sufficient to ensure the best quality of an- notations, it is also important to identify for which labels additional training might be necessary.
Quantifying the annotation complexity of an- notation campaign in such way gives under- standing of the difficulty of the task, and allows to identify incoherent understanding of the guidelines early on, and gives a more refined understanding of the quality of the annotations than considering IAA measures alone.
many annotators, who probably did not read the guidelines thoroughly, literally interpreted some persuasion technique definitions, e.g., in the context of Simplification techniques, in- stead of detecting certain logic patterns in text (see Annex A for definitions), the annotators literally interpreted the word 'simplification" and reasoned based on the base of whether the presentation of the information is too simplis- tic and certain facts were downplayed or exag- gerated, which is actually linked to a different technique, i.e., singeraattioo-mmimiaatioo
technique
abrov
all
low
difficalty
Name Calling Labelnng
ARNCL
.051
.196
.315
Gliitt by Asocciation
AR:GA
.250
.393
.333
Doult
ARDD
.426
.286
.456
Agpeal Hopoersy
AR:AH
02.5
.111
Quetioning Repozation
AR:QR
Flag Waving
LFW
.286
.667
1.00
Appeal to Authocity
RAA
222
.100
1,60
Agpeal to Values
3AVV
.150
.133
.667
Appeal Poputarity
EAP
.211
2000
1.400
Appeal Fea: Picjudic
LAF
.091
1955
.158
Causal Oversimplifiaaton
S:aa
.368
1
Consequcttal
.250
.178
False Dilemmas Cheiio
DEFDNC
Strawman
.200
.429
1
Red Herring
DERT
.500
.600
1.60
some of the media analysts who served as an- notators were often using background knowl- edge (professional bias) to make decisions whether some text fragments are instances of persuasion techniques, which was strictly pro- hibited by the guidelines; this was mainly re- lated to Simplifications and Distractions,
Whatzhutism
12W
.500
.500
1,00
Singans
C:8
.200
.333
.00
Comrenstionn Killer
C.K
.148
1100
.400
Appeal Time
C:AT
.333
.750
.333
Loaded Language
MWLLL.
J042
2148
1059
Obfiscationn Vaguensss -inffusnnn
MW.OVC
.400
Ekaggeration Mininisation
MW-EM
.2001
Repetition
MWWR
.464
.667
.400
Per-lahel disagreement of three groups of anno- tators: all, top and low and related complexity markers. The colours reflect the respective annotation complexity class. The techniques considered potentially requir- ing additional clarification or whose definitions might exhibit inconsistencies are marked with an asterisk.
Table I:
some of the annotators, in particular, media analysts were making direct link of persua- sion technique labeling with fact verification, which was not in line with the guidelines.
To sum up, for the major fraction of persuasion techniques the disagreements resulted not from sub- jective perceptions of the annotators, but mainly due to not sticking strictly to the definitions pro- vided in the 60-page guidelines and/or professional background bias that lead to misinterpretation of the persuasion technique definitions.
4.4 Disagreement sources
On top of the findings on annotation complexity we additionally summarize here our findings on the sources of disagreements and annotation complex- ity from the continuous meetings with annotators and curators:

---

5 Embedding-based IAA Assessment
database, without quantization and with cosine dis- tance (Johnson et al., 2019). While any multilin- gual embeddings could be used, we specifically use LASER embeddings (Schwenk and Douze, 2017) for simplicity reasons, i.e., our aim is to introduce new paradigm to IAA computation, and we do not aim at determining which embeddings are the best, which is exercise of more limited interest given the pace at which new embeddings emerge. Moreover, LASER embeddings do not require lan- guage specific thresholds. As such, one single cut- off to discriminate similar sentences can be used for all the languages, which is not generally the case for semantic similarity models (Isono, 2022). The drawback of these embeddings is that they are less discriminating than other models as the range of values corresponding to true positives largely intersects with the range of values corresponding to false positives.
5.1 Holistic IAA
We introduce a new measure, namely, Holistic IAA, which allows to compare annotator with any other, even if they did not annotate single docu- ment in common and annotated documents in dif- ferent languages. This metric exploits the prop- erty of multilingual aligned sentence embeddings, which are able to encode with similar vector rep- resentations sentences in different language with the same meaning, and different sentences with a similar meaning in given language.
Formally, we introduce the following holistic agreement between two annotators as 0 (a1, 02) where ot is the function that maps input texts to label for a given annotator a;; and for any two pair of strings Br is the threshold on the length ratio and 0, is the threshold on the similarity measure defined for any embedding model M using the cosine distance between the embedding vector of the input strings (we denote it with for the first letter of the word "holos" in Greek).
In a second step, for cach annotator and corre- sponding annotated text spans, we consider the set of all similar text spans. In Figure we illustrate in detail the behaviour of LASER two sample queries reporting the L2 similarity. For the more complex query Q2, all but one retrieved spans are correct, but a divergence in meaning can be seen with decreasing semantic similarity. We use cosine distance in order to avoid the length of the vector to impact the measure. Moreover, in order to avoid comparing sentences of extremely different sizes, the length of the retrieved span and the query span is constrained by threshold on the ratio of their respective lengths, i.e., Aj.
We define the set of Comparable Text Pairs (CTP) between two sets of texts X and Y as:
[F.EXX x min >0. [FNMl
M()
Using this definition and defining S(a:) as the func- tion returning all the sentences annotated by annota- tor ag we define the Holistic IAA for 2 annotators:
La (F)=2(()))
S(an) Stsap)
(CTP S(ag)
Q1 "neaomyermoaaze" unneceptable) antiriite (i,, 0.0,, invisbley, resosseniblic (it. 0.H4, unsistiiaaie), morcepmatle (ir. 0.05, unaccepubble)
insupportable (ft, unteaamblle.
Extending groups of annotators A and B, we get the more generic formulation:
02 "bout simplement, un mensongs" all simply lie) E natic and menpages (it, 0.04, it is all a lie,, jume Alsossiso (s). 0.06, cleus lie), quesa ano dagia (r. 0.06 this se he) foome minsonge (6, 0.07, an commons lie), alley wirder else grgle Lage (de. 0.8, again a big lie), Wir glouben, dass wir belogon wer dm (lc. 0.09, believe ar being lind abfiri oblomojer (pl. 0.09, lying profusely), finscennt 0.09 ratid
EnAEAAR
S.ECTT 9o9o (2)())))()))
Mc.ss. (A,B)
Star).s (.S(2)
0.09, crumbling lies), bouchée 0.09, with eailly), auntonges chonsds dh, 0.09, shameless Bes)
Ax (CTT"t 2r
8(0).8(0))
Finally, let An(D) denote the set of annotators of dataset D. We can now define the Holistic IAA value for dataset as:
Figure 2: Example of query and retrieved sentences, for each span we report the language, L2 similarity and the English translation.
M. %.
P.. (0 ()n(D))An(D)
In Table 2 we provide example of how text spans and their associated metadata can be queried and retrieved from the database. In this example, the retrieved texts all refer to the same concept as the query text, despite spelling variations one language and different languages being used. How-
In first step, the embedding for each annotated text span by each annotator computed and stored in a vector database, and associated with the following metadata: the document id, the anno- tator and the label. We use FAISS for the vector

---

Uye
Gitt
1pan
lan..
label
parameters 0g and Og. In Table we show a sample of the parameters searched, in Annex F we report the results of the full grid search performed. The correlation between strict and same is on overall higher than when comparing di ff and strict as well as same and diff, and is even perfect or near perfect for a subset of the parameters. We selected the parameter as 0: 0 and 0g 0.75 for the rest of the paper, despite these being not the optimal. Optimal parameters are too conservative and as such the CTP set was too small in order to compare all annotators or groups of annotators, and a such prevented from further studying the properties of Holistic IAA.
query
Eanatiche Gannyyde miligjjmahh
MWLL
ceply
AR.NCT.
upan
label
query
TT
MW.T.
roply
fascista
TT
MWLL
reply
Everysyy
PL
AR:NCT.
reply
01036
Puschismss
Db
MW-LL
reply
0.033
Everyan
PL
ARENCI
reply
01033
Gammetic
RU
MWR
H
reply
0.043
fnscisli
FT
MW-LL
Table 2: Example of query span and result span fetched from the database, 1.2 similarity is reported, as well as the associated labels and the annotator identi- fier.
ever, we can observe that the labels can disagree: This illustrates at the same time the difficulty of dis- tinguishing between Loaded language and Name Calling, and that some annotators are not consis- tent in their annotations. Notably, the high rate of confusion between Loaded language (MW:LL) and Name Calling (AR:NCL) is observable.
This proves that the Holistic IAA can be used proxy for the pan-document pan-annotators agree- ment for some specific set of parameters, however, without the possibility to precisely link its value to other standard TAA measures, and with the caveat that the correlation is positive yet not perfect. As such, Holistic IAA can be used mainly to comment on the qualitative difference in agreement between different subsets of annotations.
5.2 Validation: Methodology
In order to validate the approach, we perform rank correlation amalysis between the ranking computed by standard ıAA techniques and the ones with our approach using Kendall's s Tau rank correlation coef- ficient (Kendall, 1938). We consider 2 datasets: the raw annotations of the annotators, and the corpus (dataset of curated documents by curators).
miking
ranking?
cocf
suppost
0.0
0:75
stril
01.50
00
0.25
same
strict
0.80
0.0
0.75
same
dift
0.26
0.50
0.90
diff
street
-0.33
0.50
0.90
stnol
0.67
0.50
0.55
0.33
The raw annotations allow us to compute pair- wise IAA with Cohen's 's k between annotators, who have annotated the exact same documents. For each annotator, consider the ranking of the annota- tors he can be compared to and which have at least 10 annotations in common.
0.75
0.80
came
stricl
0.03
0.75
0.80
same
dift
0.30
Table 3: Rank correlation between: Cohen's 's R com-
puted on the original data (strict), Holistic IAA com- puted on the same documents as Cohen (same). Holistic IAA computed on all the other documents (diff).
Given the raw annotations dataset, we compute the Holistic IAA o value, and for each annotator we rank all the other annotators to which it can be compared to, as measured by the average level of agreement on labels for semantically similar text spans.
5.4 Validation: Error Analysis
We performed an error analysis of the confusions found using Holistic IAA: using the 33k+ confu- sions found by the approach over the dataset, for each pair of labels we evaluate up 5 alleged confusions and graded the similarity between the corresponding texts on a 3-tier scale. Two texts are considered: identical if the meaning is so close that minor nuance in text would not alter the label chosen (e.g. "opération spéciale" (fr) and "Spezial- operation" " (de) both meaning "special operation"); close if the meaning is substantially different, but semantically close enough making the label de- batable and worthy to be flagged to a curator for review, for instance one text could be more generic than the other one (e.g. "fini molto male" (it) =
5.3 Validation: Results
We compare the ranking of most "similar' annota- tors for each annotator computed using Cohen's 's K with the ranking computed using Holistic IAA on the same subset of annotators. We consider 3 rankings: strict Cohen's 's к; same ranking is done on the same set of documents and annotators the one used compute Cohen's κ; diff ranking is done on the same pair of annotators, but strictly on documents that were not jointly annotated by them. We perform a simple grid search over the hyper-

---

"it ended badly" and "durement mise à mal" (fr) =bbadly impacted"); unrelated if the meaning is unrelated - even if the texts contain the same ele- ments.
tors: ai and s; denote respectively the agreement percentage between annotators and the support at step i. For step 2, the o value is higher, and the average IAA is 1.6 pts higher, while the average intra-annotator agreement (self-agreement) is 3.5 pis higher. This demonstrates that Holistic IAA is able to capture and quantify the positive impact of the corpus-level curation.
A total of 502 data points were annotated. Note that only texts containing least one space were considered. In Table 4 report the count in each category, and the mean, min and max simi- larity measure as given by the LASER embeddings. When adding the close and identical classes, we get that in a bit more than half of the cases the ap- proach is able to correctly flag potential incoherent annotations.
In Table we illustrate the impact of exclud- ing Loaded Language (MW:LL) and Name Calling (AR: (AR:NCL) from the dataset as these labels constitute nearly half of the annotations and are frequently confused with each other by annotators in terms of absolute number (but not in proportion) as shown in Figure 1 and Table 1. We observe that the agrec- ment between annotators can be label specific. In Figure 8 we consider the whole curated dataset and measure the o value between pairs of languages. The columns a, report the value after step 1 and 2 considering the whole range of la- bels, while the columns de exclude the two labels MW and NCL. Doing so gives us an under- standing of the agreement for the lower populated labels. Please note that all the Attacks Repu- lation (AR:*) and Manipulative Wordings (MW:*) were excluded from the second step of the curation due to time constraints - except for DE and PL. The impact of the second curation step is almost always positive for all pairs of languages, except notably for one language for which the related values de- teriorate and which drags down the intra-language coherence score.
We can also see the difficulty of setting cutoff boundaries as the range of minimum and maximum semantic distance overlapping between all the classes, and with close and identical having almost the same mean boundaries. We can nevertheless ob- serve that the mean value of close is 0.75, making it reasonable candidate for f/.
These results show that about half of the annota- tions flagged by the system were indeed of interest to the curators. However, as such, the results are highly dependent on the model used. Future work will require to identify embeddings with a larger margin between the classes in order to make the work of the curators more efficient.
relation
courc
min
unrelated
257
0.145
0.896
0.613
0.900
Idestical
123
0,846
0.661
1.
Table 4: Statistics on the distance for 3 similarity classes using the LASER embeddings and cosine distance set of potential confusions flagged by Holistic IAA
Overall, when considering the corpus we observe a quality increase as measured by the o value.
curl
cur2
43
62
12
5.5 Impact of the second curation step
193778
177604
0.586
45694
183937
In order to further evidentiate the behavior of Holistsic TAA, we use it to quantify the impact of the corpus-level curation step. This step was performed par-language after the usual document- level curation step was accomplished. The data was sorted per-label and the master curators looked at the overall coherence of the annotated text-span label pairs, the context of the spans was also pro- vided. This step lead to several corrections and is understood to have boosted the overall coherence of the dataset, and should be reflected with higher o value for the corpus.
0.64
S1327
0.508
123539
0,49
10519
0.525
10210
0:45]
8375
0.131
29189
0.61
11688
0.625
27113
0.597
5185
0.647
54566
6163
01367
0.394
0.483
0,093
0.101
20tra 0.605
0,083
0.050
Table 5: Curated dataset, agreement after step and 2, where a; and s, denote respectively the value between annotators and the support at step i.
5.6 Multilingual Dataset Coherence Estimation
In Table we consider the agreement as mea- sured by Holistic IAA after step and 2 of the curation by considering the 4 most active cura-
Knowing the dataset coherence computed using standard IAA measures in a monolingual setting,

---

CUPI
CUPZ
11
s1"
the dataset is consistently higher after the second curation step vis-a-vis after the first step, suggest- ing that this new curation approach of interest to increase the quality of annotations.
0.577
191778
0.463
714
57351
0.207
598
0.544
51327
0:387
24R
0.59
10119
0.338
1426
0.451
8575
0.251
562
B
D
0.61
11688
0.268
114
C
C
0.597
3155
0,558
154
C
D
0.471
6161
0.514
140
0.625
0.517
0.380
0.729
0823
0.567
0.333
40op 2
0.560
0.592
0.756
0.762
0881
0,883
0.083
0.504
0.496
Table 7: o value for the curated data after each curation step for different values of 8g and fixed 0: =
Table 6: Curated dataset: agreement with (left) and without (right) taking into account the two classes MW: LL and NC:
lang2
a2
a2'
change
change"
19
19
0.012
0.053
0.55
0.57
+3.011
+0.02
FR
11
0.585
0.892
0.168
0.223
+0.077
-0.055
FR
PL.
0.615
0.401
0.651
0.454
-0.051
+0.035
and comparing it with values computed using Holis- tic IAA, we extrapolate from it the coherence of the entire multilingual dataset. Only two curators have jointly annotated the same set of documents while acting as annotators before the curation phase and taking on the curator role, as such we can compute the Krippendorff's is a between them, which is 0.588, a little under the recommended value. The o value between them on the same data is 0.420. A group of 3 "master curators" covered all the languages and curated most of the dataset. Their average o value the raw annotations is of 0.565. This higher value illustrates the fact that the coherence of the annotations in the final dataset is higher than when measured on the raw annotations.
RU
0:43
0.643
0.658
0.245
0.458
0.258
+0013
-0,045
0.0
PL.
0.545
0.549
0.384
0.346
40.044
.0,038
0.524
0.234
F1.
0.798
0.476
+0.027
.0.149
RU
0.688
0.818
0.512
0.255
+0.01
-0,057
RI1
R17
0.501
0.531
0.341
0.373
-0.03
+0.08
0
0.567
0.594
0.323
0.310
+0013
-0.03
inter
0.519
0.521
0.258
0.305
40.002
+0.007
intra
0.619
0.646
0.409
0.944
+0
0,025
Table 8: Curated dataset: impact of the two curation steps on the overall agreement between languages (inter) and inside languages (intra), when the labels MW:LL and AR: NCL are included (al and a2) and excluded (al" and a2")
Conclusions
We reported on the complexity of annotating per- suasion techniques in large-scale multilingual annotation campaign. We introduced the Holistic lAA paradigm,
We now consider only the curated dataset. In Fig- ure 8 we can observe that the value intra-language range has an average value of 0.538, slightly above the o value of 0.420 of the two reference annota- tors for which Krippendorff's 's a could be computed. We can conclude that the coherence of the dataset restricted to each language is above the coherence of the reference annotators.
measure to serve proxy of the estimation of inter-annotator agreement and actual corpus coherence in settings that are funda- mentally outside the scope of usual TAA measures.
We demonstrate that annotator ranking computed using this new measure is positive and can highly correlates with ranking computed using Cohen's 's Kappa in some settings. Using it, we can observe the beneficial impact of the second step of our 2- step curation phase, and also identify similarity and divergence between annotators for some subsets of labels. The experiment conducted this study sup- ports what was informally remarked regarding the estimation of the performance of the annotators and increased our confidence in the coherence of the final corpus. We believe that using Holistic IAA part of the monitoring of multilingual or monolin- gual large-scale annotation campaigns could help to spot problems by flagging potential incoherence in the labels of semantically similar sentences at an early stage. future work we envisage exploration of thresholds for finer interpretation and exploring the use of other semantic similarity models.
However, most of the inter-language o values are much lower than the intra-language values. We believe this to be due to 2 factors: 1) each curation was performed ler-language, ignoring the others, thereby increasing the self coherence of each lan- guage; 2) in the case of the diff vs. strict in Figure 3 Holistic IAA is less able to capture agreement than in the case of same vs. strict, thereby denoting limitation of our approach. This could be partially alleviated by using 'better' em- beddings. Nevertheless, even with a lower perfor- mance, a tool based on Holistic IAA to check for annotation coherence across languages would help to increase the quality of the dataset by flagging potential inconsistent annotations.
In Table we can observe that the o value for

---

Limitations
must contain all necessary information unam- biguously determine the label, which is the case in the campaign whose data was used to illustrate our approach.
Distribution Representativeness Although the un- derlying corpus of annotated news articles covers a wide range of topics as well as media from all sides of the political spectrum it should neither be seen as representative nor balanced in any specific way w.r.t. media in any country. Consequently, the distribution of the annotated persuasion techniques might, in principle, not be fully representative as well.
Ethics Statement
Biases The news articles for the creation of the underlying dataset were sampled in such way in order to have a balanced representation with spect different points of view and type of media. We also strived to engage mix of annotators with different backgrounds, i.c., both media analysts and computational linguists. Furthermore, the an- notators were explicitly instructed not take their personal feeling about the particular topic and to objectively focus on identifying whether specific persuasion techniques were used. Disregarding the aforementioned efforts, the distribution of the various persuasion techniques annotated might not perfectly reflect the broader spectrum of the media landscape in the target languages, which should be taken into account in exploiting the related sta- tistical information for any kind of analysis, etc. Analogously, the findings and statistics related to the annotation complexity are linked to the spe- cific pool of annotators engaged in the campaign, and, consequently, they should be considered approximative.
Biases Given that human data annotation involves some degree of subjectivity we created a compre- hensive 60-page annotation guidelines document to clarify important cases during the annotation process. Nevertheless, some degree of intrinsic subjectivity might have impacted the techniques picked up by the annotators during the annotation, and impacted so the distribution thereof in the fi- nal dataset. Furthermore, although the taxonomy used in this annotation campaign covers most the 'popular' techniques used in the media, we iden- tified some persuasive attempts which could not have been matched with any of the techniques in the existing taxonomy, and were tagged as OTHER (less than 3% of all annotations) and were not con- sidered in the reported work, which once again poses a certain limitation with respect to the repre- sentativeness of persuasion technique types used in the media.
Methodology Soundness Our results are limited to certain extent, in particular, the introduced IAA proof of concept
Intended Use and Misuse Potential The reported work focuses solely on sharing experience with the research community on annotating persuasion tech- niques in news articles in large campaign, analy- sis of the difficulty of annotating such techniques,
metric should be considered since certain approximations and simplifications were made and parameters were chosen, e.g., the choice for cutoff of maximal retrieved similar sen- tences, the length ratio to select sentence to be compared is constrained, and the choice of sim- ilarity metrics for computing semantic similarity that exploits specific sentence embeddings model. Different settings and choices could yield different results. Disregarding of these shortcomings, the new metric helped circumvent the limited scope and utility of classical IAA in such a large-scale multilingual campaign. We believe that the pro- posed methodology presented in this paper is too some extent generic, and would be of great interest to the community.
and ways of measuring annotation agreement and consistency across languages. The reported work is not linked to a release of the underlying annotated dataset, which subject of different publication and related ethical considerations.
Acknowledgements
We are greatly indebted to all the annotators from different organizations, who participated in the an- notation campaign, and without whom carrying out the reported study would not have been possible.
The approach considers only the text of the an- notation, as such their context ignored. This limitation mitigated in case the annotation guide- lines do not specify that the span of annotation
The work presented in this paper financed by the Joint Research Centre of the European Com- mission.

---

References
15th International Workshop on Semantic Evaluation, SemEval '21, pages 70-98.
Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani. Gio- vanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on pro- paganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Work- shop, Abu Dhabi, UAE
Ivan Habernal, Raffael Hannemann, Christian Pol- lak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argu- mentation meets serious games. In EMNLP.
Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018. Adapting serious game for fallacious argumentation to German: Pitfalls, insights, and best practices. LREC.
Alberto
Israa Jaradat. Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):184--186)
Kristina Hristakieva, Stefano Cresci, Giovanni Da San Martino, Mauro Conti, and Preslav Nakov. 2022. The spread of propaganda by coordinated communities on social media. In Proceedings of the 14th ACM Web Science Conference, WebSei '22 pages 191-201, Barcelona, Spain.
Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie M. Lukin. Jeffrey Micher. Douglas Summers- Stay, Peter Sutor, and Clare Voss. 2022. The search for agreement on logical fallacy annota- tion of an infodemic. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 4430-4438, Marseille, France. European Language Resources Association.
Fumika Isono. 2022. Language agnostic multi- lingual sentence embedding models for seman- tic search. http:://pinneraa//eeellggrgaaaap agnostic- seatiiccmmuiiligguaa-snnnne wewallleeeeeeeeebeeeeee embedding- models- for semantic-search Accessed: 2023-06-24. 24.
Giovanni Da San Martino, Alberto Barrón- Cedenoo Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceed- ings of the International Workshop on Semantic Eval- uation, SemEval '20, Barcelona, Spain.
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Associ- ation for Computational Linguistics: FMNLP 2022, pages 7180-7198 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Giovanni Da San Martino, Alberto Cerrnn,Ceeeño,, and Preslav Nakov. 2019a. Findings the NLP4IF-2019 shared task on fine-grained propaganda detection. In Proceedings of the Second Workshop on Natural Lan- guage Processing for Internet Freedom: Censorship. Disinformation, and Propaganda, pages 162-170, Hong Kong, China. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-99 93.
Giovanni Da San Martino, Seunghak Yu, Alberto Carrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grand grainer analysis of propaganda news articles. In EMNLP.
Klaus Krippendorff. 2009. Testing the reliability of content analysis data. The content analysis reader, pages 350-357.
Mary L McHugh. 2012. Interrater reliability; the kappa statistic. Biochemia medica, 22(3):276-282.
Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019c. Fine-grained analysis of propaganda in news articles. In EMNLP.
Preslav Nakov, Firoj Alam, Shaden Shaar, Giovanni
Da San Martino, and Yifan Zhang. 2021a. COVID- 19 in Bulgarian social media: Factuality, harmfulness. propaganda, and framing. Proceedings of the Inter- national Conference on Recent Advances in Natural Language Processing, RANLP '21.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. De- tecting propaganda techniques in memes. In Pro- ceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACL-IICNLP '21, pages
Preslav Nakov, Firoj Alam, Shaden Shaar, Giovanni Da San Martino, and Yifan Zhang. 2021b. second pandemic? Analysis of fake news about COVID-19 vaccines in Qatar. In Proceedings of the International Conference on Recent Advances Natural Language Processing, RANLP '21.
Dimiter Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 202 Task at SemEval-2021: Detection of persuasion tech- niques in texts and images. In Proceedings of the
Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of model of annotation. Transactions of the Association for Computational Linguisties, 2:311- 326.

---

Jakub Piskorski, Nicolas Stefanovitch. Valerie-Anne Bausier, Nicolo Faggiani, Jens Linge, Sopho Kharazi, Nikolaos Nikolaidis, Giulia Teodori, Bertrand De Longueville, Brian Doherty, Jason Gonin, Camelia Ignat, Bonka Kotseva, Eleonora Mantica, Lorena Marcaletti, Enrico Rossi, Alessio Spadaro, Marco Verile, Giovanni Da San Martino, Firoj Alam, and Preslav Nakov. 2023a. News categorization, framing and persuasion techniques: Annotation guidelines. Technical report, European Commission Joint Research Centre, Ispra (Italy).
the latter is used to justify and/or to support the statement.
Simplification: The argument excessively simpli- fies a problem, usually regarding the cause, the consequence or the existence of choices.
Distraction: The argument takes focus away from the main topic or argument to distract the reader. Call: The text is not an argument, but encour- agement to act or think in particular way. Manipulative wording: the text is not an argu- ment, but uses specific language, which contains words or phrases that are either non-ncutral, confus- ing, exaggerating, loaded, etc., in order to impact the reader emotionally.
Jakub Piskorski. Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023b. Semeval- 2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi- lingual setu. In Proceedings of the 1 17th Interna- tional Workshop on Semantic Evaluation, SemEval 2023, Toronto, Canada.
They are further subdivided into 23 fine-grained persuasion techniques. The full list of the fine- grained techniques is presented in 3, whereas some examples of text snippets representing various per- suasion techniques are provided in Figure 4.
Jakub Piskorski, Nicolas Stefanovitch, Nikolaos Niko- laidis, Giovanni Da San Martino, and Preslav Nakov, 2023c. Multilingual multifaceted understanding of online news in terms of genre, framing, and persua- sion techniques. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 3001-3022, Torouto, Canada. Association for Computational Lin- guistics.
B Dataset Statistics
In Figure 5 we provide the distribution of the per- suasion techniques per language. Name Calling and Loaded Language are by far the most popu- lated classes across all languages, and are followed by Doubt and Questioning the Reputation. In total there were approx. 9K text spans (with persuasion techniques) annotated for English (536 documents), 7.2K for French (21) documents), 5.7K for German (177 documents), 8K for Italian (303 documents), 3.8K for Polish (194 documents), and 4.1K for Russian (191 documents).
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and politi- cal fact-checking. In Proceedings of the Conference on Empirical Methods in Natural Language Process- ing. EMNLP 177, pages 2931-2931 Copenhagen, Denmark.
Holger Schwenk and Matthijs Douze. 2017. Learn- ing joint multilingual sentence representations with neural machine translation. arXiv preprint arXiv: 1rXiv:1704.041:: 704.04154.
C Annotation guidelines excerpt
Zhivar Sourati, Vishnu Priya Prasanna Venkatesh, Dar- shan Deshpande, Himanshu Rawlani, Filip Nievski, Hông-An Sandlin, and Alain Mermoud. 2022. Ro- bust and explainable identification of logical fallacies natural language arguments.
This section provides an excerpt from the anno- tation guidelines (Piskorski et al., 2023a). The following general rules should be applied when annotating persuasion techniques:
if one has doubts whether given text frag- ment contains a persuasion technique then do not annotate it, (conservative approach)
A Persuasion Techniques
The two-tier persuasion technique taxonomy has 6 -garie- categories:
select the minimal amount of text' to annotate case of doubts whether to include a longer text fragment or not,
Attack on reputation: The argument does not address the topic, but rather targets the participant (personality, experience, deeds) in order to question and/or undermine their credibility. The object of the argumentation can also refer group of indi- viduals, an organization, an object, an activity. Justification: The argument is made of two parts, a statement and an explanation or an appeal, where
'In our guidelines we do have specific rules for each of the persuasion techniques of what the annotation should include, e.g.. in case of Justification techniques the annotation should include certain appeal and the claim or idea it supports, if explicitly expressed in the immediate context, or, in case of Loaded Language only the amotionally-loaded word/phrase should he annotated, disregarding the context appears in

---

ATTACK REPUTATION
Name Calling or L Labelling [AR:NCL]: a form argumeet in which loaded labels are directed at individual, group, abject or acrivity, typically in an insulling or demeaning way, but also using labels the target audience finds desirable.
Name Calling or Lubelling: 'Fancia' Anti- Vax Rist Sparks COVID Outbreak in Australia.
Guilt by Association [A RcGAJ; attacking the opponenc or an activityy by associating with another group, activity concept that has sharp
Guilt by Association: Manohar is big supporter for equal pay for equal work. This the same policy that all those extreme feminist groups support. Extremists like Manohar thould not taken seriously. This task is quite complex. Is his professional p background, experience and the time left sufficient to accomplish the task hand?
negative comnotations for the largei audience.
Casting Doubt [AR:D]: questioning the character personal artributes of someone or something order question their general credibilitt or quality.
Casting Doubt:
Appeal Hypeeriky [AR:AH]: H]: the target of the technique attacked its reputation by changing them with fnnomstennm
Appeal to Hy Hypuccisy; How can you demand that I eat less meat to my carbon footprint you yourself drive big SUV and fly for holidays to Bali?
Questioning the Reputation [AR:QR]: the target attacked by making strong negative claims about it, focusing specially on undermining its character and moral stature rather than relying on argument about the topic.
Questioning the Reputation; I hope I presensed my argument clearly, Nove my upponent will attempt refute my argument by own fallaciouz, incokerent, illogioal version of history
JUSTIFICATION
Flag Waising: We should make America great again, and restrict the inumgraation laws.
Flng Waiving [J:W]]: justifying idea exhaling the pride of group or highlighting the benefits for that specific group.
Appeal to Aufbority [J:AA]: weight given argumeet, an idea information hy simply stating that a particular entiry considered on authority is the source of the information.
Appeal to Authority: Since the Pope said that this axpect of the doctrine true we should add i to the creed.
Appeal Popularity: Because everyone else goes away to college, it must be the right thing to do.
Appeal Popularity [J:A]]: weight is given to an argumeett or idea justifying it on the hasis that allegedly "everybndy" (or the large majocity) agrees with it or "nobady"" disagrees with
Appeal Values: It's standard practice to pay we' IIl continue adleeing to the same standards this company has always followed.
than
Appeal to Values [J:A]] a weight is given an idea by linking it to values soon by the targer audience as positive.
Appeal Fear, Prejudice [J:AF]: promotes rejects idea through the repulsion or fear of the audience lowards this iden.
to Appeal Fear, Prejudice: a great disservice to
monntoin the preteuse that there is nothing probiematical about Amoris lactitia. A moral cuinstrophe self-evidenily underway and not possible honestly to deny its couse.
DISTRACTION
Strawman [DSM]] consiats in making an impression refuting an argumentt of the opponent's 's proposition. whereas the real subject of the argument was not addressed refuted, but instead replaced with false one.
Referving to your claim that previding medicare for all citizeus would he onstly and danger five market, iufor that you don't 't care if people die from not having healtheare, so we are not going suppurt your endearour.
strawman:
Red Herring [DRHH] comsists in diverting the atiention of the undience from the main tupic being discussed, by initoducing another lopic, which is incllvvnt. Whataboutism [D:W]: technique astempts discredit opponent's is position by charging them with hypocrisy without ditectly disproving their argument.
Red Herring: Lately, there has been a lot of criticism regarding the quality of our product. We' 've decided to have new sole in response, you can buy
lower cast!.
Wh hatabontimm: nation deflects criticism of its recent human rights violations by pointing to the history of slavery in the United States.
SIMPLIFICATION
School violence hat gone up and academic performance has gome down since video gumer featuring violence were introduced. Therefore, video games with violence should be bauned, resulting in school improsement.
Causal Oversimpiiication:
Causal O Oversámpliscatton [S:CaO]: assuuming a single cause or reason when there are actually multiple causes for an
False Dilemma or No Choice [S:FDNC]: a logical fallacy that presents only two options sides when there are many options sides. In extreme, the author tells the andience exactly what actions to take. climinating other possible choices.
False Dilemma or No Choice: There is no alternotive to Pfizer Covid-19 dies.
vaccine. Either sakos
Consequential Oversimplification: If beglu restrict freedom of speech, this will encourage the government to infringe upon other fundamental rights, and eventually this will result totaliturian state make
assertion one making of some "liss' ovent/action leading to a domino-like like chain of events have some significant negative (positive) effects and consequences that appear to he Indicrous or unwarranted or with each step in the chain and more improbbble.
Consequential Oversimplifictiion [S:CoO]:
where citizens have little to no control of their lives and decisions they
Sloguns: "Immigrants welcome, racist
CALL
Conversation Killer: I'm not to naive or simplistic to believe we can eliminate wars. You con't 't chunge human nature.
Slogans (C:S]: a hrief and striking phmse often acting like emotional
appeals, that may include labeling and stereotyping.
no time engage the lucury of cooling off to take the tranquilziig drug of gradualimm. Now is the time to make real the promises of democracy, Now is desolate valley of segregation to the sunlit path of racial justice.
Appeal to Time:
Conversation Killer [AACK]: words or phrases that discourage critical
thought and mesningful discuasion about a given topic.
from the dark and
Appeal Time [C:AT]: the argument is centred around the idea that time has come for a particular action.
I Loaded I Language: They keep freding these people with trash. They should stop.
MANIPULATIVE WORDING
Obfuscation, Intentional Vagueness, Confusion: Feathers can not be durk, becouse all feuthers light!
Loaded Language LMW:LLJ: use of spocitic words and phrases with strong emotional implications (either positive negative) to influence and convince audience that an argument valid.
Exuggeration or Minimisation: From the seminaries, to the elengy, to bishops, to the corainaiis homosesuals are present at all levels, by the thousand
Obfuscation, Intentional Vagueness, Confusion [MW:OVC]: use of words that deliberately clear, vague ambiguous so that the audience may have its own interpretations.
Repetition: Hurtlocker deserves an Osear, Other fimms have posential, they do not deserve an Oscar like Hurtlocker does. The other mouies may deserve homoruble mention bus Hurtlocker deserves the Oscar.
Exaggeration Minimisation [MI: consists of either representing something an exoessive manner or making something seem less important or smaller than it really is.
Repetition [MW:R]: the speaker uses the same phrase repeatedly hopes that the repetition will lead to persunde the audience.
Figure 4: Examples of text snippets with persuasion techniques. The text fragments highlighted in bold are the actual text spans annotated.
Figure 3: Persuasion techniques taxonomy. The six goarse-grained techniques are subdivided into 23 fine- grained ones. An acronym for each technique is given in squared brackets.

---

Persuasion lechnique Distribution
The part of the text that refers For each persuasion technique we have also speci- fied what text fragment should be annotated in the document. The general rule is to annotate the min- imum amount of text that can be considered as a trigger to spot the technique, even if requires an understanding of the context that spans over more than one of the preceding sentences. Sometimes, the to-be-annotated text fragment might go beyond the boundaries of one single sentence. In the fol- lowing we briefly summarize the rules for all the techniques.
tuague sbon Miiirbattion
Dofuscation Vagueness Contusion
Tradert longrong
Apped to Time
Conserarten 12l
Whitebouttom
Hld furring
Stownan
Rolo Stiermma, Thoke
Coveal Deotttrlliiitttt
Appeall to Autharitty
ropoal Populaiity
Apsual la Valvas
Flgg Klviing
woddiiiig Reputtaiion
Name Calling or Labelling: The noun phrase, the adjective that constitutes the label and/or the name. If quotation marks are used, they should be included in the annotation as well.
Hyyporryy
Deube
curk Aysontiation
Aama celling Lubling
Pulah
in merman
Frund
Guilt by Association:
The part of text that refers to an entity and a mention of someone else (consid- ered evil/negative) doing the same or similar thing that is considered negative. The mention of the activity of the target entity might be implicit.
Figure 5: Distribution of the persuasion techniques per language (in percentage).
avoid personal bias (i.e., opinion and emo- tions) on the topic being discussed as this has nothing to do with the annotation of persua- sion techniques,
Only the text fragment that ques- tions the credibility and the object whose credibility is being questioned. There no need include the full context.
Casting Doubt:
do not exploit external knowledge to decide whether given text fragment should be tagged as a persuasion technique,
The text phrase embracing certain activity, and another one which is used as an argument to accuse the former as being a hypocrite.
Appeal to Hypocrisy:
do not confuse persuasion technique detec- tion with faet checking. A given text frag- ment might contain a claim which is known to be true, but that docs not imply there are no persuasion techniques to annotate in this particular text fragment,
Questioning the Reputation: Only the text frag- ments that refer to something negative being men- tioned about the person/group/ohject.
Flag Waving: The part of the text that refers to patriotism other group related values, and the conclusion/action it is supposed to support if it is present in the text.
often, authors use irony (not being explicitly part of the taxonomy), which in most cases serves purpose persuade the reader, most frequently to attack the reputation of someone or something. In such cases the respective persuasion technique type should be used, or other if the use of irony does not fall under any persuasion technique type in the taxonomy,
Appeal to Authority:
The part of the text that refers to the authority (and potentially some of his/her statemeen/opimiion/ccttonnn and the conclu- sion it supports, in case the latter is present in the text.
Appeal to Popularity: The part of the text that refers to something that a majority does or seems to be widely supported and/or is popular together with the conclusion it is supposed to support.
in case of quotations reporting of what given person said the annotation of the persua- sion techniques within the boundaries of that quotation should be done from the perspective of that person who is making some statement or claim (point of reference) and not from the author perspective.
Appeal to Values:
to values, and include the conclusion it is supposed to support, in case the latter is included explicitly in the text.
Appeal to Fear, Prejudice: The part of the text that

---

refers to the fears, prejudices, e.g., of something that might happen.
Strawman: When this technique is used, usually the relevant context might span across more sen- tences. However, one should only annotate the text fragment (sentence or part thereof), which intro- duces the distraction.
115
When this technique is used, usu- ally the relevant context might span across more sentences. However, one should only annotate the text fragment (sentence part thereof), which in- troduces the distraction.
Red Herring:
120
145 2006
Figure 6: Confusion matrix on the annotations, as per Holistic IAA, a minimal support of 50
Whataboutism: When this technique is used, usu- ally the relevant context might span across multiple sentences. However, one should only annotate the text fragment (sentence or part thereof) that intro- duces the distraction.
by quotation marks, include them as well.
Conversation Killer: minimal text span that triggers ending the conversation, discussion, etc.
Appeal to Time: A minimal text span referring to the argument of time that calls for some action. Both the call and the action should be annotated. Loaded Language: Only the phrase containing loaded words, the context in which they appear should not be annotated. As general rule one should consider to tag longer text fragment if and only if each of the words adds more emotional 'load' to the text fragment.
Causal Oversimplificatin: The minimal text fragment that matches the logical pattern should be annotated:
occurred after therefore, was the only cause of
caused Y; therefore, X was the only cause of (although A,B, also contributed to Y.)
or a false conclusion drawn therefrom should be an- notated, although, often not all parts of the pattern above are explicitly mentioned in the text.
Obfuscation, Intentional Vagueness, Confusion:
The minimal text fragment that introduces confu- sion: it could be word, but also longer piece of text that requires to be read in order to understand the confusion it causes.
False Dilemma or No Choice: The minimal text fragment that matches of the following logical patterns should be annotated:
(a) Black& White Fallacy:
Exaggeration or Minimisation: The text frag- ment that provides the description that downplays or exaggerates the object of criticism. The latter should be included in the annotated text as well.
There are only two alternatives and B to a given problem/task, task. It cannot be A. Therefore, the only solution is B (since A is not an option).
(b) Dictatorship
Repetition: All text fragments that repeat the same message or information that was introduced earlier. The first occurrence of the message/information is to be annotated well. If it is not clear what exactly to annotate then the entire sentence should be annotated. Furthermore, phasize that a repetition of something per se is not always a persuasion technique, but could some- times be used only to refer to a topic/issue being discussed.
The only solution to given problem/task is A. although, often not all parts of the pattern above are explicitly mentioned the text.
Consequential Oversimplification: The entire text fragment that matches the above logical pattern should be annotated:
important to em-
if A will happen then B, D, ... will happen
where:
is something is trying reject (support) D are perceived some potential negative (pasitive) consequences happening if happens.
D Confusion Matrix based on Holistic IAA
The slogan only (no need to annotate the conclusion it supports), and in case it is surrounded
Slogans:
In Figure 7 we report the confusion matrix found using Holistic IAA on the final curated dataset. De-

---

misking
ranking2
cost
010
0.75
undl
(020
0.0
0.75
SJnW
000
0.80
det
street
0.27
0.0
0.80
kame
stree
0.87
0.0
0.80
com
0.20
01
0R8
0.25
0.0
same
0.51
0.0
und
-0.35
164
0.0
0.90
came
street
0.67
0.0
0.90
cam
0.17
144
0.50
0.75
ai
Minel
0.27
0.50
0.30
0.40
0.30
0.80
same
stree
0.87
V9L
0.50
0.80
same
dif
0.20
1303 4000
MWCV MM MW000
0.50
0,85
aml
0.33
0.50
0.85
0.93
Figure 7: Confusion matrix on final corpus, as per Holis- tic IAA, for a minimal support of 10
-0.35
0.30
com
0.67
0.50
0.90
kame
0.00
0.75
0.75
4df
Street
0.27
0.87
spite overall similarity with exact confusion matrix on annotations in Figure there are few notable difference, particularly with C: CK being largely largely more confused and A being largely less confused.
0.5
0.80
sam
und
0.9
0.33
0.75
0.80
same
dif
0.30
0.75
0.85
ddff
straet
0.18
0.75
0.85
sirel
0.67
0.75
0.85
0.22
stbxt
0.75
kame
street
0.67
0.75
0.90
53me
diF
0.00
In Figure 6 we report the confusion matrix on the set of annotations. It is closer the Figure than Figure 7. This indicates that the curation process actually eliminated some common confusion in the annotations. The magnitude are different for two reasons that can not be mcasure independently: it contains less errors, and there are less overall pair-wise comparison performed as the total set of annotation considered is about two times smaller.
Table 9: Rank correlation between: Cohen's k. com-
puted on the original data (strict). Holistic IAA com- puted on the same documents Cohen (same). Holistic IAA computed on all the other documents (diff).
which maximise the rank correlation between the ranking of annotators produces by Cohen's Kappa and the one produces by Holistic TAA. We consider groups of document--nnolltorr pairs: strict, for which Cohen's к can be computed; same any document annotated by previous annotators; diff any documents not annotated jointly by previous annotators. The pairwise comparison of these sets with Kendall's Tau rank correlation is consider 3 dimensions of multi-criteria decision problem: A minimal number of 10 annotations in common is required for pair of annotator be considered, support is the total number of pairs being compared. The table with the result of the parameter search are reported Table 9.
Identifying the top and low groups of annotators
order to split the annotators into two groups, in a first time the curators based on their subjective as- sessments established groups of equal size. This was further corroborated in a second step using the following approach: the curated data was taken as considered
ground truth and the annotators classifiers, whose annotations are considered as prediction. As such we computed the miero F1 for each annotator, and ranking them along that mea- sure, the median split validated the first subjective assessment which contained a few more annotators, which all ranked the highest in the lower split. The average micro Fi score of the top and low groups are respectively of 0.603 +- 0.119 and 0.284 0.081.
F Parameter search
We conducted an exhaustive parameter search in order to determine the optimal parameter f, and A,


Reviews:
[{"review_id": "FcbVbnNr1K", "reviewer": "Reviewer_9ysf", "summary": "", "questions": "-", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the fair review. Regarding the points raised under “Reasons to reject”:\n\n“Subjectivity of the annotation task”: \n\nIn Section 4.3 we provide detailed analysis of the disagreements between different annotators vis-a-vis different persuasion techniques. While for some techniques (but few) it is indeed due to subjective perception, e.g. Loaded Language, Name Calling - due to cultural differences, we have found out that for the vast part of the techniques the high-level of disagreements (multiple meetings with annotators) was actually due to: (a) not understanding the techniques - many annotators did not read the guidelines thoroughly enough before starting the real annotations, which caused problems, for instance, as regards Simplifications, which have rather strict definitions and certain logic patterns have to be found in text (see the Annex of the paper); instead the annotators were often explicitly interpreting the word “simplification” and reasoning based on whether the presentation is too simplistic and whether certain facts were downplayed/exaggerated, which is actually a different technique, (b) professional bias - some of the media analysts who served as annotators were, unfortunately, often using background knowledge to make decision whether some text fragments are instances of persuasion techniques, which was strictly prohibited, (c) some of the annotators were making a direct link of persuasion technique labeling with claim checking and fact verification, which was again a mistake, etc. We have not provided in the paper a section on these findings due to space limitations although it could have been added in the final version of the paper to provide deeper insights on the lessons learned from the joint meetings with the annotators.   \n\n\"Computation of the holistic iaa value\":\n\nThe procedure to compute the value is described in section 5.1, however we reckon that there was no explicit formula for the computation in the submitted paper. We can easily provide such a formula: \n\no(a1, a2, n_sample, th_len, th_sim) = mean([a1(x) == a(y) for x in A(a1) for y in Intersection( A(a2) , MostSimilar(x, n_sample) ) if len(x) <= th_len * len(y) and len(x) >= (1-th_len)*len(y) and sim(x, y, “LASER”) >= th_sim])  , where A(ai) is the set of annotated strings of annotator ai, and ai(x) is a function giving the annotation of annotator ai for the input string x.\n\nPlease note, that this is an approximate value, as in the paper which considers only a subset of the annotations, and is asymmetric (the value is computed for a target annotator) using a maximal number of points to compare it with.\n\nWe can provide also an exact formula with the parameters to compute an exact value which is symmetric for both annotators (not used in the paper for the moment):\n\no(a1, a2, th_len, th_sim) = mean([a1(x) == a2(y) for x,y in A(a1) x A(a2) if len(x) <= th_len * len(y) and len(x) >= (1-th_len)*len(y) and sim(x, y, “LASER”) >= th_sim]). \n\nIf introduced we will probably have the time to compare both approximate and exact values before the camera-ready deadline.\n\n“Usefulness of the approach”: \n\nThe primary goal of our submission was to share some lessons learned in a large-scale annotation campaign and the concept of a new way to compute IAA in a scenario where annotators do not label the same documents and to compare annotations across different languages to guarantee certain level of coherence. While the introduced IAA metric is just a proof of concept, and certain  approximations and simplifications were introduced/made, e.g., constraining the length of retrieved strings and their numbers, and the choice of a specific sentence embeddings model, it helped to circumvent the limited scope and utility of IAA in such a large-scale multilingual campaign. Disregarding the fact of some limitations of the results reported in our manuscript, we believe the reported experiments might have practical value for scientists involved in suchlike large annotation campaigns, and the presented work might trigger further studies in this direction. \n\nNote on the dataset Dataset availability:  the  curated dataset with persuasion technique annotations is available to the scientific community, and has been described in another paper, which is not disclosed here for anonymity reasons. The single-annotator versions thereof used to compute the various agreement metrics are envisaged to accompany the final version of this paper or to be made available upon request.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The annotation campaign done for their experiments is massive, and the article is well written.", "reasons_to_reject": "Annotating persuasion techniques is a very subjective task. Moreover, the paper introduces the Holistic IAA metric but fails to explain how this is actually computed. Also, they conclude that Holistic IAA highly correlates with rankings computed using Cohen Kappa's in some settings, so it is not clear what the usefulness of this metric is.", "questions_to_authors": "-", "paper_topic_and_main_contributions": "This paper is about the complexity of persuasion technique annotation in a large annotation campaign involving 6 languages and approximately 40 annotators. Its main contribution is introducing a new word embedding-based annotator agreement metric called HolisticIAA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "Please see lines: 038, 144, 162, 321, 574, 642"}, "extra_text": {}}, {"review_id": "mb7JHUgvGM", "reviewer": "Reviewer_Wmuj", "summary": "", "questions": "1. Do I understand your objective correctly? Is your objective in effect to compute agreement between annotators on a larger scale, i.e., not only on the sentences they all annotate?\n\n2. If the answer to the above is 'yes', how are you proposing to assess whether the obtained Holistic IAA value is sufficient to your purposes?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. Regarding the points raised and questions:\n\nQ1 “Objective of the reported work”: \nThis work resulted from the specific need in our annotation campaign to capture agreement between annotators on a large scale, as standard IAA measures are limited in scope and utility in such a multi-lingual set-up. Hence, the answer to the direct question is indeed yes. It is important to emphasize that the proposed IAA metric is used to compute: (a)  agreement between annotators, and (b) final dataset coherency (the “dataset quality” might have been somewhat misleading wording we used in the manuscript). Instead of directly proving the appropriateness of the metric for the tasks at hand, we demonstrate the correlation of the new metric with well-established IAA metric in the community in the limited number of cases when such a comparison is possible, and we also demonstrate that after the second round of data curation the value of the new metric increases. We consider this as a qualitative demonstration of the metric to capture both IAA and coherence of the annotations. \n\nQ2 “Limitation of single-number agreement”: \nWe agree on the limitation, and we were not familiar with the article mentioned, to which we will compare our approach. However, the approach proposed in our paper tries to emulate standard IAA measure with a single number as these are, despite their limitations, very popular due to their simplicity and even mandatory when publishing a dataset in order to give a rough idea of the quality of a dataset. We think having richer representation of agreement as described in the suggested paper is a very interesting direction for an extension of the work we report on in this paper.\nAs regards subjectivity per se, we elaborate on this issue in the reply to reviewer 1, and we believe that (based on multiple rounds of meetings with the annotators) for the major fraction of the persuasion techniques the disagreements resulted mainly due to not sticking strictly to the definitions provided in the 60 pages of guidelines we prepared and bias resulting from professional background leading again to wrong comprehension of the persuasion technique definitions, e.g., simplifications have to follow a very strict logical pattern that needs to be identified in text, with which many people seemed to have problems. To conclude, we consider subjectivity as a minor factor contributing to the disagreements. We could add a section on these findings regarding subjectivity in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly built on a great deal of expertise about annotation, in particular in large-scale annotation projects. It raises some very good questions about the limitations of coefficients of agreement in such large-scale annotation projects, and illustrates some very good annotation practices - it could be useful to others aiming to run such a project.\n\n2. The method itself makes a lot of sense.\n\n3. the results obtained with the proposed method are analysed to a great depth.", "reasons_to_reject": "1. The main problem I have with this paper is that it's not completely clear to me what is the problem that the authors are trying to solve. My understanding is that they are trying to come up with a more useful way of measuring the actual agreement between annotators not just on the few examples they both annotate, but I am not completely sure this is right. \n\n2. A more general problem is that others have realized that coefficients of agreement are really a limited metric - especially in case of subjective judgments - and have tried to devise more informative approaches, although nobody as far as I know has addressed the specific issue tackled in this paper. I am especially thinking of the work on probabilistic models of annotation by Passonneau and Carpenter, and their TACL 2014 paper. In that paper, the authors argue convincingly that a single number cannot be sufficient  for the in-depth  analysis of annotation that the authors of this paper have in mind. The type of analysis they propose involve building models of each annotator, and of each item, that do allow for more insightful comparisons. I would encourage the authors to have a look at these methods and perhaps think about generalizing them. ", "questions_to_authors": "1. Do I understand your objective correctly? Is your objective in effect to compute agreement between annotators on a larger scale, i.e., not only on the sentences they all annotate?\n\n2. If the answer to the above is 'yes', how are you proposing to assess whether the obtained Holistic IAA value is sufficient to your purposes?", "paper_topic_and_main_contributions": "This paper introduces a method called 'Holistic Inter-Annotator Agreement' to compute the (percentage) agreement between each pair of annotators on the semantically most similar sentences they have both annotated. This allows the authors to compute what annotators are most similar to a particular annotator. This ranking is shown to correlate well with the ranking obtained using standard IAA metrics such as Cohen's K. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics, 2:311–326."}, "extra_text": {}}, {"review_id": "gN2femrVDO", "reviewer": "Reviewer_Jcj8", "summary": "", "questions": "I would like to see addressed the concerns raised under 'Reasons to reject'", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and bringing references we were not aware of. Regarding the points raised under “Reasons to Reject”:\n\n“The usefulness of agreement as quality indication”: \n\nWe reckon that IAA measures have several flaws, however despite these flaws they are still popular and mandatory to report when releasing new datasets. Our aim in this work rather than tackling all the flaws rightly pointed by the reviewer, is to propose one solution to tackle one specific aspect, i.e., the incomparability of annotations in different languages and different documents.\nRegarding the specific aspect of dataset quality, we are making the assumption that \"coherent\" annotation is an indicator of quality as it gives predictability. However, we could have made it more clear by reflecting it in the title and emphasizing on the concept of “coherence” instead of using the concept of “quality” (not the ideal in this context).\n\n“Bias of existing IAA metrics and impact on the evaluation”: \n\nThe limitations of Cohen’s Kappa are indeed well known, nevertheless we needed to base our evaluation on existing and well established metrics and choose Cohen's Kappa as such basis. As such, until further proofs are made (i.e., studying the correlation with other IAA measures - envisaged as future work) one could consider for the moment our approach at worst as a “multi-lingual multi-document extension of Cohen’s Kappa”, which we believe would still be interesting per-se for the community in the context of organizing such large multi-lingual annotation campaigns.\n\n“Quality of the sentence embeddings and the similarity thresholds”: \n\nThe primary goal of our submission was to share some lessons learned in a large-scale annotation campaign and to introduce the \"concept\" of a new way to compute IAA in a scenario where annotators do not label the same documents. Consequently, some approximations and simplifications were introduced, e.g., constraining the length of retrieved strings and their numbers, and the choice of the sentence embeddings model. However, we don’t claim the best set of parameters and models used, including the choice of the sentence embeddings model, but rather show that a versatile approach exists: for some applications or languages some embeddings could be more performant and the settings used in this paper should not be used, we could make clearer that these choices are task specific. We believe that in order to demonstrate the proof of concept of IAA agreement in the specifc scenario where annotators annotate different documents the choice of the underlying sentence embedding model is irrelevant. A follow-up study on comparison of different models could constitute a follow-up research direction\n\nNote on the dataset availability:  the curated dataset with persuasion technique annotations is available to the scientific community, and has been described in another paper, which is not disclosed here for anonymity reasons. The single-annotator versions thereof used to compute the various agreement metrics are envisaged to accompany the final version of this paper or to be made available upon request.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed metric allows to compute annotator agreement even if the annotators did not label the same documents, i.e., by pooling from the corpus similar sentences to those annotated by the annotators (using sentence embeddings) and computing label agreement on that set instead", "reasons_to_reject": "Inter annotator agreement statistics are useful to measure the reliability of an annotation scheme, i.e, that the coders (aka annotators) have internalized the coding instructions s.t. a sufficient level of agreement can be observed), but are not informative about the quality of a dataset. Agreement is flawed for many reasons, e.g., agreement in mistakes, agreement due to label biases, large chance agreement in datasets with skewed classes, headline measurements with no information about the quality of the individual labels, and many more. \n\nAlso, I am not convinced about using as a form of evaluation the correlation with an existing IAA metric. Mainly because these metrics are already biased in some form, e.g., the kappa paradox.\n\nLastly, the quality of the sentence embeddings and the similarity thresholds seem central to the success of the proposed metric, however, their selection is treated rather lightly in the paper.", "questions_to_authors": "I would like to see addressed the concerns raised under 'Reasons to reject'", "paper_topic_and_main_contributions": "The paper introduces a new inter annotator agreement metric,  HolisticIAA, based on sentence embedding similarity. The metric is compared against two traditional IAA metrics, Cohen's k and Krippendorff’s alpha, for the annotation process of one dataset on persuasion classification of text snippets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 537 -> Table* 6\nline 646 -> without in advance, or early"}, "extra_text": {"Missing_References": "Inter-coder agreement for computational linguistics -> offers a good read into the utility and shortcomings of IAA. \n\nThe paper above was updated and can be found part of a larger study into statistical methods for annotation analysis, part of this recent book: Statistical methods for annotation analysis\n\nThe aforementioned book starts by presenting coefficients of agreement, useful for assessing the reliability of an annotation scheme, then moves on to measures of dataset and annotator quality, to probabilistic models of annotation, in particular. On the latter topic, the following resources are also good starting points:\n\nA recent EACL tutorial on 'Aggregating and Learning from Multiple Annotators' and these two TACL papers: The Benefits of a Model of Annotation, and Comparing Bayesian models of annotation"}}]

Author rebuttal:

Meta-review: The paper discusses a new idea for assessing IAA and annotation quality in large multilingual annotation campaigns characterized by high complexity and minimal overlap in the annotated items. Specifically, the authors introduce a new annotator agreement metric based on word embeddings to better account for the difficulties inherent in human annotation of persuasion techniques and to assess the overall coherence of a dataset. Given the subjective nature of the task, investigations into annotators’ disagreement on the specific techniques can significantly help improve future research. The rebuttal phase was constructive and helped clarify several points. I am confident that the authors will enhance the revised version by incorporating many of the details mentioned in their responses. 

**Pros**
- it introduces a novel, robust method for assessing agreement in large multilingual annotation campaigns, where overlap between annotators is limited; 
- it presents a new metric as a proof of concept and adequately highlights the limitations of the approach;
- a large-scale annotation campaign was conducted (6 languages and about 40 annotators); 
- the paper reveals high expertise ;
- the work can be of considerable value for other researchers in the field;
- the article is well written and clear;
- results analysis is sufficiently thorough.

**Cons**
- greater clarity seems to be required in stating and explaining the main objectives of the paper, to avoid misunderstanding. The responses provided by the authors during the clarification phase should be integrated into the revised version; 
- a discussion on the role of subjectivity and on the main sources of disagreement should be integrated;
- the explicit formula for the new metric needs to be given in the paper;
- essential details about the experimental setting are missing (see reviewers comments). 
It is worth noting that these deficiencies have been convincingly addressed in the author responses, suggesting that they would necessitate only minor revisions in the final paper.
Decision: Accept-Main