# _VivesDebate_: A New Annotated Multilingual Corpus of Argumentation in a Debate Tournament
## 3 Annotation Methodology
### Inter-Annotator Agreement Tests  
We carried out, first, a qualitative analysis in order to validate that the team of annotators was applying the same criteria in the revision of the automatic transcriptions of debates. This analysis consisted of the revision of three files (_Debate15.csv_, _Debate11.csv_, and a debate from the previous year's edition, which was not included in our corpus) by the two annotators in parallel and the comparison of the results obtained by the senior annotators. The team met to discuss the problems arising from the comparison of the results in order to resolve doubts and inconsistencies. We devoted three sessions to this until we solved all disagreements and reached the same results in the revision of transcriptions, which explains why we revised the three files selected (one per session). The initial guidelines were updated with the new criteria established, such as following the criteria of the linguistic portal of the Catalan Audiovisual Media Corporation for spelling the named entities or writing down the main ideas in those cases in which the quality of the audio did not allow us to understand part of the message conveyed. In a nutshell, we ensure that the text of the ADUs was transcribed correctly, maintaining the linguistic variant originally used, whereas in the remaining text we applied a more superficial revision. Once the transcription of texts obtained reliable results, we initiated the segmentation task, which was by far the most difficult task in the whole annotation process. We conducted three Inter-Annotator Agreement (IAA) tests until we reached an acceptable agreement for the segmentation of the transcribed texts into ADUs (see Table 1). We used the same file (_Debate6.csv_) in the two tests. We calculated the observed agreement and the Krippendorff's alpha [29]. The criteria followed for the evaluation of the Inter-Annotator Agreement test were the following:  
* In the case of the PHASE, STANCE and argument REL_TYPE tags, we considered agreement to be reached when the annotators assigned the same value to each tag, while disagreement was considered to be when the value was different. * In the case of the ADU tag, we considered agreement to exist when the span of the ADUs matched exactly, and disagreement to exist when the span did not match at all or coincided partially. We have also conducted a third Inter-Annotator Agreement test for evaluating the ADU tag considering partial agreement. In this case, we considered agreement to exist when the span of the ADUs coincided partially (22)-(25). (22a) [la cosificacio que s'esta fent de la dona]_ADUI_  
(22b) [the objectivation of women]_ADUI_  
(23a) [**ens centrarem en** la cosificacio que s'esta fent de la dona]_ADUI_  
(23b) [**we will focus on** the objectivation of women]_ADUI_  
(24a) [el vincle que es genera entre ella i el nado que porta al seu ventre]_ADUI_ [**es tencat de manera miserable]_ADUI_  
(24b) [the bond between her and the baby she carries in her womb]_ADUI_ [**is broken in a miserable way]_ADUI_  
(25a) [ el vincle que es genera entre ella i el nado que porta al seu ventre es trencat de manera miserable]_ADUI_  
(25b) [the bond between her and the baby she carries in her womb is broken in a miserable way]_ADUI_The disagreements found are basically of two types: (a) the inclusion or omission of words at the beginning or at the end of the ADU (22) vs. (23); and (b) the segmentation of the same text into two ADUs or a single ADU (24) vs. (25), the latter being stronger disagreement than the former. For instance, one of the annotators considered 'is broken in a miserable way' to be a different ADU (24), whereas the other annotator considered this segment part of the same ADU (25). Finally, we agreed that it should be annotated as a single ADU (25), because 'is broken' is the main verb of the sentence, and the argument is that what is broken is the bond between the mother and the baby.