VivesDebate-Speech: A Corpus of Spoken Argumentation to Leverage Audio Features for Argument Mining
Authors: Ramon Ruiz-Dolz
Javier Iranzo Sanchez
Venue: EMNLP 2023 Main (2023)
Keywords: Argument Mining
Argument Segmentation
Argument Classification
Speech Corpus

Abstract: In this paper, we describe VivesDebate-Speech, a corpus of spoken argumentation created to leverage audio features for argument mining tasks. The creation of this corpus represents an important contribution to the intersection of speech processing and argument mining communities, and one of the most complete publicly available resources in this topic. Moreover, we have performed a set of first-of-their-kind experiments which show an improvement when integrating audio features into the argument mining pipeline. The provided results can be used as a baseline for future research.

VivesDebate-Speech: A Corpus of Spoken Argumentation to Leverage Audio Features for Argument Mining
Ramon Ruiz-Dolz
Javier Iranzo-Sánchez VRAIN-MLLP
Centre for Argument Technology
University of Dundee Dundee DD1 4HN, United Kingdom @odddddee
Universitat Politècnica de València 46022 València, Spain jai wairrsanuuuv... es
rruizdolz001 @dundee. ac.
Abstract
This is substantial limitation, not only for our knowledge on the impact that speech may directly have when approaching argument-baste NLP tasks, but because of the significant loss of information that happens when we only take into account the text transcript of spoken argumentation.
In this paper, we describe VivesDebute-Spee Speeech a corpus of spoken argumentation created to leverage audio features for argument mining tasks. The creation of this corpus represents an important contribution the intersection of speech processing and argument mining munities, and one of the most complete publicly available resources in this topic. Moreover, have performed a set of ffst-of-their -inddr-kidd& ex- periments which show an improvement when integrating audio features into the argument mining pipeline. The provided results can be used as baseline future research.
In this work, will focus the initial steps of argument analysis considering acoustic features, namely, the automatic identification of natural lan- guage arguments. Argument mining the area of research that studies this first step in the analy- sis natural language argumentative discourses, and it is defined as the task of automatically iden- tifying arguments and their structures from natu- ral language inputs. As surveyed in (Lawrence and Reed, 2020), argument mining can be divided into three main sub-tasks: first, the segmentation of natural language spans relevant for argumen- tative reasoning (typically defined as Argumenta- tive Discourse Units ADUs (Peldszus and Stede, 2013)); second, the classification of these units into finer-grained argumentative classes (e.g., ma- jor claims, claims, premises (Stab and Gurevych, 2014)); and third, the identification of argumenta- tive structures and relations existing between these units (e.g., inference, conflict, or rephrase (Ruiz- Dolz al., 2021a)). Therefore, our contribution is twofold. First, we create a new publicly avail- able resource for argument mining research that enables the use of audio features for argumentative purposes. Second, we present first-of ceat--ff-hhi---eee their-kind kind experiments showing that the use of acoustic infor- mation improves the performance of segmenting ADUs from natural language inputs (both audio and text).
1 Introduction
The automatic analysis of argumentation in human debates is complex problem that encompasses different challenges such as mining, computation- ally representing, automatically assessing natu- ral language arguments. Furthermore, human ar- gumentation is present in different mediums and domains such as argumentative monologues (e.g.. essays) and dialogues (c.g., debates), argumenta- tion text (e.g., opinion pieces or social network discussions) and speech (c.g., debate tournaments or political speeches), and domains such as the po- litical (Haddadan et al., 2019; Ruiz-Dolz, 2022), legal (Poudyal al., 2020), financial (Chen et al., 2022), or scientific (Al Khatib al., 2021: Bao al., 2022) among others. Thus, human argu- mentation presents a linguistically heterogeneous nature that requires us to carefully investigate and analyse all these variables in order to propose and develop argumentation systems which are robust to these variations in language. In addition to this heterogeneity, it is worth mentioning that vast majority of the publicly available resources for Natural Language Process- ing (NLP) have been created considering text fea- tures only, even if their original source comes from speech (Visser et al., 2020; Goffredo et al., 2022).
The VivesDebate-Speech Corpus
argumentation-baan
The first step in our research was the creation of new natural language argumentative corpus. In this work, we present VivesDebate- Speechh,, an argumen- tative corpus created to leverage audio features for

---

argument mining tasks. The VivesDebate-Speech has been created taking the previously annotated VivesDebate corpus (Ruiz-Dolz et al., 2021b) as a starting point.
Table I: Set-level statistics of the VivesDebate-Speee corpus. Each debate is carried out between teams, and two to four members of each team participate as speakers in the debate.
The VivesDebate corpus contains 29 professional debates in Catalan, where each debate has been comprehensively annotated. This way, it is possi- ble to capture longer-range dependencies between natural language ADUs, and to keep the chrono- logical order of the complete debate. Although the nature of the debates was speech-based argu- mentation, the VivesDebate corpus was published considering only the textual features included in the transcriptions of the debates that were used dur- ing the annotation process. this paper, we have extended the VivesDebate corpus with its corre- sponding argumentative speeches in audio format. In addition to the speech features, we also created and released the BIO (i.e., Beginning, Inside, Out- side) files for approaching the task of automatically identifying ADUs from natural language inputs (i.e., both textual and speech). The BIO files allow us to determine whether a word is the Beginning. belongs Inside, or it is Outside an ADU.
# tags
Set
debates
Duration
Train
9.8h
4605
63305
21432
Dev
3
1,3h
692
8058
3752
Test
3
1.3h
640
8413
3102
by changing the intonation and with the use of pauses. The VivesDebate-Speeee Speech is released under Creative Commons Attribution-NonCommeeeet ShareAlike 4.0 International license (CC BY-NC- SA 4.0) and can be publicly accessed from Zen- odo'.
2.1 Text
The text-based part of the VivesDebate- cor- pus consists of 29 BIO files where each word in the debate is labelled with BIO tag. This way, the BIO files created in this work enable the task of automatically identifying argumentative natu- ral language sequences existing in the complete debates annotated the VivesDebate corpus. Fur- thermore, these files represent the basis on which it has been possible to achieve the main purpose of the PressDebate-Speede corpus, i.e., to extend the textual features of the debates with their corre- sponding audio features.
The ShesDDebate-ppeed corpus is, to the best of our knowledge, the largest publicly available re- source for audio-bases argument mining. Further- more, combined with the original VivesDebale cor- pus, a wider range of NLP tasks can be approached taking the new audio features into consideration (e.g., argument evaluation or argument summarisa- tion) in addition to the standard argument mining tasks such as ADU segmentation, argument clas- sification, and the identification of argumentative relations. Compared to the size of the few pre- viously available audio-based argumentative cor- pora (Lippi and Torroni, 2016; Mestre et al., 2021) (i.e., 2 and 7 hours respectively), the VivesDehate- Speech represents a significant leap forward (i.c.. more than 12 hours) for the research community not just in size (see Table 1) but also versatil- ity. A recent study (Mancini et al., 2022) explored also multimodal argument mining, but focusing exclusively on the classification of arguments and the detection of relations. A fine- -neiinai approach to the segmentation of ADUs is not considered mostly because of the limitations of the existing resources for audio-based argument mining. This is an important limitation, given that audio features have an explicit role in spoken argumentation: ar- gument delimitation. This is achieved, for example,
We created the BIO files combining the transcrip- tions and the ADU annotation files of the VivesDe- bate? corpus. For that purpose, we performed a sequential search of each annotated ADU in the transcript file of each corresponding debate, bring- ing into consideration the chronological order of the annotated ADUs.
2.2 Speech
Once the revised transcription has been augmented with the ADU information, the transcription was force-aligned with the audio in order to obtain word level timestamps. This process was carried out us- ing the hybrid 2NN-HMM system that was previ- ously used obtain the VivesDebate transcription, implemented using TLK (del Agua al., 2014). As result of this process, we have obtained (start,end) timestamps for every (abel, pair. We split the
hhttps://doi..o org/10. htgg/1..521111 5281 /zenodo. 7762260 chttps://doi.orgg 281 http///eeoodooooee 5145655
7102681

---

corpus into train, dev, and test considering the nu- merical order of the files (i.e., Debatel-22 for train, Debate24-26 for dev, and Debate27-29 for test). The statistics for the final VivesDebate-Speene cor- pus are shown in Table 1.
Spoken Debate
3 Problem Description
Audio
Audio 2
Audio i
We approached the identification of natural lan- guage ADUs in two different ways: (i) as token classification problem, and (ii) as sequence classi- fication problem. For the first approach, we analyse our information the token level. Each token is as- signed BIO label and we model the probabilities token belonging to each of these specific label considering the n--length closest natural language contextual tokens. For the second approach, the information is analysed at the sentence level. In order to address the ADU identification task sequence classification problem need to have set of previously segmented natural language se- quences. Then, the problem is approached as 2-class classification task, discriminating argumen- tative relevant from non-argumentative natural lan- guage sequences.
Text
Text 2
Text
A-Seg
T-Seg
Full Audio
Full Text
A-CIf
T-Clf
A1
A2
An
T1
T2
Tn
Segmented Debate
Figure Overview of the proposed eascaded approach.
directly outputs the segmentation of this text into argumentative units. Second, we propose a cas- caded model composed of two sub-tasks: argument segmentation and argument classification. In the first sub-task, the discourse is segmented into in- dependent units, and then for each unit it is deter- mined if it contains argumentative information not. Both approaches produce an equivalent output, a sequence of BIO tags which then compared against the reference. This work investigates how audio features can be best incorporated into the previously described process. An overview of the proposed cascaded method is shown in Figure 1. observe, segmentation step follows the ASR step, which segments either the whole audio (A-Sen (A-Seg) or the whole text (T-Seg) into (potential) argumentative segments. A classification step then detects if the segment contains an argumentative unit, by using either the audio (A-CIf) or the text (T-CIf) contained in each segment. If efficiency is major concern, the segmentation produced by the ASR step can be re-used instead of specific segmentation step tailored for argumentation, but this could decrease the quality of the results. This process significantly differs from the E2E approach where the BIO tags are directly generated from the output of the ASR step. This way, our cascaded
4 Proposed Method
The use of audio information for argument mining presents significant advantages across 3 axes: effi- ciency, information and error propagation. Firstly, the segmentation of the raw audio into indepen- dent units is a pre-requisite for most Automatic Speech Recognition (ASR) system. If the segmen- tation produced in the ASR step is incorporated into the argument mining pipeline, we remove the need for a specific text-segmentation step, which brings significant computational and complexity savings. Secondly, the use of audio features allows to take into account relevant prosodic features such intonation and pauses which are eritical for discourse segmentation, but that are missing from a text-only representation. Lastly, the use of ASR transcriptions introduces noise into the pipeline as a result of recognition errors, which can hamper downstream performance. Working directly with the speech signal allows us to avoid this source of error propagation.
As
Two different methods leverage audio fea- tures for argument mining are explored in this pa- per. First, a standard end-to-end (E2E) approach that takes the text-based transcription of the spo- ken debate produced by the ASR as an input, and

---

model is interesting because it makes possible to analyse different combinations of audio and text features.
Table 2: Accuracy and Macro-F1 results of the argu- mentative discourse segmentation task on both dev and test sets.
The cascaded method has one significant advan- tage, which is that audio segmentation is a widely studied problem in ASR and Speech Translation (ST) for which significant breakthroughs have been achieved in the last few years. Currently, one of the best performing audio segmentation meth- ods is SHAS (Tsiamas et al., 2022). which uses a probabilistic Divide and Conquer (DAC) algo- rithm to obtain optimal segments. Furthermore, we have compared SHAS with Voice Activity Detection (VAD) baseline, as well as with the non- probabilistic VAD method (Potapezyk et al., 2019) using a Wav2 pause predictor (Baevski et al.. 2020), which performs ASR inference and then splits based on detected word boundaries (see Ap- pendix A. To complete our proposal, we have also explored text-only segmentation methods in which a Transformer-basee model is trained to de- tect boundaries between natural language segments. This way, each word can belong to two different classes, boundary or not boundary.
Model
Dev
Test
Acc.
Macro-F11
Acc.
Macro-Fl
E2E BIO-5
0.71
0.45
0.72
0.47
E2E BIO-A
0.72
0.48
0.75
0.49
T-Seg+A-CIf
0.59
0.41
0,58
0.41
T-
0.64
0.49
0.69
0.49
A-Seg+A-CIf
0.60
0.42
0.58
0.43
A-Seg+T-CIf
0.67
0.51
0.70
0.51
and test sets. The end-to-end model outputs the BIO tags directly, either from fixed length input (of which was also the best performing value), de- noted as E2F BIO-5. Alternatively, the E2E BIO-A was trained considering the natural language seg- ments produced by the SHAS-mult model instead of relying on a specific maximum length defined without any linguistic criteria. This way, it was our objective to improve the training of our end-to-end model through the use of linguistically informed ausio-based natural language segments. can be observed how this second approach leverages the audio information to improve test macro-F1 from 0.47 to 0.49.
The second stage of our cascaded method is an argument detection classifier, that decides, for each segment, if it includes argumentative content and should be kept, or be discarded otherwise. In the case that our classifier detects argumentative con- tent within segment, its first word is assigned the B label (i.e., Begin) and the rest of its words are assigned the I label (i.e., Inside). Differently, if the classifier does not detect argumentative content within a segment, all the words belonging to this segment are assigned the O label (i.e., Outside).
For the cascade model, we test both audio and text segmenters and classifiers. Similarly to the E2E case, the use of audio segmentation consis- tently improves the results. For the text classifier, moving from text segmentation (T-Seg + T-CIf) to audio segmentation (A-Seg + T-CIf) increases test macro-F1 I from 0.49 to 0.51. Likewise, when us- ing an audio classifier (A-CIf), audio segmentation improves the test results from 0.41 to 0.43 macro- Fl. However, the relatively mediocre performance of the audio classification models with respect its text counterparts stands out. We believe this could be caused due to the fact that speech classifi- cation is a harder task than text classification in our setup, because the audio classifier deals with the raw audio, whereas the text classifier uses the refer- transcriptions as input. Additionally, the pre- trained audio models might not be capable enough for some transfer learning tasks, as they have been trained with a significantly lower number of to- kens, which surely hampers language modelling capabilities.
The code used to implement all the experiments reported in this paper can be publicly accessed from GitHub Furthermore, the weights of all the text- based models resulting from our experiments can be publicly downloaded from the Huggingface* repository.
5 Results
Once the hyperparameters of each individual model have been optimised considering the experimental setup defined in the Appendix A.2, the best results for each system combination reported on Ta- ble 2. The results are consistent across the dev
hhttps///githii ctt. jairsn// WivesDebate-Spere 'E2E Model, Text Segmenter, and Text Classifier.

---

6 Conclusions
been shown here.
We have presented the VivesDebate-Spett cor- pus, which is currently the largest speech-based argument corpus. Furthermore, the experiments have shown how having access to audio informa- tion can be souree of significant improvement. Specifically, using audio segmentation instead of text-based segmentation consistently improves per- formance, both for the text and audio classifiers used in the cascade approach, as well as in the end- to-end scenario, where audio segmentation is used as decoding constraint for the model.
Acknovledgements
This work is partially supported by the Span- ish Government project PID2020-1 13416RB- I00, by the Spanish Ministry of Economy and Competitiveness (MINECO) under reference no. TIN2015-68326-R (MORE), the FPU scholarship FPU1 8/0413, and by the 'AI for Citizen Intelli- gence Coaching against Disinformation (TITAN)' project, funded by the EU Horizon 2020 research and innovation programme under grant agreement 101070658, and by UK Research and innovation under the UK governments Horizon funding guar- antee grant numbers 10040483 and 10055990.
Limitations
The goal of the experiments presented in this work is simply to highlight some of the advantages that can be gained by integrating audio information into the argumentation pipeline, but this raises questions about how should evaluation be carried out. In this paper we have simulated an ASR oracle system in order to be able to compare the extracted arguments against the reference. We have reported accuracy and Fl results by assuming that this is standard classification task, and therefore incorrect labels all incur the same cost. However, there is no definitive proof that this assumption is correct. Given that argumentative discourse is a highly complex topic that relies on complex interactions between claims, more realistic evaluation framework might be needed. Something that warrants further study, for example, is whether precision and recall are equally important, or if it better for example to have higher recall at the expense of precision. In particular, the comprehension of an argument might not be hampered by additional or redundant ADUs, but might be significantly harder if a critical piece of information is missing.
References
Khalid Al Khatib, Tirthankar Ghosal, Yufang Hou, Anita Waard, and Dayne Freitag. 2021. Argument mining for scholarly document processing: Taking stock and looking ahead. In Proceedings of the Sec- ond Workshop on Scholarly Document Processing. pages 56-65.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vee 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems, NeurIPS.
Jianzhu Bao, Jingyi Sun, Qinglin Zhu, and Ruifeng Xu. 2022. Have my arguments been replied to? argument pair extraction as machine reading comprehension. In ACI. 2022, pages 29-35.
Chung-Chi Chen, Hen-Hsen Huang, Yu-Lich Huang, Hi- roya Takamura, and Hsin-Hsi Chen. 2022. Overview of the ntcir- finnum-3 task: investor's and man- ager's fine-grained claim detection. In Proceedings of the 16th NTCIR Conference on Evaluation of In- formation Access Technologies, pages 87-91.
Likewise, in order move towards more re- alistic setting, using real ASR system instead of an oracle ASR system should be preferred. This could only be achieved by devising a new evalu- ation framework that can work even the ASR system introduces errors, but it would provide a more accurate comparison between systems using audio or text features. For the comparison show in this work, the audio systems work with the raw audio, whereas the text systems use the oracle tran- scription. In realistic setting, the errors introduced by the ASR system would make it harder for the text-based models, and therefore the gap between the two could be significantly bigger that what has
Miguel A. del Agua, Adrià Giménez, Nicolás Serrano, Jesús Madrés-Ferrees Jorge Civera, Alberto Sanchis, and Alfons Juan. 2014. The translectures upv toolkit. In Advances in Speech and Language Technologies for Iberian Languages, IberSPEECH, pages 269- 278.
Pierpaolo Goffredo, Shohreh Haddadan, Vorakit Vorak- itphan, Elena Cabrio, and Serena Villata. 2022. Fal- lacious argument classification in political debates. In Proceedings of the Thirty-First IJCAI, pages 4143- 4149.
Shohreh Haddadan, Elena Cabrio, and Serena Villata. 2019. Yes, can! mining arguments in 50 years of us presidential campaign debates. In Proceedings of ACI., pages 4684-4690.

---

John Lawrence and Chris Reed. 2020. Argument min- ing: A survey. Computational Linguistics, 45(4):765- 818.
methods in natural language processing (EMNLP). pages 46-56.
loannis Tsiamas, Gerard Gállego, José A. R. Fonol- losa, and Marta R. Costa-jussà. 2022. SHAS: Ap- proaching optimal Segmentation for End-to-End Speech Translation. In Proceedings of Interspeech 2022, pages 106-110.
Marco Lippi and Paolo Torroni. 2016. Argument min- ing from speech: Detecting claims political de- bates. In Proceedings of the Thirtieth AAAI Confer- ence on Artificial Intelligence, pages 2979-2985.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Jacky Visser, Barbara Konat, Rory Duthie, Marcin Kos- zowy, Katarzyna Budzynska, and Chris Reed. 2020. Argumentation in the 2016 us presidential elections: annotated corpora of television debates and social me- día reaction. Language Resources and Evaluation,
Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1990. 1692.
54(1):123-154.
Eleonora Mancini, Federico Ruggeri, Andrea Galassi, and Paolo Torroni. 2022. Multimodal argument min- ing: A case study in political debates. In Proceedings of the 9th Workshop on Argument Mining, pages 158- 170.
A
Appendix
A.1 Audio Segmentation Algorithms
Table shows the best performance on the dev set of the different audio segmentation methods tested. Results are reported without using an argument classifier, which is equivalent to a majority class classifier baseline, as well as an oracle classifier which assigns the most frequent class (based on the reference labels) to segment. This allows us to analyse the upper-bound of performance that could be achieve with a perfect classifier.
Rafael Mestre, Razvan Milicin, Stuart Middleton, Matt Ryan, Jiatong Zhu, and Timothy J Norman. 2021. M-arg: Multimodal argument mining dataset for po- litical debates with audio and transcripts. In Proceed- ings of the 8th Workshop on Argument Mining, pages 78-88.
Andreas Peldszus and Manfred Stede. 2013. From gument diagrams to argumentation mining in texts: A survey. International Journul of Cognitive Infor- matics and Natural Intelligence (IJCINT), 7(1):1311.
Table 3: Audio segmentation methods performance on the dev set, as measured by accuracy (Acc.) and Macro- F1.
Tomasz Potapezyk, Pawel Przybysz. Marcin Cho- chowski, and Artur Szumaczuk. 2019. Samsung's system for the IWSLT 2019 end-to-end speech trans- lation task. In Proceedings of the 16th International Conference on Spoken Language Translation, IWSLT.
Method
Classifier
Majority
Oracle
Prakash Poudyal, Jaromir Savelka, Aagie leven, Marie Francine Moens, Teresa Gonçalves, and Paulo Quaresma. 2020. Echr: legal corpus for argument mining. In Proceedings of the 7th Workshop on Ar- gument Mining, pages 67-75.
Acc.
Macro-F1
Acc.
Macro-F1
Baseline (VAD)
0.53
0.35
0.70
0.53
DAC xlsr-53
0.61
0.34
0.81
0,65
DAC xls_
0.61
0.35
0.81
0.64
Ramon Ruiz-Dolz. 2022. cascade model for argu- ment mining in japanese political discussions: the qa lab-pollinfo-3 case study. In Proceedings of the 16th NTCIR Conference Evaluation of Information Ac- cess Technologies, pages 175-180.
SHAS-es
0.64
0.37
0.83
0.66
SHAS-ca
0.64
0.36
0,82
0.64
SHAS-multi
0.65
0.37
0.83
0.66
The results highlight the strength of the SHAS method, with the SHAS-es and SHAS-multi mod- els which are working on a zero-shot scenario, out- performs the Catalan W2V models. The SHAS-ca model had insufficient training data to achieve par- ity with the zero-shot models trained on larger au- dio collections. As result of this, the SHAS-multi model was selected for the rest of the experiments. One key factor to study is the relationship be- tween the maximum segment length (in seconds) produced by the SHAS segmenter, and the perfor- mance of the downstream classifier. Longer seg- ments provide more context that can be helpful to
Ramon Ruiz-Dolz, Jose Alemany. Stella Heras Bar- berá, and Ana Gareia-Fornes. 2021a. Transformer- based models for automatic identification of argu- ment relations: cross-domain evaluation. IEEE Intelligent Systems, 36(6):62-70.
Ramon Ruiz-Dolz, Montserrat Nofre, Mariona Taulé, Stella Heras, and Ana Marcia-Fomess 2021b. Vives- debate: A new annotated multilingual corpus of argu- mentation in a dehate tournament, Applied Sciences, 11(15):7160.
Christian Stab and Iryna Gurevych. 2014. Identifying ar- gumentative discourse structures in persuasive essays. In Proceedings of the 2014 conference on empirical

---

the classification task, but a longer segment might contain significant portion of both argumentative and non-argumentative content. Figure 2 shows the performance the text classifier as a function of segment size, measured on the dev set. 5 seconds was selected as the maximum sentence length, as shorter segments did not improve results.
between segmentation to classification. For audio-only segmentation (A-Seg), we performed a comparison between VAD, DAC, and SHAS al- gorithms. We evaluated the performance of these algorithms on the dev partition, and we selected the SHAS algorithm to be integrated into the cas- caded architecture for the final evaluation of our proposal. For the hybrid DAC segmentation, two Catalan W2V ASR models tested, xlsr-53 and xis-r- In the experiments, we used the original SHAS Spanish and multilingual checkpoints. Addi- tionally, we trained a Catalan SHAS model with the Sea-Deebtt--peede train audios, as there exists no other public dataset that contains the unsegmented audios needed for training SHAS model. Regard- ing our audio-only classifier (A-CIf), Wav2 models have been finetuned on the sequence classi- fication task (i.e., hatpieent/noo-argumeeee
0.5
0.19
0.48
0.47
0.46
0.45
0.11
0.43
0.42
0.41
0.4
10
15
20
Figure 2: Dev set F1 score as a function of maximum segment length (s), SHAS-mult segmenter followed by text classifier.
A.2 Experimental Setup
Regarding the implementation of the text-based sub-tasks (see Figure 1, green modules) of our cas- caded method for argument mining, we have used RoBERTa (Liu et al., 2019) architecture pre-trained in Catalan language data*. For the segmentation (T- Seg). we experimented with a RoBERTa model for token classification, and we used the segments pro- duced by this model to measure the impact of the audio features compared to text in the segmentation part of our cascaded method. For the classification (T-CIf), we finetuned a RoBERTa-baa model for sequence classification with a two-class classifica- tion sofimax function able to discriminate between the argument and the non-argument classes. As for the training parameters used in our experiments with the RoBERTa models, we ran 50 epochs, con- sidering learning rate of le-5, and batch size of 128 samples. The best model among the 50 epochs selected based on the performance in the dev set. All the experiments have been carried out us- ing an Intel Core 17-9700k CPU with an NVIDIA RTX 3090 GPU and 32GB of RAM.
The implementation of the audio-basse sub- tasks (see Figure 1, blue modules) is quite differ-
? ssofcatalllvaa22ee--lag----
"projecte- hwwpiaaiaaaaaaaaaaaaaaaaaa basc-ca-v2 was used for all RRB based models reported in this work
Mfachhol/wwa22vcc-eeee---0000 and facehnok/wav2vec22. xls-r-1b


Reviews:
[{"review_id": "MHxgT9MA4d", "reviewer": "Reviewer_N8m1", "summary": "", "questions": "1) How do you deal with possible noise in audio, for example, mispronunciation or repeated words?\n\n2) How does VivesDebate-Speech compare to other publicly available resources for audio-based argument mining, and what are the unique contributions of this corpus besides its size?\n\n3) Why does the incorporation of audio features hurt the accuracy compared with E2E BIO (Table 2)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer N8m1,\n\nFirst of all we would like to thank you for your insightful feedback and constructive comments. In our response, we would like to address the mentioned reason to reject and the three questions with hopes of improving your impression in our work.\n\nWe would like to clarify your concern, which have led you to give a unique reason for rejecting our work. This short paper has two main contributions. The first one is the creation of VivesDebate-Speech, the largest available resource for argument mining including audio information. The second one is a set of first-of-its kind experiments in which we are able to observe a small, but consistent improvement in the performance of argument mining systems when considering acoustic features in addition to text. Therefore, it was not our objective to provide a deep analysis of different algorithms and configurations, but to generate a new corpus and provide a set of solid baselines that can be used as reference in future work done in the direction of analysing or proposing new algorithms for the task of argument mining from speech.\n\nWe would also like to give an answer to the three questions included in the review.\n\n1. The speakers know the topic of the debate far in advance, and have received training in public speaking, so most of the speeches\nare well prepared and without any \"noise\" (in the sense of hesitations, repetitions, etc). In the very few cases where there might be issues, the annotators were instructed to remove any noise from the argumentative discourse units.\n\n2. In addition to its size, the VivesDebate-Speech corpus expands the previous available speech-based argument mining corpora in means of its argumentative richness. Given the nature of the original VivesDebate corpus containing argumentative annotations of the complete lines of reasoning present in debates of an average length of 40 minutes, this argumentative richness is also reflected in the VivesDebate-Speech corpus. Making it one of the most argumentatively dense corpora publicly available.\n\n3. The incorporation of audio features does not hurt the accuracy at all, in fact what we can observe in Table 2 is that combining audio features with text feature we can consistently improve the performance of our systems in both, the E2E and the cascaded approaches. It would not be fair, however, to directly compare the E2E with the cascaded approaches since both tasks are significantly different in nature (i.e., token labelling vs. sequence labelling) and in complexity (i.e., number of classes). See Sections 3 and 4.\n\nWe hope that with this response we will be able to improve your perception on the Soundness and Excitement of our work. \n\nThank you very much again for your time and dedication,\n\nThe authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The creation of a comprehensive and versatile corpus of spoken argumentation, which provides a valuable resource for researchers in the field of audio-based argument mining, as well as the potential for new insights and advancements in this field.\n\n2) This paper highlights the advantages of integrating audio information into the argumentation pipeline, and provides a baseline for future research in this area.", "reasons_to_reject": "1) The lack of comprehensive results from experiments performed using VivesDebate-Speech, such as using models of different structures and of various sizes.", "questions_to_authors": "1) How do you deal with possible noise in audio, for example, mispronunciation or repeated words?\n\n2) How does VivesDebate-Speech compare to other publicly available resources for audio-based argument mining, and what are the unique contributions of this corpus besides its size?\n\n3) Why does the incorporation of audio features hurt the accuracy compared with E2E BIO (Table 2)?", "paper_topic_and_main_contributions": "This paper proposes VivesDebate-Speech, which augments the VivesDebate corpus with spoken argumentation in an attempt to leverage audio features for argument mining tasks. VivesDebate-Speech represents a significant leap forward for the research community in terms of size and versatility. Compared to the few previously available audio-based argumentative corpora, VivesDebate-Speech provides more than 12 hours of spoken argumentation, which doubles the size of the existing largest audio-based corpus. The paper also highlights the advantages of integrating audio information into the argumentation pipeline, and provides a baseline for future research in this area\n\nThe authors point out that human argumentation presents a linguistically heterogeneous nature that requires careful investigation, which makes it reasonable to incorporate audio features into argumentative corpora. The creation of this corpus represents an important contribution to the intersection of speech processing and argument mining communities.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JbAUrnqFYZ", "reviewer": "Reviewer_hNqj", "summary": "", "questions": "I enjoyed reading the paper very much. I have no question for the authors. I apologize to the authors if this is not valuable feedback.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer hNqj,\n\nWe would like to thank you very much for your time and effort reviewing our work. We are also very pleased to read your kind words about our work.\n\nThank you very much,\n\nThe authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting and useful corpus. It is well-written and overall it looks sound in technical terms.", "reasons_to_reject": "No reason to reject.", "questions_to_authors": "I enjoyed reading the paper very much. I have no question for the authors. I apologize to the authors if this is not valuable feedback.", "paper_topic_and_main_contributions": "The papers presents a novel corpus of *spoken* argumentation from argument mining. It also presents results from an experiment using this corpus and showing that including speech features improves the predictive performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "vEZrAdu2Ds", "reviewer": "Reviewer_s6XU", "summary": "", "questions": "A) Why did you not discuss the accuracy / why do think that F1 is the relevant metric?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer s6XU,\n\nFirst of all we would like to thank you for your insightful feedback and constructive comments. In our response, we would like to address the mentioned reason to reject and the question with hopes of improving your impression in our work.\n\nRegarding the main concern and reason to reject about the lack of description of the classification models used in our paper, all the details are provided in the appendix. Appendix A.1 Audio Segmentation Algorithms and Appendix A.2 Experimental Setup contain all the details of the different algorithms and models that we used to implement the system depicted in Figure 1.\n\nSecond, with respect to your question A), we focused our discussion on the macro F1 instead of accuracy because it is a less optimistic metric. In highly unbalanced datasets such as ours, macro F1 is more informative since a majority class classifier would be severally punished while with accuracy not. However, with the camera ready version of our paper we can extend the analysis of both metrics in the additional page.\n\nFinally, regarding your concerns about the resource description, we did not include the information of the speakers since it can be extended from the source VivesDebate corpus. Each debate has an average of 6 speakers (3 speakers per team) and each speaker has a complete continuous speech of 10-15 minutes. In our corpus there are a total of 16 teams, therefore we can estimate that 45-55 speakers were involved in these debates. However, the speaker identity was not included in any of the two corpora. \n\nRegarding your concern about the incorrect alignment, we would like to mention that the starting point for this extension are the human transcriptions of the VivesDebate corpus, so there is a 100% match between the text to be aligned and the words being said. The automatic alignment is just a matter of finding the correct timestamp for each word. This specific alignment task is sufficiently mature and we have been doing it for many years, so our models can perfectly align the audio with the text. No manual post-editing is applied after the alignment as we estimate that alignment accurary is close to 100% in this case (A manual revision of Debate 1 did not show any alignment mistakes, and we are confident that this also applies to the rest of the debates).\n\nWe hope that with this response we will be able to improve your perception on the Soundness and Excitement of our work. \n\nThank you very much again for your time and dedication,\n\nThe authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Provides a valuable resource\n - has an interesting experiment set, given the space restrictions", "reasons_to_reject": "- the classification systems used in the experiments are not described, as far as I can see. There is only Fig 1 showing the data flow.", "questions_to_authors": "A) Why did you not discuss the accuracy / why do think that F1 is the relevant metric?", "paper_topic_and_main_contributions": "This submission introduces an audio alignment for an argumentation corpus. The authors use the aligned audio for experimenting with non-textual features for argument labeling.  They find that using audio features improves segmentation for text-based classification, yielding (slightly) better annotations than when using text-based segmentation.\n\nThe two contributions (resources & experiments) are both interesting, but each of them lacking certain aspects.\n\nFor the resources, I would have liked to see a more detailed description, for example:\n - how many speakers are there? how long are the continuous segments per speaker (if you have speaker change)?\n - is the speaker identity annotated in the data?\n - how did you deal with incorrect alignments from the automatic alignment process? For resource papers, it is always important to me to give information about the data gathering process, because this information cannot be recovered later on by other researchers (in contrast to e.g. new experiments).\nSome of this information can be found in the actual data, but it is in my opinion still good to directly report it.\n\nFor the experiments, there are two aspects:\n - the authors report accuracy and F1 score.  Their baseline model has the highest accuracy, but only the F1 score is discussed in the paper (where their own model has higher scores). It is not clear to me why a metric is reported but then completely ignored. As the accuracy paints a different picture than F1, it would be good to at least have a succinct explanation for this decision\n - it is not clear to me why the authors chose to report the accuracy of a pure text-based classifier and a pure audio-based classifier but not a classifier using both as input. I am not claiming that it needs to be in the paper but using all available features is probably what most people think about first and I would add a short explanation (could very well be \"we are not interested in it\", but better formulated)\n\nI wrote these topics here instead of in \"reasons to reject\" because these are aspects where I would like to see more but at the same recognize that this is a focused short paper. If anything, it seems to me that the page restrictions are a bit too small for this content; I am e.g. also missing a more in-depth description of the classifiers you used in your experiments. Other than that, I enjoyed reading this submission.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes VivesDebate-Speech, which augments the VivesDebate corpus with spoken argumentation in an attempt to leverage audio features for argument mining tasks. The size (about double the size of a comparable audio dataset) and versatility of the corpus have shown to be very interesting for the research community, it offers the chance for AM and speech processing communities to work together, and it highlights the benefit of integrating audio features in computation argumentation tasks.

The reviewers generally agreed that the paper is well written and clear. The most significant issue brought up in the reviews was the lack of detail in the description of both the resource and the experiments (which the reviewer also sees is an issue related to space restrictions). The concern was outlined in detail and the authors thoroughly addressed each point in the author response. I believe their response should be sufficient to dampen the concerns brought up.
Decision: Accept-Main