DREAM: Deployment of Recombination and Ensembles in Argument Mining
Authors: Florian Ruosch
Cristina Sarasua
Abraham Bernstein
Venue: EMNLP 2023 Main (2023)
Keywords: Argument Mining
Recombination
Ensemble Methods

Abstract: Current approaches to Argument Mining (AM) tend to take a holistic or black-box view of the overall pipeline. This paper, in contrast, aims to provide a solution to achieve increased performance based on current components instead of independent all-new solutions. To that end, it presents the Deployment of Recombination and Ensemble methods for Argument Miners (DREAM) framework that allows for the (automated) combination of AM components. Using ensemble methods, DREAM combines sets of AM systems to improve accuracy for the four tasks in the AM pipeline. Furthermore, it leverages recombination by using different argument miners elements throughout the pipeline. Experiments with five systems previously included in a benchmark show that the systems combined with DREAM can outperform the previous best single systems in terms of accuracy measured by an AM benchmark.

DREAM:
Deployment of Recombination and Ensembles in Argument Mining
Florian Ruosch and Cristina Sarasua and Abraham Bernstein University of Zurich, Switzerland lastname@ift @in uzh.ch ch
Abstract
Moreover, Lawrence and Reed (2019) also take the same line by advocating for unifying frame- work to enable the harmonization of all AM tasks, including the format of data and results. Such a unification would be necessary to combine many systems and facilitate the integration of additional ones. Likewise. not every task receives the same amount of attention, with approaches for identi- fying argumentative relations being sparse (Al- Khatib et al., 2021). Thus, they cover a smaller range of domains or not work well across them. By using recombination, we hypothesize to in- crease coverage and find yet untapped potential. To this end, we formulate the following research question:
Current approaches Argument Mining (AM) tend take holistic or black-box view of the overall pipeline. This paper, in contrast, aims provide solution achieve increased per- formance based on current components instead of independent all-new solutions. To that end, it presents the Deployment Recombination and Ensemble methods for Argument Miners (DREAM) framework that allows for the (auto- mated) combination of AM components. Using ensemble methods, DREAM combines sets of AM systems to improve accuracy for four tasks in the AM pipeline. Furthermore, lever- ages recombination by using different argu- ment miners clements throughout the pipeline. Experiments with five systems previously in- cluded a benchmark show that the systems combined with DREAM previous best single systems in terms of accu- racy measured by an AM benchmark.
RQ. How can we leverage (re-)oombinations of Ar- gument Mining systems improve accuracy?
outperform the
Thus, we build DREAM, system that allows for the Deployment of Recombination and Ensemble methods for Argument Miners. For this en- deavor, we base our approach and the evaluation on BAM (Ruosch et al., 2022), our benchmark for Argument Mining. We reuse the performance data of five AM systems when evaluated by BAM as well as its implementation for our purposes. Ac- cordingly, we restrict the systems for the initial combinations to these five argument miners and adhere the definition of the four tasks in the AM pipeline (Lippi and Torroni, 2016a): sentence clas- sification, boundary detection, component identifi- cation, and relation prediction. Using these tools. we try to outperform the current best accuracy for every task of the AM pipeline by combining sys- tems with the following ensemble methods: voting, stacking, and bagging. Finally, we split the AM sys- tems into "modules" according to the AM pipeline, allowing their recombination to increase accuracy.
1 Introduction
A well-known and open challenge in Argument Mining (AM) is that approaches do not generalize well across domains (Lippi and Torroni, 2016a). Thus, a single system will not be able to solve the task of extracting arguments from publications across multiple research fields. Therefore, we in- vestigate the use of ensemble methods (Opitz and Maclin, 1999) find combinations that are ex- pected to help alleviate the issue.
Furthermore, the overview of AM approaches presented by Lawrence and Reed (2019) shows that papers typically introduce novel techniques or use methods for AM that have demonstrated success in other applications. As new systems tend to take the holistic view of an end-to-end pipeline (Lawrence and Reed, 2019), has become evident that novel approaches rarely investigate improvements of in- termediate steps. By introducing a system with ensemble methods and combinations, we aim to improve smaller aspects of the pipeline.
We present two main contributions in this paper. First and foremost, we build the DREAM frame- work to combine AM systems using ensemble

---

methods and recombinations. Second, show the value of such combinations, as they outperform some of the state-of-the-art systems used in the AM benchmark.
respect to identified connections between proposi- tions and use fixed set of 36 pairs.
First, the presence of discourse indicators: words such as "because" and "however", indicat- ing support- and conftict-relations, respectively, be- tween adjacent statements. These words provide good signal (precision of 1.00), but the technique fails capture most relations (recall of 0.08) due to their low number of occurrences in texts. Fur- thermore, they can not used to find relations for non -djaaejacent propositions.
The remainder of this paper is structured as follows: Section 2 presents background and the related work, and Section 3 introduces our methodology. In the ensuing Section 4, describe our experiments and their results before we evaluate them in Section 5. Then, Section discusses limitations and future work. Finally, we draw conclusions in Section 7.
The second technique is based on changes in the topic for consecutive propositions, which is as- sumed to relate to the argumentative structure in the text. The similarity of adjacent propositions is calculated using the synsets of WordNet', re- sulting a number on a scale from to 1. A preset threshold then determines whether the topic remains the same, and, that being the case, it is deduced that the propositions are connected. This approach achieves precision of 0.70 and a recall of 0.54, respectively.
2 Background
this section, we lay the foundations by describ- ing Argument Mining and presenting specific re- lated work.
2.1 Argument Mining
The field of AM is wide-ranging and has differ- ent interpretations (Wells, 2014) of what its tasks consist of. We focus the information extraction approach (Budzynska and Villata, 2015; Lippi and Torroni, 2016a); the automated analysis argu- ments in natural language text. To this endavor, we consider the AM pipeline as described by Lippi and Torroni (2016a), depicted in Figure 1. The in- put text is processed in four stages: argumentative sentence detection, argument component bound- ary detection, argument component detection, and argument structure prediction.
The third method uses argumentation schemes (Walton et al., 2008): "common patterns of human reasoning." They avoid the need for having the components and the structure of arguments already annotated by instead focusing on features of the parts of the present scheme. With list of propositions from the text and a Naive Bayes classifier, they can determine the particular scheme and, therefore, detect information about the structure of the argumentation. This results a precision of 0.82 and a recall of 0.69.
In the first step, sentences are classified as argu- mentative if they contain parts of an argument and non-argumentative otherwise. Next, the bound- identified
Finally, the techniques are combined to exploit their respective fortes. The presence of discourse indicators is used to infer connections among propositions in the first step. Subsequently, com- ponents are related after having determined argu- mentation scheme instances. Lastly, previously unconnected units are integrated based on topic similarity. Combining the methods results in an improved performance with precision and recall, increasing to 0.91 and 0.77, respectively.
aries of the argument components by segmenting the argumentative sentences. Then, these argument components are classified accord- ing to the representation of the arguments defined beforehand. Finally, the structure (i.e., relations) of the previously identified components is predicted to form an argument graph. The annotated text (in any format) is the output of the AM pipeline.
In contrast to the approach described above, we aim to provide combinations on larger scale and a pipeline for a unifying framework that allows for integrating additional components. We aim to investigate if and how combinations (of parts of) different AM system can be used to improve over- all performance. Finally, our approach also differs
2.2 Specific Related Work
Combining approaches in AM has barely received any attention in previous literature. The only ex- ception is the work of Lawrence and Reed (2015), where the authors implement and combine three different AM techniques. They are evaluated with

---

Argument component detection
Argument component boundary detection
Argumentative sentence detection
Argument structure prediction
Raw text
Annotated text
Figure 1: The Argument Mining pipeline adapted from Lippi and Torroni (2016a).
in that we do not look to combine techniques or features but rather out-of-the-box argument min- ers. This facilitates the integration of additional systems.
H1.1 There exists an ensemble of systems for ev- ery task that is more accurate than any other ensemble of systems (excluding single sys- tems).
Since already restrict the space of systems and combinations that need to explore by limiting ourselves the systems in BAM, we can test all combinations of size n, where < n S 5, because we require combinations of at least two and can combine at most all five systems. Thus, it becomes matter of running all possible systems and com- bining them using ensemble methods We accept the hypothesis if we find one more ensembles of systems that exceed all others in terms of accuracy as measured by BAM for all the tasks, impor- tant to note that these ensembles might differ for individual tasks.
3 Methodology
In this section, we first lay out our evaluation hypotheses and then describe the concept of the DREAM (re- coombinaiinno framework.
3.1 Hypotheses
We base both the evaluation and the conception on the systems, data, and results of BAM, our benchmark for Argument Mining (Ruosch et al., 2022). We utilize the five systems included in BAM (i.e., AURC, TARGER, TRABAM, ArguminSci, and MARGOT) and evaluate the recombinations using the provided implementation of the bench- mark. That means the AM pipeline is split into four tasks (Lippi and Torroni, 20)6 sentence classi- fication, boundary detection, component identifi- cation, and relation prediction. These tasks are evaluated with their respective metrics from BAM (i.e., micro F1, the boundary similarity measure defined by Fournier (2013), and FI-score).
H1.2. For some tasks in the AM pipeline, the most accurate ensemble of systems will be more accurate than the most accurate single system for this task.
For the second sub- hipohhhyothsss, we can compare the previously discovered combinations with the most accurate single system and compare their numbers for all the tasks. That means doing pair-wise comparison four times, namely once for every task, and checking whether the combinations outperform the single systems. Again, we accept the hypothesis we can confirm this for at least some of the four tasks.
To evaluate the implemented recombination sys- tem, formulate the following hypotheses de- rived from the research question and describe our approach assess their acceptance or rejection.
HI. For some tasks in the AM pipeline, ensem- bles of systems exist for which accuracy will be higher than for the most accurate single system.
H2. For some tasks in the AM pipeline, the ac- curacy for subsequent tasks will be higher if intermediate data is used that has been pro- duced by the system with the highest accuracy for the preceding task instead of its own inter- mediate data.
This hypothesis encapsulates two different but en- tangled problems: finding the optimal set of sys- tems combine and testing whether they are more accurate than the current top system. Thus, we split it into two sub-hypothexes, which are the requisites for accepting Hypothesis H1.
Subsequently, we investigate how improve single tasks and how the intermediate results influ- ence the ensuing tasks of the pipeline. Thus, we

---

hypothesize that data of higher accuracy compared the ground truth will also result in an increase in system's 's accuracy as opposed to its own in- termediate results. Again, we try out all possible pairs to answer this hypothesis. Any two systems can be combined by using one's output as the other one's input, provided that the former's accuracy was higher than the latter's at the preceding task. Considering the five systems and four tasks in the benchmark, we have to try out a maximum of 60 pairwise combinations, every system acting as "in- put provider" and "input taker" but never at the same time. We compare the new highest accuracy for every task and system to the previous results and accept this hypothesis if the new numbers are higher than the old ones.
miners lacking the capabilities to solve one or more of these lasks. Furthermore, Lawrence and Reed (2019) point out that current systems tend to take a holistic view the end-to-end pipeline. This is further emphasized by the fact that black-box models, such as neural networks and, more specif- ically, transformers, become increasingly preva- lent. While they carry the advantage of improved performance, they prevent a look into their inner workings and modularization of their features. Thus, we have access solely to the final outputs of argument miners for our framework. However, as we showed in BAM (Ruosch et al., 2022), we can reconstruct the intermediate results necessary for evaluating the tasks mentioned above of the AM pipeline. We can these reconstructed interme- diate results for the recombination effort, with the added benefit of not needing to re-train or re-run any of the systems (i.e., we only perform post-hoc combinations).
H3.
For some tasks in the AM pipeline, the accu- racy will be higher if we use an ensemble of systems and intermediate results as input pro- duced by the most accurate system (ensemble) for the preceding task.
DREAM reads the output files from the argu- ment miners and calculates the combinations ac- cording to the specified parameters. There are sev- eral different options when combining this data: the list of employed systems, the method to calculate the combination, and the targeted task.
The final hypothesis brings all possible combina- tions together. We not only allow combining sys- tems for tasks but also to "mix-and-matchh for the intermediate results in the hope of improving the accuracy of the whole pipeline. We employ the best combinations from Hypothesis H1 and com- bine them with the insights from Hypothesis H2. We compare the newly obtained accuracies to the previous best per the benchmark and accept the hypothesis if we outperform the top single system for every task.
Figure visualizes the ways we combine sys- tems. Figure 2a corresponds to what is described in Hypothesis H1: using ensemble methods to com- bine multiple systems for a single task. This is what we call Vertical Iniegration. Meanwhile. Fig- ure 2b illustrates Hypothesis H2: using different systems throughout the AM pipeline (recombina- tion). This referred to Horizontal Integration. Tying these two together, we get the Combined In- tegration, where we allow sets of systems to be used for the intermediate results fed forward in the pipeline to either other combinations or single systems.
3.2 The DREAM Framework
The basic idea behind the approach to combining multiple AM systems is simple: Employ a multi- tude of systems such that they can combine their strengths and, at the same time, balance out their weaknesses. Our framework, DREAM, is intended for the (automated) recombination of multiple AM systems according to predefined parameters. Fol- lowing the aforementioned AM pipeline by Lippi and Torroni (2016a), we first identify argumenta- tive sentences, then we identify the boundaries of the components and classify them (usually as either claim premise). Finally, predict the relations between the argumentative components (such as supports or attacks).
3.2.1 Vertical Integration
Vertical Integration gets its name because we choose systems from the "column" of options as illustrated in Figure 2a. The number of systems used for the combination can vary from a minimum of two to all available systems. The list of used systems can be either specified or the recombina- tion framework can try (all) possible combinations (including power sets). This is how we approach eettmexprrimnnttiionn for recombination.
Not every argument miner adheres to this pipeline, which results in some of the argument
As for the method to calculate the combination of results, we follow the well-estallished ensemble

---

S B
CCC
R
S
B
C
R
System 1
System
System
System 2
8 (a) Using ensemble methods task.
System
System 3
8
System 4
System 4
System 5
System
(b) Recombining systems over the whole pipeline.
Figure The two types of combinations employed in the framework.
methods (Opitz and Maclin, 1999). As simplest method, we employ hard voting scheme (Little- stone and Warmuth, 1994), where systems can op- tionally be assigned weight. It is important to note that systems may all receive uniform weight of one or be assigned arbitrary scores (e.g., bench- mark results). Then, we calculate the score of the available answers for a given item based on the systems' output and weights. Next, we use the ensemble stacking method, which trains meta- classifier on the predictions the trained argument miners produced. We employ multinomial logistic regression (Greene, 2003) as the stacking model. Our third ensemble method is bootstrap aggregat- ing (bagging) (Breiman, 1996). Using bootstrapped sets for training. we expect to strengthen the en- semble of classifiers.
argumentative sentences since, by definition, only they may contain argumentative components. The same holds for the component identification: it will only identify components whose boundaries have been detected. Lastly, the relation prediction relies on the previously identified components to find the triples (subject and object are from the set of ar- gumentative components) that constitute its output. Thus, we can see that every subsequent step in the AM pipeline depends on its predecessor's output.
3.2.3 Combined Integration
Finally, we will also allow for the Vertically Inte- grated ensemble learners to be used as the inter- mediate result creators and, thus, bring it together with Horizontal Integration to the Combined Inte- gration. Since we hypothesize that both individual Integrations increase accuracy, we hypothesized their combination exhibits mance. Thus, we make an effort find sequences of combined AM systems that further improve the accuracy of the tasks in the pipeline.
3.2.2 Horizontal Integration
higher perfor-
Next, we allow combining systems across the bor- ders of individual tasks (i.e., columns) by using intermediate results and feeding them to other sys- tems. This results in what we call Horizontal Inte- gration, since we allow the combination of differ- ent "rows", as depicted in Figure 2b. The output for all the tasks in the AM pipeline depends on the input fed into the corresponding module. Although, these modules may not be explicitly constructed such and may have be inferred due to the holistic view of AM systems (Lawrence and Reed, 2019). Still, we can generally describe the data processing in the AM pipeline. In the first step, the raw text supplied to the pipeline is split into sentences, which in turn are classified as either ar- gumentative or con-argumentattie, depending on the presence of argumentative components in them. Thus, the output of the sentence detection depends on its input because it will process (and output) no more and no less than the text it has been sup- plied with. Subsequently, the boundary detection will find the delineations of components in only the
Experiments
In this section, we discuss conducted experiments. First, we describe the setup used. Then, we ex- plain the implementation of the experiments and the subsequent evaluation.
4.1 Setup
We rely on the systems and data used by BAM (Ru- osch et al., 2022), the results of which are shown in Table 1. Thus, we consider five different AM systems that have been benchmarked using the Sci- Arg data set (Lauscher al., 2018b). It represents the only available collection of full argument an- notated scientific papers in English and builds on the Dr. Inventor data set (Fisas al., 2016). The corpus consists of publications from the field computer graphics and contains a total of 10,780

---

System
S
B
C
R
AURC
(Trautmann et al., 2020)
0.792
0.470
TARGER
(Chernodub et al., 2019)
0.653
0.483
0.656
TRABAM
(Mayer al., 2020)
0.832
0.506
0.662
0.021
ArguminSci
(Lauscher et al., 201 8a)
0.600
0.115
0.091
MARGOT
(Lippi and Torroni, 2016b)
0.454
0.097
0.133
Table 1: Results of BAM, our benchmark for Argument Mining (Ruosch et al., 2022).
sentences that have been annotated with argumen- tative components (background and own claims as well as data) and relations (contradicts, supports, semantically same, and part of)
the benchmark. It consists of range of different tools to analyze rhetorical aspects, but we only use the argument component identification functional- ity. This module uses a BiLSTM to tag tokens as one of three argumentative component types akin to the annotations in the corpus: background claim, own claim, data. Finally, MARGOT (Lippi and Torroni, 201 detects claims and evidences by analyzing the sentence structures and uses subset tree kernel (Collins and Duffy, 2002) to compare their constituency parse trees.
BAM uses individual evaluation measures for each of the pipeline tasks. For the argumenta- tive sentence classification, it employs the micro- F1 (van Rijshergen, 1979) score to avoid the skew- ing effect of a possible label imbalance. For the boundary detection task, BAM uses the boundary similarity measure as proposed by Fournier (2013), which compares the identified boundaries for two segmentations for the same text. The argumenta- tive component identification is evaluated by us- ing the Fl as implemented for the task of Named Entity Recognition (Segura-Bedmar Bedmar et al., 2013). Finally, relation prediction is treated as the classi- fication of triples (subject, predicate, object) into retrieved or missed and thus, BAM employs the F1- score. Therefore, we obtain four individual scores between 0 and 1, one per task in the AM pipeline, where bigger signifies better.
Data and code involved in the execution and subsequent evaluation are available in the project's repository.
4.2 Vertical Integration
The best results for each task using the Vertical In- tegration are presented in Table 2.3 For context, we also report the runner-up and the worst result, as in- dicated in the Result column, and provide the mean, median, and variance for each task. We round all results three decimal places for readability, ex- cept where necessary to indicate differences. Each task of the AM pipeline is represented by row, in which the accuracy (as measured by BAM), the used ensemble method, and the systems involved are indicated (the order matters therein as the first system serves as the primary to which all other annotations are aligned to). Notably, the relation prediction score R is absent since only one system performed in BAM, and thus, there is no oppor- tunity to apply an ensemble method. Also, because no system explicitly disentangles the AM pipeline into individual tasks, we perform the combination on the final output and not on task-specific anno- tations, akin to the way it is handled in BAM. We tried every possible combination of all system lists and ensemble methods to obtain the results and list the best, second best, and worst here.
We use the same five systems that have already been evaluated in the initial showcase of BAM. The first three were trained on the benchmark data set, while the last two were already pre-trained by the authors of the systems. AURC (Trautmann et al., 2020) treats AM as sequence tagging prob- lem and employs the BiLSTM model of Reimers al. (2019) to identify argumentative spans in texts. TARGER (Chernodub al., 2019) also uses a BiLSTM in conjunction with a CNN-CRF and pre-computed word embeddings to label tokens from free text input as belonging to either claims or premises. TRABAM (Mayer et al., 2020) relies on pre-trained transformers such as SciBERT (Beltagy et al., 2019) in combination with neural networks. TRABAM is the sole system in the benchmark that solves all the pipeline tasks, tagging argumenta- tive components and predicting relations between them. ArguminSci (Lauscher et al., 201 2018a) was trained on the data set that is also incorporated in
ttttps: /tttps://gitaal ifi uzh. ch/DDIS Twtoos-Public/DEEEE.. 'The full results are omitted for brevity and are available in the online repository.

---

Task
Result
Score
Method
Systems
mean: 0.739
Best
0.8419
Stacking
TARGER, AURC, MARGOT, TRABAM
S
median: 0.793
Second
0.8416
Stacking
TARGER, AURC, TRABAM
variance: 0.011
Worst
0,513
Voting
MARGOT, TARGER
mean: 0.379
Best
0,4972
Bagging
TARGER, ArguminSci. MARGOT, TRABAM
B
median: 0.457
Second
0.4971
Bagging
ArguminSci, MARGOT, TARGER, TRABAM
variance: 0.020
Worst
0.022
Voting
MARGOT, ArguminSci
mean: 0.498
Best
0.673
Voting
TRABAM, TARGER
C
median: 0.615
Second 0.671
Voting
TRABAM, Arguminšci, TARGER
variance: 0.036
Worst
0.052
Voting
MARGOT, ArguminSci
Table 2: Best, runner-up, and worst results task for Vertical Integration (mean, median, and variance refer to all results per task),²
From To
C
R
omitted. The sentence classification is not consid- ered for the Vertical Integration its input is the initial text, which is not considered intermediate result since is the same for every system.
TRABAM AURC
0.475
TRABAM TARGER
0.494
0.630
TRABAM ArguminSci
0.281
0.345
TRABAM MARGOT
0.171
0.162
4.4 Combined Integration
Table Results per system for Horizontal Integration.
In Table 4, show the results of the Combined Integration. We list the results achieved with the previously identified most accurate single system or ensemble (from the Vertical Integration) and their score for each AM pipeline task. For cach row in the table, the output has been combined with the output of the preceding row, according to the Horizontal Integration. This results in the Combined Integration.
Stacking the systems TARGER, AURC, MAR- GOT, and TRABAM using logistic regression is the most accurate ensemble for sentence classifi- cation with S = 0.8419. Bagging with TARGER, ArguminSei, MARGOT, and TRABAM achieves score of B 0.4972 for boundary detection, which is the highest among the ensembles. Com- bining the two systems TRABAM and TARGER using the hard voting scheme results in C' - 0.673 as the best score for component identification.
The ensemble of TAARER, AURC, MARGOT, and TRABAM stacked using logistic regression the most accurate for sentence classification with S 0.842. The single system TRABAM achieves the highest boundary detection score with B = 0.483. Combining TRABAM and TARGER into an ensemble using voting results in C = 0.673 as the best score for component identification. Fi- nally, TRABAM scores R 0.019 for the relation prediction. Interestingly, ensembles are only better than single systems in two out of the three AM pipeline tasks (relation prediction does not have an alternative to TRABAM).
The main insight gained from these results is that no ensemble method outperforms the others. Rather, each of the three techniques achieves the highest score for task.
4.3 Horizontal Integration
Table shows the complete results for the Hor- izontal Integration. We used the most accurate system from BAM, TRABAM, as listed the "From"-column to indicate where the intermedi- ate results originated from. These were combined with the output of the individual systems (in the "To"-colume in the respective rows by using them the template for the subsequent annotations.
5 Hypotheses Evaluation
In this section, we evaluate the hypotheses individ- ually. The results from BAM in Table serve the baseline, more specifically, the best-performing system nicknamed TRABAM in with the bold- faced numbers. It achieved the following scores for the AM tasks, where each them is on a scale from zero to one, and higher means better: sen-
TARGER combined with TRABAM scores the highest for both the boundary detection B 0.494 and component identification C 0.630. Again, due to the lack of a system to combine TRABAM with, the results for the relation prediction R are

---

Task
Score
Method
(ystem(s)
S
0.842
Stacking
TARGER, AURC, MARGOT, TRABAM
B
0,483
Single
TRABAM
C
0.673
Voting
TRABAM, TARGER
0.019
Single
TRABAM
Table 4: Results per task for Combined Integration.
tence classification S = 0.832, boundary detection
TARGER were combined using the voting method (C = 0.673), they beat the previous best achieved by TRABAM (C = 0.662), with the difference being statistically significant. This is in contrast to the boundary detection, where the best ensemble re- sult does not reach the most accurate single system: bagging TARGER, ArguminSci, MARGOT, and TRABAM scored B 0.497, while TRABAM 0.506. As
= 0.506, component identification C = 0.662, and relation prediction R = 0.021. Unlike the result reported in BAM, we use TRABAM's inter- mediate results as input for the last step, decreasing the accuracy (from R 0.318 when using the ground truth components). We compare the newly obtained scores these numbers to evaluate the hypotheses. The statistical significance testing re- sults and the correction for multiple comparisons can be found in Appendix A.
held the most accurate result explained in the previous hypothesis, the relation prediction is omitted.
Since found one of three ensembles to out- perform single systems with a statistical signifi- cance, this leads us to accept Sub-hypothesis H1.2. Moreover, this indicates a correlation between the systems' errors since they do not seem to balance out their weaknesses in all cases. An exhaustive error analysis would be necessary to reveal more detailed insights.
5.1 Hypothesis H1
This evaluation is based the outcome of the Sub- hypotheses H1.1 and H1.2. Thus, we assess these two before giving the verdict on H1.
H1.1 Before collating previous and new results, we look at the isolated findings from applying the ensemble methods. We hypothesized that there would be set of systems that the most accu- rate compared to any other combination. We can confirm this hypothesis by looking at the results produced in the experiments by using the ensemble methods. Due to the lack of a second system to combine TARGER with, ensembles can be built improve the relation prediction score R; thus, it is omitted.
H1 We based the acceptance of Hypothe- sis HI on accepting both its corresponding sub- hypotheses, which we did as explained above. This means that we also accept Hypothesis H1.
5.2 Hypothesis H2
Table shows the results of using the annotations produced by TRABAM (i.e., the most accurate sys- tem in the benchmark) as the input to subsequent steps for the other systems. The boldfaced numbers indicate improvements over the initial results with the system's own data. We can see that, except when combining TRABAM with TARGER for the component classification, we consistently outper- form the benchmark results, and the differences are all statistically significant. Akin to the previous hy- potheses, R cannot be improved as we do not have another system to feed TRABAM's intermediate results into, or vice versa. Therefore, we also omit the relation prediction from evaluating this hypoth- esis. Since we could show that using more accurate intermediate results can improve the subsequent step of the AM pipeline, we accept Hypothesis H2.
Table shows the results for each task. From we can see that the highest scores are unique num- bers. This leads us to accept Sub-hypothesis H1.1.
H1.2 This hypothesis compares the results from the benchmark and the Vertical Integration by opposing the best results from Table 1 and Ta- ble 2. For the sentence classification, the ensem- ble of TARGER, AURC, MARGOT, and TRA- BAM combined by stacking them (with logistic re- gression) slightly outperforms the previously most accurate single system TRABAM: S 0.842 and S 0.832, respectively. Statistical testing. however, reveals that the difference is not signif- icant (cf. Appendix A). For the component iden- tification where the two systems TRABAM and

---

5.3 Hypothesis I13
AM systems' common weaknesses. Also, the new analysis can incorporate the spatial and temporal costs of the recombinations, which was omitted in this paper. In the future, should the number of ar- gument mining systems considerably increase, the framework could be extended to include predic- tor to choose the sets and sequences of argument miners for given document that lead to opti- mal accuracy improvement. This would involve developing a cost function.
This hypothesis merges the Vertical and the Hori- zontal Integration into the Combined Integration to improve the accuracy for all tasks in the pipeline by also allowing intermediate data produced by en- semble methods. The results are shown in Table with the boldfaced numbers indicating the tasks for which a new highest accuracy was achieved: S = 0.842 and C' - 0.673 outperform the previ- ous best single systems from BAM, but only the latter being statistically significantly different. This is in contrast to B = 0.183 and R = 0.019, where the former did not perform better, and the latter even lowered the score. Still, we have evidence that Combined Integration can be used improve at least some tasks in the AM pipeline. Thus, we accept Hypothesis H3.
7 Conclusions
This paper presented DREAM, framework for the Deployment of Recombination and Ensemble methods for Argument Miners. Our work focuses on improving accuracy in Argument Mining (AM) and addresses the need for incremental improve- ments as opposed to current approaches, which tend to provide all-new solutions (Lawrence and Reed, 2019). With the DREAM framework, we implemented a flexible and automated approach for (re crombiing AM systems. It offers the Vertical Integration (using ensemble methods for a single task), the Horizontal Integration (using different systems throughout the pipeline), and, finally, the Combined Integration (allowing sets of systems for the intermediate data).
6 Limitations and Future Work
The major limitation of this work is that we imple- ment post-hoc combinations. The reasons for not re-training the systems are two-fold. First, out of practicality to facilitate the addition of new AM systems and existing ensembles. Second, to set the scope of this research as opposed to works that look to explicitly fuse models such as neural networks by entangling the final classification layer such as described in Ribeiro et al. (2020). The latter opens up the future work of applying these techniques to the current five AM systems and mixing their latent representations, opposed only their outputs. Another limitation is that all the included sys- tems take holistic view of the AM pipeline, and none is explicitly split into the four modules we infer for the ensemble methods. Given the success of treejcommintioons of components in other do- mains, this paper can, hence, be seen as a call to action to systematically explore the effectiveness of functional components of the AM pipeline and share these for re-use by others. Indeed, more broadly, the limited availability of AM systems and benchmark datasets hampered our ability to system- atically compare a larger design space of system (component) combinations and limits the general- izability of our findings to other dotains/datasets.
Our findings confirmed the hypotheses formu- lated in this work. We showed that ensemble meth- ods (Opitz and Maclin, 1999) could be used to improve accuracy for specific tasks in the AM pipeline. Furthermore, we demonstrated that re- combination by using intermediate data from the most accurate system could lead to higher accu- racy in the subsequent task. Finally, highlighted the potential of deploying ensemble methods and recombination for AM. We hope this work will contribute to the further improvement of otatc-of- the-art and better generalizing AM systems across domains, a prevalent and well aell-acknowledged prob- lem (Lippi and Torroni, 201
A Acknowledgmentss
This research was partially funded by the Swiss National Science Foundation (SNSF) under Project "CrowdAlytics" (Grant Number 184994). The au- thors would also like to thank Luca Rossetto for his inputs and the anonymous reviewers for their constructive feedback.
The plans for future efforts in this direction in- clude two main points. As the next step, aim to conduct an error analysis and explore the influences of the systems on the results. This will help iden- tify the strengths and weaknesses of the individual systems and may provide insights about current

---

References
Anne Lauscher, Goran Glavaš, and Kai Eckert. 2018a. ArguminSci: A Tool for Analyzing Argumentation and Rhetorical Aspects in Scientific Writing. Pro- ceedings of the Sth Workshop on Argument Mining, pages 22-28.
Khalid Al-Khatib, Tirthankar Ghosal, Yufang Hou, Anita de Waard, and Dayne Freitag. 2021. Argu- ment Mining for Scholarly Document Processing: Taking Stock and Looking Ahead. In Proceedings of the Second Workshop on Scholarly Document Pro- cessing, pages 56-65. Association for Computational Linguistics.
Anne Lauscher, Goran Glavaš, and Simone Paolo Ponzetto. 2018b. An Argument-Annotate Corpus of Scientific Publications. Proceedings of the 5th Workshop on Argument Mining, pages 40-46.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SeiB- ERT: A pretrained language model for scientific text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing TEMNLP-IJCNIAL pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguisties.
John Lawrence and Chris Reed. 2015, Combining Ar- gument Mining Techniques. Proceedings of the 2nd Workshop on Argumentation Mining, pages 127-136.
John Lawrence and Chris Reed. 2019. Argument Mining: A Survey. Computational Linguistics. (August):1-55.
Marco Lippi and Paolo Torroni. 20l Argumentation Mining: State of the Art and Emerging Trends, ACM Transactions Internet Technology, 16(2):1-25.
Yoav Benjamini and Yosef Hochberg. 1995, Control- ling the false discovery rate: practical and pow- erful approach to multiple testing. Journal of the Royal statistical society: series B (Metodologicca), 57(1):289-300
Marco Lippi and Paolo Torroni. 201 MARGOT: A web server for argumentation mining. Expert Sys- tems with Applications, 65:292-303.
Leo Breiman, 1996. Bagging predictors. Machine learning, 24: 123-14.4
Nick Littlestone and Manfred K Warmuth. 1994. The weighted majority algorithm. Information and com- putation, 108(2)212-261.
Katarzyna Budzynska and Serena Villata. 2015, Argu- ment Mining. IEEE Intelligent Informatics Bulletin, 17(1):1-6.
Tobias Mayer, Elena Cabrio, and Serena Villata. 2020. Transformer-based argument mining for healthcare applications. In Frontiers in Artificial Intelligence and Applications. volume 325, pages 2108-218 2215.
Artem Chernodub, Oleksiy Oliynyk, Philipp Hei- denreich, Alexander Bondarenko, Matthias Hagen, Chris Biemann, and Alexander Panchenko. 2019. TARGER: Neural Argument Mining Your Finger- tips. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 195-200.
David Opitz and Richard Maclin. 1999. Popular en- semble methods: An empirical study. Journal of artificial intelligence research, 11:169-198.
Nils Reimers, Benjamin Schiller, Tilman Beck, Jo- hannes Daxenberger. Christian Stab, and Iryna Gurevych. 2019. Classification and Clustering of Arguments with Contextualized Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 567-578, Florence, Italy.
Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kerrels over discrete structures, and the voted perceptron. In Pro- ceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 263-270.
Beatriz Fisas, Francesco Ronzano, and Horacio Saggion. 2016. A multi-layered annotated corpus of scien- tific papers. Proceedings of the Tenth International Conference Language Resources and Evaluation (LREC'16), 16), pages 308 1-088.
Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. 2020. Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs. Transactions of the Association for Compu- tational Linguistics, 8:589-600 6044
Chris Fournier. 2013. Evaluating text segmentation us- ing boundary edit distance. In Proceedings of the 5 Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1702-1712.
Florian Ruosch, Cristina Sarasua, and Abraham Bern- stein. 2022. BAM: Benchmarking Argument Min- ing on Scientific Documents. In Proceedings of the Workshop on Scientific Document Understanding co- located with 36th AAAI Conference on Artificial In- telligence (AAAI 2022).
William H Greene. 2003. Econometric analysis. Pear- son Education India.
Isabel Segura- Bedmar, Paloma Martínez Fernández, and María Herrero Zazo. 2013. Semeval-2013 task 9: Ex- traction of drug-drug interactions from biomedical texts (ddiextraction 2013). Association for Computa- tional Linguistics.
Steffen Herbold. 2020. Autorank: A python package for automated ranking of classifiers. Journal of Open Source Software, 5148)

---

Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schlitze, and Iryna Gurevych. 2020. Fine-Grained Argument Unit Recognition and Clas- sification. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9048-9058
C for TRABAM and TARGER (Voting) vs. TRABAM
The statistical analysis was conducted for populations with 12 paired samples. The wase significance level of the tests is al- pha=0.050. We rejected the null hypothesis that the population is normal for the population C-V- trabam+targer (p=0.024). Therefore, we assume that not all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and one of them is not normal, we use Wilcoxon's signed rank test to determine the differences in the central tendency and report the median (MD) and the median absolute deviation (MAD) for each population. We reject the null hy- pothesis (p=0.021) of Wilcoxon's signed rank test that population C-trabam--ter (MD=0.671+-0.0) MAD=0.038) is not greater than population C-V- trabam+targer (MD=0.691+-0.045 MAD=0.017). Therefore, we assume that the median of C-V- trabam+turger is significantly larger than the me- dian value of C-trabam-test with a small effect size (g4mma=-0.461).
CJ van Rijshergen. 1979. Information retrieval.
Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation schemes. Cam- bridge University Press.
Simon Wells. 2014. Argument Mining: Was Ist Das? Proceedings of the 14th International Work-
shop on Computational Models of Natural Argument (CMNA14), Krakow, Poland.
A
Appendix
A.1 Statistical Significance Testing
We report the p-values where we claim to outper- form a previously best result (indicated in bold in the original tables) in Table 5, Table 6, and Table 7, Bold numbers indicate statistically significant dif- ferences (p < 0.05), while denotes small, ** medium, and *** large effect size. The testing for the statistical significance of the results was conducted with Autorank (Herbold, 2020). The de- tailed reports on the conducted tests for statistical significance, including all procedures and assump- tions testing, are shown below.
B for TRABAM and AURC (Recombination) vs. AURC The statistical analysis was conducted for 2 populations with 12 paired samples. The family- significance level of the tests is al- pha=0.050. We failed to reject the null hypoth- esis that the population is normal for all popula- tions (minimal observed p-value=0.052). There- fore, we assume that all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and both populations are normal, the t-test to determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations B-aurc-test (M=0.028+00.009. SD=0.012) and SB-RR(trabam)+<(urg (M=0.488+- 0.036, SD=0.047) are equal. Therefore, we assume that the mean value of (133R-((rabam) is sig- nificantly larger than the mean value of B-aure-test with a large effect size (d=-13.252).
S for TARGER, AURC, MARGOT, and TRA- BAM (Stacking) vs. TRABAM The statistical analysis was conducted for 2 populations with 12 paired samples. The family-wise significance level of the tests alpha=0.050. We failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.328). Therefore, we assume that all populations are nor- mal. No check for homogeneity was required be- cause we only have two populations. Because we have only two populations and both populations are normal, we use the t-test to determine differ- ences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We failed to reject the null hypothesis (p=0.180) of the paired
t-test that the mean values of the populations C- trabam-test (M-0.834+000041, SD=0.054) and SB- Aungerraurc--aaggottrra (M=0.838+-0.039, SD=0.052) are are equal. Therefore, we assume that there is no statistically significant difference between the mean values of the populations.
B for TRABAM and TARGER (Recombination) vs. TARGER The statistical analysis was con- ducted for 2 populations with 12 paired samples. The family-wise significance level of the tests is alpha=0.050. We failed to reject the null hypoth-

---

Task
Method
Systems
Previous best
p-value
S
Stacking
TARGER, AURC, MARGOT, TRABAM
TRABAM
0.180
C
Voting
TRABAM, TARGER
TRABAM
0.021
Table 5: The p-values corresponding to the results reported in Table 2.
From To
B
C
R
TRABAM AURC
0.000 ##*
TRABAM TARGER
0.018
TRABAM ArguminSci
0.000 #*
0.000
#**
TRABAM MARGOT
0.000 #
0.000
Table 6: The p-values corresponding to the results reported in Table 3.
esis that the population normal for all popula- tions (minimal observed p-valuc=0.189)) There- fore, we assume that all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and both populations are normal, use the t-test determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.018) the paired t-test that the mean values of the populations C-targer-test (M=0.485+-0.0477 SD=0.063) and C-R- (((((raaam))(trreer)) (M=0.504+- 0.042, SD=0.056) are equal. Therefore, we assume that the mean value of C-R- C-R-(trabam)++ (tarber)rr(ttrrr)) is significantly larger than the mean value of C-targer- test with a small effect size (d=-0.313).
fore, we assume that the mean value of C-R- (raaaa))[ (arg)))isnrrnnii is significantly larger than the mean value of C-arguminsci-- with a large effect size (d=-5.227).
C for TRABAM and ArguminSci (Recombina- tion) vs. ArguminSci The statistical analysis was conducted for 2 populations with 12 paired samples. The family-wise significance level of the tests is alpha=0.050. We failed to reject the null hypothesis that the population is normal for all pop- ulations (minimal observed p-value=0.681), There- fore, assume that all populations normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-arguminsci-tess (M=0.093+- 0.016, SD=0.021) and (-R-(trabmm)+ (M-0.344+-0.044 SD=0.054) are equal. There- fore, we assume that the mean value of C-R- (trabam)+ (arguminsci) is significantly larger than the mean value of C-argumiinsci-ten test with large effect size (d=-6.140).
B for TRABAM and ArguminSei (Recombina- tion) vs. ArguminSei The statistical analysis was conducted for 2 populations with 12 paired samples. The family-wise significance level of the tests is alpha=0.050. We failed to reject the null hypothesis that the population is normal for all pop- ulations (minimal observed p-valuc=0.120 There- fore, we assume that all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-arguminsci-test (M=0. 102+- 0.013, SD=0.018) and C-R- (M-0.287+-0.035. SD=0.047) are equal. There-
B for MARGOT and TRABAM (Recombina- tion) vs. MARGOT The statistical analysis was conducted for 2 populations with 12 paired sam- plcs. The family-wis significance level of the is alpha=0.050. We failed reject the null hypothesis that the population is normal for all pop- ulations (minimal observed p-value=0. 133). There- fore, we assume that all populations are normal. No check for homogeneity was required because we

---

Task
Method
Systems
Previous best
p-value
S
Stacking
TARGER, AURC, MARGOT, TRABAM
TRABAM
0.180
C
Voting
TRABAM, TARGER
TRABAM
0.021
Table 7: The p-values corresponding to the results reported in Table 4.
Hypothesis
p-value
Rank
(i/m)a
B
B B-aure-test
SB (B-R((rrbaam)((urr))
1.933E-12
0.006
C
C-arguminsci-test
2.133E-09
2
0.013
B
C-arguminsci-test
(-Be(rrrbam))(agguunnnss)))
1.866E-8
3
0.019
B
C-margot-test
(-R-(raabam)((marro))))
2.032E-07
0.025 4
C
C-margot-test
C-R (raaarraaa)))(maarro)))
4.266E-04
5
0.031
B
C-targer-test
(-R-(traaam)<(aarger)
1.822E-02
6
0.038
C
C-trabam-test
C- C-V-trabam-+targer
2.124E-02
7
0.044
S
C-trabam-test
SB- J0-S-arggr--auuuttngoottt--obbb
1.795E-01
0.050 8
Table 8: Calculations of the Benjamini-Hochhee procedure.
only have two populations. Because we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.000) of the paired t-test that the mean val- ues of the populations C-margot-test (M=0.098+- 0.014, SD=0.019) and ((a-((rraam))<mmarrn)) (M=0.1714-0.020, SD=0.026) are equal. There- fore, we assume that the mean value of C-R- (mabao)((maroo))) is significantly larger than the value of C-margot-test with large effect size (d=-3.210).
(M=0.1644-0.029, SD=0.039) are equal. There- fore, we assume that the mean value of C-R- (margo))(marrot)) is significantly larger than the mean value of C-margot-test with medium effect size (d=-0.743).
S for TARGER, AURC, MARGOT, and TRA- BAM (Stacking) vs. TRABAM The statistical analysis was conducted for populations with 12 paired samples. The family-wise significance level of the tests is alpha=0.050. We failed reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.328) Therefore, we assume that all populations are nor- mal. No check for homogeneity was required be- cause we only have two populations. Because we have only populations and both populations are normal, we use the t-test to determine differ- ences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We failed to reject the null hypothesis (p=0.1 180) of the paired t-test that the mean values of the populations C- trabam-test 1M-0.834+000041 SD=0.054) and SB- (M=0.838+000039
for MARGOT and TRABAM (Recombina- tion) vs. MARGOT The statistical analysis was conducted for 2 populations with 12 paired sam- ples. The family-wise significance level of the tests is alpha=0.050. We failed to reject the null hypothesis that the population is normal for all pop- ulations (minimal observed p-valuc=0.347), There- fore, we assume that all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M) and the standard deviation (SD) for each population. We reject the null hypothesis (p=0.000) of the paired t-test that the mean val- ues of the populations C-margot-test (M=0.135+- 0.031, SD=0.042) and C-R- (trabam)+(marggn)))
S- S-uuger-aurr-mmrrrttttti SD=0.052) are are equal. Therefore, we assume that there is no statistically significant difference between the mean values of the populations.
C for (TRABAM and TARGER (Voting)) and TRABAM (Recombination) TRABAM The statistical analysis was conducted for 2 popula- tions with 12 paired samples. The family-wisc significance level of the tests alpha=0.050. We

---

rejected the null hypothesis that the population is normal for the population C-R-(trabam)+(C-VV trabam+targer) (p=0.024). Therefore, we assume that not all populations are normal. No check for homogeneity was required because we only have two populations. Because we have only two populations and one of them is not normal, we use Wilcoxon's signed rank test to determine the differences in the central tendency and report the median (MD) and the median absolute deviation (MAD) for each population. We reject the null hypothesis (p=0.021) of Wilcoxon's signed rank test that population C-trabam-test (MD=0.671+ +- 0.062, MAD=0.038) is not greater than population the median of is significantly larger than the median value of C-traban-test with small effect size (gamma=- 0.461).
(MD=0.691+ +- 0.048, MAD=0.017). Therefore, we assume that
A.2 Multiple Comparisons Problem
With the results from the tests above, we correct for the multiple comparisons problem by using the Benjamini-Hochhere procedure (Benjamini and Hochberg. 1995) with critical value of a 0.05. Table 8 shows the details of the calculations. From it, see that all the differences still statistically significant, even after correcting for the multiple comparisons problem.


Reviews:
[{"review_id": "LGSTQbX5da", "reviewer": "Reviewer_1CYs", "summary": "", "questions": "A. Even from my lack of expert knowledge on the subject, a question\nintrigues me. The authors claim that combining approaches in AM has\nbarely received any attention in previous literature. Is there any\nreason for it? Does it have something to do with the reviewer's item A\nin the previous section \"Reasons to reject\" ?\n\nIn relation to the authors' responses, the fact that the only costs to\nconsider are the adjustment and prediction of the combined methods\ndoes not in any way detract from their estimation, given that The\npractical interest of any computational process is determined by its\ncosts, both temporal and spatial. On the other hand, time costs cannot\nbe evaluated based on approximate estimates resulting from physical\ntiming (i.e. seconds), dependent on a multitude of external factors,\nbut rather based on formally defined time units. Even assuming that\nthe cost of training and running sets is several magnitudes less than\nthe cost of individual Argument Miners, the authors should make this\npoint as best they can.\n\nRegarding the question of what could be the reasons why combining\napproaches in AM has barely received any attention in previous\nliterature, the authors should reflect their response in the text.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your valuable comments.\n\nFirst, we would like to address your concern regarding the lack of reporting computational and temporal costs. Since we explicitly factored out the training and execution costs of the involved systems (see Section 4.1 Setup), we only consider the costs of combining them. Thus, the only costs to be considered are the fitting and prediction of the combination methods, and, therefore, no comparison can and should be made to the original systems. Given the magnitude of the data (the train and test data set contain approximately 242,000 tokens combined), the temporal costs for the ensemble methods were in the range of minutes (or even seconds). The computing time for all possible combinations (n = 309) for the Vertical Integration was 32 minutes and 35.16 seconds (resulting in an average of 6.33 seconds per), and for all results (n = 7) of the Horizontal Integration, it took 16.82 seconds (or 2.40 seconds on average). For the Combined Integration, no additional computations were necessary since it was merely a matter of selecting the best-performing combinations or single systems from the given results.\nWe did not mention this in the paper because the cost to train and execute ensembles is several magnitudes smaller than the cost of the individual Argument Miners (seconds versus days, see the BAM paper by Ruosch et al. (2022)). To fully address your concern, we will gladly make use of the additional page for the camera-ready version and add a paragraph on all the computational costs (in terms of execution times).\n\nSecond, to answer your question about combining approaches in AM barely receiving any attention. We do not have any evidence (apart from the lack of results when searching for these terms in literature) and can only speculate. One hint has been given in a very popular survey for the field of Argument Mining by Lawrence and Reed (2020): new publications tend to take a holistic view of the pipeline and, therefore, always present new approaches from scratch. This might have to do with the repeated advent of new techniques in Deep Learning and Large Language Models and people trying out those instead of focusing on improving existing methods. Nonetheless, if we take Ferrari et al. (2019) seriously, who showed that some advances of deep learning pipelines in the recommender systems domain may not always generalize well to other settings, and assume that the finding also holds for other domains then pursuing a recombination approach may be fruitful supporting our call to arms. \n\nAgain, thank you very much for taking the time to review our paper. We hope that we addressed all your concerns that stood in the way of giving the paper a more positive rating and would greatly appreciate it if the updated score reflected this.\n\nReferences\n* Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. Proceedings of the 13th ACM Conference on Recommender Systems, 101–109.\n* John Lawrence and Chris Reed. 2019. Argument Mining: A Survey. Computational Linguistics, (August):1–55.\n* Florian Ruosch, Cristina Sarasua, and Abraham Bernstein. 2022. BAM: Benchmarking Argument Mining on Scientific Documents. In Proceedings of the Workshop on Scientific Document Understanding co-located with 36th AAAI Conference on Artificial Intelligence (AAAI 2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I am not a specialist in the field, but precisely for that reason I\nparticularly appreciated the clarity of the writing and the (apparent)\nrigor of the testing protocol. In this sense, the results of the\nexperimental tests that support the proposal have been convincing to\nme, which is why I understand that the paper could have sufficient\nquality to be accepted.", "reasons_to_reject": "At no time do the authors estimate the computational and temporal\ncosts of their proposal, nor can they therefore compare them with\nthose of their competitors in the experimental tests. We are not talking about\na less important issue, but a determining one that can limit the\ninterest of the proposal to the theoretical framework.", "questions_to_authors": "A. Even from my lack of expert knowledge on the subject, a question\nintrigues me. The authors claim that combining approaches in AM has\nbarely received any attention in previous literature. Is there any\nreason for it? Does it have something to do with the reviewer's item A\nin the previous section \"Reasons to reject\" ?\n\nIn relation to the authors' responses, the fact that the only costs to\nconsider are the adjustment and prediction of the combined methods\ndoes not in any way detract from their estimation, given that The\npractical interest of any computational process is determined by its\ncosts, both temporal and spatial. On the other hand, time costs cannot\nbe evaluated based on approximate estimates resulting from physical\ntiming (i.e. seconds), dependent on a multitude of external factors,\nbut rather based on formally defined time units. Even assuming that\nthe cost of training and running sets is several magnitudes less than\nthe cost of individual Argument Miners, the authors should make this\npoint as best they can.\n\nRegarding the question of what could be the reasons why combining\napproaches in AM has barely received any attention in previous\nliterature, the authors should reflect their response in the text.", "paper_topic_and_main_contributions": "The authors introduce an Argument Mining (AM) framework that allows\nfor the automated combination of AM components in order to improve\naccuracy for the tasks in the overall pipeline, the goal being\ngeneralize well across domains by combining the strengths and\nbalance out the weaknesses.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 489: Horizontal Intergration --> Horizontal Integration"}, "extra_text": {}}, {"review_id": "w85TgzZrPX", "reviewer": "Reviewer_tpwx", "summary": "", "questions": "A. What are the results of tests for statistical significance? - answered in the rebuttal, thanks\n\nB. There is a possibility that the best results are just due to randomness (the Multiple Comparisons problem). What are the results after applying corrections such as Bonferroni correction? What are the results on a different test set? - answered in the rebuttal, thanks", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your valuable comments.\n\nWe would like to rebut and answer the following concerns that you raised in your review in this particular order: (I) the status of ensemble methods in Argument Mining, (II a) the statistical significance testing and (II b) the multiple comparisons problem, (III) the lack of additional test sets, and (IV) the poor scientific soundness.\n\n(I) Ensemble Methods in Argument Mining\n\nWe never claim that the results we obtained are surprising, but rather our goal is to improve the state-of-the-art and provide a framework to do so even further. To the best of our knowledge, we are the first to employ ensemble methods for the task of Argument Mining, also evidenced by the relatively short list of related works.\nMoreover, we see this paper as “a call to action to systematically explore the effectiveness of functional components of the AM pipeline and share these for re-use by others”, as emphasized in the Limitations and Future Work section.\n\n(II) Statistical Significance\n\n(a) Testing\n\nAccording to your comment, we performed the tests for statistical significance of the newly obtained results compared to the previous best runs from the BAM paper (Ruosch et al. 2022) using this benchmark. We report the p-values in tables corresponding to the ones in the DREAM paper. We find that, apart from one instance, all newly found best results are indeed statistically significantly different and, therefore, indeed valid. These calculations are to be added to the appendix of the camera-ready paper.\n\n\\* denotes small effect size, ** medium, and *** large effect size; bold indicates statistically significant difference (p < 0.05)\n- Corresponding to Table 2\n| Task | Ensemble                    | Previous best | p-value |\n|------|-----------------------------|---------------|---------|\n| S    | S-targer+aurc+margot+trabam | TRABAM        | 0.180   |\n| C    | V-trabam+targer             | TRABAM        | **0.021** * |\n- Corresponding to Table 3\n| From   | To         | B         | C         | R |\n|--------|------------|-----------|-----------|---|\n| TRABAM | AURC       | **0.000** *** | -         | - |\n| TRABAM | TARGER     | **0.018** *   | -         | - |\n| TRABAM | ArguminSci | **0.000** *** | **0.000** *** | - |\n| TRABAM | MARGOT     | **0.000** *** | **0.000** **  | - |\n- Corresponding to Table 4\n| Task | Ensemble                    | Previous best | p-value |\n|------|-----------------------------|---------------|---------|\n| S    | S-targer+aurc+margot+trabam | TRABAM        | 0.180   |\n| C    | V-trabam+targer             | TRABAM        | **0.021** * |\n\nA detailed report on the conducted tests for statistical significance, including all procedures and assumptions testing, is included at the end of the rebuttal.\n\n(b) Multiple Comparisons Problem\n\nWith the results from the tests above, we correct for the multiple comparisons problem by using the Benjamini-Hochberg procedure with a critical value of Q = 0.05. The table with the details of the calculations can be found below. From it, we see that all the differences are still statistically significant, even after correcting for the multiple comparisons problem. These calculations are to be added to the appendix of the camera-ready paper.\n\nBold p-values in the table indicate that they are statistically significant after the correction.\n| Hypothesis                                     | p-value       | Rank | (i/m)Q |\n|------------------------------------------------|---------------|------|--------|\n| B B-aurc-test SB-R-(trabam)+(aurc)             | **1.933E-12** | 1    | 0.006  |\n| C C-arguminsci-test C-R-(trabam)+(arguminsci)  | **2.133E-09** | 2    | 0.013  |\n| B C-arguminsci-test C-R-(trabam)+(arguminsci)  | **1.866E-08** | 3    | 0.019  |\n| B C-margot-test C-R-(trabam)+(margot)          | **2.032E-07** | 4    | 0.025  |\n| C C-margot-test C-R-(trabam)+(margot)          | **4.266E-04** | 5    | 0.031  |\n| B C-targer-test C-R-(trabam)+(targer)          | **1.822E-02** | 6    | 0.038  |\n| C C-trabam-test C-V-trabam+targer              | **2.124E-02** | 7    | 0.044  |\n| S C-trabam-test SB-S-targer+aurc+margot+trabam | 1.795E-01     | 8    | 0.050  |\n\n(III) Lack of Additional Test Sets\n\nUnfortunately, including an additional test set would take more time than we have in the rebuttal period. We would need to obtain and pre-process a new data set, then train, execute, and evaluate all the systems as well as their combinations. Even when we only consider training the systems on new data, we are looking at several days of computing (see the run times of the systems in Table 1 of the BAM paper (Ruosch et al. 2022)). This is simply not feasible at the moment.\nAs to why we only had one test set in the data to begin with lies in the fact that BAM is currently the only unifying benchmark in Argument Mining that produces comparable results. Its test data set only consists of one annotated corpus of scientific papers originally created by Lauscher et al. (2018) (Sci-Arg).  Note that we are unaware of any other dataset of scholarly papers annotated for Argument Mining. We do agree that the community urgently needs to get together and collect a suite of datasets – it is a crucial step in improving the performance of Argument Mining approaches. \n\n(IV) Poor Soundness\n\nWe thank the reviewer for the very positive verdict given under “Reasons To Accept”, including the mention of a “methodical approach”. We understand that the poor rating of soundness has to do with the lack of statistical tests for the significance of the obtained results. We hope that the reviewer is satisfied with the deliberations and p-values given above and, therefore, is inclined to increase the appropriate score.\n\nAgain, thank you very much for taking the time to review our paper. We hope that we addressed all your concerns that stood in the way of giving the paper a more positive rating and would greatly appreciate it if the updated score would reflect this.\n\nReferences\n* Florian Ruosch, Cristina Sarasua, and Abraham Bernstein. 2022. BAM: Benchmarking Argument Mining on Scientific Documents. In Proceedings of the Workshop on Scientific Document Understanding co-located with 36th AAAI Conference on Artificial Intelligence (AAAI 2022).\n\n\nHere is the detailed report on the conducted tests for statistical significance, including all procedures and assumptions testing:\n\n************ S C-trabam-test SB-S-targer+aurc+margot+trabam ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.328). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe failed to reject the null hypothesis (p=0.180) of the paired t-test that the mean values of the populations C-trabam-test (M=0.834+-0.041, SD=0.054) and SB-S-targer+aurc+margot+trabam (M=0.838+-0.039, SD=0.052) are are equal. Therefore, we assume that there is no statistically significant difference between the mean values of the populations.\n\n************ C C-trabam-test C-V-trabam+targer ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe rejected the null hypothesis that the population is normal for the population C-V-trabam+targer (p=0.024). Therefore, we assume that not all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and one of them is not normal, we use Wilcoxon's signed rank test to determine the differences in the central tendency and report the median (MD) and the median absolute deviation (MAD) for each population.\nWe reject the null hypothesis (p=0.021) of Wilcoxon's signed rank test that population C-trabam-test (MD=0.671+-0.062, MAD=0.038) is not greater than population C-V-trabam+targer (MD=0.691+-0.048, MAD=0.017). Therefore, we assume that the median of C-V-trabam+targer is significantly larger than the median value of C-trabam-test with a small effect size (gamma=-0.461).\n\n************ B B-aurc-test SB-R-(trabam)+(aurc) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.052). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations B-aurc-test (M=0.028+-0.009, SD=0.012) and SB-R-(trabam)+(aurc) (M=0.488+-0.036, SD=0.047) are equal. Therefore, we assume that the mean value of SB-R-(trabam)+(aurc) is significantly larger than the mean value of B-aurc-test with a large effect size (d=-13.252).\n\n************ B C-targer-test C-R-(trabam)+(targer) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.189). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.018) of the paired t-test that the mean values of the populations C-targer-test (M=0.485+-0.047, SD=0.063) and C-R-(trabam)+(targer) (M=0.504+-0.042, SD=0.056) are equal. Therefore, we assume that the mean value of C-R-(trabam)+(targer) is significantly larger than the mean value of C-targer-test with a small effect size (d=-0.313).\n\n************ B C-arguminsci-test C-R-(trabam)+(arguminsci) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.120). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-arguminsci-test (M=0.102+-0.013, SD=0.018) and C-R-(trabam)+(arguminsci) (M=0.287+-0.035, SD=0.047) are equal. Therefore, we assume that the mean value of C-R-(trabam)+(arguminsci) is significantly larger than the mean value of C-arguminsci-test with a large effect size (d=-5.227).\n\n************ C C-arguminsci-test C-R-(trabam)+(arguminsci) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.681). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-arguminsci-test (M=0.093+-0.016, SD=0.021) and C-R-(trabam)+(arguminsci) (M=0.344+-0.040, SD=0.054) are equal. Therefore, we assume that the mean value of C-R-(trabam)+(arguminsci) is significantly larger than the mean value of C-arguminsci-test with a large effect size (d=-6.140).\n\n************ B C-margot-test C-R-(trabam)+(margot) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.133). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-margot-test (M=0.098+-0.014, SD=0.019) and C-R-(trabam)+(margot) (M=0.171+-0.020, SD=0.026) are equal. Therefore, we assume that the mean value of C-R-(trabam)+(margot) is significantly larger than the mean value of C-margot-test with a large effect size (d=-3.210).\n\n************ C C-margot-test C-R-(trabam)+(margot) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.347). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe reject the null hypothesis (p=0.000) of the paired t-test that the mean values of the populations C-margot-test (M=0.135+-0.031, SD=0.042) and C-R-(trabam)+(margot) (M=0.164+-0.029, SD=0.039) are equal. Therefore, we assume that the mean value of C-R-(trabam)+(margot) is significantly larger than the mean value of C-margot-test with a medium effect size (d=-0.743).\n\n************ S C-trabam-test SB-S-targer+aurc+margot+trabam ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe failed to reject the null hypothesis that the population is normal for all populations (minimal observed p-value=0.328). Therefore, we assume that all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and both populations are normal, we use the t-test to determine differences between the mean values of the populations and report the mean value (M)and the standard deviation (SD) for each population.\nWe failed to reject the null hypothesis (p=0.180) of the paired t-test that the mean values of the populations C-trabam-test (M=0.834+-0.041, SD=0.054) and SB-S-targer+aurc+margot+trabam (M=0.838+-0.039, SD=0.052) are are equal. Therefore, we assume that there is no statistically significant difference between the mean values of the populations.\n\n************ C C-trabam-test C-R-(trabam)+(C-V-trabam+targer) ************\nThe statistical analysis was conducted for 2 populations with 12 paired samples.\nThe family-wise significance level of the tests is alpha=0.050.\nWe rejected the null hypothesis that the population is normal for the population C-R-(trabam)+(C-V-trabam+targer) (p=0.024). Therefore, we assume that not all populations are normal.\nNo check for homogeneity was required because we only have two populations.\nBecause we have only two populations and one of them is not normal, we use Wilcoxon's signed rank test to determine the differences in the central tendency and report the median (MD) and the median absolute deviation (MAD) for each population.\nWe reject the null hypothesis (p=0.021) of Wilcoxon's signed rank test that population C-trabam-test (MD=0.671+-0.062, MAD=0.038) is not greater than population C-R-(trabam)+(C-V-trabam+targer) (MD=0.691+-0.048, MAD=0.017). Therefore, we assume that the median of C-R-(trabam)+(C-V-trabam+targer) is significantly larger than the median value of C-trabam-test with a small effect size (gamma=-0.461).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The code is available for reproducibility.\n\nB. Very good and detailed explanation of the approach.\n\nC. A methodical approach to test the various ensemble configurations.", "reasons_to_reject": "A. Ensemble approaches have been proven to improve over individual approaches in a wide range of tasks. The fact that it appears to be the case in this task as well is not surprising.\n\nAfter the authors rebuttal, the following is no longer applicable:\nB. It is unclear whether the conclusions are valid. It appears that the experiments are based on a single test set and therefore the results might suffer of the Multiple Comparisons problem: Given that multiple experiments have been made by exhausting all possible ensemble configurations, it is possible that the best results are due to a coincidence. The experiments did not include tests of statistical significance or statistical corrections to compensate for this.", "questions_to_authors": "A. What are the results of tests for statistical significance? - answered in the rebuttal, thanks\n\nB. There is a possibility that the best results are just due to randomness (the Multiple Comparisons problem). What are the results after applying corrections such as Bonferroni correction? What are the results on a different test set? - answered in the rebuttal, thanks", "paper_topic_and_main_contributions": "This paper presents a framework for the use of ensemble approaches for various phases of he task of argument mining. The authors experiment with various ensemble setups and evaluate the results with existing data (BAM).\n\nCode is available in a code repository.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None (but this is not my area of research).", "Typos_Grammar_Style_and_Presentation_Improvements": "The quality of writing is very good and I did not find any typos or English errors."}, "extra_text": {}}, {"review_id": "JKkSrdzZnz", "reviewer": "Reviewer_9Pww", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your valuable comments and very positive feedback. We would like to address your single concern, which poses a reason to reject. We had to keep the content of the paper to a minimum because of the page limit restriction. We can gladly extend the Related Work section with more elaborate explanations of the stages of the Argument Mining pipeline and the related components using the additional page for the camera-ready version.\nWe expect to discuss a good dozen references in such a section. Note, however, that we are unaware of any other approach exploring the combination of existing approaches. So the related work would focus on different views of Argument Mining as a task, the existing general approaches to solve it, and an overview of “what works and what does not” regarding classifiers as well as features.\n\nAgain, thank you very much for taking the time to review our paper. We hope that we addressed all your concerns that stood in the way of giving the paper a more positive rating and would greatly appreciate it if the updated score reflected this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting approach that allows modules to be recombined from components. This method is certainly transferable for other tasks.\n\nData and code are available on an anonymous GitHub account", "reasons_to_reject": "The Related Work chapter refers to only one work. Perhaps one could introduce the components that are included in the evaluation here?\n(This is currently in \"Setup\").", "questions_to_authors": "", "paper_topic_and_main_contributions": "The goal of this research is to provide a system for argument mining (AM) with ensemble methods, to test argument mining modules and allow for new combinations. The authors want to find the optimal combination of modules for the AM tasks and then test, whether this also produces the best result overall.\n\nThe methodology:\nUse of 5 systems for AM\n- Division of the AM pipeline into 4 tasks: \n\t- sentence classification: sentence tokenization and classification to argumentative/non-argumentative.\n\t- boundary detection: find the delineations of components in only the argumentative sentences\n\t- component identification: identify components whose boundaries have been detected\n\t- relation prediction: relies on the previously identified components to find the triples (subject and object are from the set of argumentative components) that constitute its output\n- combine whole systems by giving a weight\n- combine single modules by passing intermediate results to other systems\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper advances the state of the art in argument mining by showing that a pipelined approach, with ensembling, can outperform current end-to-end systems.

Reviewers were mostly ambivalent about this paper in terms of excitement, largely because the main contribution seems to be experimental results showing the advantages of a less-commonly used approach to this task. Reviewers also noted that improvements from using ensembles was not surprising or innovative, and that were results were only provided for a single dataset. However, they reported that the code to reproduce these experiments would be useful, and that these results would inform future work.

In terms of soundness, some initial concerns were raised, but reviewers seem largely satisfied with the authors' response, including an update that would correct for multiple comparisons. The one exception to this is Reviewer 1CYs's concerns about computational costs, but this seems like a somewhat secondary concern.
Decision: Accept-Main