Question: What is the main evaluation metric?
Evidence: In the following experiment, we focus on automatic identification of arguments in the discourse. Our approach is based on supervised and semi-supervised machine learning methods on the gold data Toulmin dataset introduced in section SECREF51 .
An argument consists of different components (such as premises, backing, etc.) which are implicitly linked to the claim. In principle one document can contain multiple independent arguments. However, only 4% of the documents in our dataset contain arguments for both sides of the issue. Thus we simplify the task and assume there is only one argument per document.
Given the low inter-annotator agreement on the pathos dimension (Table TABREF77 ), we focus solely on recognizing the logical dimension of argument. The pathos dimension of argument remains an open problem for a proper modeling as well as its later recognition.
Since the smallest annotation unit is a token and the argument components do not overlap, we approach identification of argument components as a sequence labeling problem. We use the BIO encoding, so each token belongs to one of the following 11 classes: O (not a part of any argument component), Backing-B, Backing-I, Claim-B, Claim-I, Premise-B, Premise-I, Rebuttal-B, Rebuttal-I, Refutation-B, Refutation-I. This is the minimal encoding that is able to distinguish two adjacent argument components of the same type. In our data, 48% of all adjacent argument components of the same type are direct neighbors (there are no "O" tokens in between).
We report Macro- INLINEFORM0 score and INLINEFORM1 scores for each of the 11 classes as the main evaluation metric. This evaluation is performed on the token level, and for each token the predicted label must exactly match the gold data label (classification of tokens into 11 classes).
As instances for the sequence labeling model, we chose sentences rather than tokens. During our initial experiments, we observed that building a sequence labeling model for recognizing argument components as sequences of tokens is too fine-grained, as a single token does not convey enough information that could be encoded as features for a machine learner. However, as discussed in section UID73 , the annotations were performed on data pre-segmented to sentences and annotating tokens was necessary only when the sentence segmentation was wrong or one sentence contained multiple argument components. Our corpus consists of 3899 sentences, from which 2214 sentences (57%) contain no argument component. From the remaining ones, only 50 sentences (1%) have more than one argument component. Although in 19 cases (0.5%) the sentence contains a Claim-Premise pair which is an important distinction from the argumentation perspective, given the overall small number of such occurrences, we simplify the task by treating each sentence as if it has either one argument component or none.
The approximation with sentence-level units is explained in the example in Figure FIGREF112 . In order to evaluate the expected performance loss using this approximation, we used an oracle that always predicts the correct label for the unit (sentence) and evaluated it against the true labels (recall that the evaluation against the true gold labels is done always on token level). We lose only about 10% of macro INLINEFORM0 score (0.906) and only about 2% of accuracy (0.984). This performance is still acceptable, while allowing to model sequences where the minimal unit is a sentence.
Table TABREF114 shows the distribution of the classes in the gold data Toulmin, where the labeling was already mapped to the sentences. The little presence of rebuttal and refutation (4 classes account only for 3.4% of the data) makes this dataset very unbalanced.
We chose SVMhmm BIBREF111 implementation of Structural Support Vector Machines for sequence labeling. Each sentence ( INLINEFORM0 ) is represented as a vector of real-valued features.
We defined the following feature sets:
FS0: Baseline lexical features
word uni-, bi-, and tri-grams (binary)
FS1: Structural, morphological, and syntactic features
First and last 3 tokens. Motivation: these tokens may contain discourse markers or other indicators for argument components, such as “therefore” and “since” for premises or “think” and “believe” for claims.
Relative position in paragraph and relative position in document. Motivation: We expect that claims are more likely to appear at the beginning or at the end of the document.
Number of POS 1-3 grams, dependency tree depth, constituency tree production rules, and number of sub-clauses. Based on BIBREF113 .
FS2: Topic and sentiment features
30 features taken from a vector representation of the sentence obtained by using Gibbs sampling on LDA model BIBREF114 , BIBREF115 with 30 topics trained on unlabeled data from the raw corpus. Motivation: Topic representation of a sentence might be valuable for detecting off-topic sentences, namely non-argument components.
Scores for five sentiment categories (from very negative to very positive) obtained from Stanford sentiment analyzer BIBREF116 . Motivation: Claims usually express opinions and carry sentiment.
FS3: Semantic, coreference, and discourse features
Binary features from Clear NLP Semantic Role Labeler BIBREF117 . Namely, we extract agent, predicate + agent, predicate + agent + patient + (optional) negation, argument type + argument value, and discourse marker which are based on PropBank semantic role labels. Motivation: Exploit the semantics of Capturing the semantics of the sentences.
Binary features from Stanford Coreference Chain Resolver BIBREF118 , e.g., presence of the sentence in a chain, transition type (i.e., nominal–pronominal), distance to previous/next sentences in the chain, or number of inter-sentence coreference links. Motivation: Presence of coreference chains indicates links outside the sentence and thus may be informative, for example, for classifying whether the sentence is a part of a larger argument component.
Results of a PTDB-style discourse parser BIBREF119 , namely the type of discourse relation (explicit, implicit), presence of discourse connectives, and attributions. Motivation: It has been claimed that discourse relations play a role in argumentation mining BIBREF120 .
FS4: Embedding features
300 features from word embedding vectors using word embeddings trained on part of the Google News dataset BIBREF121 . In particular, we sum up embedding vectors (dimensionality 300) of each word, resulting into a single vector for the entire sentence. This vector is then directly used as a feature vector. Motivation: Embeddings helped to achieve state-of-the-art results in various NLP tasks BIBREF116 , BIBREF122 .
Except the baseline lexical features, all feature types are extracted not only for the current sentence INLINEFORM0 , but also for INLINEFORM1 preceding and subsequent sentences, namely INLINEFORM2 , INLINEFORM3 , INLINEFORM4 INLINEFORM5 , INLINEFORM6 , where INLINEFORM7 was empirically set to 4. Each feature is then represented with a prefix to determine its relative position to the current sequence unit.
Let us first discuss the upper bounds of the system. Performance of the three human annotators is shown in the first column of Table TABREF139 (results are obtained from a cumulative confusion matrix). The overall Macro- INLINEFORM0 score is 0.602 (accuracy 0.754). If we look closer at the different argument components, we observe that humans are good at predicting claims, premises, backing and non-argumentative text (about 0.60-0.80 INLINEFORM1 ), but on rebuttal and refutation they achieve rather low scores. Without these two components, the overall human Macro- INLINEFORM2 would be 0.707. This trend follows the inter-annotator agreement scores, as discussed in section UID75 .
In our experiments, the feature sets were combined in the bottom-up manner, starting with the simple lexical features (FS0), adding structural and syntactic features (FS1), then adding topic and sentiment features (FS2), then features reflecting the discourse structure (FS3), and finally enriched with completely unsupervised latent vector space representation (FS4). In addition, we were gradually removing the simple features (e.g., without lexical features, without syntactic features, etc.) to test the system with more “abstract” feature sets (feature ablation). The results are shown in Table TABREF139 .
The overall best performance (Macro- INLINEFORM0 0.251) was achieved using the rich feature sets (01234 and 234) and significantly outperformed the baseline as well as other feature sets. Classification of non-argumentative text (the "O" class) yields about 0.7 INLINEFORM1 score even in the baseline setting. The boundaries of claims (Cla-B), premises (Pre-B), and backing (Bac-B) reach in average lower scores then their respective inside tags (Cla-I, Pre-I, Bac-I). It can be interpreted such that the system is able to classify that a certain sentence belongs to a certain argument component, but the distinction whether it is a beginning of the argument component is harder. The very low numbers for rebuttal and refutation have two reasons. First, these two argument components caused many disagreements in the annotations, as discussed in section UID86 , and were hard to recognize for the humans too. Second, these four classes have very few instances in the corpus (about 3.4%, see Table TABREF114 ), so the classifier suffers from the lack of training data.
The results for the in-domain cross validation scenario are shown in Table TABREF140 . Similarly to the cross-validation scenario, the overall best results were achieved using the largest feature set (01234). For mainstreaming and red-shirting, the best results were achieved using only the feature set 4 (embeddings). These two domains contain also fewer documents, compared to other domains (refer to Table TABREF71 ). We suspect that embeddings-based features convey important information when not enough in-domain data are available. This observation will become apparent in the next experiment.
The cross-domain experiments yield rather poor results for most of the feature combinations (Table TABREF141 ). However, using only feature set 4 (embeddings), the system performance increases rapidly, so it is even comparable to numbers achieved in the in-domain scenario. These results indicate that embedding features generalize well across domains in our task of argument component identification. We leave investigating better performing vector representations, such as paragraph vectors BIBREF123 , for future work.
Error analysis based on the probabilistic confusion matrix BIBREF124 shown in Table TABREF142 reveals further details. About a half of the instances for each class are misclassified as non-argumentative (the "O" prediction).
Backing-B is often confused with Premise-B (12%) and Backing-I with Premise-I (23%). Similarly, Premise-I is misclassified as Backing-I in 9%. This shows that distinguishing between backing and premises is not easy because these two components are similar such that they support the claim, as discussed in section UID86 . We can also see that the misclassification is consistent among *-B and *-I tags.
Rebuttal is often misclassified as Premise (28% for Rebuttal-I and 18% for Rebuttal-B; notice again the consistency in *-B and *-I tags). This is rather surprising, as one would expect that rebuttal would be confused with a claim, because its role is to provide an opposing view.
Refutation-B and Refutation-I is misclassified as Premise-I in 19% and 27%, respectively. This finding confirms the discussion in section UID86 , because the role of refutation is highly context-dependent. In a pragmatic perspective, it is put forward to indirectly support the claim by attacking the rebuttal, thus having a similar function to the premise.
We manually examined miss-classified examples produced the best-performing system to find out which phenomena pose biggest challenges. Properly detecting boundaries of argument components caused problems, as shown in Figure FIGREF146 (a). This goes in line with the granularity annotation difficulties discussed in section UID86 . The next example in Figure FIGREF146 (b) shows that even if boundaries of components were detected precisely, the distinction between premise and backing fails. The example also shows that in some cases, labeling on clause level is required (left-hand side claim and premise) but the approximation in the system cannot cope with this level of detail (as explained in section UID111 ). Confusing non-argumentative text and argument components by the system is sometimes plausible, as is the case of the last rhetorical question in Figure FIGREF146 (c). On the other hand, the last example in Figure FIGREF146 (d) shows that some claims using figurative language were hard to be identified. The complete predictions along with the gold data are publicly available.
SVMhmm offers many hyper-parameters with suggested default values, from which three are of importance. Parameter INLINEFORM0 sets the order of dependencies of transitions in HMM, parameter INLINEFORM1 sets the order of dependencies of emissions in HMM, and parameter INLINEFORM2 represents a trading-off slack versus magnitude of the weight-vector. For all experiments, we set all the hyper-parameters to their default values ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 ). Using the best performing feature set from Table TABREF139 , we experimented with a grid search over different values ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) but the results did not outperform the system trained with default parameter values.
The INLINEFORM0 scores might seem very low at the first glance. One obvious reason is the actual performance of the system, which gives a plenty of room for improvement in the future. But the main cause of low INLINEFORM2 numbers is the evaluation measure — using 11 classes on the token level is very strict, as it penalizes a mismatch in argument component boundaries the same way as a wrongly predicted argument component type. Therefore we also report two another evaluation metrics that help to put our results into a context.
Krippendorff's INLINEFORM0 — It was also used for evaluating inter-annotator agreement (see section UID75 ).
Boundary similarity BIBREF125 — Using this metric, the problem is treated solely as a segmentation task without recognizing the argument component types.
As shown in Table TABREF157 (the Macro- INLINEFORM0 scores are repeated from Table TABREF139 ), the best-performing system achieves 0.30 score using Krippendorf's INLINEFORM1 , which is in the middle between the baseline and the human performance (0.48) but is considered as poor from the inter-annotator agreement point of view BIBREF54 . The boundary similarity metrics is not directly suitable for evaluating argument component classification, but reveals a sub-task of finding the component boundaries. The best system achieved 0.32 on this measure. Vovk2013MT used this measure to annotate argument spans and his annotators achieved 0.36 boundary similarity score. Human annotators in BIBREF125 reached 0.53 boundary similarity score.
The overall performance of the system is also affected by the accuracy of individual NPL tools used for extracting features. One particular problem is that the preprocessing models we rely on (POS, syntax, semantic roles, coreference, discourse; see section UID115 ) were trained on newswire corpora, so one has to expect performance drop when applied on user-generated content. This is however a well-known issue in NLP BIBREF126 , BIBREF127 , BIBREF128 .
To get an impression of the actual performance of the system on the data, we also provide the complete output of our best performing system in one PDF document together with the gold annotations in the logos dimension side by side in the accompanying software package. We believe this will help the community to see the strengths of our model as well as possible limitations of our current approaches.
We report Macro-F1 score and F1 scores for each of the 11 classes as the main evaluation metric.
Answer: The authors report Macro-F1 score and F1 scores for each of the 11 classes as the main evaluation metric.