Introduction
In recent years we have witnessed a great surge in activity in the area of computational argument analysis (e.g. BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 ), and the emergence of dedicated venues such as the ACL Argument Mining workshop series starting in 2014 BIBREF4 .
Argumentative relation classification is a sub-task of argument analysis that aims to determine relations between argumentative units A and B, for example, A supports B; A attacks B. Consider the following argumentative units (1) and (2), given the topic (0) “Marijuana should be legalized”:
This example is modeled in Figure FIGREF3 .
It is clear that (1) has a negative stance towards the topic and (2) has a positive stance towards the topic. Moreover, we can say that (2) attacks (1). In discourse, such a relation is often made explicit through discourse markers: (1). However, (2); On the one hand (1), on the other (2); (1), although (2); Admittedly, (2); etc. In the absence of such markers we must determine this relation by assessing the semantics of the individual argumentative units, including (often implicit) world knowledge about how they are related to each other.
In this work, we show that argumentative relation classifiers – when provided with textual context surrounding an argumentative unit's span – are very prone to neglect the actual textual content of the EAU span. Instead they heavily rely on contextual markers, such as conjunctions or adverbials, as a basis for prediction. We argue that a system's capacity of predicting the correct relation based on the argumentative units' content is important in many circumstances, e.g., when an argumentative debate crosses document boundaries. For example, the prohibition of marijuana debate extends across populations and countries – argumentative units for this debate can be recovered from thousands of documents scattered across the world wide web. As a consequence, argumentative relation classification systems should not be (immensely) dependent on contextual clues – in the discussed cross-document setting these clues may even be misleading for such a system, since source and target arguments can be embedded in different textual contexts (e.g., when (1) and (2) stem from different documents it is easy to imagine a textual context where (2) is not introduced by however but instead by an `inverse' form such as e.g. moreover).
Related Work
It is well-known that the rhetorical and argumentative structure of texts bear great similarities. For example, BIBREF5 , BIBREF6 , BIBREF0 observe that elementary discourse units (EDUs) in RST BIBREF7 share great similarity with elementary argumentative units (EAUs) in argumentation analysis. BIBREF8 experiment with a modified version of the Microtext corpus BIBREF9 , which is an extensively annotated albeit small corpus. Similar to us, they separate argumentative units from discursive contextual markers. While BIBREF8 conduct a human evaluation to investigate the separation of Logos and Pathos aspects of arguments, our work investigates how (de-)contextualization of argumentative units affects automatic argumentative relation classification models.
Argumentative Relation Prediction: Models and Features
In this section, we describe different formulations of the argumentative relation classification task and describe features used by our replicated model. In order to test our hypotheses, we propose to group all features into three distinct types.
Models
Now, we introduce a classification of three different prediction models used in the argumentative relation prediction literature. We will inspect all of them and show that all can suffer from severe issues when focusing (too much) on the context.
The model INLINEFORM0 adopts a discourse parsing view on argumentative relation prediction and predicts one outgoing edge for an argumentative unit (one-outgoing edge). Model INLINEFORM1 assumes a connected graph with argumentative units and is tasked with predicting edge labels for unit tuples (labeling relations in a graph). Finally, a model INLINEFORM2 is given two (possibly) unrelated argumentative units and is tasked with predicting connections as well as edge labels (joint edge prediction and labeling).
BIBREF13 divide the task into relation prediction INLINEFORM0 and relation class assignment INLINEFORM1 : DISPLAYFORM0
DISPLAYFORM0
which the authors describe as argumentative relation identification ( INLINEFORM0 ) and stance detection ( INLINEFORM1 ). In their experiments, INLINEFORM2 , i.e., no distinction is made between features that access only the argument content (EAU span) or only the EAU's embedding context, and some features also consider both (e.g., discourse features). This model adopts a parsing view on argumentative relation classification: every unit is allowed to have only one type of outgoing relation (this follows trivially from the fact that INLINEFORM3 has only one input). Applying such a model to argumentative attack and support relations might impose unrealistic constraints on the resulting argumentation graph: A given premise might in fact attack or support several other premises. The approach may suffice for the case of student argumentative essays, where EAUs are well-framed in a discourse structure, but seems overly restrictive for many other scenarios.
Another way of framing the task, is to learn a function DISPLAYFORM0
Here, an argumentative unit is allowed to be in a attack or support relation to multiple other EAUs. Yet, both INLINEFORM0 and INLINEFORM1 assume that inputs are already linked and only the class of the link is unknown.
Thus, we might also model the task in a three-class classification setting to learn a more general function that performs relation prediction and classification jointly (see also, e.g., BIBREF10 ): DISPLAYFORM0
The model described by Eq. EQREF22 is the most general one: not only does it assume a graph view on argumentative units and their relations (as does Eq. EQREF20 ); in model formulation (Eq. EQREF22 ), an argumentative unit can have no or multiple support or attack relations. It naturally allows for cases where an argumentative unit INLINEFORM0 (supports INLINEFORM1 INLINEFORM2 attacks INLINEFORM3 INLINEFORM4 is-unrelated-to INLINEFORM5 ). Given a set of EAUs mined from different documents, this model enables us to construct a full-fledged argumentation graph.
Feature implementation
Our feature implementation follows the feature descriptions for Stance recognition and link identification in BIBREF13 . These features and variations of them have been used successfully in several successive works (cf. BIBREF1 , BIBREF16 , BIBREF15 ).
For any model the features are indexed by INLINEFORM0 . We create a function INLINEFORM1 which maps from feature indices to feature types. In other words, INLINEFORM2 tells us, for any given feature, whether it is content-based ( INLINEFORM3 ), content-ignorant ( INLINEFORM4 ) or full access ( INLINEFORM5 ). The features for, e.g., the joint prediction model INLINEFORM6 of type INLINEFORM7 ( INLINEFORM8 ) can then simply be described as INLINEFORM9 . Recall that features computed on the basis of the EAU span are content-based ( INLINEFORM10 ), features from the EAU-surrounding text are content-ignorant ( INLINEFORM11 ) and features computed from both are denoted by full-access ( INLINEFORM12 ). Details on the extraction of features are provided below.
These consist of boolean values indicating whether a certain word appears in the argumentative source or target EAU or both (and separately, their contexts). More precisely, for any classification instance we extract uni-grams from within the span of the EAU (if INLINEFORM0 ) or solely from the sentence-context surrounding the EAUs (if INLINEFORM1 ). Words which occur in both bags are only visible in the full-access setup INLINEFORM2 and are modeled as binary indicators.
Such features consist of syntactic production rules extracted from constituency trees – they are modelled analogously to the lexical features as a bag of production rules. To make a clear division between features derived from the EAU embedding context and features derived from within the EAU span, we divide the constituency tree in two parts, as is illustrated in Figure FIGREF26 .
If the EAU is embedded in a covering sentence, we cut the syntax tree at the corresponding edge ( in Figure FIGREF26 ). In this example, the content-ignorant ( INLINEFORM0 ) bag-of-word production rule representation includes the rules INLINEFORM1 and INLINEFORM2 . Analogously to the lexical features, the production rules are modeled as binary indicator features.
These features describe shallow statistics such as the ratio of argumentative unit tokens compared to sentence tokens or the position of the argumentative unit in the paragraph. We set these features to zero for the content representation of the argumentative unit and replicate those features that allow us to treat the argumentative unit as a black-box. For example, in the content-based ( INLINEFORM0 ) system that has access only to the EAU, we can compute the #tokens in the EAU, but not the #tokens in EAU divided by #tokens in the sentence. The latter feature is only accessible in the full access system variants. Hence, in the content-based ( INLINEFORM1 ) system most of these statistics are set to zero since they cannot be computed by considering only the EAU span.
For the content-based representation we retrieve only discourse relations that are confined within the span of the argumentative unit. In the very frequent case that discourse features cross the boundaries of embedding context and EAU span, we only take them into account for INLINEFORM0 .
We use the element-wise sum of 300-dimensional pre-trained GloVe vectors BIBREF24 corresponding to the words within the EAU span ( INLINEFORM0 ) and the words of the EAU-surrounding context ( INLINEFORM1 ). Additionally, we compute the element-wise subtraction of the source EAU vector from the target EAU vector, with the aim of modelling directions in distributional space, similarly to BIBREF25 . Words with no corresponding pre-trained word vector and empty sequences (e.g., no preceding context available) are treated as a zero-vector.
Tree-based sentiment annotations are sentiment scores assigned to nodes in constituency parse trees BIBREF26 . We represent these scores by a one-hot vector of dimension 5 (5 is very positive, 1 is very negative). We determine the contextual ( INLINEFORM0 ) sentiment by looking at the highest possible node of the context which does not contain the EAU (ADVP in Figure FIGREF26 ). The sentiment for an EAU span ( INLINEFORM1 ) is assigned to the highest possible node covering the EAU span which does not contain the context sub-tree (S in Figure FIGREF26 ). The full-access ( INLINEFORM2 ) score is assigned to the lowest possible node which covers both the EAU span and its surrounding context (S' in Figure FIGREF26 ). Next to the sentiment scores for the selected tree nodes and analogously to the word embeddings, we also calculate the element-wise subtraction of the one-hot sentiment source vectors from the one-hot sentiment target vectors. This results in three additional vectors corresponding to INLINEFORM3 , INLINEFORM4 and INLINEFORM5 difference vectors.
Results
Our first step towards our main experiments is to replicate the competitive argumentative relation classifier of BIBREF13 , BIBREF1 . Hence, for comparison purposes, we first formulate the task exactly as it was done in this prior work, using the model formulation in Eq. EQREF17 , which determines the type of outgoing edge from a source (i.e., tree-like view).
The results in Table TABREF38 confirm the results of BIBREF13 and suggest that we successfully replicated a large proportion of their features.
The results for all three prediction settings (one outgoing edge: INLINEFORM0 , support/attack: INLINEFORM1 and support/attack/neither: INLINEFORM2 ) across all type variables ( INLINEFORM3 , INLINEFORM4 and INLINEFORM5 ) are displayed in Table TABREF39 . All models significantly outperform the majority baseline with respect to macro F1. Intriguingly, the content-ignorant models ( INLINEFORM6 ) always perform significantly better than the models which only have access to the EAUs' content ( INLINEFORM7 , INLINEFORM8 ). In the most general task formulation ( INLINEFORM9 ), we observe that INLINEFORM10 even significantly outperforms the model which has maximum access (seeing both EAU spans and surrounding contexts: INLINEFORM11 ).
At first glance, the results of the purely EAU focused systems ( INLINEFORM0 ) are disappointing, since they fall far behind their competitors. On the other hand, their F1 scores are not devastatingly bad. The strong most-frequent-class baseline is significantly outperformed by the content-based ( INLINEFORM1 ) system, across all three prediction settings.
In summary our findings are as follows: (i) models which see the EAU span (content-based, INLINEFORM0 ) are significantly outperformed by models that have no access to the span itself (content-ignorant, INLINEFORM1 ) across all settings; (ii) in two of three prediction settings ( INLINEFORM2 and INLINEFORM3 ), the model which only has access to the context even outperforms the model that has access to all information in the input. The fact that using features derived exclusively from the EAU embedding context ( INLINEFORM4 ) can lead to better results than using a full feature-system ( INLINEFORM5 ) suggests that some information from the EAU can even be harmful. Why this is the case, we cannot answer exactly. A plausible cause might be related to the smaller dimension of the feature space, which makes the SVM less likely to overfit. Still, this finding comes as a surprise and calls for further investigation in future work.
A system for argumentative relation classification can be applied in one of two settings: single-document or cross-document, as illustrated in Figure FIGREF42 :
in the first case (top), a system is tasked to classify EAUs that appear linearly in one document – here contextual clues can often highlight the relationship between two units. This is the setting we have been considering up to now. However, in the second scenario (bottom), we have moved away from the closed single-document setting and ask the system to classify two EAUs extracted from different document contexts. This setting applies, for instance, when we are mining arguments from multiple sources.
In both cases, however, a system that relies more on contextual clues than on the content expressed in the EAUs is problematic: in the single-document setting, such a system will rely on discourse indicators – whether or not they are justified by content – and can thus easily be fooled.
In the cross-document setting, discourse-based indicators – being inherently defined with respect to their internal document context – do not have a defined rhetorical function with respect to EAUs in a separate document and thus a system that has learned to rely on such markers within a single-document setting can be seriously misled.
We believe that the cross-document setting should be an important goal in argumentation analysis, since it generalizes better to many debates of interest, where EAUs can be found scattered across thousands of documents. For example, for the topic of legalizing marijuana, EAUs may be mined from millions of documents and thus their relations may naturally extend across document boundaries. If a system learns to over-proportionally attend to the EAUs' surrounding contexts it is prone to making many errors.
In what follows we are simulating the effects that an overly context-sensitive classifier could have in a cross-document setting, by modifying our experimental setting, and study the effects on the different model types: In one setup – we call it randomized-context – we systematically distort the context of our testing instances by exchanging the context in a randomized manner; in the other setting – called no-context, we are deleting the context around the ADUs to be classified. Randomized-context simulates an open world debate where argumentative units may occur in different contexts, sometimes with discourse markers indicating an opposite class. In other words, in this setting we want to examine effects when porting a context-sensitive system to a multi-document setting. For example, as seen in Figure FIGREF42 , the context of an argumentative unit may change from “However” to “Moreover” – which can happen naturally in open debates.
The results are displayed in Figure FIGREF43 . In the standard setting (Figure FIGREF43 ), the models that have access to the context besides the content ( INLINEFORM0 ) and the models that are only allowed to access the context ( INLINEFORM1 ), always perform better than the content-based models ( INLINEFORM2 ) (bars above zero). However, when we randomly flip contexts of the test instances (Figure FIGREF43 ), or suppress them entirely (Figure FIGREF43 ), the opposite picture emerges: the content-based models always outperform the other models. For some classes (support, INLINEFORM3 ) the difference can exceed 50 F1 percentage points. These two studies, where testing examples are varied regarding their context (randomized-context or no-context) simulates what can be expected if we apply our systems for relation class assignment to EAUs stemming from heterogeneous sources. While the performances of a purely content-based model naturally stays stable, the performance of the other systems decrease notably – they perform worse than the content-based model.
We calculate the ANOVA classification F scores of the features with respect to our three task formulations INLINEFORM0 and INLINEFORM1 . The F percentiles of features extracted from the EAU surrounding text ( INLINEFORM2 ) and features extracted from the EAU span ( INLINEFORM3 ), are displayed in Figure FIGREF50 .
It clearly stands out that features obtained from the EAU surrounding context ( INLINEFORM0 ) are assigned much higher scores compared to features stemming from the EAU span ( INLINEFORM1 ). This holds true for all three task formulations and provides further evidence that models – when given the option – put a strong focus on contextual clues while neglecting the information provided by the EAU span itself.
Discussion
While competitive systems for argumentative relation classification are considered to be robust, our experiments have shown that despite confidence-inspiring scores on unseen testing data, such systems can easily be fooled – they can deliver strong performance scores although the classifier does not have access to the content of the EAUs. In this respect, we have provided evidence that there is a danger in case models focus too much on rhetorical indicators, in detriment of the context. Thus, the following question arises: How can we prevent argumentation models from modeling arguments or argumentative units and their relations in overly naïve ways? A simple and intuitive way is to dissect EAUs from their surrounding document context. Models trained on data that is restricted to the EAUs' content will be forced to focus on the content of EAUs. We believe that this will enhance the robustness of such models and allows them to generalize to cross-document argument relation classification. The corpus of student essays makes such transformations straightforward: only the EAUs were annotated (e.g., “However, INLINEFORM0 A INLINEFORM1 ”). If annotations extend over the EAUs (e.g., only full sentences are annotated, “ INLINEFORM2 However, A INLINEFORM3 ”), such transformations could be performed automatically after a discourse parsing step. When inspecting the student essays corpus, we further observed that an EAU mining step should involve coreference resolution to better capture relations between EAUs that involve anaphors (e.g., “Exercising makes you feel better” and “It INLINEFORM4 increases endorphin levels”).
Thus, in order to conduct real-world end-to-end argumentation relation mining for a given topic, we envision a system that addresses three steps: (i) mining of EAUs and (ii) replacement of pronouns in EAUs with referenced entities (e.g., INLINEFORM0 ). Finally (iii), given the cross product of mined EAUs we can apply a model of type INLINEFORM1 to construct a full-fledged argumentation graph, possibly spanning multiple documents. We have shown that in order to properly perform step (iii), we need stronger models that are able to better model EAU contents. Hence, we encourage the argumentation community to test their systems on a decontextualized version of the student essays, including the proposed – and possibly further extended – testing setups, to challenge the semantic representation and reasoning capacities of argument analysis models. This will lead to more realistic performance estimates and increased robustness of systems when addressing desirable multi-document tasks.
Conclusion
We have shown that systems which put too much focus on discourse information may be easily fooled – an issue which has severe implications when systems are applied to cross-document argumentative relation classification tasks. The strong reliance on contextual clues is also problematic in single-document contexts, where systems can run a risk of assigning relation labels relying on contextual and rhetorical effects – instead of focusing on content. Hence, we propose that researchers test their argumentative relation classification systems on two alternative versions of the StudentEssay data that reflect different access levels. (i) EAU-span only, where systems only see the EAU spans and (ii) context-only, where systems can only see the EAU-surrounding context. These complementary settings will (i) challenge the semantic capacities of a system, and (ii) unveil the extent to which a system is focusing on the discourse context when making decisions. We will offer our testing environments to the research community through a platform that provides datasets and scripts and a table to trace the results of content-based systems.
Acknowledgments
This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant no. GRK 1994/1 and by the Leibniz ScienceCampus “Empirical Linguistics and Computational Language Modeling”, supported by the Leibniz Association under grant no. SAS-2015-IDS-LWC and by the Ministry of Science, Research, and Art of Baden-Württemberg.

Question: Do they report results only on English data?
Answer: Unanswerable

Question: How do they demonstrate the robustness of their results?
Answer: performances of a purely content-based model naturally stays stable

Question: What baseline and classification systems are used in experiments?
Answer: BIBREF13, majority baseline

Question: How are the EAU text spans annotated?
Answer: Answer with content missing: (Data and pre-processing section) The data is suited for our experiments because the annotators were explicitly asked to provide annotations on a clausal level.

Question: How are elementary argumentative units defined?
Answer: Unanswerable