Question: What did the authors use as a measure of annotation reliability?
Evidence: The goal of this study was to annotate documents on a detailed level with respect to an argumentation model. First, we will present the annotation scheme. Second, we will describe the annotation process. Finally, we will evaluate the agreement and draw some conclusions.
Given the theoretical background briefly introduced in section SECREF2 , we motivate our selection of the argumentation model by the following requirements. First, the scope of this work is to capture argumentation within a single document, thus focusing on micro-level models. Second, there should exist empirical evidence that such a model has been used for analyzing argumentation in previous works, so it is likely to be suitable for our purposes of argumentative discourse analysis in user-generated content. Regarding the first requirement, two typical examples of micro-level models are the Toulmin's model BIBREF36 and Walton's schemes BIBREF55 . Let us now elaborate on the second requirement.
Walton's argumentation schemes are claimed to be general and domain independent. Nevertheless, evidence from the computational linguistics field shows that the schemes lack coverage for analyzing real argumentation in natural language texts. In examining real-world political argumentation from BIBREF56 , Walton.2012 found out that 37.1% of the arguments collected did not fit any of the fourteen schemes they chose so they created new schemes ad-hoc. Cabrio2013b selected five argumentation schemes from Walton and map these patterns to discourse relation categories in the Penn Discourse TreeBank (PDTB) BIBREF48 , but later they had to define two new argumentation schemes that they discovered in PDTB. Similarly, Song.et.al.2014 admitted that the schemes are ambiguous and hard to directly apply for annotation, therefore they modified the schemes and created new ones that matched the data.
Although Macagno.Konstantinidou.2012 show several examples of two argumentation schemes applied to few selected arguments in classroom experiments, empirical evidence presented by Anthony.Kim.2014 reveals many practical and theoretical difficulties of annotating dialogues with schemes in classroom deliberation, providing many details on the arbitrary selection of the sub-set of the schemes, the ambiguity of the scheme definitions, concluding that the presence of the authors during the experiment was essential for inferring and identifying the argument schemes BIBREF57 .
Although this model (refer to section SECREF21 ) was designed to be applicable to real-life argumentation, there are numerous studies criticizing both the clarity of the model definition and the differentiation between elements of the model. Ball1994 claims that the model can be used only for the most simple arguments and fails on the complex ones. Also Freeman1991 and other argumentation theorists criticize the usefulness of Toulmin's framework for the description of real-life argumentative texts. However, others have advocated the model and claimed that it can be applied to the people's ordinary argumenation BIBREF58 , BIBREF59 .
A number of studies (outside the field of computational linguistics) used Toulmin's model as their backbone argumentation framework. Chambliss1995 experimented with analyzing 20 written documents in a classroom setting in order to find the argument patterns and parts. Simosi2003 examined employees' argumentation to resolve conflicts. Voss2006 analyzed experts' protocols dealing with problem-solving.
The model has also been used in research on computer-supported collaborative learning. Erduran2004 adapt Toulmin's model for coding classroom argumentative discourse among teachers and students. Stegmann2011 builds on a simplified Toulmin's model for scripted construction of argument in computer-supported collaborative learning. Garcia-Mila2013 coded utterances into categories from Toulmin's model in persuasion and consensus-reaching among students. Weinberger.Fischer.2006 analyze asynchronous discussion boards in which learners engage in an argumentative discourse with the goal to acquire knowledge. For coding the argument dimension, they created a set of argumentative moves based on Toulmin's model. Given this empirical evidence, we decided to build upon the Toulmin's model.
In this annotation task, a sequence of tokens (e.g. a phrase, a sentence, or any arbitrary text span) is labeled with a corresponding argument component (such as the claim, the grounds, and others). There are no explicit relations between these annotation spans as the relations are implicitly encoded in the pragmatic function of the components in the Toulmin's model.
In order to prove the suitability of the Toulmin's model, we analyzed 40 random documents from the gold data persuasive dataset using the original Toulmin's model as presented in section SECREF21 . We took into account sever criteria for assessment, such as frequency of occurrence of the components or their importance for the task. We proposed some modifications of the model based on the following observations.
Authors do not state the degree of cogency (the probability of their claim, as proposed by Toulmin). Thus we omitted qualifier from the model due to its absence in the data.
The warrant as a logical explanation why one should accept the claim given the evidence is almost never stated. As pointed out by BIBREF37 , “data are appealed to explicitly, warrants implicitly.” This observation has also been made by Voss2006. Also, according to [p. 205]Eemeren.et.al.1987, the distinction of warrant is perfectly clear only in Toulmin’s examples, but the definitions fail in practice. We omitted warrant from the model.
Rebuttal is a statement that attacks the claim, thus playing a role of an opposing view. In reality, the authors often attack the presented rebuttals by another counter-rebuttal in order to keep the whole argument's position consistent. Thus we introduced a new component – refutation – which is used for attacking the rebuttal. Annotation of refutation was conditioned of explicit presence of rebuttal and enforced by the annotation guidelines. The chain rebuttal–refutation is also known as the procatalepsis figure in rhetoric, in which the speaker raises an objection to his own argument and then immediately answers it. By doing so, the speaker hopes to strengthen the argument by dealing with possible counter-arguments before the audience can raise them BIBREF43 .
The claim of the argument should always reflect the main standpoint with respect to the discussed controversy. We observed that this standpoint is not always explicitly expressed, but remains implicit and must be inferred by the reader. Therefore, we allow the claim to be implicit. In such a case, the annotators must explicitly write down the (inferred) stance of the author.
By definition, the Toulmin's model is intended to model single argument, with the claim in its center. However, we observed in our data, that some authors elaborate on both sides of the controversy equally and put forward an argument for each side (by argument here we mean the claim and its premises, backings, etc.). Therefore we allow multiple arguments to be annotated in one document. At the same time, we restrained the annotators from creating complex argument hierarchies.
Toulmin's grounds have an equivalent role to a premise in the classical view on an argument BIBREF15 , BIBREF60 in terms that they offer the reasons why one should accept the standpoint expressed by the claim. As this terminology has been used in several related works in the argumentation mining field BIBREF7 , BIBREF61 , BIBREF62 , BIBREF11 , we will keep this convention and denote the grounds as premises.
One of the main critiques of the original Toulmin's model was the vague distinction between grounds, warrant, and backing BIBREF63 , BIBREF64 , BIBREF65 . The role of backing is to give additional support to the warrant, but there is no warrant in our model anymore. However, what we observed during the analysis, was a presence of some additional evidence. Such evidence does not play the role of the grounds (premises) as it is not meant as a reason supporting the claim, but it also does not explain the reasoning, thus is not a warrant either. It usually supports the whole argument and is stated by the author as a certain fact. Therefore, we extended the scope of backing as an additional support to the whole argument.
The annotators were instructed to distinguish between premises and backing, so that premises should cover generally applicable reasons for the claim, whereas backing is a single personal experience or statements that give credibility or attribute certain expertise to the author. As a sanity check, the argument should still make sense after removing backing (would be only considered “weaker”).
We call the model as a modified Toulmin's model. It contains five argument components, namely claim, premise, backing, rebuttal, and refutation. When annotating a document, any arbitrary token span can be labeled with an argument component; the components do not overlap. The spans are not known in advance and the annotator thus chooses the span and the component type at the same time. All components are optional (they do not have to be present in the argument) except the claim, which is either explicit or implicit (see above). If a token span is not labeled by any argument component, it is not considered as a part of the argument and is later denoted as none (this category is not assigned by the annotators).
An example analysis of a forum post is shown in Figure FIGREF65 . Figure FIGREF66 then shows a diagram of the analysis from that example (the content of the argument components was shortened or rephrased).
The annotation experiment was split into three phases. All documents were annotated by three independent annotators, who participated in two training sessions. During the first phase, 50 random comments and forum posts were annotated. Problematic cases were resolved after discussion and the guidelines were refined. In the second phase, we wanted to extend the range of annotated registers, so we selected 148 comments and forum posts as well as 41 blog posts. After the second phase, the annotation guidelines were final.
In the final phase, we extended the range of annotated registers and added newswire articles from the raw corpus in order to test whether the annotation guidelines (and inherently the model) is general enough. Therefore we selected 96 comments/forum posts, 8 blog posts, and 8 articles for this phase. A detailed inter-annotator agreement study on documents from this final phase will be reported in section UID75 .
The annotations were very time-consuming. In total, each annotator spent 35 hours by annotating in the course of five weeks. Discussions and consolidation of the gold data took another 6 hours. Comments and forum posts required on average of 4 minutes per document to annotate, while blog posts and articles on average of 14 minutes per document. Examples of annotated documents from the gold data are listed in Appendix UID158 .
We discarded 11 documents out of the total 351 annotated documents. Five forum posts, although annotated as persuasive in the first annotation study, were at a deeper look a mixture of two or more posts with missing quotations, therefore unsuitable for analyzing argumentation. Three blog posts and two articles were found not to be argumentative (the authors took no stance to the discussed controversy) and one article was an interview, which the current model cannot capture (a dialogical argumentation model would be required).
For each of the 340 documents, the gold standard annotations were obtained using the majority vote. If simple majority voting was not possible (different boundaries of the argument component together with a different component label), the gold standard was set after discussion among the annotators. We will refer to this corpus as the gold standard Toulmin corpus. The distribution of topics and registers in this corpus in shown in Table TABREF71 , and Table TABREF72 presents some lexical statistics.
Based on pre-studies, we set the minimal unit for annotation as token. The documents were pre-segmented using the Stanford Core NLP sentence splitter BIBREF69 embedded in the DKPro Core framework BIBREF70 . Annotators were asked to stick to the sentence level by default and label entire pre-segmented sentences. They should switch to annotations on the token level only if (a) a particular sentence contained more than one argument component, or (b) if the automatic sentence segmentation was wrong. Given the “noise” in user-generated Web data (wrong or missing punctuation, casing, etc.), this was often the case.
Annotators were also asked to rephrase (summarize) each annotated argument component into a simple statement when applicable, as shown in Figure FIGREF66 . This was used as a first sanity checking step, as each argument component is expected to be a coherent discourse unit. For example, if a particular occurrence of a premise cannot be summarized/rephrased into one statement, this may require further splitting into two or more premises.
For the actual annotations, we developed a custom-made web-based application that allowed users to switch between different granularity of argument components (tokens or sentences), to annotate the same document in different argument “dimensions” (logos and pathos), and to write summary for each annotated argument component.
As a measure of annotation reliability, we rely on Krippendorff's unitized alpha ( INLINEFORM0 ) BIBREF71 . To the best of our knowledge, this is the only agreement measure that is applicable when both labels and boundaries of segments are to be annotated.
Although the measure has been used in related annotation works BIBREF61 , BIBREF7 , BIBREF72 , there is one important detail that has not been properly communicated. The INLINEFORM0 is computed over a continuum of the smallest units, such as tokens. This continuum corresponds to a single document in the original Krippendorff's work. However, there are two possible extensions to multiple documents (a corpus), namely (a) to compute INLINEFORM1 for each document first and then report an average value, or (b) to concatenate all documents into one large continuum and compute INLINEFORM2 over it. The first approach with averaging yielded extremely high the standard deviation of INLINEFORM3 (i.e., avg. = 0.253; std. dev. = 0.886; median = 0.476 for the claim). This says that some documents are easy to annotate while others are harder, but interpretation of such averaged value has no evidence either in BIBREF71 or other papers based upon it. Thus we use the other methodology and treat the whole corpus as a single long continuum (which yields in the example of claim 0.541 INLINEFORM4 ).
Table TABREF77 shows the inter-annotator agreement as measured on documents from the last annotation phase (see section UID67 ). The overall INLINEFORM0 for all register types, topics, and argument components is 0.48 in the logos dimension (annotated with the modified Toulmin's model). Such agreement can be considered as moderate by the measures proposed by Landis.Koch.1977, however, direct interpretation of the agreement value lacks consensus BIBREF54 . Similar inter-annotator agreement numbers were achieved in the relevant works in argumentation mining (refer to Table TABREF33 in section SECREF31 ; although most of the numbers are not directly comparable, as different inter-annotator metrics were used on different tasks).
There is a huge difference in INLINEFORM0 regarding the registers between comments + forums posts ( INLINEFORM1 0.60, Table TABREF77 a) and articles + blog posts ( INLINEFORM2 0.09, Table TABREF77 b) in the logos dimension. If we break down the value with respect to the individual argument components, the agreement on claim and premise is substantial in the case of comments and forum posts (0.59 and 0.69, respectively). By contrast, these argument components were annotated only with a fair agreement in articles and blog posts (0.22 and 0.24, respectively).
As can be also observed from Table TABREF77 , the annotation agreement in the logos dimension varies regarding the document topic. While it is substantial/moderate for prayer in schools (0.68) or private vs. public schools (0.44), for some topics it remains rather slight, such as in the case of redshirting (0.14) or mainstreaming (0.08).
First, we examine the disagreement in annotations by posing the following research question: are there any measurable properties of the annotated documents that might systematically cause low inter-annotator agreement? We use Pearson's correlation coefficient between INLINEFORM0 on each document and the particular property under investigation. We investigated the following set of measures.
Full sentence coverage ratio represents a ratio of argument component boundaries that are aligned to sentence boundaries. The value is 1.0 if all annotations in the particular document are aligned to sentences and 0.0 if no annotations match the sentence boundaries. Our hypothesis was that automatic segmentation to sentences was often incorrect, therefore annotators had to switch to the token level annotations and this might have increased disagreement on boundaries of the argument components.
Document length, paragraph length and average sentence length. Our hypotheses was that the length of documents, paragraphs, or sentences negatively affects the agreement.
Readability measures. We tested four standard readability measures, namely Ari BIBREF73 , Coleman-Liau BIBREF74 , Flesch BIBREF75 , and Lix BIBREF76 to find out whether readability of the documents plays any role in annotation agreement.
Correlation results are listed in Table TABREF82 . We observed the following statistically significant ( INLINEFORM0 ) correlations. First, document length negatively correlates with agreement in comments. The longer the comment was the lower the agreement was. Second, average paragraph length negatively correlates with agreement in blog posts. The longer the paragraphs in blogs were, the lower agreement was reached. Third, all readability scores negatively correlate with agreement in the public vs. private school domain, meaning that the more complicated the text in terms of readability is, the lower agreement was reached. We observed no significant correlation in sentence coverage and average sentence length measures. We cannot draw any general conclusion from these results, but we can state that some registers and topics, given their properties, are more challenging to annotate than others.
Another qualitative analysis of disagreements between annotators was performed by constructing a probabilistic confusion matrix BIBREF77 on the token level. The biggest disagreements, as can be seen in Table TABREF85 , is caused by rebuttal and refutation confused with none (0.27 and 0.40, respectively). This is another sign that these two argument components were very hard to annotate. As shown in Table TABREF77 , the INLINEFORM5 was also low – 0.08 for rebuttal and 0.17 for refutation.
We analyzed the annotations and found the following phenomena that usually caused disagreements between annotators.
Each argument component (e.g., premise or backing) should express one consistent and coherent piece of information, for example a single reason in case of the premise (see Section UID73 ). However, the decision whether a longer text should be kept as a single argument component or segmented into multiple components is subjective and highly text-specific.
While rhetorical questions have been researched extensively in linguistics BIBREF78 , BIBREF79 , BIBREF80 , BIBREF81 , their role in argumentation represents a substantial research question BIBREF82 , BIBREF83 , BIBREF84 , BIBREF85 , BIBREF86 . Teninbaum.2011 provides a brief history of rhetorical questions in persuasion. In short, rhetorical questions should provoke the reader. From the perspective of our argumentation model, rhetorical questions might fall both into the logos dimension (and thus be labeled as, e.g., claim, premise, etc.) or into the pathos dimension (refer to Section SECREF20 ). Again, the decision is usually not clear-cut.
As introduced in section UID55 , rebuttal attacks the claim by presenting an opponent's view. In most cases, the rebuttal is again attacked by the author using refutation. From the pragmatic perspective, refutation thus supports the author's stance expressed by the claim. Therefore, it can be easily confused with premises, as the function of both is to provide support for the claim. Refutation thus only takes place if it is meant as a reaction to the rebuttal. It follows the discussed matter and contradicts it. Such a discourse is usually expressed as:
[claim: My claim.] [rebuttal: On the other hand, some people claim XXX which makes my claim wrong.] [refutation: But this is not true, because of YYY.]
However, the author might also take the following defensible approach to formulate the argument:
[rebuttal: Some people claim XXX-1 which makes my claim wrong.] [refutation: But this is not true, because of YYY-1.] [rebuttal: Some people claim XXX-2 which makes my claim wrong.] [refutation: But this is not true, because of YYY-2.] [claim: Therefore my claim.]
If this argument is formulated without stating the rebuttals, it would be equivalent to the following:
[premise: YYY-1.] [premise: YYY-2.] [claim: Therefore my claim.]
This example shows that rebuttal and refutation represent a rhetorical device to produce arguments, but the distinction between refutation and premise is context-dependent and on the functional level both premise and refutation have very similar role – to support the author's standpoint. Although introducing dialogical moves into monological model and its practical consequences, as described above, can be seen as a shortcoming of our model, this rhetoric figure has been identified by argumentation researchers as procatalepsis BIBREF43 . A broader view on incorporating opposing views (or lack thereof) is discussed under the term confirmation bias by BIBREF21 who claim that “[...] people are trying to convince others. They are typically looking for arguments and evidence to confirm their own claim, and ignoring negative arguments and evidence unless they anticipate having to rebut them.” The dialectical attack of possible counter-arguments may thus strengthen one's own argument.
One possible solution would be to refrain from capturing this phenomena completely and to simplify the model to claims and premises, for instance. However, the following example would then miss an important piece of information, as the last two clauses would be left un-annotated. At the same time, annotating the last clause as premise would be misleading, because it does not support the claim (in fact, it supports it only indirectly by attacking the rebuttal; this can be seen as a support is considered as an admissible extension of abstract argument graph by BIBREF87 ).
Doc#422 (forumpost, homeschooling) [claim: I try not to be anti-homeschooling, but... it's just hard for me.] [premise: I really haven't met any homeschoolers who turned out quite right, including myself.] I apologize if what I'm saying offends any of you - that's not my intention, [rebuttal: I know that there are many homeschooled children who do just fine,] but [refutation: that hasn't been my experience.]
To the best of our knowledge, these context-dependent dialogical properties of argument components using Toulmin's model have not been solved in the literature on argumentation theory and we suggest that these observations should be taken into account in the future research in monological argumentation.
Appeal to emotion, sarcasm, irony, or jokes are common in argumentation in user-generated Web content. We also observed documents in our data that were purely sarcastic (the pathos dimension), therefore logical analysis of the argument (the logos dimension) would make no sense. However, given the structure of such documents, some claims or premises might be also identified. Such an argument is a typical example of fallacious argumentation, which intentionally pretends to present a valid argument, but its persuasion is conveyed purely for example by appealing to emotions of the reader BIBREF88 .
We present some statistics of the annotated data that are important from the argumentation research perspective. Regardless of the register, 48% of claims are implicit. This means that the authors assume that their standpoint towards the discussed controversy can be inferred by the reader and give only reasons for that standpoint. Also, explicit claims are mainly written just once, only in 3% of the documents the claim was rephrased and occurred multiple times.
In 6% of the documents, the reasons for an implicit claim are given only in the pathos dimension, making the argument purely persuasive without logical argumentation.
The “myside bias”, defined as a bias against information supporting another side of an argument BIBREF89 , BIBREF90 , can be observed by the presence of rebuttals to the author's claim or by formulating arguments for both sides when the overall stance is neutral. While 85% of the documents do not consider any opposing side, only 8% documents present a rebuttal, which is then attacked by refutation in 4% of the documents. Multiple rebuttals and refutations were found in 3% of the documents. Only 4% of the documents were overall neutral and presented arguments for both sides, mainly in blog posts.
We were also interested whether mitigating linguistic devices are employed in the annotated arguments, namely in their main stance-taking components, the claims. Such devices typically include parenthetical verbs, syntactic constructions, token agreements, hedges, challenge questions, discourse markers, and tag questions, among others BIBREF91 . In particular, [p. 1]Kaltenbock.et.al.2010 define hedging as a discourse strategy that reduces the force or truth of an utterance and thus reduces the risk a speaker runs when uttering a strong or firm assertion or other speech act. We manually examined the use of hedging in the annotated claims.
Our main observation is that hedging is used differently across topics. For instance, about 30-35% of claims in homeschooling and mainstreaming signal the lack of a full commitment to the expressed stance, in contrast to prayer in schools (15%) or public vs. private schools (about 10%). Typical hedging cues include speculations and modality (“If I have kids, I will probably homeschool them.”), statements as neutral observations (“It's not wrong to hold the opinion that in general it's better for kids to go to school than to be homeschooled.”), or weasel phrases BIBREF92 (“In some cases, inclusion can work fantastically well.”, “For the majority of the children in the school, mainstream would not have been a suitable placement.”).
On the other hand, most claims that are used for instance in the prayer in schools arguments are very direct, without trying to diminish its commitment to the conveyed belief (for example, “NO PRAYER IN SCHOOLS!... period.”, “Get it out of public schools”, “Pray at home.”, or “No organized prayers or services anywhere on public school board property - FOR ANYONE.”). Moreover, some claims are clearly offensive, persuading by direct imperative clauses towards the opponents/audience (“TAKE YOUR KIDS PRIVATE IF YOU CARE AS I DID”, “Run, don't walk, to the nearest private school.”) or even accuse the opponents for taking a certain stance (“You are a bad person if you send your children to private school.”).
These observations are consistent with the findings from the first annotation study on persuasion (see section UID48 ), namely that some topics attract heated argumentation where participant take very clear and reserved standpoints (such as prayer in schools or private vs. public schools), while discussions about other topics are rather milder. It has been shown that the choices a speaker makes to express a position are informed by their social and cultural background, as well as their ability to speak the language BIBREF93 , BIBREF94 , BIBREF91 . However, given the uncontrolled settings of the user-generated Web content, we cannot infer any similar conclusions in this respect.
We investigated premises across all topics in order to find the type of support used in the argument. We followed the approach of Park.Cardie.2014, who distinguished three types of propositions in their study, namely unverifiable, verifiable non-experiential, and verifiable experiential.
Verifiable non-experiential and verifiable experiential propositions, unlike unverifiable propositions, contain an objective assertion, where objective means “expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations.” Such assertions have truth values that can be proved or disproved with objective evidence; the correctness of the assertion or the availability of the objective evidence does not matter BIBREF8 . A verifiable proposition can further be distinguished as experiential or not, depending on whether the proposition is about the writer's personal state or experience or something non-experiential. Verifiable experiential propositions are sometimes referred to as anectotal evidence, provide the novel knowledge that readers are seeking BIBREF8 .
Table TABREF97 shows the distribution of the premise types with examples for each topic from the annotated corpus. As can be seen in the first row, arguments in prayer in schools contain majority (73%) of unverifiable premises. Closer examination reveals that their content vary from general vague propositions to obvious fallacies, such as a hasty generalization, straw men, or slippery slope. As Nieminen.Mustonen.2014 found out, fallacies are very common in argumentation about religion-related issues. On the other side of the spectrum, arguments about redshirting rely mostly on anecdotal evidence (61% of verifiable experiential propositions). We will discuss the phenomena of narratives in argumentation in more detail later in section UID98 . All the topics except private vs. public schools exhibit similar amount of verifiable non-experiential premises (9%–22%), usually referring to expert studies or facts. However, this type of premises has usually the lowest frequency.
Manually analyzing argumentative discourse and reconstructing (annotating) the underlying argument structure and its components is difficult. As [p. 267]Reed2006 point out, “the analysis of arguments is often hard, not only for students, but for experts too.” According to [p. 81]Harrell.2011b, argumentation is a skill and “even for simple arguments, untrained college students can identify the conclusion but without prompting are poor at both identifying the premises and how the premises support the conclusion.” [p. 81]Harrell.2011 further claims that “a wide literature supports the contention that the particular skills of understanding, evaluating, and producing arguments are generally poor in the population of people who have not had specific training and that specific training is what improves these skills.” Some studies, for example, show that students perform significantly better on reasoning tasks when they have learned to identify premises and conclusions BIBREF95 or have learned some standard argumentation norms BIBREF96 .
One particular extra challenge in analyzing argumentation in Web user-generated discourse is that the authors produce their texts probably without any existing argumentation theory or model in mind. We assume that argumentation or persuasion is inherent when users discuss controversial topics, but the true reasons why people participate in on-line communities and what drives their behavior is another research question BIBREF97 , BIBREF98 , BIBREF99 , BIBREF100 . When the analyzed texts have a clear intention to produce argumentative discourse, such as in argumentative essays BIBREF7 , the argumentation is much more explicit and a substantially higher inter-annotator agreement can be achieved.
The model seems to be suitable for short persuasive documents, such as comments and forum posts. Its applicability to longer documents, such as articles or blog posts, is problematic for several reasons.
The argument components of the (modified) Toulmin's model and their roles are not expressive enough to capture argumentation that not only conveys the logical structure (in terms of reasons put forward to support the claim), but also relies heavily on the rhetorical power. This involves various stylistic devices, pervading narratives, direct and indirect speech, or interviews. While in some cases the argument components are easily recognizable, the vast majority of the discourse in articles and blog posts does not correspond to any distinguishable argumentative function in the logos dimension. As the purpose of such discourse relates more to rhetoric than to argumentation, unambiguous analysis of such phenomena goes beyond capabilities of the current argumentation model. For a discussion about metaphors in Toulmin's model of argumentation see, e.g., BIBREF102 , BIBREF103 .
Articles without a clear standpoint towards the discussed controversy cannot be easily annotated with the model either. Although the matter is viewed from both sides and there might be reasons presented for either of them, the overall persuasive intention is missing and fitting such data to the argumentation framework causes disagreements. One solution might be to break the document down to paragraphs and annotate each paragraph separately, examining argumentation on a different level of granularity.
As introduced in section SECREF20 , there are several dimensions of an argument. The Toulmin's model focuses solely on the logos dimension. We decided to ignore the ethos dimension, because dealing with the author's credibility remains unclear, given the variety of the source web data. However, exploiting the pathos dimension of an argument is prevalent in the web data, for example as an appeal to emotions. Therefore we experimented with annotating appeal to emotions as a separate category independent of components in the logos dimension. We defined some features for the annotators how to distinguish appeal to emotions. Figurative language such as hyperbole, sarcasm, or obvious exaggerating to “spice up” the argument are the typical signs of pathos. In an extreme case, the whole argument might be purely emotional, as in the following example.
Doc#1698 (comment, prayer in schools) [app-to-emot: Prayer being removed from school is just the leading indicator of a nation that is ‘Falling Away’ from Jehovah. [...] And the disasters we see today are simply God’s finger writing on the wall: Mene, mene, Tekel, Upharsin; that is, God has weighed America in the balances, and we’ve been found wanting. No wonder 50 million babies have been aborted since 1973. [...]]
We kept annotations on the pathos dimension as simple as possible (with only one appeal to emotions label), but the resulting agreement was unsatisfying ( INLINEFORM0 0.30) even after several annotation iterations. Appeal to emotions is considered as a type of fallacy BIBREF104 , BIBREF18 . Given the results, we assume that more carefully designed approach to fallacy annotation should be applied. To the best of our knowledge, there have been very few research works on modeling fallacies similarly to arguments on the discourse level BIBREF105 . Therefore the question, in which detail and structure fallacies should be annotated, remains open. For the rest of the paper, we thus focus on the logos dimension solely.
Some of the educational topics under examination relate to young children (e.g., redshirting or mainstreaming); therefore we assume that the majority of participants in discussions are their parents. We observed that many documents related to these topics contain narratives. Sometimes the story telling is meant as a support for the argument, but there are documents where the narrative has no intention to persuade and is simply a story sharing.
There is no widely accepted theory of the role of narratives among argumentation scholars. According to Fisher.1987, humans are storytellers by nature, and the “reason” in argumentation is therefore better understood in and through the narratives. He found that good reasons often take the form of narratives. Hoeken.Fikkers.2014 investigated how integration of explicit argumentative content into narratives influences issue-relevant thinking and concluded that identifying with the character being in favor of the issue yielded a more positive attitude toward the issue. In a recent research, Bex.2011 proposes an argumentative-narrative model of reasoning with evidence, further elaborated in BIBREF106 ; also Niehaus.et.al.2012 proposes a computational model of narrative persuasion.
Stemming from another research field, LeytonEscobar2014 found that online community members who use and share narratives have higher participation levels and that narratives are useful tools to build cohesive cultures and increase participation. Betsch.et.al.2010 examined influencing vaccine intentions among parents and found that narratives carry more weight than statistics.
As a measure of annotation reliability, we rely on Krippendorff’s unitized alpha (αU) (Krippendorff 2004).
Answer: The authors use Krippendorff’s unitized alpha (αU) as the measure of annotation reliability.