Question: What is the Araucaria presented by Reed and Rowe in 2004?
Evidence: The argumentation mining field has been evolving very rapidly in the recent years, resulting into several workshops co-located with major NLP conferences. We first present related works with a focus on annotations and then review experiments with classifying argument components, schemes, or relations.
One of the first papers dealing with annotating argumentative discourse was Argumentative Zoning for scientific publications BIBREF42 . Later, Teufel.et.al.2009 extended the original 7 categories to 15 and annotated 39 articles from two domains, where each sentence is assigned a category. The obtained Fleiss' INLINEFORM0 was 0.71 and 0.65. In their approach, they tried to deliberately ignore the domain knowledge and rely only on general, rhetorical and logical aspect of the annotated texts. By contrast to our work, argumentative zoning is specific to scientific publications and has been developed solely for that task.
Reed.Rowe.2004 presented Araucaria, a tool for argumentation diagramming which supports both convergent and linked arguments, missing premises (enthymemes), and refutations. They also released the AracuariaDB corpus which has later been used for experiments in the argumentation mining field. However, the creation of the dataset in terms of annotation guidelines and reliability is not reported – these limitations as well as its rather small size have been identified BIBREF10 .
Biran.Rambow.2011 identified justifications for subjective claims in blog threads and Wikipedia talk pages. The data were annotated with claims and their justifications reaching INLINEFORM0 0.69, but a detailed description of the annotation approach was missing.
[p. 1078]Schneider.et.al.2013b annotated Wikipedia talk pages about deletion using 17 Walton's schemes BIBREF43 , reaching a moderate agreement (Cohen's INLINEFORM0 0.48) and concluded that their analysis technique can be reused, although “it is intensive and difficult to apply.”
Stab.Gurevych.2014 annotated 90 argumentative essays (about 30k tokens), annotating claims, major claims, and premises and their relations (support, attack). They reached Krippendorff's INLINEFORM0 0.72 for argument components and Krippendorff's INLINEFORM1 0.81 for relations between components.
Rosenthal2012 annotated sentences that are opinionated claims, in which the author expresses a belief that should be adopted by others. Two annotators labeled sentences as claims without any context and achieved Cohen's INLINEFORM0 0.50 (2,000 sentences from LiveJournal) and 0.56 (2,000 sentences from Wikipedia).
Aharoni.et.al.2014 performed an annotation study in order to find context-dependent claims and three types of context-dependent evidence in Wikipedia, that were related to 33 controversial topics. The claim and evidence were annotated in 104 articles. The average Cohen's INLINEFORM0 between a group of 20 expert annotators was 0.40. Compared to our work, the linguistic properties of Wikipedia are qualitatively different from other user-generated content, such as blogs or user comments BIBREF44 .
Wacholder.et.al.2014 annotated “argument discourse units” in blog posts and criticized the Krippendorff's INLINEFORM0 measure. They proposed a new inter-annotator metric by taking the most overlapping part of one annotation as the “core” and all annotations as a “cluster”. The data were extended by Ghosh2014, who annotated “targets” and “callouts” on the top of the units.
Park.Cardie.2014 annotated about 10k sentences from 1,047 documents into four types of argument propositions with Cohen's INLINEFORM0 0.73 on 30% of the dataset. Only 7% of the sentences were found to be non-argumentative.
Faulkner2014 used Amazon Mechanical Turk to annotate 8,179 sentences from student essays. Three annotators decided whether the given sentence offered reasons for or against the main prompt of the essay (or no reason at all; 66% of the sentences were found to be neutral and easy to identify). The achieved Cohen's INLINEFORM0 was 0.70.
The research has also been active on non-English datasets. Goudas.et.al.2014 focused on user-generated Greek texts. They selected 204 documents and manually annotated sentences that contained an argument (760 out of 16,000). They distinguished claims and premises, but the claims were always implicit. However, the annotation agreement was not reported, neither was the number of annotators or the guidelines. A study on annotation of arguments was conducted by Peldszus.Stede.2013, who evaluate agreement among 26 “naive" annotators (annotators with very little training). They manually constructed 23 German short texts, each of them contains exactly one central claim, two premises, and one objection (rebuttal or undercut) and analyzed annotator agreement on this artificial data set. Peldszus.2014 later achieved higher inter-rater agreement with expert annotators on an extended version of the same data. Kluge.2014 built a corpus of argumentative German Web documents, containing 79 documents from 7 educational topics, which were annotated by 3 annotators according to the claim-premise argumentation model. The corpus comprises 70,000 tokens and the inter-annotator agreement was 0.40 (Krippendorff's INLINEFORM0 ). Houy.et.al.2013 targeted argumentation mining of German legal cases.
Table TABREF33 gives an overview of annotation studies with their respective argumentation model, domain, size, and agreement. It also contains other studies outside of computational linguistics and few proposals and position papers.
Arguments in the legal domain were targeted in BIBREF11 . Using argumentation formalism inspired by Walton.2012, they employed multinomial Naive Bayes classifier and maximum entropy model for classifying argumentative sentences on the AraucariaDB corpus BIBREF45 . The same test dataset was used by Feng.Hirst.2011, who utilized the C4.5 decision classifier. Rooney.et.al.2012 investigated the use of convolution kernel methods for classifying whether a sentence belongs to an argumentative element or not using the same corpus.
Stab.Gurevych.2014b classified sentences to four categories (none, major claim, claim, premise) using their previously annotated corpus BIBREF7 and reached 0.72 macro- INLINEFORM0 score. In contrast to our work, their documents are expected to comply with a certain structure of argumentative essays and are assumed to always contain argumentation.
Biran.Rambow.2011 identified justifications on the sentence level using a naive Bayes classifier over a feature set based on statistics from the RST Treebank, namely n-grams which were manually processed by deleting n-grams that “seemed irrelevant, ambiguous or domain-specific.”
Llewellyn2014 experimented with classifying tweets into several argumentative categories, namely claims and counter-claims (with and without evidence) and verification inquiries previously annotated by Procter.et.al.2013. They used unigrams, punctuations, and POS as features in three classifiers.
Park.Cardie.2014 classified propositions into three classes (unverifiable, verifiable non-experimental, and verifiable experimental) and ignored non-argumentative texts. Using multi-class SVM and a wide range of features (n-grams, POS, sentiment clue words, tense, person) they achieved Macro INLINEFORM0 0.69.
Peldszus.2014 experimented with a rather complex labeling schema of argument segments, but their data were artificially created for their task and manually cleaned, such as removing segments that did not meet the criteria or non-argumentative segments.
In the first step of their two-phase approach, Goudas.et.al.2014 sampled the dataset to be balanced and identified argumentative sentences with INLINEFORM0 0.77 using the maximum entropy classifier. For identifying premises, they used BIO encoding of tokens and achieved INLINEFORM1 score 0.42 using CRFs.
Saint-Dizier.2012 developed a Prolog engine using a lexicon of 1300 words and a set of 78 hand-crafted rules with the focus on a particular argument structure “reasons supporting conclusions” in French.
Taking the dialogical perspective, Cabrio.Villata.2012 built upon an argumentation framework proposed by Dung.1995 which models arguments within a graph structure and provides a reasoning mechanism for resolving accepted arguments. For identifying support and attack, they relied on existing research on textual entailment BIBREF46 , namely using the off-the-shelf EDITS system. The test data were taken from a debate portal Debatepedia and covered 19 topics. Evaluation was performed in terms of measuring the acceptance of the “main argument" using the automatically recognized entailments, yielding INLINEFORM0 score about 0.75. By contrast to our work which deals with micro-level argumentation, the Dung's model is an abstract framework intended to model dialogical argumentation.
Finding a bridge between existing discourse research and argumentation has been targeted by several researchers. Peldszus2013a surveyed literature on argumentation and proposed utilization of Rhetorical Structure Theory (RST) BIBREF47 . They claimed that RST is by its design well-suited for studying argumentative texts, but an empirical evidence has not yet been provided. Penn Discourse Tree Bank (PDTB) BIBREF48 relations have been under examination by argumentation mining researchers too. Cabrio2013b examined a connection between five Walton's schemes and discourse markers in PDTB, however an empirical evaluation is missing.
Reed and Rowe (2004) presented Araucaria, a tool for argumentation diagramming that supports both convergent and linked arguments, missing premises (enthymemes), and refutations.
Answer: a tool for argumentation diagramming that supports both convergent and linked arguments, missing premises (enthymemes), and refutations.