prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 1: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 
Context 2: Introduction
Relation classification is the task of assigning sentences with two marked entities to a predefined set of relations  The sentence “We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2> ”, for example, expresses the relation Entity-Destination(e1,e2)  While early research mostly focused on support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 , recent research showed performance improvements by applying neural networks (NNs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 on the benchmark data from SemEval 2010 shared task 8 BIBREF8  
This study investigates two different types of NNs: recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as well as their combination  We make the following contributions:
(1) We propose extended middle context, a new context representation for CNNs for relation classification  The extended middle context uses all parts of the sentence (the relation arguments, left of the relation arguments, between the arguments, right of the arguments) and pays special attention to the middle part 
(2) We present connectionist bi-directional RNN models which are especially suited for sentence classification tasks since they combine all intermediate hidden layers for their final decision  Furthermore, the ranking loss function is introduced for the RNN model optimization which has not been investigated in the literature for relation classification before 
(3) Finally, we combine CNNs and RNNs using a simple voting scheme and achieve new state-of-the-art results on the SemEval 2010 benchmark dataset 
Related Work
In 2010, manually annotated data for relation classification was released in the context of a SemEval shared task BIBREF8   Shared task participants used, i a , support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1   Recently, their results on this data set were outperformed by applying NNs BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6  
zeng2014 built a CNN based only on the context between the relation arguments and extended it with several lexical features  kim2014 and others used convolutional filters of different sizes for CNNs  nguyen applied this to relation classification and obtained improvements over single filter sizes  deSantos2015 replaced the softmax layer of the CNN with a ranking layer  They showed improvements and published the best result so far on the SemEval dataset, to our knowledge 
socher used another NN architecture for relation classification: recursive neural networks that built recursive sentence representations based on syntactic parsing  In contrast, zhang investigated a temporal structured RNN with only words as input  They used a bi-directional model with a pooling layer on top 
Convolutional Neural Networks (CNN)
CNNs perform a discrete convolution on an input matrix with a set of different filters  For NLP tasks, the input matrix represents a sentence: Each column of the matrix stores the word embedding of the corresponding word  By applying a filter with a width of, e g , three columns, three neighboring words (trigram) are convolved  Afterwards, the results of the convolution are pooled  Following collobertWeston, we perform max-pooling which extracts the maximum value for each filter and, thus, the most informative n-gram for the following steps  Finally, the resulting values are concatenated and used for classifying the relation expressed in the sentence 
Input: Extended Middle Context
One of our contributions is a new input representation especially designed for relation classification  The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context  Since in most cases the middle context contains the most relevant information for the relation, we want to focus on it but not ignore the other regions completely  Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context  Due to the repetition of the middle context, we force the network to pay special attention to it  The two contexts are processed by two independent convolutional and max-pooling layers  After pooling, the results are concatenated to form the sentence representation  Figure FIGREF3 depicts this procedure  It shows an examplary sentence: “He had chest pain and <e1>headaches</e1> from <e2>mold</e2> in the bedroom ” If we only considered the middle context “from”, the network might be tempted to predict a relation like Entity-Origin(e1,e2)  However, by also taking the left and right context into account, the model can detect the relation Cause-Effect(e2,e1)  While this could also be achieved by integrating the whole context into the model, using the whole context can have disadvantages for longer sentences: The max pooling step can easily choose a value from a part of the sentence which is far away from the mention of the relation  With splitting the context into two parts, we reduce this danger  Repeating the middle context increases the chance for the max pooling step to pick a value from the middle context 
Convolutional Layer
Following previous work (e g , BIBREF5 , BIBREF6 ), we use 2D filters spanning all embedding dimensions  After convolution, a max pooling operation is applied that stores only the highest activation of each filter  We apply filters with different window sizes 2-5 (multi-windows) as in BIBREF5 , i e  spanning a different number of input words 
Recurrent Neural Networks (RNN)
Traditional RNNs consist of an input vector, a history vector and an output vector  Based on the representation of the current input word and the previous history vector, a new history is computed  Then, an output is predicted (e g , using a softmax layer)  In contrast to most traditional RNN architectures, we use the RNN for sentence modeling, i e , we predict an output vector only after processing the whole sentence and not after each word  Training is performed using backpropagation through time BIBREF9 which unfolds the recurrent computations of the history vector for a certain number of time steps  To avoid exploding gradients, we use gradient clipping with a threshold of 10 BIBREF10  
Input of the RNNs
Initial experiments showed that using trigrams as input instead of single words led to superior results  Hence, at timestep INLINEFORM0 we do not only give word INLINEFORM1 to the model but the trigram INLINEFORM2 by concatenating the corresponding word embeddings 
Connectionist Bi-directional RNNs
Especially for relation classification, the processing of the relation arguments might be easier with knowledge of the succeeding words  Therefore in bi-directional RNNs, not only a history vector of word INLINEFORM0 is regarded but also a future vector  This leads to the following conditioned probability for the history INLINEFORM1 at time step INLINEFORM2 : DISPLAYFORM0
Thus, the network can be split into three parts: a forward pass which processes the original sentence word by word (Equation EQREF6 ); a backward pass which processes the reversed sentence word by word (Equation ); and a combination of both (Equation )  All three parts are trained jointly  This is also depicted in Figure FIGREF7  
Combining forward and backward pass by adding their hidden layer is similar to BIBREF7   We, however, also add a connection to the previous combined hidden layer with weight INLINEFORM0 to be able to include all intermediate hidden layers into the final decision of the network (see Equation )  We call this “connectionist bi-directional RNN” 
In our experiments, we compare this RNN with uni-directional RNNs and bi-directional RNNs without additional hidden layer connections 
Word Representations
Words are represented by concatenated vectors: a word embedding and a position feature vector 
Pretrained word embeddings  In this study, we used the word2vec toolkit BIBREF11 to train embeddings on an English Wikipedia from May 2014
Context 3: Introduction
The art of argumentation has been studied since the early work of Aristotle, dating back to the 4th century BC BIBREF0   It has been exhaustively examined from different perspectives, such as philosophy, psychology, communication studies, cognitive science, formal and informal logic, linguistics, computer science, educational research, and many others  In a recent and critically well-acclaimed study, Mercier Sperber 2011 even claim that argumentation is what drives humans to perform reasoning  From the pragmatic perspective, argumentation can be seen as a verbal activity oriented towards the realization of a goal BIBREF1 or more in detail as a verbal, social, and rational activity aimed at convincing a reasonable critic of the acceptability of a standpoint by putting forward a constellation of one or more propositions to justify this standpoint BIBREF2  
Analyzing argumentation from the computational linguistics point of view has very recently led to a new field called argumentation mining BIBREF3   Despite the lack of an exact definition, researchers within this field usually focus on analyzing discourse on the pragmatics level and applying a certain argumentation theory to model and analyze textual data at hand 
Our motivation for argumentation mining stems from a practical information seeking perspective from the user-generated content on the Web  For example, when users search for information in user-generated Web content to facilitate their personal decision making related to controversial topics, they lack tools to overcome the current information overload  One particular use-case example dealing with a forum post discussing private versus public schools is shown in Figure FIGREF4   Here, the lengthy text on the left-hand side is transformed into an argument gist on the right-hand side by (i) analyzing argument components and (ii) summarizing their content  Figure FIGREF5 shows another use-case example, in which users search for reasons that underpin certain standpoint in a given controversy (which is homeschooling in this case)  In general, the output of automatic argument analysis performed on the large scale in Web data can provide users with analyzed arguments to a given topic of interest, find the evidence for the given controversial standpoint, or help to reveal flaws in argumentation of others 
Satisfying the above-mentioned information needs cannot be directly tackled by current methods for, e g , opinion mining, questions answering, or summarization and requires novel approaches within the argumentation mining field  Although user-generated Web content has already been considered in argumentation mining, many limitations and research gaps can be identified in the existing works  First, the scope of the current approaches is restricted to a particular domain or register, e g , hotel reviews BIBREF5 , Tweets related to local riot events BIBREF6 , student essays BIBREF7 , airline passenger rights and consumer protection BIBREF8 , or renewable energy sources BIBREF9   Second, not all the related works are tightly connected to argumentation theories, resulting into a gap between the substantial research in argumentation itself and its adaptation in NLP applications  Third, as an emerging research area, argumentation mining still suffers from a lack of labeled corpora, which is crucial for designing, training, and evaluating the algorithms  Although some works have dealt with creating new data sets, the reliability (in terms of inter-annotator agreement) of the annotated resources is often unknown BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14  
Annotating and automatically analyzing arguments in unconstrained user-generated Web discourse represent challenging tasks  So far, the research in argumentation mining “has been conducted on domains like news articles, parliamentary records and legal documents, where the documents contain well-formed explicit arguments, i e , propositions with supporting reasons and evidence present in the text” BIBREF8   [p  50]Boltuzic Snajder 2014 point out that “unlike in debates or other more formal argumentation sources, the arguments provided by the users, if any, are less formal, ambiguous, vague, implicit, or often simply poorly worded ” Another challenge stems from the different nature of argumentation theories and computational linguistics  Whereas computational linguistics is mainly descriptive, the empirical research that is carried out in argumentation theories does not constitute a test of the theoretical model that is favored, because the model of argumentation is a normative instrument for assessing the argumentation BIBREF15   So far, no fully fledged descriptive argumentation theory based on empirical research has been developed, thus feasibility of adapting argumentation models to the Web discourse represents an open issue 
These challenges can be formulated into the following research questions:
In this article, we push the boundaries of the argumentation mining field by focusing on several novel aspects  We tackle the above-mentioned research questions as well as the previously discussed challenges and issues  First, we target user-generated Web discourse from several domains across various registers, to examine how argumentation is communicated in different contexts  Second, we bridge the gap between argumentation theories and argumentation mining through selecting the argumenation model based on research into argumentation theories and related fields in communication studies or psychology  In particular, we adapt normative models from argumentation theory to perform empirical research in NLP and support our application of argumentation theories with an in-depth reliability study  Finally, we use state-of-the-art NLP techniques in order to build robust computational models for analyzing arguments that are capable of dealing with a variety of genres on the Web 
Our contributions
We create a new corpus which is, to the best of our knowledge, the largest corpus that has been annotated within the argumentation mining field to date  We choose several target domains from educational controversies, such as homeschooling, single-sex education, or mainstreaming  A novel aspect of the corpus is its coverage of different registers of user-generated Web content, such as comments to articles, discussion forum posts, blog posts, as well as professional newswire articles 
Since the data come from a variety of sources and no assumptions about its actual content with respect to argumentation can be drawn, we conduct two extensive annotation studies  In the first study, we tackle the problem of relatively high “noise” in the retrieved data  In particular, not all of the documents are related to the given topics in a way that makes them candidates for further deep analysis of argumentation (this study results into 990 annotated documents)  In the second study, we discuss the selection of an appropriate argumentation model based on evidence in argumentation research and propose a model that is suitable for analyzing micro-level argumention in user-generated Web content  Using this model, we annotate 340 documents (approx  90,000 tokens), reaching a substantial inter-annotator agreement  We provide a hand-analysis of all the phenomena typical to argumentation that are prevalent in our data  These findings may also serve as empirical evidence to issues that are on the spot of current argumentation research 
From the computational perspective, we experiment on the annotated data using various machine learning methods in order to extract argument structure from documents  We propose several novel feature sets and identify configurations that run best in in-domain and cross-domain scenarios  To foster research in the community, we provide the annotated data as well as all the experimental software under free license 
The rest of the article is structured as follows  First, we provide an essential background in argumentation theory in section SECREF2   Section SECREF3 surveys related work in several areas  Then we introduce the dataset and two annotation studies in section SECREF4   Section SECREF5 presents our experimental work and discusses the results and errors and section SECREF6 concludes this article 
Theoretical background
Let us first present some definitions of the term argumentation itself  [p  3]Ketcham 1917 defines argumentation as “the art of persuading others to think or act in a definite way  It includes all writing and speaking which is persuasive in form ” According to MacEwan 1898, “argumentation is the process of proving or disproving a proposition  Its purpose is to induce a new belief, to establish truth or combat error in the mind of another ” [p  2]Freeley Steinberg 2008 narrow the scope of argumentation to “reason giving in communicative situations by people whose purpose is the justification of acts, beliefs, attitudes, and values ” Although these definitions vary, the purpose of argumentation remains the same – to persuade others 
We would like to stress that our perception of argumentation goes beyond somehow limited giving reasons BIBREF17 , BIBREF18   Rather, we see the goal of argumentation as to persuade BIBREF19 , BIBREF20 , BIBREF21   Persuasion can be defined as a successful intentional effort at influencing another's mental state through communication in a circumstance in which the persuadee has some measure of freedom BIBREF22 , although, as OKeefe2011 points out, there is no correct or universally-endorsed definition of either `persuasion' or `argumentation'  However, broader understanding of argumentation as a means of persuasion allows us to take into account not only reasoned discourse, but also non-reasoned mechanisms of influence, such as emotional appeals BIBREF23  
Having an argument as a product within the argumentation process, we should now define it  One typical definition is that an argument is a claim supported by reasons BIBREF24   The term claim has been used since 1950's, introduced by Toulmin 1958, and in argumentation theory it is a synonym for standpoint or point of view  It refers to what is an issue in the sense what is being argued about  The presence of a standpoint is thus crucial for argumentation analysis  However, the claim as well as other parts of the argument might be implicit; this is known as enthymematic argumentation, which is rather usual in ordinary argumentative discourse BIBREF25  
One fundamental problem with the definition and formal description of arguments and argumentation is that there is no agreement even among argumentation theorists  As [p  29]vanEmeren et al 2014 admit in their very recent and exhaustive survey of the field, ”as yet, there is no unitary theory of argumentation that encompasses the logical, dialectical, and rhetorical dimensions of argumentation and is universally accepted  The current state of the art in argumentation theory is characterized by the coexistence of a variety of theoretical perspectives and approaches, which differ considerably from each other in conceptualization, scope, and theoretical refinement ”
Argumentation models
Despite the missing consensus on the ultimate argumentation theory, various argumentation models have been proposed that capture argumentation on different levels  Argumentation models abstract from the language level to a concept level that stresses the links between the different components of an argument or how arguments relate to each other BIBREF26   Bentahar et al 2010 propose a taxonomy of argumentation models, that is horizontally divided into three categories – micro-level models, macro-level models, and rhetorical models 
In this article, we deal with argumentation on the micro-level (also called argumentation as a product or monological models)  Micro-level argumentation focuses on the structure of a single argument  By contrast, macro-level models (also called dialogical models) and rhetorical models highlight the process of argumentation in a dialogue BIBREF27   In other words, we examine the structure of a single argument produced by a single author in term of its components, not the relations that can exist among arguments and their authors in time  A detailed discussion of these different perspectives can be found, e g , in BIBREF28 , BIBREF29 , BIBREF30 , BIBREF1 , BIBREF31 , BIBREF32  
Dimensions of argument
The above-mentioned models focus basically only on one dimension of the argument, namely the logos dimension  According to the classical Aristotle's theory BIBREF0 , argument can exist in three dimensions, which are logos, pathos, and ethos  Logos dimension represents a proof by reason, an attempt to persuade by establishing a logical argument  For example, syllogism belongs to this argumentation dimension BIBREF34 , BIBREF25   Pathos dimension makes use of appealing to emotions of the receiver and impacts its cognition BIBREF35   Ethos dimension of argument relies on the credibility of the arguer  This distinction will have practical impact later in section SECREF51 which deals with argumentation on the Web 
Original Toulmin's model
We conclude the theoretical section by presenting one (micro-level) argumentation model in detail – a widely used conceptual model of argumentation introduced by Toulmin 1958, which we will henceforth denote as the Toulmin's original model  This model will play an important role later in the annotation studies (section SECREF51 ) and experimental work (section SECREF108 )  The model consists of six parts, referred as argument components, where each component plays a distinct role 
is an assertion put forward publicly for general acceptance BIBREF38 or the conclusion we seek to establish by our arguments BIBREF17  
It is the evidence to establish the foundation of the claim BIBREF24 or, as simply put by Toulmin, “the data represent what we have to go on ” BIBREF37   The name of this concept was later changed to grounds in BIBREF38  
The role of warrant is to justify a logical inference from the grounds to the claim 
is a set of information that stands behind the warrant, it assures its trustworthiness 
limits the degree of certainty under which the argument should be accepted  It is the degree of force which the grounds confer on the claim in virtue of the warrant BIBREF37  
presents a situation in which the claim might be defeated 
A schema of the Toulmin's original model is shown in Figure FIGREF29   The lines and arrows symbolize implicit relations between the components  An example of an argument rendered using the Toulmin's scheme can be seen in Figure FIGREF30  
We believe that this theoretical overview should provide sufficient background for the argumentation mining research covered in this article; for further references, we recommend for example BIBREF15  
Related work in computational linguistics
We structure the related work into three sub-categories, namely argumentation mining, stance detection, and persuasion and on-line dialogs, as these areas are closest to this article's focus  For a recent overview of general discourse analysis see BIBREF39   Apart from these, research on computer-supported argumentation has been also very active; see, e g , BIBREF40 for a survey of various models and argumentation formalisms from the educational perspective or BIBREF41 which examines argumentation in the Semantic Web 
Argumentation Mining
The argumentation mining field has been evolving very rapidly in the recent years, resulting into several workshops co-located with major NLP conferences  We first present related works with a focus on annotations and then review experiments with classifying argument components, schemes, or relations 
One of the first papers dealing with annotating argumentative discourse was Argumentative Zoning for scientific publications BIBREF42   Later, Teufel et al 2009 extended the original 7 categories to 15 and annotated 39 articles from two domains, where each sentence is assigned a category  The obtained Fleiss' INLINEFORM0 was 0 71 and 0 65  In their approach, they tried to deliberately ignore the domain knowledge and rely only on general, rhetorical and logical aspect of the annotated texts  By contrast to our work, argumentative zoning is specific to scientific publications and has been developed solely for that task 
Reed Rowe 2004 presented Araucaria, a tool for argumentation diagramming which supports both convergent and linked arguments, missing premises (enthymemes), and refutations  They also released the AracuariaDB corpus which has later been used for experiments in the argumentation mining field  However, the creation of the dataset in terms of annotation guidelines and reliability is not reported – these limitations as well as its rather small size have been identified BIBREF10  
Biran Rambow 2011 identified justifications for subjective claims in blog threads and Wikipedia talk pages  The data were annotated with claims and their justifications reaching INLINEFORM0 0 69, but a detailed description of the annotation approach was missing 
[p  1078]Schneider et al 2013b annotated Wikipedia talk pages about deletion using 17 Walton's schemes BIBREF43 , reaching a moderate agreement (Cohen's INLINEFORM0 0 48) and concluded that their analysis technique can be reused, although “it is intensive and difficult to apply ”
Stab Gurevych 2014 annotated 90 argumentative essays (about 30k tokens), annotating claims, major claims, and premises and their relations (support, attack)  They reached Krippendorff's INLINEFORM0 0 72 for argument components and Krippendorff's INLINEFORM1 0 81 for relations between components 
Rosenthal2012 annotated sentences that are opinionated claims, in which the author expresses a belief that should be adopted by others  Two annotators labeled sentences as claims without any context and achieved Cohen's INLINEFORM0 0 50 (2,000 sentences from LiveJournal) and 0 56 (2,000 sentences from Wikipedia) 
Aharoni et al 2014 performed an annotation study in order to find context-dependent claims and three types of context-dependent evidence in Wikipedia, that were related to 33 controversial topics  The claim and evidence were annotated in 104 articles  The average Cohen's INLINEFORM0 between a group of 20 expert annotators was 0 40  Compared to our work, the linguistic properties of Wikipedia are qualitatively different from other user-generated content, such as blogs or user comments BIBREF44  
Wacholder et al 2014 annotated “argument discourse units” in blog posts and criticized the Krippendorff's INLINEFORM0 measure  They proposed a new inter-annotator metric by taking the most overlapping part of one annotation as the “core” and all annotations as a “cluster”  The data were extended by Ghosh2014, who annotated “targets” and “callouts” on the top of the units 
Park Cardie 2014 annotated about 10k sentences from 1,047 documents into four types of argument propositions with Cohen's INLINEFORM0 0 73 on 30% of the dataset  Only 7% of the sentences were found to be non-argumentative 
Faulkner2014 used Amazon Mechanical Turk to annotate 8,179 sentences from student essays  Three annotators decided whether the given sentence offered reasons for or against the main prompt of the essay (or no reason at all; 66% of the sentences were found to be neutral and easy to identify)  The achieved Cohen's INLINEFORM0 was 0 70 
The research has also been active on non-English datasets  Goudas et al 2014 focused on user-generated Greek texts  They selected 204 documents and manually annotated sentences that contained an argument (760 out of 16,000)  They distinguished claims and premises, but the claims were always implicit  However, the annotation agreement was not reported, neither was the number of annotators or the guidelines  A study on annotation of arguments was conducted by Peldszus Stede 2013, who evaluate agreement among 26 “naive" annotators (annotators with very little training)  They manually constructed 23 German short texts, each of them contains exactly one central claim, two premises, and one objection (rebuttal or undercut) and analyzed annotator agreement on this artificial data set  Peldszus 2014 later achieved higher inter-rater agreement with expert annotators on an extended version of the same data  Kluge 2014 built a corpus of argumentative German Web documents, containing 79 documents from 7 educational topics, which were annotated by 3 annotators according to the claim-premise argumentation model  The corpus comprises 70,000 tokens and the inter-annotator agreement was 0 40 (Krippendorff's INLINEFORM0 )  Houy et al 2013 targeted argumentation mining of German legal cases 
Table TABREF33 gives an overview of annotation studies with their respective argumentation model, domain, size, and agreement  It also contains other studies outside of computational linguistics and few proposals and position papers 
Arguments in the legal domain were targeted in BIBREF11   Using argumentation formalism inspired by Walton 2012, they employed multinomial Naive Bayes classifier and maximum entropy model for classifying argumentative sentences on the AraucariaDB corpus BIBREF45   The same test dataset was used by Feng Hirst 2011, who utilized the C4 5 decision classifier  Rooney et al 2012 investigated the use of convolution kernel methods for classifying whether a sentence belongs to an argumentative element or not using the same corpus 
Stab Gurevych 2014b classified sentences to four categories (none, major claim, claim, premise) using their previously annotated corpus BIBREF7 and reached 0 72 macro- INLINEFORM0 score  In contrast to our work, their documents are expected to comply with a certain structure of argumentative essays and are assumed to always contain argumentation 
Biran Rambow 2011 identified justifications on the sentence level using a naive Bayes classifier over a feature set based on statistics from the RST Treebank, namely n-grams which were manually processed by deleting n-grams that “seemed irrelevant, ambiguous or domain-specific ”
Llewellyn2014 experimented with classifying tweets into several argumentative categories, namely claims and counter-claims (with and without evidence) and verification inquiries previously annotated by Procter et al 2013  They used unigrams, punctuations, and POS as features in three classifiers 
Park Cardie 2014 classified propositions into three classes (unverifiable, verifiable non-experimental, and verifiable experimental) and ignored non-argumentative texts  Using multi-class SVM and a wide range of features (n-grams, POS, sentiment clue words, tense, person) they achieved Macro INLINEFORM0 0 69 
Peldszus 2014 experimented with a rather complex labeling schema of argument segments, but their data were artificially created for their task and manually cleaned, such as removing segments that did not meet the criteria or non-argumentative segments 
In the first step of their two-phase approach, Goudas et al 2014 sampled the dataset to be balanced and identified argumentative sentences with INLINEFORM0 0 77 using the maximum entropy classifier  For identifying premises, they used BIO encoding of tokens and achieved INLINEFORM1 score 0 42 using CRFs 
Saint-Dizier 2012 developed a Prolog engine using a lexicon of 1300 words and a set of 78 hand-crafted rules with the focus on a particular argument structure “reasons supporting conclusions” in French 
Taking the dialogical perspective, Cabrio Villata 2012 built upon an argumentation framework proposed by Dung 1995 which models arguments within a graph structure and provides a reasoning mechanism for resolving accepted arguments  For identifying support and attack, they relied on existing research on textual entailment BIBREF46 , namely using the off-the-shelf EDITS system  The test data were taken from a debate portal Debatepedia and covered 19 topics  Evaluation was performed in terms of measuring the acceptance of the “main argument" using the automatically recognized entailments, yielding INLINEFORM0 score about 0 75  By contrast to our work which deals with micro-level argumentation, the Dung's model is an abstract framework intended to model dialogical argumentation 
Finding a bridge between existing discourse research and argumentation has been targeted by several researchers  Peldszus2013a surveyed literature on argumentation and proposed utilization of Rhetorical Structure Theory (RST) BIBREF47   They claimed that RST is by its design well-suited for studying argumentative texts, but an empirical evidence has not yet been provided  Penn Discourse Tree Bank (PDTB) BIBREF48 relations have been under examination by argumentation mining researchers too  Cabrio2013b examined a connection between five Walton's schemes and discourse markers in PDTB, however an empirical evaluation is missing 
Stance detection
Research related to argumentation mining also involves stance detection  In this case, the whole document (discussion post, article) is assumed to represent the writer's standpoint to the discussed topic  Since the topic is stated as a controversial question, the author is either for or against it 
Somasundaran Wiebe 2009 built a computational model for recognizing stances in dual-topic debates about named entities in the electronic products domain by combining preferences learned from the Web data and discourse markers from PDTB BIBREF48   Hasan Ng 2013 determined stance in on-line ideological debates on four topics using data from createdebate com, employing supervised machine learning and features ranging from n-grams to semantic frames  Predicting stance of posts in Debatepedia as well as external articles using a probabilistic graphical model was presented in BIBREF49   This approach also employed sentiment lexicons and Named Entity Recognition as a preprocessing step and achieved accuracy about 0 80 in binary prediction of stances in debate posts 
Recent research has involved joint modeling, taking into account information about the users, the dialog sequences, and others  Hasan Ng 2012 proposed machine learning approach to debate stance classification by leveraging contextual information and author's stances towards the topic  Qiu et al 2013 introduced a computational debate side model to cluster posts or users by sides for general threaded discussions using a generative graphical model employing words from various subjectivity lexicons as well as all adjectives and adverbs in the posts  Qiu Jiang 2013 proposed a graphical model for viewpoint discovery in discussion threads  Burfoot et al 2011 exploited the informal citation structure in U S  Congressional floor-debate transcripts and use a collective classification which outperforms methods that consider documents in isolation 
Some works also utilize argumentation-motivated features  Park et al 2011 dealt with contentious issues in Korean newswire discourse  Although they annotate the documents with “argument frames”, the formalism remains unexplained and does not refer to any existing research in argumentation  Walker et al 2012b incorporated features with some limited aspects of the argument structure, such as cue words signaling rhetorical relations between posts, POS generalized dependencies, and a representation of the parent post (context) to improve stance classification over 14 topics from convinceme net 
Online persuasion
Another stream of research has been devoted to persuasion in online media, which we consider as a more general research topic than argumentation 
Schlosser 2011 investigated persuasiveness of online reviews and concluded that presenting two sides is not always more helpful and can even be less persuasive than presenting one side  Mohammadi et al 2013 explored persuasiveness of speakers in YouTube videos and concluded that people are perceived more persuasive in video than in audio and text  Miceli et al 2006 proposed a computational model that attempts to integrate emotional and non-emotional persuasion  In the study of Murphy 2001, persuasiveness was assigned to 21 articles (out of 100 manually preselected) and four of them are later analyzed in detail for comparing the perception of persuasion between expert and students  Bernard et al 2012 experimented with children's perception of discourse connectives (namely with “because”) to link statements in arguments and found out that 4- and 5-years-old and adults are sensitive to the connectives  Le 2004 presented a study of persuasive texts and argumentation in newspaper editorials in French 
A coarse-grained view on dialogs in social media was examined by Bracewell et al 2013, who proposed a set of 15 social acts (such as agreement, disagreement, or supportive behavior) to infer the social goals of dialog participants and presented a semi-supervised model for their classification  Their social act types were inspired by research in psychology and organizational behavior and were motivated by work in dialog understanding  They annotated a corpus in three languages using in-house annotators and achieved INLINEFORM0 in the range from 0 13 to 0 53 
Georgila et al 2011 focused on cross-cultural aspects of persuasion or argumentation dialogs  They developed a novel annotation scheme stemming from different literature sources on negotiation and argumentation as well as from their original analysis of the phenomena  The annotation scheme is claimed to cover three dimensions of an utterance, namely speech act, topic, and response or reference to a previous utterance  They annotated 21 dialogs and reached Krippendorff's INLINEFORM0 between 0 38 and 0 57 
Given the broad landscape of various approaches to argument analysis and persuasion studies presented in this section, we would like to stress some novel aspects of the current article  First, we aim at adapting a model of argument based on research by argumentation scholars, both theoretical and empirical  We pose several pragmatical constraints, such as register independence (generalization over several registers)  Second, our emphasis is put on reliable annotations and sufficient data size (about 90k tokens)  Third, we deal with fairly unrestricted Web-based sources, so additional steps of distinguishing whether the texts are argumentative are required  Argumentation mining has been a rapidly evolving field with several major venues in 2015  We encourage readers to consult an upcoming survey article by Lippi Torroni 2016 or the proceedings of the 2nd Argumentation Mining workshop BIBREF50 to keep up with recent developments  However, to the best of our knowledge, the main findings of this article have not yet been made obsolete by any related work 
Annotation studies and corpus creation
This section describes the process of data selection, annotation, curation, and evaluation with the goal of creating a new corpus suitable for argumentation mining research in the area of computational linguistics  As argumentation mining is an evolving discipline without established and widely-accepted annotation schemes, procedures, and evaluation, we want to keep this overview detailed to ensure full reproducibility of our approach  Given the wide range of perspectives on argumentation itself BIBREF15 , variety of argumentation models BIBREF27 , and high costs of discourse or pragmatic annotations BIBREF48 , creating a new, reliable corpus for argumentation mining represents a substantial effort 
A motivation for creating a new corpus stems from the various use-cases discussed in the introduction, as well as some research gaps pointed in section SECREF1 and further discussed in the survey in section SECREF31 (e g , domain restrictions, missing connection to argumentation theories, non-reported reliability or detailed schemes) 
Topics and registers
As a main field of interest in the current study, we chose controversies in education  One distinguishing feature of educational topics is their breadth of sub-topics and points of view, as they attract researchers, practitioners, parents, students, or policy-makers  We assume that this diversity leads to the linguistic variability of the education topics and thus represents a challenge for NLP  In a cooperation with researchers from the German Institute for International Educational Research we identified the following current controversial topics in education in English-speaking countries: (1) homeschooling, (2) public versus private schools, (3) redshirting — intentionally delaying the entry of an age-eligible child into kindergarten, allowing their child more time to mature emotionally and physically BIBREF51 , (4) prayer in schools — whether prayer in schools should be allowed and taken as a part of education or banned completely, (5) single-sex education — single-sex classes (males and females separate) versus mixed-sex classes (“co-ed”), and (6) mainstreaming — including children with special needs into regular classes 
Since we were also interested in whether argumentation differs across registers, we included four different registers — namely (1) user comments to newswire articles or to blog posts, (2) posts in discussion forums (forum posts), (3) blog posts, and (4) newswire articles  Throughout this work, we will refer to each article, blog post, comment, or forum posts as a document  This variety of sources covers mainly user-generated content except newswire articles which are written by professionals and undergo an editing procedure by the publisher  Since many publishers also host blog-like sections on their portals, we consider as blog posts all content that is hosted on personal blogs or clearly belong to a blog category within a newswire portal 
Raw corpus statistics
Given the six controversial topics and four different registers, we compiled a collection of plain-text documents, which we call the raw corpus  It contains 694,110 tokens in 5,444 documents  As a coarse-grained analysis of the data, we examined the lengths and the number of paragraphs (see Figure FIGREF43 )  Comments and forum posts follow a similar distribution, being shorter than 300 tokens on average  By contrast, articles and blogs are longer than 400 tokens and have 9 2 paragraphs on average  The process of compiling the raw corpus and its further statistics are described in detail in Appendix UID158  
Annotation study 1: Identifying persuasive documents in forums and comments
The goal of this study was to select documents suitable for a fine-grained analysis of arguments  In a preliminary study on annotating argumentation using a small sample (50 random documents) of forum posts and comments from the raw corpus, we found that many documents convey no argumentation at all, even in discussions about controversies  We observed that such contributions do not intend to persuade; these documents typically contain story-sharing, personal worries, user interaction (asking questions, expressing agreement), off-topic comments, and others  Such characteristics are typical to on-line discussions in general, but they have not been examined with respect to argumentation or persuasion  Indeed, we observed that there are (1) documents that are completely unrelated and (2) documents that are related to the topic, but do not contain any argumentation  This issue has been identified among argumentation theorist; for example as external relevance by Paglieri Castelfranchia 2014  Similar findings were also confirmed in related literature in argumentation mining, however never tackled empirically BIBREF53 , BIBREF8 These documents are thus not suitable for analyzing argumentation 
In order to filter documents that are suitable for argumentation annotation, we defined a binary document-level classification task  The distinction is made between either persuasive documents or non-persuasive (which includes all other sorts of texts, such as off-topic, story sharing, unrelated dialog acts, etc ) 
The two annotated categories were on-topic persuasive and non-persuasive  Three annotators with near-native English proficiency annotated a set of 990 documents (a random subset of comments and forum posts) reaching 0 59 Fleiss' INLINEFORM0   The final label was selected by majority voting  The annotation study took on average of 15 hours per annotator with approximately 55 annotated documents per hour  The resulting labels were derived by majority voting  Out of 990 documents, 524 (53%) were labeled as on-topic persuasive  We will refer to this corpus as gold data persuasive 
We examined all disagreements between annotators and discovered some typical problems, such as implicitness or topic relevance  First, the authors often express their stance towards the topic implicitly, so it must be inferred by the reader  To do so, certain common-ground knowledge is required  However, such knowledge heavily depends on many aspects, such as the reader's familiarity with the topic or her cultural background, as well as the context of the source website or the discussion forum thread  This also applies for sarcasm and irony  Second, the decision whether a particular topic is persuasive was always made with respect to the controversial topic under examination  Some authors shift the focus to a particular aspect of the given controversy or a related issue, making the document less relevant 
We achieved moderate agreement between the annotators, although the definition of persuasiveness annotation might seem a bit fuzzy  We found different amounts of persuasion in the specific topics  For instance, prayer in schools or private vs  public schools attract persuasive discourse, while other discussed controversies often contain non-persuasive discussions, represented by redshirting and mainstreaming  Although these two topics are also highly controversial, the participants of on-line discussions seem to not attempt to persuade but they rather exchange information, support others in their decisions, etc  This was also confirmed by socio-psychological researchers  Ammari et al 2014 show that parents of children with special needs rely on discussion sites for accessing information and social support and that, in particular, posts containing humor, achievement, or treatment suggestions are perceived to be more socially appropriate than posts containing judgment, violence, or social comparisons  According to Nicholson Leask 2012, in the online forum, parents of autistic children were seen to understand the issue because they had lived it  Assuming that participants in discussions related to young kids (e g , redshirting, or mainstreaming) are usually females (mothers), the gender can also play a role  In a study of online persuasion, Guadagno Cialdini 2002 conclude that women chose to bond rather than compete (women feel more comfortable cooperating, even in a competitive environment), whereas men are motivated to compete if necessary to achieve independence 
Annotation study 2: Annotating micro-structure of arguments
The goal of this study was to annotate documents on a detailed level with respect to an argumentation model  First, we will present the annotation scheme  Second, we will describe the annotation process  Finally, we will evaluate the agreement and draw some conclusions 
Given the theoretical background briefly introduced in section SECREF2 , we motivate our selection of the argumentation model by the following requirements  First, the scope of this work is to capture argumentation within a single document, thus focusing on micro-level models  Second, there should exist empirical evidence that such a model has been used for analyzing argumentation in previous works, so it is likely to be suitable for our purposes of argumentative discourse analysis in user-generated content  Regarding the first requirement, two typical examples of micro-level models are the Toulmin's model BIBREF36 and Walton's schemes BIBREF55   Let us now elaborate on the second requirement 
Walton's argumentation schemes are claimed to be general and domain independent  Nevertheless, evidence from the computational linguistics field shows that the schemes lack coverage for analyzing real argumentation in natural language texts  In examining real-world political argumentation from BIBREF56 , Walton 2012 found out that 37 1% of the arguments collected did not fit any of the fourteen schemes they chose so they created new schemes ad-hoc  Cabrio2013b selected five argumentation schemes from Walton and map these patterns to discourse relation categories in the Penn Discourse TreeBank (PDTB) BIBREF48 , but later they had to define two new argumentation schemes that they discovered in PDTB  Similarly, Song et al 2014 admitted that the schemes are ambiguous and hard to directly apply for annotation, therefore they modified the schemes and created new ones that matched the data 
Although Macagno Konstantinidou 2012 show several examples of two argumentation schemes applied to few selected arguments in classroom experiments, empirical evidence presented by Anthony Kim 2014 reveals many practical and theoretical difficulties of annotating dialogues with schemes in classroom deliberation, providing many details on the arbitrary selection of the sub-set of the schemes, the ambiguity of the scheme definitions, concluding that the presence of the authors during the experiment was essential for inferring and identifying the argument schemes BIBREF57  
Although this model (refer to section SECREF21 ) was designed to be applicable to real-life argumentation, there are numerous studies criticizing both the clarity of the model definition and the differentiation between elements of the model  Ball1994 claims that the model can be used only for the most simple arguments and fails on the complex ones  Also Freeman1991 and other argumentation theorists criticize the usefulness of Toulmin's framework for the description of real-life argumentative texts  However, others have advocated the model and claimed that it can be applied to the people's ordinary argumenation BIBREF58 , BIBREF59  
A number of studies (outside the field of computational linguistics) used Toulmin's model as their backbone argumentation framework  Chambliss1995 experimented with analyzing 20 written documents in a classroom setting in order to find the argument patterns and parts  Simosi2003 examined employees' argumentation to resolve conflicts  Voss2006 analyzed experts' protocols dealing with problem-solving 
The model has also been used in research on computer-supported collaborative learning  Erduran2004 adapt Toulmin's model for coding classroom argumentative discourse among teachers and students  Stegmann2011 builds on a simplified Toulmin's model for scripted construction of argument in computer-supported collaborative learning  Garcia-Mila2013 coded utterances into categories from Toulmin's model in persuasion and consensus-reaching among students  Weinberger Fischer 2006 analyze asynchronous discussion boards in which learners engage in an argumentative discourse with the goal to acquire knowledge  For coding the argument dimension, they created a set of argumentative moves based on Toulmin's model  Given this empirical evidence, we decided to build upon the Toulmin's model 
In this annotation task, a sequence of tokens (e g  a phrase, a sentence, or any arbitrary text span) is labeled with a corresponding argument component (such as the claim, the grounds, and others)  There are no explicit relations between these annotation spans as the relations are implicitly encoded in the pragmatic function of the components in the Toulmin's model 
In order to prove the suitability of the Toulmin's model, we analyzed 40 random documents from the gold data persuasive dataset using the original Toulmin's model as presented in section SECREF21   We took into account sever criteria for assessment, such as frequency of occurrence of the components or their importance for the task  We proposed some modifications of the model based on the following observations 
Authors do not state the degree of cogency (the probability of their claim, as proposed by Toulmin)  Thus we omitted qualifier from the model due to its absence in the data 
The warrant as a logical explanation why one should accept the claim given the evidence is almost never stated  As pointed out by BIBREF37 , “data are appealed to explicitly, warrants implicitly ” This observation has also been made by Voss2006  Also, according to [p  205]Eemeren et al 1987, the distinction of warrant is perfectly clear only in Toulmin’s examples, but the definitions fail in practice  We omitted warrant from the model

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain the new context represetation?
Context 2:  We only considered words appearing more than 100 times and added a special PADDING token for convolution  This results in an embedding training text of about 485,000 terms and INLINEFORM0 tokens  During model training, the embeddings are updated 
Position features  We incorporate randomly initialized position embeddings similar to zeng2014, nguyen and deSantos2015  In our RNN experiments, we investigate different possibilities of integrating position information: position embeddings, position embeddings with entity presence flags (flags indicating whether the current word is one of the relation arguments), and position indicators BIBREF7  
Objective Function: Ranking Loss
Ranking  We applied the ranking loss function proposed in deSantos2015 to train our models  It maximizes the distance between the true label INLINEFORM0 and the best competitive label INLINEFORM1 given a data point INLINEFORM2   The objective function is DISPLAYFORM0
with INLINEFORM0 and INLINEFORM1 being the scores for the classes INLINEFORM2 and INLINEFORM3 respectively  The parameter INLINEFORM4 controls the penalization of the prediction errors and INLINEFORM5 and INLINEFORM6 are margins for the correct and incorrect classes  Following deSantos2015, we set INLINEFORM7   We do not learn a pattern for the class Other but increase its difference to the best competitive label by using only the second summand in Equation EQREF10 during training 
Experiments and Results
We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8   It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other)  8,000 sentences have been distributed as training set and 2,717 sentences served as test set  For evaluation, we applied the official scoring script and report the macro F1 score which also served as the official result of the shared task 
RNN and CNN models were implemented with theano BIBREF12 , BIBREF13   For all our models, we use L2 regularization with a weight of 0 0001  For CNN training, we use mini batches of 25 training examples while we perform stochastic gradient descent for the RNN  The initial learning rates are 0 2 for the CNN and 0 01 for the RNN  We train the models for 10 (CNN) and 50 (RNN) epochs without early stopping  As activation function, we apply tanh for the CNN and capped ReLU for the RNN  For tuning the hyperparameters, we split the training data into two parts: 6 5k (training) and 1 5k (development) sentences  We also tuned the learning rate schedule on dev 
Beside of training single models, we also report ensemble results for which we combined the presented single models with a voting process 
Performance of CNNs
As a baseline system, we implemented a CNN similar to the one described by zeng2014  It consists of a standard convolutional layer with filters with only one window size, followed by a softmax layer  As input it uses the middle context  In contrast to zeng2014, our CNN does not have an additional fully connected hidden layer  Therefore, we increased the number of convolutional filters to 1200 to keep the number of parameters comparable  With this, we obtain a baseline result of 73 0  After including 5 dimensional position features, the performance was improved to 78 6 (comparable to 78 9 as reported by zeng2014 without linguistic features) 
In the next step, we investigate how this result changes if we successively add further features to our CNN: multi-windows for convolution (window sizes: 2,3,4,5 and 300 feature maps each), ranking layer instead of softmax and our proposed extended middle context  Table TABREF12 shows the results  Note that all numbers are produced by CNNs with a comparable number of parameters  We also report F1 for increasing the word embedding dimensionality from 50 to 400  The position embedding dimensionality is 5 in combination with 50 dimensional word embeddings and 35 with 400 dimensional word embeddings  Our results show that especially the ranking layer and the embedding size have an important impact on the performance 
Performance of RNNs
As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence  With this model, we achieve an F1 score of 61 2 on the SemEval test set 
Afterwards, we investigate the impact of different position features on the performance of uni-directional RNNs (position embeddings, position embeddings concatenated with a flag indicating whether the current word is an entity or not, and position indicators BIBREF7 )  The results indicate that position indicators (i e  artificial words that indicate the entity presence) perform the best on the SemEval data  We achieve an F1 score of 73 4 with them  However, the difference to using position embeddings with entity flags is not statistically significant 
Similar to our CNN experiments, we successively vary the RNN models by using bi-directionality, by adding connections between the hidden layers (“connectionist”), by applying ranking instead of softmax to predict the relation and by increasing the word embedding dimension to 400 
The results in Table TABREF14 show that all of these variations lead to statistically significant improvements  Especially the additional hidden layer connections and the integration of the ranking layer have a large impact on the performance 
Combination of CNNs and RNNs
Finally, we combine our CNN and RNN models using a voting process  For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes  In case of a tie, we pick one of the most frequent classes randomly  The combination achieves an F1 score of 84 9 which is better than the performance of the two NN types alone  It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations 
Comparison with State of the Art
Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models  Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features 
Conclusion
In this paper, we investigated different features and architectural choices for convolutional and recurrent neural networks for relation classification without using any linguistic features  For convolutional neural networks, we presented a new context representation for relation classification  Furthermore, we introduced connectionist recurrent neural networks for sentence classification tasks and performed the first experiments with ranking recurrent neural networks  Finally, we showed that even a simple combination of convolutional and recurrent neural networks improved results  With our neural models, we achieved new state-of-the-art results on the SemEval 2010 task 8 benchmark data 
Acknowledgments
Heike Adel is a recipient of the Google European Doctoral Fellowship in Natural Language Processing and this research is supported by this fellowship 
This research was also supported by Deutsche Forschungsgemeinschaft: grant SCHU 2246/4-2 

ground_truth: They use two independent convolutional and max-pooling layers on (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. They concatenated the two results after pooling to get the new context representation.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What argument components do the ML methods aim to identify?
Context 3: 
Rebuttal is a statement that attacks the claim, thus playing a role of an opposing view  In reality, the authors often attack the presented rebuttals by another counter-rebuttal in order to keep the whole argument's position consistent  Thus we introduced a new component – refutation – which is used for attacking the rebuttal  Annotation of refutation was conditioned of explicit presence of rebuttal and enforced by the annotation guidelines  The chain rebuttal–refutation is also known as the procatalepsis figure in rhetoric, in which the speaker raises an objection to his own argument and then immediately answers it  By doing so, the speaker hopes to strengthen the argument by dealing with possible counter-arguments before the audience can raise them BIBREF43  
The claim of the argument should always reflect the main standpoint with respect to the discussed controversy  We observed that this standpoint is not always explicitly expressed, but remains implicit and must be inferred by the reader  Therefore, we allow the claim to be implicit  In such a case, the annotators must explicitly write down the (inferred) stance of the author 
By definition, the Toulmin's model is intended to model single argument, with the claim in its center  However, we observed in our data, that some authors elaborate on both sides of the controversy equally and put forward an argument for each side (by argument here we mean the claim and its premises, backings, etc )  Therefore we allow multiple arguments to be annotated in one document  At the same time, we restrained the annotators from creating complex argument hierarchies 
Toulmin's grounds have an equivalent role to a premise in the classical view on an argument BIBREF15 , BIBREF60 in terms that they offer the reasons why one should accept the standpoint expressed by the claim  As this terminology has been used in several related works in the argumentation mining field BIBREF7 , BIBREF61 , BIBREF62 , BIBREF11 , we will keep this convention and denote the grounds as premises 
One of the main critiques of the original Toulmin's model was the vague distinction between grounds, warrant, and backing BIBREF63 , BIBREF64 , BIBREF65   The role of backing is to give additional support to the warrant, but there is no warrant in our model anymore  However, what we observed during the analysis, was a presence of some additional evidence  Such evidence does not play the role of the grounds (premises) as it is not meant as a reason supporting the claim, but it also does not explain the reasoning, thus is not a warrant either  It usually supports the whole argument and is stated by the author as a certain fact  Therefore, we extended the scope of backing as an additional support to the whole argument 
The annotators were instructed to distinguish between premises and backing, so that premises should cover generally applicable reasons for the claim, whereas backing is a single personal experience or statements that give credibility or attribute certain expertise to the author  As a sanity check, the argument should still make sense after removing backing (would be only considered “weaker”) 
We call the model as a modified Toulmin's model  It contains five argument components, namely claim, premise, backing, rebuttal, and refutation  When annotating a document, any arbitrary token span can be labeled with an argument component; the components do not overlap  The spans are not known in advance and the annotator thus chooses the span and the component type at the same time  All components are optional (they do not have to be present in the argument) except the claim, which is either explicit or implicit (see above)  If a token span is not labeled by any argument component, it is not considered as a part of the argument and is later denoted as none (this category is not assigned by the annotators) 
An example analysis of a forum post is shown in Figure FIGREF65   Figure FIGREF66 then shows a diagram of the analysis from that example (the content of the argument components was shortened or rephrased) 
The annotation experiment was split into three phases  All documents were annotated by three independent annotators, who participated in two training sessions  During the first phase, 50 random comments and forum posts were annotated  Problematic cases were resolved after discussion and the guidelines were refined  In the second phase, we wanted to extend the range of annotated registers, so we selected 148 comments and forum posts as well as 41 blog posts  After the second phase, the annotation guidelines were final 
In the final phase, we extended the range of annotated registers and added newswire articles from the raw corpus in order to test whether the annotation guidelines (and inherently the model) is general enough  Therefore we selected 96 comments/forum posts, 8 blog posts, and 8 articles for this phase  A detailed inter-annotator agreement study on documents from this final phase will be reported in section UID75  
The annotations were very time-consuming  In total, each annotator spent 35 hours by annotating in the course of five weeks  Discussions and consolidation of the gold data took another 6 hours  Comments and forum posts required on average of 4 minutes per document to annotate, while blog posts and articles on average of 14 minutes per document  Examples of annotated documents from the gold data are listed in Appendix UID158  
We discarded 11 documents out of the total 351 annotated documents  Five forum posts, although annotated as persuasive in the first annotation study, were at a deeper look a mixture of two or more posts with missing quotations, therefore unsuitable for analyzing argumentation  Three blog posts and two articles were found not to be argumentative (the authors took no stance to the discussed controversy) and one article was an interview, which the current model cannot capture (a dialogical argumentation model would be required) 
For each of the 340 documents, the gold standard annotations were obtained using the majority vote  If simple majority voting was not possible (different boundaries of the argument component together with a different component label), the gold standard was set after discussion among the annotators  We will refer to this corpus as the gold standard Toulmin corpus  The distribution of topics and registers in this corpus in shown in Table TABREF71 , and Table TABREF72 presents some lexical statistics 
Based on pre-studies, we set the minimal unit for annotation as token  The documents were pre-segmented using the Stanford Core NLP sentence splitter BIBREF69 embedded in the DKPro Core framework BIBREF70   Annotators were asked to stick to the sentence level by default and label entire pre-segmented sentences  They should switch to annotations on the token level only if (a) a particular sentence contained more than one argument component, or (b) if the automatic sentence segmentation was wrong  Given the “noise” in user-generated Web data (wrong or missing punctuation, casing, etc ), this was often the case 
Annotators were also asked to rephrase (summarize) each annotated argument component into a simple statement when applicable, as shown in Figure FIGREF66   This was used as a first sanity checking step, as each argument component is expected to be a coherent discourse unit  For example, if a particular occurrence of a premise cannot be summarized/rephrased into one statement, this may require further splitting into two or more premises 
For the actual annotations, we developed a custom-made web-based application that allowed users to switch between different granularity of argument components (tokens or sentences), to annotate the same document in different argument “dimensions” (logos and pathos), and to write summary for each annotated argument component 
As a measure of annotation reliability, we rely on Krippendorff's unitized alpha ( INLINEFORM0 ) BIBREF71   To the best of our knowledge, this is the only agreement measure that is applicable when both labels and boundaries of segments are to be annotated 
Although the measure has been used in related annotation works BIBREF61 , BIBREF7 , BIBREF72 , there is one important detail that has not been properly communicated  The INLINEFORM0 is computed over a continuum of the smallest units, such as tokens  This continuum corresponds to a single document in the original Krippendorff's work  However, there are two possible extensions to multiple documents (a corpus), namely (a) to compute INLINEFORM1 for each document first and then report an average value, or (b) to concatenate all documents into one large continuum and compute INLINEFORM2 over it  The first approach with averaging yielded extremely high the standard deviation of INLINEFORM3 (i e , avg  = 0 253; std  dev  = 0 886; median = 0 476 for the claim)  This says that some documents are easy to annotate while others are harder, but interpretation of such averaged value has no evidence either in BIBREF71 or other papers based upon it  Thus we use the other methodology and treat the whole corpus as a single long continuum (which yields in the example of claim 0 541 INLINEFORM4 ) 
Table TABREF77 shows the inter-annotator agreement as measured on documents from the last annotation phase (see section UID67 )  The overall INLINEFORM0 for all register types, topics, and argument components is 0 48 in the logos dimension (annotated with the modified Toulmin's model)  Such agreement can be considered as moderate by the measures proposed by Landis Koch 1977, however, direct interpretation of the agreement value lacks consensus BIBREF54   Similar inter-annotator agreement numbers were achieved in the relevant works in argumentation mining (refer to Table TABREF33 in section SECREF31 ; although most of the numbers are not directly comparable, as different inter-annotator metrics were used on different tasks) 
There is a huge difference in INLINEFORM0 regarding the registers between comments + forums posts ( INLINEFORM1 0 60, Table TABREF77 a) and articles + blog posts ( INLINEFORM2 0 09, Table TABREF77 b) in the logos dimension  If we break down the value with respect to the individual argument components, the agreement on claim and premise is substantial in the case of comments and forum posts (0 59 and 0 69, respectively)  By contrast, these argument components were annotated only with a fair agreement in articles and blog posts (0 22 and 0 24, respectively) 
As can be also observed from Table TABREF77 , the annotation agreement in the logos dimension varies regarding the document topic  While it is substantial/moderate for prayer in schools (0 68) or private vs  public schools (0 44), for some topics it remains rather slight, such as in the case of redshirting (0 14) or mainstreaming (0 08) 
First, we examine the disagreement in annotations by posing the following research question: are there any measurable properties of the annotated documents that might systematically cause low inter-annotator agreement? We use Pearson's correlation coefficient between INLINEFORM0 on each document and the particular property under investigation  We investigated the following set of measures 
Full sentence coverage ratio represents a ratio of argument component boundaries that are aligned to sentence boundaries  The value is 1 0 if all annotations in the particular document are aligned to sentences and 0 0 if no annotations match the sentence boundaries  Our hypothesis was that automatic segmentation to sentences was often incorrect, therefore annotators had to switch to the token level annotations and this might have increased disagreement on boundaries of the argument components 
Document length, paragraph length and average sentence length  Our hypotheses was that the length of documents, paragraphs, or sentences negatively affects the agreement 
Readability measures  We tested four standard readability measures, namely Ari BIBREF73 , Coleman-Liau BIBREF74 , Flesch BIBREF75 , and Lix BIBREF76 to find out whether readability of the documents plays any role in annotation agreement 
Correlation results are listed in Table TABREF82   We observed the following statistically significant ( INLINEFORM0 ) correlations  First, document length negatively correlates with agreement in comments  The longer the comment was the lower the agreement was  Second, average paragraph length negatively correlates with agreement in blog posts  The longer the paragraphs in blogs were, the lower agreement was reached  Third, all readability scores negatively correlate with agreement in the public vs  private school domain, meaning that the more complicated the text in terms of readability is, the lower agreement was reached  We observed no significant correlation in sentence coverage and average sentence length measures  We cannot draw any general conclusion from these results, but we can state that some registers and topics, given their properties, are more challenging to annotate than others 
Another qualitative analysis of disagreements between annotators was performed by constructing a probabilistic confusion matrix BIBREF77 on the token level  The biggest disagreements, as can be seen in Table TABREF85 , is caused by rebuttal and refutation confused with none (0 27 and 0 40, respectively)  This is another sign that these two argument components were very hard to annotate  As shown in Table TABREF77 , the INLINEFORM5 was also low – 0 08 for rebuttal and 0 17 for refutation 
We analyzed the annotations and found the following phenomena that usually caused disagreements between annotators 
Each argument component (e g , premise or backing) should express one consistent and coherent piece of information, for example a single reason in case of the premise (see Section UID73 )  However, the decision whether a longer text should be kept as a single argument component or segmented into multiple components is subjective and highly text-specific 
While rhetorical questions have been researched extensively in linguistics BIBREF78 , BIBREF79 , BIBREF80 , BIBREF81 , their role in argumentation represents a substantial research question BIBREF82 , BIBREF83 , BIBREF84 , BIBREF85 , BIBREF86   Teninbaum 2011 provides a brief history of rhetorical questions in persuasion  In short, rhetorical questions should provoke the reader  From the perspective of our argumentation model, rhetorical questions might fall both into the logos dimension (and thus be labeled as, e g , claim, premise, etc ) or into the pathos dimension (refer to Section SECREF20 )  Again, the decision is usually not clear-cut 
As introduced in section UID55 , rebuttal attacks the claim by presenting an opponent's view  In most cases, the rebuttal is again attacked by the author using refutation  From the pragmatic perspective, refutation thus supports the author's stance expressed by the claim  Therefore, it can be easily confused with premises, as the function of both is to provide support for the claim  Refutation thus only takes place if it is meant as a reaction to the rebuttal  It follows the discussed matter and contradicts it  Such a discourse is usually expressed as:
[claim: My claim ] [rebuttal: On the other hand, some people claim XXX which makes my claim wrong ] [refutation: But this is not true, because of YYY ]
However, the author might also take the following defensible approach to formulate the argument:
[rebuttal: Some people claim XXX-1 which makes my claim wrong ] [refutation: But this is not true, because of YYY-1 ] [rebuttal: Some people claim XXX-2 which makes my claim wrong ] [refutation: But this is not true, because of YYY-2 ] [claim: Therefore my claim ]
If this argument is formulated without stating the rebuttals, it would be equivalent to the following:
[premise: YYY-1 ] [premise: YYY-2 ] [claim: Therefore my claim ]
This example shows that rebuttal and refutation represent a rhetorical device to produce arguments, but the distinction between refutation and premise is context-dependent and on the functional level both premise and refutation have very similar role – to support the author's standpoint  Although introducing dialogical moves into monological model and its practical consequences, as described above, can be seen as a shortcoming of our model, this rhetoric figure has been identified by argumentation researchers as procatalepsis BIBREF43   A broader view on incorporating opposing views (or lack thereof) is discussed under the term confirmation bias by BIBREF21 who claim that “[   ] people are trying to convince others  They are typically looking for arguments and evidence to confirm their own claim, and ignoring negative arguments and evidence unless they anticipate having to rebut them ” The dialectical attack of possible counter-arguments may thus strengthen one's own argument 
One possible solution would be to refrain from capturing this phenomena completely and to simplify the model to claims and premises, for instance  However, the following example would then miss an important piece of information, as the last two clauses would be left un-annotated  At the same time, annotating the last clause as premise would be misleading, because it does not support the claim (in fact, it supports it only indirectly by attacking the rebuttal; this can be seen as a support is considered as an admissible extension of abstract argument graph by BIBREF87 ) 
Doc#422 (forumpost, homeschooling) [claim: I try not to be anti-homeschooling, but    it's just hard for me ] [premise: I really haven't met any homeschoolers who turned out quite right, including myself ] I apologize if what I'm saying offends any of you - that's not my intention, [rebuttal: I know that there are many homeschooled children who do just fine,] but [refutation: that hasn't been my experience ]
To the best of our knowledge, these context-dependent dialogical properties of argument components using Toulmin's model have not been solved in the literature on argumentation theory and we suggest that these observations should be taken into account in the future research in monological argumentation 
Appeal to emotion, sarcasm, irony, or jokes are common in argumentation in user-generated Web content  We also observed documents in our data that were purely sarcastic (the pathos dimension), therefore logical analysis of the argument (the logos dimension) would make no sense  However, given the structure of such documents, some claims or premises might be also identified  Such an argument is a typical example of fallacious argumentation, which intentionally pretends to present a valid argument, but its persuasion is conveyed purely for example by appealing to emotions of the reader BIBREF88  
We present some statistics of the annotated data that are important from the argumentation research perspective  Regardless of the register, 48% of claims are implicit  This means that the authors assume that their standpoint towards the discussed controversy can be inferred by the reader and give only reasons for that standpoint  Also, explicit claims are mainly written just once, only in 3% of the documents the claim was rephrased and occurred multiple times 
In 6% of the documents, the reasons for an implicit claim are given only in the pathos dimension, making the argument purely persuasive without logical argumentation 
The “myside bias”, defined as a bias against information supporting another side of an argument BIBREF89 , BIBREF90 , can be observed by the presence of rebuttals to the author's claim or by formulating arguments for both sides when the overall stance is neutral  While 85% of the documents do not consider any opposing side, only 8% documents present a rebuttal, which is then attacked by refutation in 4% of the documents  Multiple rebuttals and refutations were found in 3% of the documents  Only 4% of the documents were overall neutral and presented arguments for both sides, mainly in blog posts 
We were also interested whether mitigating linguistic devices are employed in the annotated arguments, namely in their main stance-taking components, the claims  Such devices typically include parenthetical verbs, syntactic constructions, token agreements, hedges, challenge questions, discourse markers, and tag questions, among others BIBREF91   In particular, [p  1]Kaltenbock et al 2010 define hedging as a discourse strategy that reduces the force or truth of an utterance and thus reduces the risk a speaker runs when uttering a strong or firm assertion or other speech act  We manually examined the use of hedging in the annotated claims 
Our main observation is that hedging is used differently across topics  For instance, about 30-35% of claims in homeschooling and mainstreaming signal the lack of a full commitment to the expressed stance, in contrast to prayer in schools (15%) or public vs  private schools (about 10%)  Typical hedging cues include speculations and modality (“If I have kids, I will probably homeschool them ”), statements as neutral observations (“It's not wrong to hold the opinion that in general it's better for kids to go to school than to be homeschooled ”), or weasel phrases BIBREF92 (“In some cases, inclusion can work fantastically well ”, “For the majority of the children in the school, mainstream would not have been a suitable placement ”) 
On the other hand, most claims that are used for instance in the prayer in schools arguments are very direct, without trying to diminish its commitment to the conveyed belief (for example, “NO PRAYER IN SCHOOLS!    period ”, “Get it out of public schools”, “Pray at home ”, or “No organized prayers or services anywhere on public school board property - FOR ANYONE ”)  Moreover, some claims are clearly offensive, persuading by direct imperative clauses towards the opponents/audience (“TAKE YOUR KIDS PRIVATE IF YOU CARE AS I DID”, “Run, don't walk, to the nearest private school ”) or even accuse the opponents for taking a certain stance (“You are a bad person if you send your children to private school ”) 
These observations are consistent with the findings from the first annotation study on persuasion (see section UID48 ), namely that some topics attract heated argumentation where participant take very clear and reserved standpoints (such as prayer in schools or private vs  public schools), while discussions about other topics are rather milder  It has been shown that the choices a speaker makes to express a position are informed by their social and cultural background, as well as their ability to speak the language BIBREF93 , BIBREF94 , BIBREF91   However, given the uncontrolled settings of the user-generated Web content, we cannot infer any similar conclusions in this respect 
We investigated premises across all topics in order to find the type of support used in the argument  We followed the approach of Park Cardie 2014, who distinguished three types of propositions in their study, namely unverifiable, verifiable non-experiential, and verifiable experiential 
Verifiable non-experiential and verifiable experiential propositions, unlike unverifiable propositions, contain an objective assertion, where objective means “expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations ” Such assertions have truth values that can be proved or disproved with objective evidence; the correctness of the assertion or the availability of the objective evidence does not matter BIBREF8   A verifiable proposition can further be distinguished as experiential or not, depending on whether the proposition is about the writer's personal state or experience or something non-experiential  Verifiable experiential propositions are sometimes referred to as anectotal evidence, provide the novel knowledge that readers are seeking BIBREF8  
Table TABREF97 shows the distribution of the premise types with examples for each topic from the annotated corpus  As can be seen in the first row, arguments in prayer in schools contain majority (73%) of unverifiable premises  Closer examination reveals that their content vary from general vague propositions to obvious fallacies, such as a hasty generalization, straw men, or slippery slope  As Nieminen Mustonen 2014 found out, fallacies are very common in argumentation about religion-related issues  On the other side of the spectrum, arguments about redshirting rely mostly on anecdotal evidence (61% of verifiable experiential propositions)  We will discuss the phenomena of narratives in argumentation in more detail later in section UID98   All the topics except private vs  public schools exhibit similar amount of verifiable non-experiential premises (9%–22%), usually referring to expert studies or facts  However, this type of premises has usually the lowest frequency 
Manually analyzing argumentative discourse and reconstructing (annotating) the underlying argument structure and its components is difficult  As [p  267]Reed2006 point out, “the analysis of arguments is often hard, not only for students, but for experts too ” According to [p  81]Harrell 2011b, argumentation is a skill and “even for simple arguments, untrained college students can identify the conclusion but without prompting are poor at both identifying the premises and how the premises support the conclusion ” [p  81]Harrell 2011 further claims that “a wide literature supports the contention that the particular skills of understanding, evaluating, and producing arguments are generally poor in the population of people who have not had specific training and that specific training is what improves these skills ” Some studies, for example, show that students perform significantly better on reasoning tasks when they have learned to identify premises and conclusions BIBREF95 or have learned some standard argumentation norms BIBREF96  
One particular extra challenge in analyzing argumentation in Web user-generated discourse is that the authors produce their texts probably without any existing argumentation theory or model in mind  We assume that argumentation or persuasion is inherent when users discuss controversial topics, but the true reasons why people participate in on-line communities and what drives their behavior is another research question BIBREF97 , BIBREF98 , BIBREF99 , BIBREF100   When the analyzed texts have a clear intention to produce argumentative discourse, such as in argumentative essays BIBREF7 , the argumentation is much more explicit and a substantially higher inter-annotator agreement can be achieved 
The model seems to be suitable for short persuasive documents, such as comments and forum posts  Its applicability to longer documents, such as articles or blog posts, is problematic for several reasons 
The argument components of the (modified) Toulmin's model and their roles are not expressive enough to capture argumentation that not only conveys the logical structure (in terms of reasons put forward to support the claim), but also relies heavily on the rhetorical power  This involves various stylistic devices, pervading narratives, direct and indirect speech, or interviews  While in some cases the argument components are easily recognizable, the vast majority of the discourse in articles and blog posts does not correspond to any distinguishable argumentative function in the logos dimension  As the purpose of such discourse relates more to rhetoric than to argumentation, unambiguous analysis of such phenomena goes beyond capabilities of the current argumentation model  For a discussion about metaphors in Toulmin's model of argumentation see, e g , BIBREF102 , BIBREF103  
Articles without a clear standpoint towards the discussed controversy cannot be easily annotated with the model either  Although the matter is viewed from both sides and there might be reasons presented for either of them, the overall persuasive intention is missing and fitting such data to the argumentation framework causes disagreements  One solution might be to break the document down to paragraphs and annotate each paragraph separately, examining argumentation on a different level of granularity 
As introduced in section SECREF20 , there are several dimensions of an argument  The Toulmin's model focuses solely on the logos dimension  We decided to ignore the ethos dimension, because dealing with the author's credibility remains unclear, given the variety of the source web data  However, exploiting the pathos dimension of an argument is prevalent in the web data, for example as an appeal to emotions  Therefore we experimented with annotating appeal to emotions as a separate category independent of components in the logos dimension  We defined some features for the annotators how to distinguish appeal to emotions  Figurative language such as hyperbole, sarcasm, or obvious exaggerating to “spice up” the argument are the typical signs of pathos  In an extreme case, the whole argument might be purely emotional, as in the following example 
Doc#1698 (comment, prayer in schools) [app-to-emot: Prayer being removed from school is just the leading indicator of a nation that is ‘Falling Away’ from Jehovah  [   ] And the disasters we see today are simply God’s finger writing on the wall: Mene, mene, Tekel, Upharsin; that is, God has weighed America in the balances, and we’ve been found wanting  No wonder 50 million babies have been aborted since 1973  [   ]]
We kept annotations on the pathos dimension as simple as possible (with only one appeal to emotions label), but the resulting agreement was unsatisfying ( INLINEFORM0 0 30) even after several annotation iterations  Appeal to emotions is considered as a type of fallacy BIBREF104 , BIBREF18   Given the results, we assume that more carefully designed approach to fallacy annotation should be applied  To the best of our knowledge, there have been very few research works on modeling fallacies similarly to arguments on the discourse level BIBREF105   Therefore the question, in which detail and structure fallacies should be annotated, remains open  For the rest of the paper, we thus focus on the logos dimension solely 
Some of the educational topics under examination relate to young children (e g , redshirting or mainstreaming); therefore we assume that the majority of participants in discussions are their parents  We observed that many documents related to these topics contain narratives  Sometimes the story telling is meant as a support for the argument, but there are documents where the narrative has no intention to persuade and is simply a story sharing 
There is no widely accepted theory of the role of narratives among argumentation scholars  According to Fisher 1987, humans are storytellers by nature, and the “reason” in argumentation is therefore better understood in and through the narratives  He found that good reasons often take the form of narratives  Hoeken Fikkers 2014 investigated how integration of explicit argumentative content into narratives influences issue-relevant thinking and concluded that identifying with the character being in favor of the issue yielded a more positive attitude toward the issue  In a recent research, Bex 2011 proposes an argumentative-narrative model of reasoning with evidence, further elaborated in BIBREF106 ; also Niehaus et al 2012 proposes a computational model of narrative persuasion 
Stemming from another research field, LeytonEscobar2014 found that online community members who use and share narratives have higher participation levels and that narratives are useful tools to build cohesive cultures and increase participation  Betsch et al 2010 examined influencing vaccine intentions among parents and found that narratives carry more weight than statistics 
Summary of annotation studies
This section described two annotation studies that deal with argumentation in user-generated Web content on different levels of detail  In section SECREF44 , we argued for a need of document-level distinction of persuasiveness  We annotated 990 comments and forum posts, reaching moderate inter-annotator agreement (Fleiss' INLINEFORM0 0 59)  Section SECREF51 motivated the selection of a model for micro-level argument annotation, proposed its extension based on pre-study observations, and outlined the annotation set-up  This annotation study resulted into 340 documents annotated with the modified Toulmin's model and reached moderate inter-annotator agreement in the logos dimension (Krippendorff's INLINEFORM1 0 48)  These results make the annotated corpora suitable for training and evaluation computational models and each of these two annotation studies will have their experimental counterparts in the following section 
Experiments
This section presents experiments conducted on the annotated corpora introduced in section SECREF4   We put the main focus on identifying argument components in the discourse  To comply with the machine learning terminology, in this section we will use the term domain as an equivalent to a topic (remember that our dataset includes six different topics; see section SECREF38 ) 
We evaluate three different scenarios  First, we report ten-fold cross validation over a random ordering of the entire data set  Second, we deal with in-domain ten-fold cross validation for each of the six domains  Third, in order to evaluate the domain portability of our approach, we train the system on five domains and test on the remaining one for all six domains (which we report as cross-domain validation) 
Identification of argument components
In the following experiment, we focus on automatic identification of arguments in the discourse  Our approach is based on supervised and semi-supervised machine learning methods on the gold data Toulmin dataset introduced in section SECREF51  
An argument consists of different components (such as premises, backing, etc ) which are implicitly linked to the claim  In principle one document can contain multiple independent arguments  However, only 4% of the documents in our dataset contain arguments for both sides of the issue  Thus we simplify the task and assume there is only one argument per document 
Given the low inter-annotator agreement on the pathos dimension (Table TABREF77 ), we focus solely on recognizing the logical dimension of argument  The pathos dimension of argument remains an open problem for a proper modeling as well as its later recognition 
Since the smallest annotation unit is a token and the argument components do not overlap, we approach identification of argument components as a sequence labeling problem  We use the BIO encoding, so each token belongs to one of the following 11 classes: O (not a part of any argument component), Backing-B, Backing-I, Claim-B, Claim-I, Premise-B, Premise-I, Rebuttal-B, Rebuttal-I, Refutation-B, Refutation-I  This is the minimal encoding that is able to distinguish two adjacent argument components of the same type  In our data, 48% of all adjacent argument components of the same type are direct neighbors (there are no "O" tokens in between) 
We report Macro- INLINEFORM0 score and INLINEFORM1 scores for each of the 11 classes as the main evaluation metric  This evaluation is performed on the token level, and for each token the predicted label must exactly match the gold data label (classification of tokens into 11 classes) 
As instances for the sequence labeling model, we chose sentences rather than tokens  During our initial experiments, we observed that building a sequence labeling model for recognizing argument components as sequences of tokens is too fine-grained, as a single token does not convey enough information that could be encoded as features for a machine learner  However, as discussed in section UID73 , the annotations were performed on data pre-segmented to sentences and annotating tokens was necessary only when the sentence segmentation was wrong or one sentence contained multiple argument components  Our corpus consists of 3899 sentences, from which 2214 sentences (57%) contain no argument component  From the remaining ones, only 50 sentences (1%) have more than one argument component  Although in 19 cases (0 5%) the sentence contains a Claim-Premise pair which is an important distinction from the argumentation perspective, given the overall small number of such occurrences, we simplify the task by treating each sentence as if it has either one argument component or none 
The approximation with sentence-level units is explained in the example in Figure FIGREF112   In order to evaluate the expected performance loss using this approximation, we used an oracle that always predicts the correct label for the unit (sentence) and evaluated it against the true labels (recall that the evaluation against the true gold labels is done always on token level)  We lose only about 10% of macro INLINEFORM0 score (0 906) and only about 2% of accuracy (0 984)  This performance is still acceptable, while allowing to model sequences where the minimal unit is a sentence 
Table TABREF114 shows the distribution of the classes in the gold data Toulmin, where the labeling was already mapped to the sentences  The little presence of rebuttal and refutation (4 classes account only for 3 4% of the data) makes this dataset very unbalanced 
We chose SVMhmm BIBREF111 implementation of Structural Support Vector Machines for sequence labeling  Each sentence ( INLINEFORM0 ) is represented as a vector of real-valued features 
We defined the following feature sets:
FS0: Baseline lexical features
word uni-, bi-, and tri-grams (binary)
FS1: Structural, morphological, and syntactic features
First and last 3 tokens  Motivation: these tokens may contain discourse markers or other indicators for argument components, such as “therefore” and “since” for premises or “think” and “believe” for claims 
Relative position in paragraph and relative position in document  Motivation: We expect that claims are more likely to appear at the beginning or at the end of the document 
Number of POS 1-3 grams, dependency tree depth, constituency tree production rules, and number of sub-clauses  Based on BIBREF113  
FS2: Topic and sentiment features
30 features taken from a vector representation of the sentence obtained by using Gibbs sampling on LDA model BIBREF114 , BIBREF115 with 30 topics trained on unlabeled data from the raw corpus  Motivation: Topic representation of a sentence might be valuable for detecting off-topic sentences, namely non-argument components 
Scores for five sentiment categories (from very negative to very positive) obtained from Stanford sentiment analyzer BIBREF116   Motivation: Claims usually express opinions and carry sentiment 
FS3: Semantic, coreference, and discourse features
Binary features from Clear NLP Semantic Role Labeler BIBREF117   Namely, we extract agent, predicate + agent, predicate + agent + patient + (optional) negation, argument type + argument value, and discourse marker which are based on PropBank semantic role labels  Motivation: Exploit the semantics of Capturing the semantics of the sentences 
Binary features from Stanford Coreference Chain Resolver BIBREF118 , e g , presence of the sentence in a chain, transition type (i e , nominal–pronominal), distance to previous/next sentences in the chain, or number of inter-sentence coreference links  Motivation: Presence of coreference chains indicates links outside the sentence and thus may be informative, for example, for classifying whether the sentence is a part of a larger argument component 
Results of a PTDB-style discourse parser BIBREF119 , namely the type of discourse relation (explicit, implicit), presence of discourse connectives, and attributions  Motivation: It has been claimed that discourse relations play a role in argumentation mining BIBREF120  
FS4: Embedding features
300 features from word embedding vectors using word embeddings trained on part of the Google News dataset BIBREF121   In particular, we sum up embedding vectors (dimensionality 300) of each word, resulting into a single vector for the entire sentence  This vector is then directly used as a feature vector  Motivation: Embeddings helped to achieve state-of-the-art results in various NLP tasks BIBREF116 , BIBREF122  
Except the baseline lexical features, all feature types are extracted not only for the current sentence INLINEFORM0 , but also for INLINEFORM1 preceding and subsequent sentences, namely INLINEFORM2 , INLINEFORM3 , INLINEFORM4 INLINEFORM5 , INLINEFORM6 , where INLINEFORM7 was empirically set to 4  Each feature is then represented with a prefix to determine its relative position to the current sequence unit 
Let us first discuss the upper bounds of the system  Performance of the three human annotators is shown in the first column of Table TABREF139 (results are obtained from a cumulative confusion matrix)  The overall Macro- INLINEFORM0 score is 0 602 (accuracy 0 754)  If we look closer at the different argument components, we observe that humans are good at predicting claims, premises, backing and non-argumentative text (about 0 60-0 80 INLINEFORM1 ), but on rebuttal and refutation they achieve rather low scores  Without these two components, the overall human Macro- INLINEFORM2 would be 0 707  This trend follows the inter-annotator agreement scores, as discussed in section UID75  
In our experiments, the feature sets were combined in the bottom-up manner, starting with the simple lexical features (FS0), adding structural and syntactic features (FS1), then adding topic and sentiment features (FS2), then features reflecting the discourse structure (FS3), and finally enriched with completely unsupervised latent vector space representation (FS4)  In addition, we were gradually removing the simple features (e g , without lexical features, without syntactic features, etc ) to test the system with more “abstract” feature sets (feature ablation)  The results are shown in Table TABREF139  
The overall best performance (Macro- INLINEFORM0 0 251) was achieved using the rich feature sets (01234 and 234) and significantly outperformed the baseline as well as other feature sets  Classification of non-argumentative text (the "O" class) yields about 0 7 INLINEFORM1 score even in the baseline setting  The boundaries of claims (Cla-B), premises (Pre-B), and backing (Bac-B) reach in average lower scores then their respective inside tags (Cla-I, Pre-I, Bac-I)  It can be interpreted such that the system is able to classify that a certain sentence belongs to a certain argument component, but the distinction whether it is a beginning of the argument component is harder  The very low numbers for rebuttal and refutation have two reasons  First, these two argument components caused many disagreements in the annotations, as discussed in section UID86 , and were hard to recognize for the humans too  Second, these four classes have very few instances in the corpus (about 3 4%, see Table TABREF114 ), so the classifier suffers from the lack of training data 
The results for the in-domain cross validation scenario are shown in Table TABREF140   Similarly to the cross-validation scenario, the overall best results were achieved using the largest feature set (01234)  For mainstreaming and red-shirting, the best results were achieved using only the feature set 4 (embeddings)  These two domains contain also fewer documents, compared to other domains (refer to Table TABREF71 )  We suspect that embeddings-based features convey important information when not enough in-domain data are available  This observation will become apparent in the next experiment 
The cross-domain experiments yield rather poor results for most of the feature combinations (Table TABREF141 )  However, using only feature set 4 (embeddings), the system performance increases rapidly, so it is even comparable to numbers achieved in the in-domain scenario  These results indicate that embedding features generalize well across domains in our task of argument component identification  We leave investigating better performing vector representations, such as paragraph vectors BIBREF123 , for future work 
Error analysis based on the probabilistic confusion matrix BIBREF124 shown in Table TABREF142 reveals further details  About a half of the instances for each class are misclassified as non-argumentative (the "O" prediction) 
Backing-B is often confused with Premise-B (12%) and Backing-I with Premise-I (23%)  Similarly, Premise-I is misclassified as Backing-I in 9%  This shows that distinguishing between backing and premises is not easy because these two components are similar such that they support the claim, as discussed in section UID86   We can also see that the misclassification is consistent among *-B and *-I tags 
Rebuttal is often misclassified as Premise (28% for Rebuttal-I and 18% for Rebuttal-B; notice again the consistency in *-B and *-I tags)  This is rather surprising, as one would expect that rebuttal would be confused with a claim, because its role is to provide an opposing view 
Refutation-B and Refutation-I is misclassified as Premise-I in 19% and 27%, respectively  This finding confirms the discussion in section UID86 , because the role of refutation is highly context-dependent  In a pragmatic perspective, it is put forward to indirectly support the claim by attacking the rebuttal, thus having a similar function to the premise 
We manually examined miss-classified examples produced the best-performing system to find out which phenomena pose biggest challenges  Properly detecting boundaries of argument components caused problems, as shown in Figure FIGREF146 (a)  This goes in line with the granularity annotation difficulties discussed in section UID86   The next example in Figure FIGREF146 (b) shows that even if boundaries of components were detected precisely, the distinction between premise and backing fails  The example also shows that in some cases, labeling on clause level is required (left-hand side claim and premise) but the approximation in the system cannot cope with this level of detail (as explained in section UID111 )  Confusing non-argumentative text and argument components by the system is sometimes plausible, as is the case of the last rhetorical question in Figure FIGREF146 (c)  On the other hand, the last example in Figure FIGREF146 (d) shows that some claims using figurative language were hard to be identified  The complete predictions along with the gold data are publicly available 
SVMhmm offers many hyper-parameters with suggested default values, from which three are of importance  Parameter INLINEFORM0 sets the order of dependencies of transitions in HMM, parameter INLINEFORM1 sets the order of dependencies of emissions in HMM, and parameter INLINEFORM2 represents a trading-off slack versus magnitude of the weight-vector  For all experiments, we set all the hyper-parameters to their default values ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 )  Using the best performing feature set from Table TABREF139 , we experimented with a grid search over different values ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) but the results did not outperform the system trained with default parameter values 
The INLINEFORM0 scores might seem very low at the first glance  One obvious reason is the actual performance of the system, which gives a plenty of room for improvement in the future  But the main cause of low INLINEFORM2 numbers is the evaluation measure — using 11 classes on the token level is very strict, as it penalizes a mismatch in argument component boundaries the same way as a wrongly predicted argument component type  Therefore we also report two another evaluation metrics that help to put our results into a context 
Krippendorff's INLINEFORM0 — It was also used for evaluating inter-annotator agreement (see section UID75 ) 
Boundary similarity BIBREF125 — Using this metric, the problem is treated solely as a segmentation task without recognizing the argument component types 
As shown in Table TABREF157 (the Macro- INLINEFORM0 scores are repeated from Table TABREF139 ), the best-performing system achieves 0 30 score using Krippendorf's INLINEFORM1 , which is in the middle between the baseline and the human performance (0 48) but is considered as poor from the inter-annotator agreement point of view BIBREF54   The boundary similarity metrics is not directly suitable for evaluating argument component classification, but reveals a sub-task of finding the component boundaries  The best system achieved 0 32 on this measure  Vovk2013MT used this measure to annotate argument spans and his annotators achieved 0 36 boundary similarity score  Human annotators in BIBREF125 reached 0 53 boundary similarity score 
The overall performance of the system is also affected by the accuracy of individual NPL tools used for extracting features  One particular problem is that the preprocessing models we rely on (POS, syntax, semantic roles, coreference, discourse; see section UID115 ) were trained on newswire corpora, so one has to expect performance drop when applied on user-generated content  This is however a well-known issue in NLP BIBREF126 , BIBREF127 , BIBREF128  
To get an impression of the actual performance of the system on the data, we also provide the complete output of our best performing system in one PDF document together with the gold annotations in the logos dimension side by side in the accompanying software package  We believe this will help the community to see the strengths of our model as well as possible limitations of our current approaches 
Conclusions
Let us begin with summarizing answers to the research questions stated in the introduction  First, as we showed in section UID55 , existing argumentation theories do offer models for capturing argumentation in user-generated content on the Web  We built upon the Toulmin's model and proposed some extensions 
Second, as compared to the negative experiences with annotating using Walton's schemes (see sections UID52 and SECREF31 ), our modified Toulmin's model offers a trade-off between its expressiveness and annotation reliability  However, we found that the capabilities of the model to capture argumentation depend on the register and topic, the length of the document, and inherently on the literary devices and structures used for expressing argumentation as these properties influenced the agreement among annotators 
Third, there are aspects of online argumentation that lack their established theoretical counterparts, such as rhetorical questions, figurative language, narratives, and fallacies in general  We tried to model some of them in the pathos dimension of argument (section UID103 ), but no satisfying agreement was reached  Furthermore, we dealt with a step that precedes argument analysis by filtering documents given their persuasiveness with respect to the controversy  Finally, we proposed a computational model based on machine learning for identifying argument components (section SECREF108 )  In this identification task, we experimented with a wide range of linguistically motivated features and found that (1) the largest feature set (including n-grams, structural features, syntactic features, topic distribution, sentiment distribution, semantic features, coreference feaures, discourse features, and features based on word embeddings) performs best in both in-domain and all-data cross validation, while (2) features based only on word embeddings yield best results in cross-domain evaluation 
Since there is no one-size-fits-all argumentation theory to be applied to actual data on the Web, the argumentation model and an annotation scheme for argumentation mining is a function of the task requirements and the corpus properties  Its selection should be based on the data at hand and the desired application  Given the proposed use-case scenarios (section SECREF1 ) and the results of our annotation study (section SECREF51 ), we recommend a scheme based on Toulmin's model for short documents, such as comments or forum posts 

ground_truth: claim, premise, backing, rebuttal, and refutation

claim, premise, backing, rebuttal, refutation