prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 1: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 
Context 2: Introduction
Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP)  It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0  
Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP  Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis  Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc ) either explicitly or implicitly referred to in the text is considered  Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0  
Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled  The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2   In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates  In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments  For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4   In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues  The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5   Stance detection on a corpus of student essays is considered in BIBREF6   After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6   In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web 
Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed  The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0   In BIBREF9 , a stance-community detection approach called SCIFNET is proposed  SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks  A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing  The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10   Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described  Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0  
In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available  The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included  We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features 
To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets  Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content 
The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information  Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions  Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary 
A Stance Detection Data Set
We have decided to consider tweets about popular sports clubs as our domain for stance detection  Considerable amounts of tweets are being published for sports-related events at every instant  Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey  As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs 
In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11   We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs  Then, we have annotated the stance information in the tweets for these targets as Favor or Against  Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals 
For the purposes of the current study, we have not annotated any tweets with the Neither class  This stance class and even finer-grained classes can be considered in further annotation studies  We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised  Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly 
At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2  Hence, our data set is a balanced one although it is currently limited in size  The corresponding stance annotations are made publicly available at http://ceng metu edu tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset csv in Comma Separated Values (CSV) format  The file contains three columns with the corresponding headers  The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against 
To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish  Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish  Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues 
Stance Detection Experiments Using SVM Classifiers
It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0   In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features  Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets  The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13  
We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel  The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure 
The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set  The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class  This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets  The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets  Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i e , better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0   Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes  Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set  On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class  Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems  Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature 
We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0   However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results  This observation may be due to the relatively limited size of the tweet data set employed  Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set  Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work 
With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams  The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2  
When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0 5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1 8%  Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system  We leave other ways of exploiting hashtags for stance detection as a future work 
To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets 
Future Prospects
Future work based on the current study includes the following:
Conclusion
Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis  It is the detection of stance within text towards a target which may be explicitly specified in the text or not  In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey  The corresponding annotations are made publicly-available for research purposes  To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set  Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature  10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems  Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets 
Context 3: Introduction
From a group of small users at the time of its inception in 2009, Quora has evolved in the last few years into one of the largest community driven Q&A sites with diverse user communities  With the help of efficient content moderation/review policies and active in-house review team, efficient Quora bots, this site has emerged into one of the largest and reliable sources of Q&A on the Internet  On Quora, users can post questions, follow questions, share questions, tag them with relevant topics, follow topics, follow users apart from answering, commenting, upvoting/downvoting etc  The integrated social structure at the backbone of it and the topical organization of its rich content have made Quora unique with respect to other Q&A sites like Stack Overflow, Yahoo! Answers etc  and these are some of the prime reasons behind its popularity in recent times  Quality question posting and getting them answered are the key objectives of any Q&A site  In this study we focus on the answerability of questions on Quora, i e , whether a posted question shall eventually get answered  In Quora, the questions with no answers are referred to as “open questions”  These open questions need to be studied separately to understand the reason behind their not being answered or to be precise, are there any characteristic differences between `open' questions and the answered ones  For example, the question “What are the most promising advances in the treatment of traumatic brain injuries?” was posted on Quora on 23rd June, 2011 and got its first answer after almost 2 years on 22nd April, 2013  The reason that this question remained open so long might be the hardness of answering it and the lack of visibility and experts in the domain  Therefore, it is important to identify the open questions and take measures based on the types - poor quality questions can be removed from Quora and the good quality questions can be promoted so that they get more visibility and are eventually routed to topical experts for better answers 
Characterization of the questions based on question quality requires expert human interventions often judging if a question would remain open based on factors like if it is subjective, controversial, open-ended, vague/imprecise, ill-formed, off-topic, ambiguous, uninteresting etc  Collecting judgment data for thousands of question posts is a very expensive process  Therefore, such an experiment can be done only for a small set of questions and it would be practically impossible to scale it up for the entire collection of posts on the Q&A site  In this work, we show that appropriate quantification of various linguistic activities can naturally correspond to many of the judgment factors mentioned above (see table 2 for a collection of examples)  These quantities encoding such linguistic activities can be easily measured for each question post and thus helps us to have an alternative mechanism to characterize the answerability on the Q&A site 
There are several research works done in Q&A focusing on content of posts  BIBREF0 exploit community feedback to identify high quality content on Yahoo! Answers  BIBREF1 use textual features to predict answer quality on Yahoo! Answers  BIBREF2 , investigate predictors of answer quality through a comparative, controlled field study of user responses  BIBREF3 study the problem of how long questions remain unanswered  BIBREF4 propose a prediction model on how many answers a question shall receive

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which sports clubs are the targets?
Context 3:  BIBREF5 analyze and predict unanswered questions on Yahoo Answers  BIBREF6 study question quality in Yahoo! Answers 
Dataset description
We obtained our Quora dataset BIBREF7 through web-based crawls between June 2014 to August 2014  This crawling exercise has resulted in the accumulation of a massive Q&A dataset spanning over a period of over four years starting from January 2010 to May 2014  We initiated crawling with 100 questions randomly selected from different topics so that different genre of questions can be covered  The crawling of the questions follow a BFS pattern through the related question links  We obtained 822,040 unique questions across 80,253 different topics with a total of 1,833,125 answers to these questions  For each question, we separately crawl their revision logs that contain different types of edit information for the question and the activity log of the question asker 
Linguistic activities on Quora
In this section, we identify various linguistic activities on Quora and propose quantifications of the language usage patterns in this Q&A site  In particular, we show that there exists significant differences in the linguistic structure of the open and the answered questions  Note that most of the measures that we define are simple, intuitive and can be easily obtained automatically from the data (without manual intervention)  Therefore the framework is practical, inexpensive and highly scalable 
Content of a question text is important to attract people and make them engage more toward it  The linguistic structure (i e , the usage of POS tags, the use of Out-of-Vocabulary words, character usage etc ) one adopts are key factors for answerability of questions  We shall discuss the linguistic structure that often represents the writing style of a question asker 
In fig 1 (a), we observe that askers of open questions generally use more no  of words compared to answered questions  To understand the nature of words (standard English words or chat-like words frequently used in social media) used in the text, we compare the words with GNU Aspell dictionary to see whether they are present in the dictionary or not  We observe that both open questions and answered questions follow similar distribution (see fig 1 (b))  Part-of-Speech (POS) tags are indicators of grammatical aspects of texts  To observe how the Part-of-Speech tags are distributed in the question texts, we define a diversity metric  We use the standard CMU POS tagger BIBREF8 for identifying the POS tags of the constituent words in the question

ground_truth: Galatasaray, Fenerbahçe

Galatasaray , Fenerbahçe 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do the answered questions measure for the usefulness of the answer?
Context 3:  We define the POS tag diversity (POSDiv) of a question $q_i$ as follows: $POSDiv(q_i) = -\sum _{j \in pos_{set}}p_j\times \log (p_j)$ where $p_j$ is the probability of the $j^{th}$ POS in the set of POS tags  Fig 1 (c) shows that the answered questions have lower POS tag diversity compared to open questions  Question texts undergo several edits so that their readability and the engagement toward them are enhanced  It is interesting to identify how far such edits can make the question different from the original version of it  To capture this phenomena, we have adopted ROUGE-LCS recall BIBREF9 from the domain of text summarization  Higher the recall value, lesser are the changes in the question text  From fig 1 (d), we observe that open questions tend to have higher recall compared to the answered ones which suggests that they have not gone through much of text editing thus allowing for almost no scope of readability enhancement 
Psycholinguistic analysis:
The way an individual talks or writes, give us clue to his/her linguistic, emotional, and cognitive states  A question asker's linguistic, emotional, cognitive states are also revealed through the language he/she uses in the question text  In order to capture such psycholinguistic aspects of the asker, we use Linguistic Inquiry and Word Count (LIWC) BIBREF10 that analyzes various emotional, cognitive, and structural components present in individuals' written texts  LIWC takes a text document as input and outputs a score for the input for each of the LIWC categories such as linguistic (part-of-speech of the words, function words etc ) and psychological categories (social, anger, positive emotion, negative emotion, sadness etc ) based on the writing style and psychometric properties of the document  In table 1 , we perform a comparative analysis of the asker's psycholinguistic state while asking an open question and an answered question 
Askers of open questions use more function words, impersonal pronouns, articles on an average whereas asker of answered questions use more personal pronouns, conjunctions and adverbs to describe their questions  Essentially, open questions lack content words compared to answered questions which, in turn, affects the readability of the question  As far as the psychological aspects are concerned, answered question askers tend to use more social, family, human related words on average compared to an open question asker  The open question askers express more positive emotions whereas the answered question asker tend to express more negative emotions in their texts  Also, answered question askers are more emotionally involved and their questions reveal higher usage of anger, sadness, anxiety related words compared to that of open questions  Open questions, on the other hand, contains more sexual, body, health related words which might be reasons why they do not attract answers 
In table 2 , we show a collection of examples of open questions to illustrate that many of the above quantities based on the linguistic activities described in this section naturally correspond to the factors that human judges consider responsible for a question remaining unanswered  This is one of the prime reasons why these quantities qualify as appropriate indicators of answerability 
Prediction model
In this section, we describe the prediction framework in detail  Our goal is to predict whether a given question after a time period $t$ will be answered or not 
Linguistic styles of the question asker
The content and way of posing a question is important to attract answers  We have observed in the previous section that these linguistic as well as psycholinguistic aspects of the question asker are discriminatory factors  For the prediction, we use the following features:

ground_truth: No