prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the tagging scheme employed?
Context 1: Introduction
There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings  It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2   Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8   A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect  Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9   Consider the following two examples:
The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns  Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut" 
Pun detection is the task of detecting whether there is a pun residing in the given text  The goal of pun location is to find the exact word appearing in the text that implies more than one meanings  Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5   Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training  For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun  Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used 
Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks  In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model  The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag  Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint  Statistics on the corpora also show that a pun tends to appear in the second half of a context  To capture such a structural property, we also incorporate word position knowledge into our structured prediction model  Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks  To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach 
Problem Definition
We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }:
INLINEFORM0 tag means the current word is not a pun 
INLINEFORM0 tag means the current word is a pun 
If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun 
The contexts have the characteristic that each context contains a maximum of one pun BIBREF9   In other words, there exists only one pun if the given sentence is detected as the one containing a pun  Otherwise, there is no pun residing in the text  To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 } 
INLINEFORM0 tag indicates that the current word appears before the pun in the given context 
INLINEFORM0 tag highlights the current word is a pun 
INLINEFORM0 tag indicates that the current word appears after the pun 
We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text 
Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure  Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively  Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut " Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8   Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8   Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach 
Model
Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15   In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling  Our model architecture is illustrated in Figure FIGREF8 with a running example  Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora  Our model is then trained on pairs of INLINEFORM3  
Input  The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb  To capture this characteristic, we consider lexical features at the character level  Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences  Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces 
We also observe that a pun tends to appear at the end of a sentence  Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively  We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information  A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20   In this work, we directly adopt the value of the binary indicator as part of the input 
The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model 
Tagging  The input is then fed into a BiLSTM network, which will be able to capture contextual information  For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1   The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15   The conditional probability is defined as:
where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5   During training, we minimize the negative log-likelihood summed over all training instances:
where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set  During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 :
This search process can be done efficiently using the Viterbi algorithm 
Datasets and Settings
We evaluate our model on two benchmark datasets BIBREF9   The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun  The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun  We notice there is no standard splitting information provided for both datasets  Thus we apply 10-fold cross validation  To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end 
For each fold, we randomly select 10% of the instances from the training set for development  Word embeddings are initialized with the 100-dimensional Glove BIBREF21   The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training  The pre-trained word embeddings are not updated during training  The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300  We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0 015 
For the pun detection task, if the predicted tag sequence contains at least one INLINEFORM0 tag, we regard the output (i e , the prediction of our pun detection model) for this task as true, otherwise false  For the pun location task, a predicted pun is regarded as correct if and only if it is labeled as the gold pun in the dataset  As to pun location, to make fair comparisons with prior studies, we only consider the instances that are labeled as the ones containing a pun  We report precision, recall and INLINEFORM1 score in Table TABREF11   A list of prior works that did not employ joint learning are also shown in the first block of Table TABREF11  
Results
We also implemented a baseline model based on conditional random fields (CRF), where features like POS tags produced by the Stanford POS tagger BIBREF27 , n-grams, label transitions, word suffixes and relative position to the end of the text are considered  We can see that our model with the INLINEFORM0 tagging scheme yields new state-of-the-art INLINEFORM1 scores on pun detection and competitive results on pun location, compared to baselines that do not adopt joint learning in the first block  For location on heterographic puns, our model's performance is slightly lower than the system of BIBREF25 , which is a rule-based locator  Compared to CRF, we can see that our model, either with the INLINEFORM2 or the INLINEFORM3 scheme, yields significantly higher recall on both detection and location tasks, while the precisions are relatively close  This demonstrates the effectiveness of BiLSTM, which learns the contextual features of given texts – such information appears to be helpful in recalling more puns 
Compared to the INLINEFORM0 scheme, the INLINEFORM1 tagging scheme is able to yield better performance on these two tasks  After studying outputs from these two approaches, we found that one leading source of error for the INLINEFORM2 approach is that there exist more than one words in a single instance that are assigned with the INLINEFORM3 tag  However, according to the description of pun in BIBREF9 , each context contains a maximum of one pun  Thus, such a useful structural constraint is not well captured by the simple approach based on the INLINEFORM4 tagging scheme  On the other hand, by applying the INLINEFORM5 tagging scheme, such a constraint is properly captured in the model  As a result, the results for such a approach are significantly better than the approach based on the INLINEFORM6 tagging scheme, as we can observe from the table  Under the same experimental setup, we also attempted to exclude word position features  Results are given by INLINEFORM7 - INLINEFORM8   It is expected that the performance of pun location drops, since such position features are able to capture the interesting property that a pun tends to appear in the second half of a sentence  While such knowledge is helpful for the location task, interestingly, a model without position knowledge yields improved performance on the pun detection task  One possible reason is that detecting whether a sentence contains a pun is not concerned with such word position information 
Additionally, we conduct experiments over sentences containing a pun only, namely 1,607 and 1,271 instances from homographic and heterographic pun corpora separately  It can be regarded as a “pipeline” method where the classifier for pun detection is regarded as perfect  Following the prior work of BIBREF4 , we apply 10-fold cross validation  Since we are given that all input sentences contain a pun, we only report accumulated results on pun location, denoted as Pipeline in Table TABREF11   Compared with our approaches, the performance of such an approach drops significantly  On the other hand, such a fact demonstrates that the two task, detection and location of puns, can reinforce each other  These figures demonstrate the effectiveness of our sequence labeling method to detect and locate English puns in a joint manner 
Error Analysis
We studied the outputs from our system and make some error analysis  We found the errors can be broadly categorized into several types, and we elaborate them here  1) Low word coverage: since the corpora are relatively small, there exist many unseen words in the test set  Learning the representations of such unseen words is challenging, which affects the model's performance  Such errors contribute around 40% of the total errors made by our system  2) Detection errors: we found many errors are due to the model's inability to make correct pun detection  Such inability harms both pun detection and pun location  Although our approach based on the INLINEFORM0 tagging scheme yields relatively higher scores on the detection task, we still found that 40% of the incorrectly predicted instances fall into this group  3) Short sentences: we found it was challenging for our model to make correct predictions when the given text is short  Consider the example “Superglue! Tom rejoined," here the word rejoined is the corresponding pun  However, it would be challenging to figure out the pun with such limited contextual information 
Related Work
Most existing systems address pun detection and location separately  BIBREF22 applied word sense knowledge to conduct pun detection  BIBREF24 trained a bidirectional RNN classifier for detecting homographic puns  Next, a knowledge-based approach is adopted to find the exact pun  Such a system is not applicable to heterographic puns  BIBREF28 applied Google n-gram and word2vec to make decisions  The phonetic distance via the CMU Pronouncing Dictionary is computed to detect heterographic puns  BIBREF10 used the hidden Markov model and a cyclic dependency network with rich features to detect and locate puns  BIBREF23 used a supervised approach to pun detection and a weakly supervised approach to pun location based on the position within the context and part of speech features  BIBREF25 proposed a rule-based system for pun location that scores candidate words according to eleven simple heuristics  Two systems are developed to conduct detection and location separately in the system known as UWAV BIBREF3   The pun detector combines predictions from three classifiers  The pun locator considers word2vec similarity between every pair of words in the context and position to pinpoint the pun  The state-of-the-art system for homographic pun location is a neural method BIBREF4 , where the word senses are incorporated into a bidirectional LSTM model  This method only supports the pun location task on homographic puns  Another line of research efforts related to this work is sequence labeling, such as POS tagging, chunking, word segmentation and NER  The neural methods have shown their effectiveness in this task, such as BiLSTM-CNN BIBREF13 , GRNN BIBREF29 , LSTM-CRF BIBREF30 , LSTM-CNN-CRF BIBREF14 , LM-LSTM-CRF BIBREF15  
In this work, we combine pun detection and location tasks as a single sequence labeling problem  Inspired by the work of BIBREF15 , we also adopt a LSTM-CRF with character embeddings to make labeling decisions 
Conclusion
In this paper, we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective  We observe that each text in our corpora contains a maximum of one pun  Hence, we design a novel tagging scheme to incorporate such a constraint  Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase  We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task, but was not explored in previous works  Furthermore, unlike many previous approaches, our approach, though simple, is generally applicable to both heterographic and homographic puns  Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective 
Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns  Research on puns for other languages such as Chinese is still under-explored, which could also be an interesting direction for our future studies 
Acknowledgments
We would like to thank the three anonymous reviewers for their thoughtful and constructive comments  This work is supported by Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 Project MOE2017-T2-1-156, and is partially supported by SUTD project PIE-SGP-AI-2018-01 
Context 2: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 

ground_truth: A new tagging scheme that tags the words before and after the pun as well as the pun words.

a new tagging scheme consisting of three tags, namely { INLINEFORM0 }

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 4: Introduction
Writing errors can occur in many different forms – from relatively simple punctuation and determiner errors, to mistakes including word tense and form, incorrect collocations and erroneous idioms  Automatically identifying all of these errors is a challenging task, especially as the amount of available annotated data is very limited  Rei2016 showed that while some error detection algorithms perform better than others, it is additional training data that has the biggest impact on improving performance 
Being able to generate realistic artificial data would allow for any grammatically correct text to be transformed into annotated examples containing writing errors, producing large amounts of additional training examples  Supervised error generation systems would also provide an efficient method for anonymising the source corpus – error statistics from a private corpus can be aggregated and applied to a different target text, obscuring sensitive information in the original examination scripts  However, the task of creating incorrect data is somewhat more difficult than might initially appear – naive methods for error generation can create data that does not resemble natural errors, thereby making downstream systems learn misleading or uninformative patterns 
Previous work on artificial error generation (AEG) has focused on specific error types, such as prepositions and determiners BIBREF0 , BIBREF1 , or noun number errors BIBREF2   Felice2014a investigated the use of linguistic information when generating artificial data for error correction, but also restricting the approach to only five error types  There has been very limited research on generating artificial data for all types, which is important for general-purpose error detection systems  For example, the error types investigated by Felice2014a cover only 35 74% of all errors present in the CoNLL 2014 training dataset, providing no additional information for the majority of errors 
In this paper, we investigate two supervised approaches for generating all types of artificial errors  We propose a framework for generating errors based on statistical machine translation (SMT), training a model to translate from correct into incorrect sentences  In addition, we describe a method for learning error patterns from an annotated corpus and transplanting them into error-free text  We evaluate the effect of introducing artificial data on two error detection benchmarks  Our results show that each method provides significant improvements over using only the available training set, and a combination of both gives an absolute improvement of 4 3% in INLINEFORM0 , without requiring any additional annotated data 
Error Generation Methods
We investigate two alternative methods for AEG  The models receive grammatically correct text as input and modify certain tokens to produce incorrect sequences  The alternative versions of each sentence are aligned using Levenshtein distance, allowing us to identify specific words that need to be marked as errors  While these alignments are not always perfect, we found them to be sufficient for practical purposes, since alternative alignments of similar sentences often result in the same binary labeling  Future work could explore more advanced alignment methods, such as proposed by felice-bryant-briscoe 
In Section SECREF4 , this automatically labeled data is then used for training error detection models 
Machine Translation
We treat AEG as a translation task – given a correct sentence as input, the system would learn to translate it to contain likely errors, based on a training corpus of parallel data  Existing SMT approaches are already optimised for identifying context patterns that correspond to specific output sequences, which is also required for generating human-like errors  The reverse of this idea, translating from incorrect to correct sentences, has been shown to work well for error correction tasks BIBREF2 , BIBREF3 , and round-trip translation has also been shown to be promising for correcting grammatical errors BIBREF4  
Following previous work BIBREF2 , BIBREF5 , we build a phrase-based SMT error generation system  During training, error-corrected sentences in the training data are treated as the source, and the original sentences written by language learners as the target  Pialign BIBREF6 is used to create a phrase translation table directly from model probabilities  In addition to default features, we add character-level Levenshtein distance to each mapping in the phrase table, as proposed by Felice:2014-CoNLL  Decoding is performed using Moses BIBREF7 and the language model used during decoding is built from the original erroneous sentences in the learner corpus  The IRSTLM Toolkit BIBREF8 is used for building a 5-gram language model with modified Kneser-Ney smoothing BIBREF9  
Pattern Extraction
We also describe a method for AEG using patterns over words and part-of-speech (POS) tags, extracting known incorrect sequences from a corpus of annotated corrections  This approach is based on the best method identified by Felice2014a, using error type distributions; while they covered only 5 error types, we relax this restriction and learn patterns for generating all types of errors 
The original and corrected sentences in the corpus are aligned and used to identify short transformation patterns in the form of (incorrect phrase, correct phrase)  The length of each pattern is the affected phrase, plus up to one token of context on both sides  If a word form changes between the incorrect and correct text, it is fully saved in the pattern, otherwise the POS tags are used for matching 
For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern:
(VVD shop_VV0 II, VVD shopping_VVG II)
After collecting statistics from the background corpus, errors can be inserted into error-free text  The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence  We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data  For each input sentence, we first decide how many errors will be generated (using probabilities from the background corpus) and attempt to create them by sampling from the collection of applicable patterns  This process is repeated until all the required errors have been generated or the sentence is exhausted  During generation, we try to balance the distribution of error types as well as keeping the same proportion of incorrect and correct sentences as in the background corpus BIBREF10   The required POS tags were generated with RASP BIBREF11 , using the CLAWS2 tagset 
Error Detection Model
We construct a neural sequence labeling model for error detection, following the previous work BIBREF12 , BIBREF13   The model receives a sequence of tokens as input and outputs a prediction for each position, indicating whether the token is correct or incorrect in the current context  The tokens are first mapped to a distributed vector space, resulting in a sequence of word embeddings

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the baselines?
Context 3: Introduction
The task of speculation detection and scope resolution is critical in distinguishing factual information from speculative information  This has multiple use-cases, like systems that determine the veracity of information, and those that involve requirement analysis  This task is particularly important to the biomedical domain, where patient reports and medical articles often use this feature of natural language  This task is commonly broken down into two subtasks: the first subtask, speculation cue detection, is to identify the uncertainty cue in a sentence, while the second subtask: scope resolution, is to identify the scope of that cue  For instance, consider the example:
It might rain tomorrow 
The speculation cue in the sentence above is ‘might’ and the scope of the cue ‘might’ is ‘rain tomorrow’  Thus, the speculation cue is the word that expresses the speculation, while the words affected by the speculation are in the scope of that cue 
This task was the CoNLL-2010 Shared Task (BIBREF0), which had 3 different subtasks  Task 1B was speculation cue detection on the BioScope Corpus, Task 1W was weasel identification from Wikipedia articles, and Task 2 was speculation scope resolution from the BioScope Corpus  For each task, the participants were provided the train and test set, which is henceforth referred to as Task 1B CoNLL and Task 2 CoNLL throughout this paper 
For our experimentation, we use the sub corpora of the BioScope Corpus (BIBREF1), namely the BioScope Abstracts sub corpora, which is referred to as BA, and the BioScope Full Papers sub corpora, which is referred to as BF  We also use the SFU Review Corpus (BIBREF2), which is referred to as SFU 
This subtask of natural language processing, along with another similar subtask, negation detection and scope resolution, have been the subject of a body of work over the years  The approaches used to solve them have evolved from simple rule-based systems (BIBREF3) based on linguistic information extracted from the sentences, to modern deep-learning based methods  The Machine Learning techniques used varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), while the deep learning approaches included Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)  Figures FIGREF1 and FIGREF1 contain a summary of the papers addressing speculation detection and scope resolution (BIBREF13, BIBREF5, BIBREF9, BIBREF3, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF6, BIBREF11, BIBREF18, BIBREF10, BIBREF19, BIBREF7, BIBREF4, BIBREF8) 
Inspired by the most recent approach of applying BERT to negation detection and scope resolution (BIBREF12), we take this approach one step further by performing a comparative analysis of three popular transformer-based architectures: BERT (BIBREF20), XLNet (BIBREF21) and RoBERTa (BIBREF22), applied to speculation detection and scope resolution  We evaluate the performance of each model across all datasets via the single dataset training approach, and report all scores including inter-dataset scores (i e  train on one dataset, evaluate on another) to test the generalizability of the models  This approach outperforms all existing systems on the task of speculation detection and scope resolution  Further, we jointly train on multiple datasets and obtain improvements over the single dataset training approach on most datasets 
Contrary to results observed on benchmark GLUE tasks, we observe XLNet consistently outperforming RoBERTa  To confirm this observation, we apply these models to the negation detection and scope resolution task, and observe a continuity in this trend, reporting state-of-the-art results on three of four datasets on the negation scope resolution task 
The rest of the paper is organized as follows: In Section 2, we provide a detailed description of our methodology and elaborate on the experimentation details  In Section 3, we present our results and analysis on the speculation detection and scope resolution task, using the single dataset and the multiple dataset training approach  In Section 4, we show the results of applying XLNet and RoBERTa on negation detection and scope resolution and propose a few reasons to explain why XLNet performs better than RoBERTa  Finally, the future scope and conclusion is mentioned in Section 5 
Methodology and Experimental Setup
We use the methodology by Khandelwal and Sawant (BIBREF12), and modify it to support experimentation with multiple models 
For Speculation Cue Detection:
Input Sentence: It might rain tomorrow 
True Labels: Not-A-Cue, Cue, Not-A-Cue, Not-A-Cue 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
1 – Normal Cue 2 – Multiword Cue 3 – Not a cue 4 – Pad token
Thus, the preprocessed sequence is as follows:
Input Sentence: [It, might, rain, tomorrow]
True Labels: [3,1,3,3]
Then, we preprocess the input using the tokenizer for the model being used (BERT, XLNet or RoBERTa): splitting each word into one or more tokens, and converting each token to it’s corresponding tokenID, and padding it to the maximum input length of the model  Thus,
Input Sentence: [wtt(It), wtt(might), wtt(rain), wtt(tom), wtt(## or), wtt(## row), wtt(〈pad 〉),wtt(〈pad 〉)   ]
True Labels: [3,1,3,3,3,3,4,4,4,4,   ]
The word ‘tomorrow' has been split into 3 tokens, ‘tom', ‘##or' and ‘##row'  The function to convert the word to tokenID is represented by wtt 
For Speculation Scope Resolution:
If a sentence has multiple cues, each cue's scope will be resolved individually 
Input Sentence: It might rain tomorrow 
True Labels: Out-Of-Scope, Out-Of-Scope, In-Scope, In-Scope 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
0 – Out-Of-Scope 1 – In-Scope
Thus, the preprocessed sequence is as follows:
True Scope Labels: [0,0,1,1]
As for cue detection, we preprocess the input using the tokenizer for the model being used  Additionally, we need to indicate which cue's scope we want to find in the input sentence  We do this by inserting a special token representing the token type (according to the cue detection annotation schema) before the cue word whose scope is being resolved  Here, we want to find the scope of the cue ‘might'  Thus,
Input Sentence: [wtt(It), wtt(〈token[1]〉), wtt(might), wtt(rain), wtt(tom), wtt(##or), wtt(##row), wtt(〈pad〉), wtt(〈pad〉)   ]
True Scope Labels: [0,0,1,1,1,1,0,0,0,0,   ]
Now, the preprocessed input for cue detection and similarly for scope detection is fed as input to our model as follows:
X = Model (Input)
Y = W*X + b
The W matrix is a matrix of size n_hidden x num_classes (n_hidden is the size of the representation of a token within the model)  These logits are fed to the loss function 
We use the following variants of each model:
BERT: bert-base-uncaseds3 amazonaws com/models huggingface co/bert/bert-base-uncased tar gz (The model used by BIBREF12)
RoBERTa: roberta-bases3 amazonaws com/models huggingface co/bert/roberta-base-pytorch_model bin (RoBERTa-base does not have an uncased variant)
XLNet: xlnet-base-caseds3 amazonaws com/models huggingface co/bert/xlnet-base-cased-pytorch_model bin (XLNet-base does not have an uncased variant)
The output of the model is a vector of probabilities per token  The loss is calculated for each token, by using the output vector and the true label for that token  We use class weights for the loss function, setting the weight for label 4 to 0 and all other labels to 1 (for cue detection only) to avoid training on padding token’s output 
We calculate the scores (Precision, Recall, F1) for the model per word of the input sentence, not per token that was fed to the model, as the tokens could be different for different models leading to inaccurate scores  For the above example, we calculate the output label for the word ‘tomorrow', not for each token it was split into (‘tom', ‘##or' and ‘##row')  To find the label for each word from the tokens it was split into, we experiment with 2 methods:
Average: We average the output vectors (softmax probabilities) for each token that the word was split into by the model's tokenizer  In the example above, we average the output of ‘tom', ‘##or' and ‘##row' to get the output for ‘tomorrow'  Then, we take an argmax over the resultant vector  This is then compared with the true label for the original word 
First Token: Here, we only consider the first token's probability vector (among all tokens the word was split into) as the output for that word, and get the label by an argmax over this vector  In the example above, we would consider the output vector corresponding to the token ‘tom' as the output for the word ‘tomorrow' 
For cue detection, the results are reported for the Average method only, while we report the scores for both Average and First Token for Scope Resolution 
For fair comparison, we use the same hyperparameters for the entire architecture for all 3 models  Only the tokenizer and the model are changed for each model  All other hyperparameters are kept same  We finetune the models for 60 epochs, using early stopping with a patience of 6 on the F1 score (word level) on the validation dataset  We use an initial learning rate of 3e-5, with a batch size of 8  We use the Categorical Cross Entropy loss function 
We use the Huggingface’s Pytorch Transformer library (BIBREF23) for the models and train all our models on Google Colaboratory 
Results: Speculation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset  For the speculation detection and scope resolution subtasks using single-dataset training, we report the results as an average of 5 runs of the model  For training the model on multiple datasets, we perform a 70-15-15 split of each training dataset, after which the train and validation part of the individual datasets are merged while the scores are reported for the test part of the individual datasets, which is not used for training or validation  We report the results as an average of 3 runs of the model  Figure FIGREF8 contains results for speculation cue detection and scope resolution when trained on a single dataset  All models perform the best when trained on the same dataset as they are evaluated on, except for BF, which gets the best results when trained on BA  This is because of the transfer learning capabilities of the models and the fact that BF is a smaller dataset than BA (BF: 2670 sentences, BA: 11871 sentences)  For speculation cue detection, there is lesser generalizability for models trained on BF or BA, while there is more generalizability for models trained on SFU  This could be because of the different nature of the biomedical domain 
Figure FIGREF11 contains the results for speculation detection and scope resolution for models trained jointly on multiple datasets  We observe that training on multiple datasets helps the performance of all models on each dataset, as the quantity of data available to train the model increases  We also observe that XLNet consistently outperforms BERT and RoBERTa  To confirm this observation, we apply the 2 models to the related task of negation detection and scope resolution
Negation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock)  The results for BERT are taken from BIBREF12  The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance  Figure FIGREF14 contains results for negation cue detection and scope resolution  We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets  Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks  A few possible reasons for this trend are:
Domain specificity, as both negation and speculation are closely related subtasks  Further experimentation on different tasks is needed to verify this 
Most benchmark tasks are sentence classification tasks, whereas the subtasks we experiment on are sequence labelling tasks  Given the pre-training objective of XLNet (training on permutations of the input), it may be able to capture long-term dependencies better, essential for sequence labelling tasks 
We work with the base variants of the models, while most results are reported with the large variants of the models 
Conclusion and Future Scope
In this paper, we expanded on the work of Khandelwal and Sawant (BIBREF12) by looking at alternative transfer-learning models and experimented with training on multiple datasets  On the speculation detection task, we obtained a gain of 0 42 F1 points on BF, 1 98 F1 points on BA and 0 29 F1 points on SFU, while on the scope resolution task, we obtained a gain of 8 06 F1 points on BF, 4 27 F1 points on BA and 11 87 F1 points on SFU, when trained on a single dataset  While training on multiple datasets, we observed a gain of 10 6 F1 points on BF and 1 94 F1 points on BA on the speculation detection task and 2 16 F1 points on BF and 0 25 F1 points on SFU on the scope resolution task over the single dataset training approach  We thus significantly advance the state-of-the-art for speculation detection and scope resolution  On the negation scope resolution task, we applied the XLNet and RoBERTa and obtained a gain of 3 16 F1 points on BF, 0 06 F1 points on BA and 0 3 F1 points on SFU  Thus, we demonstrated the usefulness of transformer-based architectures in the field of negation and speculation detection and scope resolution  We believe that a larger and more general dataset would go a long way in bolstering future research and would help create better systems that are not domain-specific 

ground_truth: varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)